Skip to content

armgit / ente

Public Imported from GitHub
0

πŸ’š End-to-end encrypted cloud for everything.

[photos] Accelerate internal Rust OCR with shared GPU models

laurenspriemf6e82ac769dd
diff --git a/infra/ml/playground/optimizations/README.md b/infra/ml/playground/optimizations/README.md
index d5ad854a08..89c468eea8 100644
--- a/infra/ml/playground/optimizations/README.md
+++ b/infra/ml/playground/optimizations/README.md
@@ -5,6 +5,97 @@ ML models. The generated CDN artifacts are written under `models/`; ONNX files
are intentionally gitignored, while `model_manifest.json` records the
reproducible output metadata.
+## OCR models
+
+Generate one shared PP-OCRv5 detector, classifier, and recognizer for CoreML and
+native WebGPU with:
+
+```sh
+uv run --no-project --with numpy==2.5.3 --with onnx==1.22.0 python \
+ infra/ml/playground/optimizations/optimize_ocr_models.py \
+ --source-dir infra/ml/playground/.cache/ocr-sources \
+ --output-dir infra/ml/playground/optimizations/models/ocr
+```
+
+Missing source files are downloaded from `https://models.ente.com/PP-OCRv5`.
+The script verifies every source hash and requires every generated model to
+match the SHA-256 of its qualified artifact. It writes `det.onnx`, `cls.onnx`,
+`rec.onnx`, the unchanged dictionary, and `ocr_model_manifest.json`. Both
+platforms download identical model bytes. Upload the ONNX files to new versioned
+URLs; the recognizer has a new output contract and must not replace the old URL.
+
+The selected transformations are:
+
+- Detection: fold the final transposed-convolution bias and batch normalization
+ into the head weights. Preserve the original backbone affine expressions.
+- Classification: canonicalize scalar affine constants and express 18 expanded
+ activations as `HardSigmoid(alpha=1/6, beta=0.5) * x`.
+- Recognition: canonicalize scalar affine constants, fold input affine operations
+ into 12 unpadded convolutions, and replace four expanded normalization blocks
+ with `LayerNormalization` using the original parameters and epsilon.
+- Detection and recognition: expand `HardSwish` into
+ `HardSigmoid(alpha=1/6, beta=0.5) * x` for CoreML compatibility.
+- Recognition: pad the final weight matrix's 18,385 output columns to 18,432 for
+ the existing WebGPU runtime's vectorized matrix multiplication, then slice
+ back to 18,385 before the original bias and softmax.
+- Recognition: reduce the original probabilities to the first winning token
+ index and its probability for each time step, using FP32 arithmetic supported
+ by both GPU providers.
+
+The recognizer returns FP32 `[N,T,2]`, with index first and probability second.
+For probabilities `p`, maximum `m`, vocabulary size `V`, and token index `i`, the
+compact head computes `min(i + V * ceil(m - p[i]))`. Since probabilities are in
+`[0,1]`, the ceiling is zero for maxima and one for every lower value. The
+minimum therefore selects the first tied maximum without integer casts or
+boolean selection. The other output is the original maximum probability.
+See the [ONNX Ceil definition](https://onnx.ai/onnx/operators/onnx__Ceil.html).
+
+The shared models retain FP32 parameters, vocabulary, CTC blank/repeat handling,
+confidence averaging, and character-span rules. These transformations preserve
+the mathematical operations but can introduce small floating-point differences.
+
+The Rust OCR caller keeps its 960-pixel longest-side detector limit and existing
+recognition crop sizes. It specializes sessions for actual dimensions without
+changing input pixels and retains two detector shapes, six classifier batch
+sizes, and eight recently used recognizer shapes.
+CoreML uses MLProgram, CPUAndGPU, and disables low-precision GPU accumulation.
+WebGPU uses NHWC internally for detection and NCHW for recognition. GPU session
+construction requires full provider coverage; failures retry on a single CPU
+thread. OCR never selects XNNPACK. Android classification uses the single-thread
+CPU because it was faster than WebGPU on Pixel 8; it uses the same classifier
+file as CoreML. Internal iOS uses CoreML for all three stages.
+
+Qualification uses unchanged ONNX Runtime 1.28.1 on Pixel 8 with native
+Dawn/Vulkan WebGPU and Apple M4 with CoreML. The saved model comparison covers
+317 crop sources from 17 unique photos, 21,256 recognition time steps, 320
+classification inputs, and 9,383,936 detector probabilities. Neither provider
+changed any winning token, orientation decision, or detector threshold decision
+in those cases. Every CoreML case formed one complete partition using GPU
+compute devices; both providers completed with CPU fallback disabled. Separate
+compact output checks matched all 144 rows exactly, including ties, uniform
+probabilities, and adjacent FP32 values. These checks do not establish bit-identical outputs on
+all images or performance on an iPhone. Smaller detector inputs were excluded
+because they missed text in the supplied photos.
+
+The shared artifacts produced these warm native inference medians on 10
+September 2026. Each range contains two trial medians, with 40 calls per trial
+and the first 10 discarded. Runs use one CPU inference thread and include the
+normal CPU input/output transfer; model loading, compilation, image decoding,
+and Rust preprocessing/postprocessing are excluded. CoreML measurements are
+from an Apple M4 Mac, not an iPhone.
+
+| Stage and NCHW input | M4 CoreML GPU | Pixel 8 native WebGPU |
+| --- | ---: | ---: |
+| Empty-image detection, 1Γ—3Γ—960Γ—704 | 10.4–10.9 ms | 128.6–132.2 ms |
+| Empty-image detection, 1Γ—3Γ—960Γ—960 | 13.3–13.6 ms | 173.3–175.0 ms |
+| Classification, 6Γ—3Γ—48Γ—192 | 1.4–2.0 ms | 19.3–33.9 ms |
+| Recognition, 6Γ—3Γ—48Γ—320 | 9.5–9.6 ms | 104.1–104.8 ms |
+| Recognition, 6Γ—3Γ—48Γ—640 | 16.8–17.0 ms | 189.5–190.4 ms |
+
+The same classifier took 8.7–8.8 ms on the Pixel's single-thread CPU, which is
+why the Android caller keeps that stage on CPU. Both uniform-gray detection
+inputs produced no pixels at or above the 0.3 threshold.
+
## Rebuilding the models
Run from the repository root:
diff --git a/infra/ml/playground/optimizations/models/ocr/.gitignore b/infra/ml/playground/optimizations/models/ocr/.gitignore
new file mode 100644
index 0000000000..6801ebbe54
--- /dev/null
+++ b/infra/ml/playground/optimizations/models/ocr/.gitignore
@@ -0,0 +1 @@
+ppocrv5_dict.txt
diff --git a/infra/ml/playground/optimizations/models/ocr/ocr_model_manifest.json b/infra/ml/playground/optimizations/models/ocr/ocr_model_manifest.json
new file mode 100644
index 0000000000..56f26722f7
--- /dev/null
+++ b/infra/ml/playground/optimizations/models/ocr/ocr_model_manifest.json
@@ -0,0 +1,93 @@
+{
+ "source_base_url": "https://models.ente.com/PP-OCRv5",
+ "source_sha256": {
+ "cls.onnx": "f4bb53707100c5f3d59ba834eb05bb400369f20aed35d4b26807b1bfadd2a70e",
+ "det.onnx": "d7fe3ea74652890722c0f4d02458b7261d9f5ae6c92904d05707c9eb155c7924",
+ "ppocrv5_dict.txt": "d1979e9f794c464c0d2e0b70a7fe14dd978e9dc644c0e71f14158cdf8342af1b",
+ "rec.onnx": "bf66820f48fa99f779974c4df78e5274a9d8e0458c4137e8c5357e40e2c3faf2"
+ },
+ "onnx": "1.22.0",
+ "models": [
+ {
+ "file": "det.onnx",
+ "sha256": "3e658f85236f1984e186070048d7cdc99ddbb84e12c91a6fd80922b85bd9487a",
+ "bytes": 4750896,
+ "operators": [
+ "Add",
+ "BatchNormalization",
+ "Concat",
+ "Constant",
+ "Conv",
+ "ConvTranspose",
+ "GlobalAveragePool",
+ "HardSigmoid",
+ "Mul",
+ "Relu",
+ "Reshape",
+ "Resize",
+ "Sigmoid"
+ ],
+ "output": "unchanged"
+ },
+ {
+ "file": "cls.onnx",
+ "sha256": "8675903197a1bab7060e830e9bcc63d29c5cf7d372735338bd9501f829e94ed1",
+ "bytes": 574875,
+ "operators": [
+ "Add",
+ "BatchNormalization",
+ "Cast",
+ "Concat",
+ "Constant",
+ "Conv",
+ "GlobalAveragePool",
+ "HardSigmoid",
+ "MatMul",
+ "MaxPool",
+ "Mul",
+ "Relu",
+ "Reshape",
+ "Shape",
+ "Slice",
+ "Softmax"
+ ],
+ "output": "unchanged"
+ },
+ {
+ "file": "rec.onnx",
+ "sha256": "667691bb173a94106b595aa87c7d176b53b68cf831b6ffa09569018754265e57",
+ "bytes": 16686404,
+ "operators": [
+ "Add",
+ "AveragePool",
+ "BatchNormalization",
+ "Ceil",
+ "Concat",
+ "Constant",
+ "Conv",
+ "Div",
+ "GlobalAveragePool",
+ "HardSigmoid",
+ "LayerNormalization",
+ "MatMul",
+ "Mul",
+ "Pow",
+ "ReduceMax",
+ "ReduceMean",
+ "ReduceMin",
+ "Relu",
+ "Reshape",
+ "Shape",
+ "Sigmoid",
+ "Slice",
+ "Softmax",
+ "Sqrt",
+ "Squeeze",
+ "Sub",
+ "Transpose",
+ "Unsqueeze"
+ ],
+ "output": "FP32 [N,T,2]: first winning index, original winning probability"
+ }
+ ]
+}
diff --git a/infra/ml/playground/optimizations/optimize_ocr_models.py b/infra/ml/playground/optimizations/optimize_ocr_models.py
new file mode 100644
index 0000000000..72284d76cd
--- /dev/null
+++ b/infra/ml/playground/optimizations/optimize_ocr_models.py
@@ -0,0 +1,547 @@
+import argparse
+import copy
+import hashlib
+import json
+from pathlib import Path
+from urllib.request import urlopen
+
+import numpy as np
+import onnx
+from onnx import helper, numpy_helper
+
+SOURCE_BASE_URL = "https://models.ente.com/PP-OCRv5"
+SOURCE_HASHES = {
+ "cls.onnx": "f4bb53707100c5f3d59ba834eb05bb400369f20aed35d4b26807b1bfadd2a70e",
+ "det.onnx": "d7fe3ea74652890722c0f4d02458b7261d9f5ae6c92904d05707c9eb155c7924",
+ "ppocrv5_dict.txt": "d1979e9f794c464c0d2e0b70a7fe14dd978e9dc644c0e71f14158cdf8342af1b",
+ "rec.onnx": "bf66820f48fa99f779974c4df78e5274a9d8e0458c4137e8c5357e40e2c3faf2",
+}
+OUTPUT_HASHES = {
+ "det.onnx": "3e658f85236f1984e186070048d7cdc99ddbb84e12c91a6fd80922b85bd9487a",
+ "cls.onnx": "8675903197a1bab7060e830e9bcc63d29c5cf7d372735338bd9501f829e94ed1",
+ "rec.onnx": "667691bb173a94106b595aa87c7d176b53b68cf831b6ffa09569018754265e57",
+}
+
+
+def constant_arrays(model):
+ result = {v.name: numpy_helper.to_array(v) for v in model.graph.initializer}
+ for node in model.graph.node:
+ if node.op_type == "Constant":
+ for attr in node.attribute:
+ if attr.name == "value":
+ result[node.output[0]] = numpy_helper.to_array(attr.t)
+ return result
+
+
+def remove_unused_nodes(model):
+ producers = {v: node for node in model.graph.node for v in node.output}
+ needed = set()
+ pending = [v.name for v in model.graph.output]
+ while pending:
+ name = pending.pop()
+ if name in needed:
+ continue
+ needed.add(name)
+ if name in producers:
+ pending.extend(producers[name].input)
+ nodes = [n for n in model.graph.node if any(v in needed for v in n.output)]
+ del model.graph.node[:]
+ model.graph.node.extend(nodes)
+ for field in ["initializer", "value_info"]:
+ values = [v for v in getattr(model.graph, field) if v.name in needed]
+ del getattr(model.graph, field)[:]
+ getattr(model.graph, field).extend(values)
+ return model
+
+
+def canonicalize_scalar_affines(model):
+ values = constant_arrays(model)
+ inferred = onnx.shape_inference.infer_shapes(model)
+ shapes = {v.name: v.type.tensor_type.shape for v in inferred.graph.value_info}
+ count = 0
+ for node in model.graph.node:
+ if node.op_type not in ["Mul", "Add"]:
+ continue
+ indices = [
+ i for i, n in enumerate(node.input) if n in values and values[n].size == 1
+ ]
+ if len(indices) != 1:
+ continue
+ index = indices[0]
+ value = values[node.input[index]].reshape(())
+ if value.dtype != np.float32:
+ continue
+ replacement = value if node.op_type == "Mul" else None
+ shape = shapes.get(node.output[0])
+ if (
+ node.op_type == "Add"
+ and shape is not None
+ and (len(shape.dim) == 4)
+ and shape.dim[1].dim_value
+ ):
+ replacement = np.full(
+ (shape.dim[1].dim_value, 1, 1), value, dtype=np.float32
+ )
+ if replacement is None:
+ continue
+ name = f"modelopt_affine_{count}"
+ model.graph.initializer.append(numpy_helper.from_array(replacement, name))
+ node.input[:] = [node.input[1 - index], name]
+ count += 1
+ return (remove_unused_nodes(model), count)
+
+
+def scalar_affine_input(node, values):
+ if node.op_type not in ["Mul", "Add"]:
+ return None
+ indices = [
+ i
+ for i, n in enumerate(node.input)
+ if n in values and values[n].size and np.all(values[n] == values[n].flat[0])
+ ]
+ if len(indices) != 1:
+ return None
+ index = indices[0]
+ return (node.input[1 - index], float(values[node.input[index]].flat[0]))
+
+
+def fold_input_affine(model):
+ values = constant_arrays(model)
+ producers = {v: n for n in model.graph.node for v in n.output}
+ count = 0
+ for node in model.graph.node:
+ if node.op_type != "Conv" or node.input[1] not in values:
+ continue
+ attrs = {a.name: helper.get_attribute_value(a) for a in node.attribute}
+ if any(attrs.get("pads", [])) or attrs.get("auto_pad", b"NOTSET") != b"NOTSET":
+ continue
+ current = node.input[0]
+ scale, bias, steps = (1.0, 0.0, 0)
+ while current in producers:
+ parent = producers[current]
+ affine = scalar_affine_input(parent, values)
+ if affine is None:
+ break
+ current, value = affine
+ if parent.op_type == "Mul":
+ scale *= value
+ else:
+ bias += scale * value
+ steps += 1
+ if not steps:
+ continue
+ weights = values[node.input[1]]
+ old_bias = (
+ values[node.input[2]]
+ if len(node.input) > 2
+ else np.zeros(weights.shape[0], dtype=np.float32)
+ )
+ new_weights = (weights.astype(np.float64) * scale).astype(np.float32)
+ new_bias = (
+ old_bias.astype(np.float64)
+ + weights.astype(np.float64).sum(axis=tuple(range(1, weights.ndim))) * bias
+ ).astype(np.float32)
+ weight_name, bias_name = (
+ f"modelopt_input_weight_{count}",
+ f"modelopt_input_bias_{count}",
+ )
+ model.graph.initializer.extend(
+ [
+ numpy_helper.from_array(new_weights, weight_name),
+ numpy_helper.from_array(new_bias, bias_name),
+ ]
+ )
+ node.input[:] = [current, weight_name, bias_name]
+ count += 1
+ return (remove_unused_nodes(model), count)
+
+
+def align_recognizer_projection(model, width):
+ values = constant_arrays(model)
+ node = next(n for n in model.graph.node if n.name == "MatMul.12")
+ weights = values[node.input[1]]
+ assert weights.shape == (120, 18385)
+ padded = np.pad(weights, ((0, 0), (0, width - 18385)))
+ model.graph.initializer.extend(
+ [
+ numpy_helper.from_array(padded, "modelopt_projection_weight"),
+ numpy_helper.from_array(
+ np.array([0], dtype=np.int64), "modelopt_slice_start"
+ ),
+ numpy_helper.from_array(
+ np.array([18385], dtype=np.int64), "modelopt_slice_end"
+ ),
+ numpy_helper.from_array(
+ np.array([2], dtype=np.int64), "modelopt_slice_axis"
+ ),
+ ]
+ )
+ node.input[1] = "modelopt_projection_weight"
+ output = node.output[0]
+ node.output[0] = "modelopt_padded_projection"
+ index = next((i for i, n in enumerate(model.graph.node) if n.name == node.name))
+ model.graph.node.insert(
+ index + 1,
+ helper.make_node(
+ "Slice",
+ [
+ node.output[0],
+ "modelopt_slice_start",
+ "modelopt_slice_end",
+ "modelopt_slice_axis",
+ ],
+ [output],
+ name="modelopt_projection_slice",
+ ),
+ )
+ return remove_unused_nodes(model)
+
+
+def fuse_recognizer_layer_norms(model):
+ model = onnx.version_converter.convert_version(model, 17)
+ values = constant_arrays(model)
+ specs = [
+ ("Add.187", "Add.191", "layer_norm_1", "helper.constant.26"),
+ ("Add.197", "Add.201", "layer_norm_2", "helper.constant.33"),
+ ("Add.207", "Add.211", "layer_norm_3", "helper.constant.49"),
+ ("Add.217", "Add.221", "layer_norm_4", "helper.constant.56"),
+ ]
+ count = 0
+ for source, output, prefix, epsilon in specs:
+ index = next((i for i, n in enumerate(model.graph.node) if output in n.output))
+ weight_name, bias_name = (
+ f"{prefix}_modelopt_weight",
+ f"{prefix}_modelopt_bias",
+ )
+ model.graph.initializer.extend(
+ [
+ numpy_helper.from_array(
+ values[prefix + ".w_0"].reshape(-1), weight_name
+ ),
+ numpy_helper.from_array(values[prefix + ".b_0"].reshape(-1), bias_name),
+ ]
+ )
+ model.graph.node[index].CopyFrom(
+ helper.make_node(
+ "LayerNormalization",
+ [source, weight_name, bias_name],
+ [output],
+ name=prefix + "_modelopt",
+ axis=-1,
+ epsilon=float(values[epsilon].item()),
+ stash_type=1,
+ )
+ )
+ count += 1
+ return (remove_unused_nodes(model), count)
+
+
+def expand_hardswish(model):
+ nodes = []
+ for node in model.graph.node:
+ if node.op_type != "HardSwish":
+ nodes.append(node)
+ continue
+ name = node.output[0] + "_modelopt_gate"
+ nodes.extend(
+ [
+ helper.make_node(
+ "HardSigmoid",
+ [node.input[0]],
+ [name],
+ name=node.name + "_gate",
+ alpha=1 / 6,
+ beta=0.5,
+ ),
+ helper.make_node(
+ "Mul",
+ [node.input[0], name],
+ list(node.output),
+ name=node.name + "_multiply",
+ ),
+ ]
+ )
+ del model.graph.node[:]
+ model.graph.node.extend(nodes)
+ return model
+
+
+def compact_recognizer_output(model):
+ output = model.graph.output[0].name
+ model.graph.initializer.extend(
+ [
+ numpy_helper.from_array(
+ np.arange(18385, dtype=np.float32).reshape(1, 1, -1), "shared_positions"
+ ),
+ numpy_helper.from_array(
+ np.array(18385, dtype=np.float32), "shared_sentinel"
+ ),
+ ]
+ )
+ model.graph.node.extend(
+ [
+ helper.make_node(
+ "ReduceMax",
+ [output],
+ ["shared_score"],
+ axes=[2],
+ keepdims=1,
+ name="shared_score",
+ ),
+ helper.make_node(
+ "Sub", ["shared_score", output], ["shared_gap"], name="shared_gap"
+ ),
+ helper.make_node(
+ "Ceil", ["shared_gap"], ["shared_nonwinner"], name="shared_nonwinner"
+ ),
+ helper.make_node(
+ "Mul",
+ ["shared_nonwinner", "shared_sentinel"],
+ ["shared_penalty"],
+ name="shared_penalty",
+ ),
+ helper.make_node(
+ "Add",
+ ["shared_positions", "shared_penalty"],
+ ["shared_candidates"],
+ name="shared_candidates",
+ ),
+ helper.make_node(
+ "ReduceMin",
+ ["shared_candidates"],
+ ["shared_index"],
+ axes=[2],
+ keepdims=1,
+ name="shared_index",
+ ),
+ helper.make_node(
+ "Concat",
+ ["shared_index", "shared_score"],
+ ["shared_output"],
+ axis=2,
+ name="shared_output",
+ ),
+ ]
+ )
+ del model.graph.output[:]
+ model.graph.output.append(
+ helper.make_tensor_value_info(
+ "shared_output", onnx.TensorProto.FLOAT, ["N", "T", 2]
+ )
+ )
+ return remove_unused_nodes(model)
+
+
+def fold_detector_head(model):
+ values = constant_arrays(model)
+ nodes = list(model.graph.node)
+ for node in nodes:
+ if (
+ node.op_type == "Reshape"
+ and node.input[0] in values
+ and (node.input[1] in values)
+ ):
+ values[node.output[0]] = values[node.input[0]].reshape(
+ values[node.input[1]]
+ )
+ replacements = {}
+ for ordinal, node in enumerate([n for n in nodes if n.op_type == "ConvTranspose"]):
+ attrs = {a.name: helper.get_attribute_value(a) for a in node.attribute}
+ assert attrs["group"] == 1 and attrs["kernel_shape"] == [2, 2]
+ assert attrs["strides"] == [2, 2] and (not any(attrs["pads"]))
+ add = next(n for n in nodes if node.output[0] in n.input and n.op_type == "Add")
+ bias_input = next(v for v in add.input if v != node.output[0])
+ weight = values[node.input[1]].astype(np.float64)
+ bias = values[bias_input].reshape(-1).astype(np.float64)
+ output = add.output[0]
+ bn = next(
+ (
+ n
+ for n in nodes
+ if output in n.input and n.op_type == "BatchNormalization"
+ ),
+ None,
+ )
+ if bn is not None:
+ attrs_bn = {a.name: helper.get_attribute_value(a) for a in bn.attribute}
+ assert not attrs_bn.get("training_mode", 0)
+ scale, offset, mean, variance = [
+ values[v].astype(np.float64) for v in bn.input[1:]
+ ]
+ factor = scale / np.sqrt(variance + attrs_bn["epsilon"])
+ weight *= factor[None, :, None, None]
+ bias = (bias - mean) * factor + offset
+ output = bn.output[0]
+ weight_name, bias_name = (
+ f"modelopt_deconv_weight_{ordinal}",
+ f"modelopt_deconv_bias_{ordinal}",
+ )
+ fused = copy.deepcopy(node)
+ fused.input[:] = [node.input[0], weight_name, bias_name]
+ fused.output[:] = [output]
+ replacements[node.name] = [fused]
+ model.graph.initializer.extend(
+ [
+ numpy_helper.from_array(weight.astype(np.float32), weight_name),
+ numpy_helper.from_array(bias.astype(np.float32), bias_name),
+ ]
+ )
+ replacements[add.name] = []
+ if bn is not None:
+ replacements[bn.name] = []
+ del model.graph.node[:]
+ for node in nodes:
+ model.graph.node.extend(replacements.get(node.name, [node]))
+ return remove_unused_nodes(model)
+
+
+def rewrite_classifier_hardswish(model):
+ values = constant_arrays(model)
+ producers = {v: n for n in model.graph.node for v in n.output}
+ nodes = []
+ count = 0
+ for node in model.graph.node:
+ matched = False
+ if (
+ node.op_type == "Div"
+ and node.input[1] in values
+ and np.all(values[node.input[1]] == 6)
+ ):
+ multiply = producers.get(node.input[0])
+ if multiply is not None and multiply.op_type == "Mul":
+ for index in [0, 1]:
+ clip = producers.get(multiply.input[index])
+ source = multiply.input[1 - index]
+ if clip is None or clip.op_type != "Clip" or len(clip.input) != 3:
+ continue
+ if (
+ clip.input[1] not in values
+ or clip.input[2] not in values
+ or (not np.all(values[clip.input[1]] == 0))
+ or (not np.all(values[clip.input[2]] == 6))
+ ):
+ continue
+ add = producers.get(clip.input[0])
+ if add is None or add.op_type != "Add" or source not in add.input:
+ continue
+ other = add.input[1 - list(add.input).index(source)]
+ if other not in values or not np.all(values[other] == 3):
+ continue
+ gate = f"modelopt_cls_gate_{count}"
+ nodes.extend(
+ [
+ helper.make_node(
+ "HardSigmoid",
+ [source],
+ [gate],
+ name=gate,
+ alpha=1 / 6,
+ beta=0.5,
+ ),
+ helper.make_node(
+ "Mul",
+ [source, gate],
+ list(node.output),
+ name=node.name + "_modelopt",
+ ),
+ ]
+ )
+ count += 1
+ matched = True
+ break
+ if not matched:
+ nodes.append(node)
+ del model.graph.node[:]
+ model.graph.node.extend(nodes)
+ return (remove_unused_nodes(model), count)
+
+
+def load_sources(directory):
+ models = {}
+ directory.mkdir(parents=True, exist_ok=True)
+ for name, expected in SOURCE_HASHES.items():
+ path = directory / name
+ if not path.exists():
+ with urlopen(f"{SOURCE_BASE_URL}/{name}", timeout=120) as response:
+ data = response.read()
+ if hashlib.sha256(data).hexdigest() != expected:
+ raise ValueError(f"Unexpected SHA-256 for downloaded {name}")
+ path.write_bytes(data)
+ if hashlib.sha256(path.read_bytes()).hexdigest() != expected:
+ raise ValueError(f"Unexpected SHA-256 for {path}")
+ if path.suffix == ".onnx":
+ models[path.stem] = onnx.load(path)
+ return models
+
+
+def optimize(models):
+ recognition, _ = canonicalize_scalar_affines(copy.deepcopy(models["rec"]))
+ recognition, folded = fold_input_affine(recognition)
+ recognition, normalized = fuse_recognizer_layer_norms(recognition)
+ if folded != 12 or normalized != 4:
+ raise ValueError(
+ f"Unexpected recognizer rewrites: folded={folded!r}, normalized={normalized!r}"
+ )
+ recognition = expand_hardswish(recognition)
+ recognition = align_recognizer_projection(recognition, 18432)
+ recognition = compact_recognizer_output(recognition)
+ detection = fold_detector_head(copy.deepcopy(models["det"]))
+ detection = expand_hardswish(detection)
+ classification, _ = canonicalize_scalar_affines(copy.deepcopy(models["cls"]))
+ classification, activations = rewrite_classifier_hardswish(classification)
+ if activations != 18:
+ raise ValueError(f"Unexpected classifier rewrites: activations={activations!r}")
+ return {
+ "det.onnx": detection,
+ "cls.onnx": classification,
+ "rec.onnx": recognition,
+ }
+
+
+def main():
+ parser = argparse.ArgumentParser()
+ parser.add_argument("--source-dir", type=Path, required=True)
+ parser.add_argument("--output-dir", type=Path, required=True)
+ args = parser.parse_args()
+ if onnx.__version__ != "1.22.0":
+ raise ValueError("Reproducible model output requires onnx==1.22.0")
+ sources = load_sources(args.source_dir)
+ records = []
+ for name, model in optimize(sources).items():
+ onnx.checker.check_model(model)
+ data = model.SerializeToString()
+ digest = hashlib.sha256(data).hexdigest()
+ if digest != OUTPUT_HASHES[name]:
+ raise ValueError(
+ f"Generated model differs from qualified artifact: {name} ({digest})"
+ )
+ path = args.output_dir / name
+ path.parent.mkdir(parents=True, exist_ok=True)
+ path.write_bytes(data)
+ records.append(
+ {
+ "file": name,
+ "sha256": digest,
+ "bytes": len(data),
+ "operators": sorted({node.op_type for node in model.graph.node}),
+ "output": "FP32 [N,T,2]: first winning index, original winning probability"
+ if name == "rec.onnx"
+ else "unchanged",
+ }
+ )
+ dictionary = args.source_dir / "ppocrv5_dict.txt"
+ (args.output_dir / dictionary.name).write_bytes(dictionary.read_bytes())
+ metadata = {
+ "source_base_url": SOURCE_BASE_URL,
+ "source_sha256": SOURCE_HASHES,
+ "onnx": onnx.__version__,
+ "models": records,
+ }
+ (args.output_dir / "ocr_model_manifest.json").write_text(
+ json.dumps(metadata, indent=2) + "\n"
+ )
+ print(json.dumps(records, indent=2))
+
+
+if __name__ == "__main__":
+ main()
diff --git a/mobile/apps/photos/changes/ocr-gpu-internal.md b/mobile/apps/photos/changes/ocr-gpu-internal.md
new file mode 100644
index 0000000000..83b2b3d571
--- /dev/null
+++ b/mobile/apps/photos/changes/ocr-gpu-internal.md
@@ -0,0 +1 @@
+- Internal users can test GPU-accelerated text recognition on Android and iOS.
diff --git a/mobile/apps/photos/lib/services/machine_learning/ocr/rust_ocr_backend.dart b/mobile/apps/photos/lib/services/machine_learning/ocr/rust_ocr_backend.dart
index d2744d2cfe..dfcde2461b 100644
--- a/mobile/apps/photos/lib/services/machine_learning/ocr/rust_ocr_backend.dart
+++ b/mobile/apps/photos/lib/services/machine_learning/ocr/rust_ocr_backend.dart
@@ -5,12 +5,14 @@ import "package:path/path.dart" as p;
import "package:path_provider/path_provider.dart";
import "package:photos/services/machine_learning/ocr/ocr_backend.dart";
import "package:photos/services/machine_learning/ocr/ocr_models.dart";
+import "package:photos/services/machine_learning/webgpu_execution_policy.dart";
+import "package:photos/src/rust/api/ml_indexing_api.dart";
import "package:photos/src/rust/api/ocr_api.dart";
import "package:synchronized/synchronized.dart";
class RustOcrBackend implements OcrBackend {
static final _logger = Logger("RustOcrBackend");
- static const _modelVersion = "pp-ocrv5";
+ static const _modelVersion = "pp-ocrv5-gpu-v1";
final _engineLock = Lock();
OcrEngine? _engine;
@@ -50,6 +52,9 @@ class RustOcrBackend implements OcrBackend {
return;
}
try {
+ await setMlExecutionConfig(
+ enableWebgpu: await webGpuExecutionPolicy.isEligible(),
+ );
_engine = await OcrEngine.create(
assetsDir: assetsDir,
includeRecognizer: includeRecognizer,
diff --git a/mobile/apps/photos/lib/services/machine_learning/ocr_service.dart b/mobile/apps/photos/lib/services/machine_learning/ocr_service.dart
index fc0f05a1ec..82247bfd7d 100644
--- a/mobile/apps/photos/lib/services/machine_learning/ocr_service.dart
+++ b/mobile/apps/photos/lib/services/machine_learning/ocr_service.dart
@@ -52,12 +52,9 @@ class OcrService {
if (!_rustOcrEnabled()) {
return OcrBackendKind.legacy;
}
- if (_isAndroid) {
+ if (_isAndroid || _isIOS) {
return OcrBackendKind.rust;
}
- if (_isIOS) {
- return OcrBackendKind.vision;
- }
return OcrBackendKind.legacy;
}
diff --git a/rust/crates/ml/src/ocr/assets.rs b/rust/crates/ml/src/ocr/assets.rs
index 0466e7f6cd..5698d696a7 100644
--- a/rust/crates/ml/src/ocr/assets.rs
+++ b/rust/crates/ml/src/ocr/assets.rs
@@ -6,40 +6,44 @@ use ente_assets::{Asset, AssetFile, AssetStore};
use super::OcrModelPaths;
use crate::error::{MlError, MlResult};
-const MODEL_BASE_URL: &str = "https://models.ente.com/PP-OCRv5";
const MODELS: &str = "models";
struct OcrModelFile {
key: &'static str,
name: &'static str,
+ url: &'static str,
size: u64,
sha256: &'static str,
}
const DETECTION: OcrModelFile = OcrModelFile {
- key: "ppocrv5_det",
+ key: "ppocrv5_det_gpu_v1",
name: "det.onnx",
- size: 4_748_769,
- sha256: "d7fe3ea74652890722c0f4d02458b7261d9f5ae6c92904d05707c9eb155c7924",
+ url: "https://entedevassets.priem.dev/det_opt.onnx",
+ size: 4_750_896,
+ sha256: "3e658f85236f1984e186070048d7cdc99ddbb84e12c91a6fd80922b85bd9487a",
};
const CLASSIFICATION: OcrModelFile = OcrModelFile {
- key: "ppocrv5_cls",
+ key: "ppocrv5_cls_gpu_v1",
name: "cls.onnx",
- size: 582_663,
- sha256: "f4bb53707100c5f3d59ba834eb05bb400369f20aed35d4b26807b1bfadd2a70e",
+ url: "https://entedevassets.priem.dev/cls_opt.onnx",
+ size: 574_875,
+ sha256: "8675903197a1bab7060e830e9bcc63d29c5cf7d372735338bd9501f829e94ed1",
};
const RECOGNITION: OcrModelFile = OcrModelFile {
- key: "ppocrv5_rec",
+ key: "ppocrv5_rec_gpu_v1",
name: "rec.onnx",
- size: 16_517_247,
- sha256: "bf66820f48fa99f779974c4df78e5274a9d8e0458c4137e8c5357e40e2c3faf2",
+ url: "https://entedevassets.priem.dev/rec_opt.onnx",
+ size: 16_686_404,
+ sha256: "667691bb173a94106b595aa87c7d176b53b68cf831b6ffa09569018754265e57",
};
const DICTIONARY: OcrModelFile = OcrModelFile {
key: "ppocrv5_dict",
name: "ppocrv5_dict.txt",
+ url: "https://models.ente.com/PP-OCRv5/ppocrv5_dict.txt",
size: 74_012,
sha256: "d1979e9f794c464c0d2e0b70a7fe14dd978e9dc644c0e71f14158cdf8342af1b",
};
@@ -47,10 +51,6 @@ const DICTIONARY: OcrModelFile = OcrModelFile {
const CATALOG: [&OcrModelFile; 4] = [&DETECTION, &CLASSIFICATION, &RECOGNITION, &DICTIONARY];
impl OcrModelFile {
- fn url(&self) -> String {
- format!("{MODEL_BASE_URL}/{}", self.name)
- }
-
fn asset(&self) -> Asset {
#[expect(
clippy::expect_used,
@@ -60,7 +60,7 @@ impl OcrModelFile {
vec![MODELS.to_string(), self.key.to_string()],
AssetFile {
name: self.name.to_string(),
- url: self.url(),
+ url: self.url.to_string(),
size: self.size,
sha256: self.sha256.to_string(),
},
@@ -111,7 +111,7 @@ pub async fn ensure_models(
"ocr models: downloading {} ({} bytes) from {}",
file.name,
file.size,
- file.url()
+ file.url
);
}
let start = Instant::now();
@@ -178,6 +178,26 @@ mod tests {
assert!(!is_detector_downloaded(&store));
}
+ #[test]
+ fn legacy_models_do_not_satisfy_the_optimized_catalog() {
+ let root = tempfile::tempdir().unwrap();
+ let store = AssetStore::new(root.path());
+ for (model, legacy_key) in [
+ (&DETECTION, "ppocrv5_det"),
+ (&CLASSIFICATION, "ppocrv5_cls"),
+ (&RECOGNITION, "ppocrv5_rec"),
+ ] {
+ let legacy = OcrModelFile {
+ key: legacy_key,
+ ..*model
+ };
+ cache_model(&store, &legacy);
+ assert!(!store.is_downloaded(&model.asset()));
+ assert_ne!(model.path(&store), legacy.path(&store));
+ }
+ assert!(!is_detector_downloaded(&store));
+ }
+
#[tokio::test]
async fn cached_detector_can_be_prepared_without_recognition_models() {
let root = tempfile::tempdir().unwrap();
diff --git a/rust/crates/ml/src/ocr/classify.rs b/rust/crates/ml/src/ocr/classify.rs
index d6e1fd06d3..b68a3db4b4 100644
--- a/rust/crates/ml/src/ocr/classify.rs
+++ b/rust/crates/ml/src/ocr/classify.rs
@@ -1,12 +1,12 @@
use std::sync::{Mutex, PoisonError};
+use super::session::{OcrModel, OcrSession};
use super::tensor::{BgrNormalization, prepare_crops, write_bgr_planes};
use crate::cv;
use crate::cv::image::ImageU8;
use crate::error::{MlError, MlResult};
-use crate::onnx::{ExecutionMode, OnnxSession, PreparedF32Input, SessionRunError, run_f32};
+use crate::onnx::{PreparedF32Input, SessionRunError, run_f32};
-const MODEL_NAMESPACE: &str = "ocr-classification";
const BATCH_SIZE: usize = 6;
const INPUT_HEIGHT: i32 = 48;
const INPUT_WIDTH: i32 = 192;
@@ -32,17 +32,13 @@ pub(crate) struct AngleDecision {
}
pub(crate) struct AngleClassifier {
- session: Mutex<OnnxSession>,
+ session: Mutex<OcrSession>,
}
impl AngleClassifier {
pub(crate) fn new(model_path: &str) -> Self {
Self {
- session: Mutex::new(OnnxSession::new(
- model_path,
- MODEL_NAMESPACE,
- ExecutionMode::CpuOnly,
- )),
+ session: Mutex::new(OcrSession::new(model_path, OcrModel::Classification)),
}
}
@@ -55,12 +51,9 @@ impl AngleClassifier {
let count = batch.len() as i64;
let expected_shape = [count, CLASS_COUNT as i64];
let mut session = self.session.lock().unwrap_or_else(PoisonError::into_inner);
- let (values, _usage) = session.run(|session| {
- let (shape, values) = run_f32(
- session,
- &input,
- [count, 3, i64::from(INPUT_HEIGHT), i64::from(INPUT_WIDTH)],
- )?;
+ let input_shape = [count, 3, i64::from(INPUT_HEIGHT), i64::from(INPUT_WIDTH)];
+ let (values, _usage) = session.run(input_shape, |session| {
+ let (shape, values) = run_f32(session, &input, input_shape)?;
if shape != expected_shape {
return Err(SessionRunError::from(MlError::CorruptModel(format!(
"angle classifier produced output shape {shape:?}, expected {expected_shape:?}"
diff --git a/rust/crates/ml/src/ocr/detect.rs b/rust/crates/ml/src/ocr/detect.rs
index 10f1216234..cf440933ea 100644
--- a/rust/crates/ml/src/ocr/detect.rs
+++ b/rust/crates/ml/src/ocr/detect.rs
@@ -4,13 +4,13 @@ use super::Point;
use super::geometry::{
clip_to_bounds, mean_inside_quad, min_area_rect, min_edge, order_corners, scale_points, unclip,
};
+use super::session::{OcrModel, OcrSession};
use super::tensor::{BgrNormalization, write_bgr_planes};
use crate::cv;
use crate::cv::image::{Contour, ImageU8};
use crate::error::{MlError, MlResult};
-use crate::onnx::{ExecutionMode, OnnxSession, PreparedF32Input, SessionRunError, run_f32};
+use crate::onnx::{PreparedF32Input, SessionRunError, run_f32};
-const MODEL_NAMESPACE: &str = "ocr-detection";
const MAX_INPUT_SIDE: i32 = 960;
const INPUT_STRIDE: i32 = 32;
const BITMAP_THRESHOLD: f32 = 0.3;
@@ -40,17 +40,13 @@ pub(crate) struct TextDetection {
}
pub(crate) struct TextDetector {
- session: Mutex<OnnxSession>,
+ session: Mutex<OcrSession>,
}
impl TextDetector {
pub(crate) fn new(model_path: &str) -> Self {
Self {
- session: Mutex::new(OnnxSession::new(
- model_path,
- MODEL_NAMESPACE,
- ExecutionMode::CpuOnly,
- )),
+ session: Mutex::new(OcrSession::new(model_path, OcrModel::Detection)),
}
}
@@ -82,9 +78,9 @@ impl TextDetector {
fn infer(&self, input: &PreparedF32Input, width: i32, height: i32) -> MlResult<Vec<f32>> {
let expected_shape = [1i64, 1, i64::from(height), i64::from(width)];
let mut session = self.session.lock().unwrap_or_else(PoisonError::into_inner);
- let (values, _usage) = session.run(|session| {
- let (shape, values) =
- run_f32(session, input, [1, 3, i64::from(height), i64::from(width)])?;
+ let input_shape = [1, 3, i64::from(height), i64::from(width)];
+ let (values, _usage) = session.run(input_shape, |session| {
+ let (shape, values) = run_f32(session, input, input_shape)?;
if shape != expected_shape {
return Err(SessionRunError::from(MlError::CorruptModel(format!(
"text detector produced output shape {shape:?}, expected {expected_shape:?}"
diff --git a/rust/crates/ml/src/ocr/mod.rs b/rust/crates/ml/src/ocr/mod.rs
index 4021d58980..ddaf78c382 100644
--- a/rust/crates/ml/src/ocr/mod.rs
+++ b/rust/crates/ml/src/ocr/mod.rs
@@ -7,6 +7,7 @@ mod detect;
mod dictionary;
mod geometry;
mod recognize;
+mod session;
mod source;
mod tensor;
diff --git a/rust/crates/ml/src/ocr/recognize.rs b/rust/crates/ml/src/ocr/recognize.rs
index 2772c5a694..b0dec5eff8 100644
--- a/rust/crates/ml/src/ocr/recognize.rs
+++ b/rust/crates/ml/src/ocr/recognize.rs
@@ -9,11 +9,10 @@ use crate::cv;
use crate::cv::image::ImageU8;
use crate::error::{MlError, MlResult};
use crate::onnx::{
- BorrowedFloatTensor, ExecutionMode, FloatTensorData, OnnxSession, PreparedF32Input,
- with_prepared_float_output,
+ BorrowedFloatTensor, FloatTensorData, PreparedF32Input, with_prepared_float_output,
};
-const MODEL_NAMESPACE: &str = "ocr-recognition";
+use super::session::{OcrModel, OcrSession};
const REC_VOCABULARY_SIZE: usize = 18385;
const REC_HEIGHT: i32 = 48;
const REC_BASE_WIDTH: i32 = 320;
@@ -61,18 +60,14 @@ impl LazyDictionary {
}
pub(crate) struct TextRecognizer {
- session: Mutex<OnnxSession>,
+ session: Mutex<OcrSession>,
dictionary: LazyDictionary,
}
impl TextRecognizer {
pub(crate) fn new(model_path: &str, dictionary_path: &str) -> Self {
Self {
- session: Mutex::new(OnnxSession::new(
- model_path,
- MODEL_NAMESPACE,
- ExecutionMode::CpuOnly,
- )),
+ session: Mutex::new(OcrSession::new(model_path, OcrModel::Recognition)),
dictionary: LazyDictionary::new(dictionary_path),
}
}
@@ -106,7 +101,7 @@ impl TextRecognizer {
i64::from(layout.target_width),
];
let mut session = self.session.lock().unwrap_or_else(PoisonError::into_inner);
- let (recognized, _usage) = session.run(|session| {
+ let (recognized, _usage) = session.run(input_shape, |session| {
with_prepared_float_output(session, &input, input_shape, |shape, values| match values {
BorrowedFloatTensor::F32(values) => {
decode_output(shape, values, layout, dictionary)
@@ -129,20 +124,59 @@ fn decode_output<'a, T>(
where
&'a [T]: FloatTensorData,
{
- let output = SequenceOutput::new(shape, values, layout.content_widths.len(), dictionary.len())?;
- Ok(layout
+ let compact = shape.get(2) == Some(&2);
+ let values_per_step = if compact { 2 } else { dictionary.len() };
+ let output = SequenceOutput::new(shape, values, layout.content_widths.len(), values_per_step)?;
+ layout
.content_widths
.iter()
.zip(output.sequences())
.map(|(&content_width, logits)| {
- ctc_decode(
- logits,
- output.vocabulary,
- dictionary,
- layout.padding_scale(content_width),
- )
+ let padding_scale = layout.padding_scale(content_width);
+ if compact {
+ let best = compact_steps(logits, dictionary.len())?;
+ Ok(decode_steps(&best, dictionary, padding_scale))
+ } else {
+ Ok(ctc_decode(
+ logits,
+ output.vocabulary,
+ dictionary,
+ padding_scale,
+ ))
+ }
})
- .collect())
+ .collect()
+}
+
+fn compact_steps<'a, T>(values: &'a [T], vocabulary: usize) -> MlResult<Vec<StepBest>>
+where
+ &'a [T]: FloatTensorData,
+{
+ values
+ .as_chunks::<2>()
+ .0
+ .iter()
+ .map(|step| {
+ let step = step.as_slice();
+ let index = step.value(0);
+ let probability = step.value(1);
+ if !index.is_finite()
+ || index < 0.0
+ || index >= vocabulary as f32
+ || index.fract() != 0.0
+ || !probability.is_finite()
+ || probability < 0.0
+ {
+ return Err(MlError::CorruptModel(format!(
+ "text recognizer produced invalid index/probability [{index}, {probability}]"
+ )));
+ }
+ Ok(StepBest {
+ index: index as usize,
+ probability,
+ })
+ })
+ .collect()
}
fn recognize_in_batches(
@@ -354,7 +388,11 @@ where
&'a [T]: FloatTensorData,
{
let best = best_per_step(logits, vocabulary);
- let spans: Vec<CharacterSpan> = character_runs(&best)
+ decode_steps(&best, dictionary, padding_scale)
+}
+
+fn decode_steps(best: &[StepBest], dictionary: &[String], padding_scale: f32) -> Recognition {
+ let spans: Vec<CharacterSpan> = character_runs(best)
.iter()
.filter_map(|run| run.span(best.len(), dictionary, padding_scale))
.collect();
@@ -534,6 +572,55 @@ mod tests {
assert_eq!(recognition, Recognition::default());
}
+ #[test]
+ fn compact_output_preserves_text_confidences_and_character_spans() {
+ let sequences = [
+ [(1, 0.9), (1, 0.7), (0, 0.8), (1, 0.6), (2, 0.5), (3, 0.9)],
+ [(0, 0.9), (0, 0.8), (0, 0.7), (0, 0.6), (0, 0.5), (0, 0.9)],
+ ];
+ let full: Vec<_> = sequences
+ .iter()
+ .flat_map(|steps| logits(4, steps))
+ .collect();
+ let compact: Vec<_> = sequences
+ .iter()
+ .flatten()
+ .flat_map(|&(index, probability)| [index as f32, probability])
+ .collect();
+ let layout = BatchLayout {
+ target_width: 320,
+ content_widths: vec![160, 320],
+ };
+ let expected = decode_output(&[2, 6, 4], &full, &layout, &dictionary()).unwrap();
+ let actual = decode_output(&[2, 6, 2], &compact, &layout, &dictionary()).unwrap();
+ assert_eq!(actual, expected);
+ assert_eq!(actual[0].text, "aabc");
+ assert_eq!(actual[1], Recognition::default());
+ }
+
+ #[test]
+ fn compact_output_rejects_invalid_indices_probabilities_and_shapes() {
+ let layout = BatchLayout {
+ target_width: 320,
+ content_widths: vec![320],
+ };
+ for values in [
+ [-1.0, 0.9],
+ [4.0, 0.9],
+ [1.5, 0.9],
+ [f32::NAN, 0.9],
+ [f32::INFINITY, 0.9],
+ [1.0, f32::NAN],
+ [1.0, f32::INFINITY],
+ [1.0, -0.1],
+ ] {
+ assert!(decode_output(&[1, 1, 2], &values, &layout, &dictionary()).is_err());
+ }
+ for shape in [[2, 1, 2], [1, 0, 2], [1, 2, 2]] {
+ assert!(decode_output(&shape, &[1.0, 0.9], &layout, &dictionary()).is_err());
+ }
+ }
+
#[test]
fn repeats_separated_by_a_blank_produce_two_characters() {
let steps = [(1, 0.9), (1, 0.9), (0, 0.9), (1, 0.6)];
diff --git a/rust/crates/ml/src/ocr/session.rs b/rust/crates/ml/src/ocr/session.rs
new file mode 100644
index 0000000000..71c7c39cdc
--- /dev/null
+++ b/rust/crates/ml/src/ocr/session.rs
@@ -0,0 +1,168 @@
+use std::collections::VecDeque;
+
+use crate::error::{MlError, MlResult};
+use crate::onnx::{
+ ExecutionMode, GpuOptions, OnnxSession, ProviderUsage, SessionHandle, SessionRunResult,
+};
+
+#[derive(Clone, Copy)]
+pub(super) enum OcrModel {
+ Detection,
+ Classification,
+ Recognition,
+}
+
+impl OcrModel {
+ fn cache_capacity(self) -> usize {
+ match self {
+ Self::Detection => 2,
+ Self::Classification => 6,
+ Self::Recognition => 8,
+ }
+ }
+
+ fn namespace(self) -> &'static str {
+ match self {
+ Self::Detection => "ocr-detection",
+ Self::Classification => "ocr-classification",
+ Self::Recognition => "ocr-recognition",
+ }
+ }
+
+ fn mode(self) -> ExecutionMode {
+ if matches!(self, Self::Classification)
+ && !cfg!(any(target_os = "ios", target_os = "macos"))
+ {
+ ExecutionMode::CpuOnly
+ } else {
+ ExecutionMode::GpuPreferred
+ }
+ }
+
+ fn gpu_options(self, [count, _, height, width]: [i64; 4]) -> GpuOptions {
+ let dimensions = match self {
+ Self::Detection => vec![
+ ("DynamicDimension.0", count),
+ ("DynamicDimension.1", height),
+ ("DynamicDimension.2", width),
+ ],
+ Self::Classification => vec![
+ ("p2o.DynamicDimension.0", count),
+ ("p2o.DynamicDimension.1", height),
+ ("p2o.DynamicDimension.2", width),
+ ],
+ Self::Recognition => vec![("DynamicDimension.0", count), ("DynamicDimension.1", width)],
+ };
+ GpuOptions {
+ dimensions,
+ #[cfg(any(target_os = "android", target_os = "linux", target_os = "windows"))]
+ prefer_nhwc: matches!(self, Self::Detection),
+ }
+ }
+}
+
+pub(super) struct OcrSession {
+ model_path: String,
+ model: OcrModel,
+ sessions: VecDeque<([i64; 4], OnnxSession)>,
+}
+
+impl OcrSession {
+ pub(super) fn new(model_path: &str, model: OcrModel) -> Self {
+ Self {
+ model_path: model_path.to_owned(),
+ model,
+ sessions: VecDeque::new(),
+ }
+ }
+
+ pub(super) fn run<T>(
+ &mut self,
+ shape: [i64; 4],
+ operation: impl FnMut(&mut SessionHandle) -> SessionRunResult<T>,
+ ) -> MlResult<(T, ProviderUsage)> {
+ self.session_for(shape)?.run(operation)
+ }
+
+ fn session_for(&mut self, shape: [i64; 4]) -> MlResult<&mut OnnxSession> {
+ let mode = self.model.mode();
+ let key = if mode == ExecutionMode::CpuOnly {
+ [0; 4]
+ } else {
+ shape
+ };
+ if let Some(index) = self.sessions.iter().position(|(cached, _)| *cached == key) {
+ self.sessions.make_contiguous()[..=index].rotate_right(1);
+ } else {
+ if self.sessions.len() == self.model.cache_capacity() {
+ self.sessions.pop_back();
+ }
+ let [n, c, h, w] = key;
+ let namespace = format!("{}-{n}x{c}x{h}x{w}-gpu-f32-v1", self.model.namespace());
+ let mut session = OnnxSession::new(&self.model_path, &namespace, mode);
+ if mode == ExecutionMode::GpuPreferred {
+ session = session
+ .with_gpu_options(self.model.gpu_options(shape))
+ .with_unvalidated_acceleration();
+ }
+ self.sessions.push_front((key, session));
+ }
+ self.sessions
+ .front_mut()
+ .map(|(_, session)| session)
+ .ok_or_else(|| MlError::Runtime("OCR session cache is empty".to_owned()))
+ }
+}
+
+#[cfg(test)]
+mod tests {
+ use super::*;
+
+ #[test]
+ fn reuses_shapes_and_evicts_the_least_recently_used_session() {
+ let mut cache = OcrSession::new("rec.onnx", OcrModel::Recognition);
+ let first = [6, 3, 48, 320];
+ cache.session_for(first).unwrap().initialize_load_state();
+ for width in 321..328 {
+ cache.session_for([6, 3, 48, width]).unwrap();
+ }
+ assert!(cache.session_for(first).unwrap().has_load_state());
+ cache.session_for([1, 3, 48, 320]).unwrap();
+ assert_eq!(cache.sessions.len(), OcrModel::Recognition.cache_capacity());
+ assert!(cache.session_for(first).unwrap().has_load_state());
+ assert!(
+ !cache
+ .sessions
+ .iter()
+ .any(|(shape, _)| *shape == [6, 3, 48, 321])
+ );
+ }
+
+ #[test]
+ fn models_override_their_actual_spatial_dimensions() {
+ assert_eq!(
+ OcrModel::Detection.gpu_options([1, 3, 960, 704]).dimensions,
+ vec![
+ ("DynamicDimension.0", 1),
+ ("DynamicDimension.1", 960),
+ ("DynamicDimension.2", 704)
+ ]
+ );
+ assert_eq!(
+ OcrModel::Recognition
+ .gpu_options([3, 3, 48, 641])
+ .dimensions,
+ vec![("DynamicDimension.0", 3), ("DynamicDimension.1", 641)]
+ );
+ assert_eq!(
+ OcrModel::Classification
+ .gpu_options([3, 3, 48, 192])
+ .dimensions,
+ vec![
+ ("p2o.DynamicDimension.0", 3),
+ ("p2o.DynamicDimension.1", 48),
+ ("p2o.DynamicDimension.2", 192)
+ ]
+ );
+ }
+}
diff --git a/rust/crates/ml/src/onnx.rs b/rust/crates/ml/src/onnx.rs
index ae18754305..6e2255895e 100644
--- a/rust/crates/ml/src/onnx.rs
+++ b/rust/crates/ml/src/onnx.rs
@@ -24,6 +24,13 @@ pub(crate) use tensor::{
use providers::{ExecutionProvider, ProviderPlan};
+#[derive(Clone, Debug)]
+pub(crate) struct GpuOptions {
+ pub(crate) dimensions: Vec<(&'static str, i64)>,
+ #[cfg(any(target_os = "android", target_os = "linux", target_os = "windows"))]
+ pub(crate) prefer_nhwc: bool,
+}
+
#[derive(Clone, Copy, Debug, PartialEq, Eq)]
enum AccelerationValidation {
GoldenRequired,
@@ -58,6 +65,7 @@ pub(crate) struct OnnxSession {
model_namespace: String,
mode: ExecutionMode,
validation: AccelerationValidation,
+ gpu_options: Option<GpuOptions>,
provider_plan: Option<ProviderPlan>,
session: Option<(Session, ExecutionProvider)>,
first_run_canary: Option<webgpu::ArmedCanary>,
@@ -70,6 +78,7 @@ impl OnnxSession {
model_namespace: model_namespace.to_string(),
mode,
validation: AccelerationValidation::GoldenRequired,
+ gpu_options: None,
provider_plan: None,
session: None,
first_run_canary: None,
@@ -83,6 +92,11 @@ impl OnnxSession {
self
}
+ pub(crate) fn with_gpu_options(mut self, options: GpuOptions) -> Self {
+ self.gpu_options = Some(options);
+ self
+ }
+
pub(crate) fn model_path(&self) -> &str {
&self.model_path
}
@@ -152,8 +166,13 @@ impl OnnxSession {
let model_name = model_file_label(model_path);
log::info!("loading {model_name} with {:?} execution", self.mode);
let started_at = std::time::Instant::now();
- let (loaded, execution_provider) =
- build_next_session(model_path, provider_plan, model_namespace, self.validation)?;
+ let (loaded, execution_provider) = build_next_session(
+ model_path,
+ provider_plan,
+ model_namespace,
+ self.validation,
+ self.gpu_options.as_ref(),
+ )?;
log::info!(
"loaded {model_name} with {execution_provider:?} in {:?}",
started_at.elapsed()
@@ -276,9 +295,15 @@ fn build_next_session(
plan: &mut ProviderPlan,
model_namespace: &str,
validation: AccelerationValidation,
+ gpu_options: Option<&GpuOptions>,
) -> MlResult<(LoadedSession, ExecutionProvider)> {
let result = providers::run_provider_plan(plan, |execution_provider| {
- let attempt = providers::provider_attempt(execution_provider, model_path, model_namespace);
+ let attempt = providers::provider_attempt(
+ execution_provider,
+ model_path,
+ model_namespace,
+ gpu_options,
+ );
if attempt.execution_provider() == ExecutionProvider::WebGpu {
#[cfg(any(target_os = "android", target_os = "linux", target_os = "windows"))]
{
@@ -287,6 +312,7 @@ fn build_next_session(
model_namespace,
attempt,
validation,
+ gpu_options,
);
}
#[cfg(not(any(target_os = "android", target_os = "linux", target_os = "windows")))]
@@ -294,7 +320,7 @@ fn build_next_session(
}
let coreml_cache_dir = attempt.coreml_cache_dir().map(Path::to_path_buf);
- match build_and_validate_session(model_path, attempt, validation) {
+ match build_and_validate_session(model_path, attempt, validation, gpu_options) {
Ok(session) => {
if let Some(cache_dir) = coreml_cache_dir {
coreml_cache::finalize(&cache_dir, model_path);
@@ -348,6 +374,7 @@ fn build_cpu_session(model_path: &str) -> MlResult<Session> {
&mut plan,
"golden-tooling",
AccelerationValidation::GoldenRequired,
+ None,
)
.map(|(loaded, _)| loaded.session)
}
@@ -359,6 +386,7 @@ fn build_and_validate_session(
model_path: &str,
attempt: providers::ProviderAttempt,
_validation: AccelerationValidation,
+ gpu_options: Option<&GpuOptions>,
) -> MlResult<Session> {
#[cfg(any(
target_os = "android",
@@ -369,7 +397,7 @@ fn build_and_validate_session(
))]
let execution_provider = attempt.execution_provider();
- let session = match providers::build_session(model_path, attempt) {
+ let session = match providers::build_session(model_path, attempt, gpu_options) {
Ok(session) => session,
Err(error) => {
#[cfg(any(
@@ -410,6 +438,7 @@ fn build_webgpu_session_with_canary(
model_namespace: &str,
attempt: providers::ProviderAttempt,
validation: AccelerationValidation,
+ gpu_options: Option<&GpuOptions>,
) -> MlResult<LoadedSession> {
// Fail closed: without a durable failure record, a crash during the
// attempt would go unnoticed and the crash loop protection would be lost.
@@ -448,7 +477,7 @@ fn build_webgpu_session_with_canary(
}
}
}
- let mut session = match providers::build_session(model_path, attempt) {
+ let mut session = match providers::build_session(model_path, attempt, gpu_options) {
Ok(session) => session,
Err(error) => {
record_provider_attempt_failure(
diff --git a/rust/crates/ml/src/onnx/providers.rs b/rust/crates/ml/src/onnx/providers.rs
index 0f85ef0cbf..596cdf04d8 100644
--- a/rust/crates/ml/src/onnx/providers.rs
+++ b/rust/crates/ml/src/onnx/providers.rs
@@ -39,7 +39,7 @@ use super::golden_test;
use super::model_file_label;
#[cfg(any(target_os = "android", target_os = "linux", target_os = "windows"))]
use super::webgpu;
-use super::{AccelerationValidation, session_load_error};
+use super::{AccelerationValidation, GpuOptions, session_load_error};
use crate::error::MlResult;
#[cfg(any(target_os = "ios", target_os = "macos"))]
@@ -48,6 +48,7 @@ const ENABLE_PERSISTENT_COREML_CACHE: bool = true;
#[derive(Clone, Copy, Debug, PartialEq, Eq)]
pub(crate) enum ExecutionMode {
PlatformDefault,
+ GpuPreferred,
CpuOnly,
}
@@ -57,10 +58,6 @@ pub(crate) enum ExecutionMode {
pub(super) enum ExecutionProvider {
CoreMl,
WebGpu,
- #[cfg_attr(
- not(any(test, target_os = "android")),
- expect(dead_code, reason = "XNNPACK is only selected on Android")
- )]
Xnnpack,
Cpu,
}
@@ -79,6 +76,10 @@ impl ProviderPlan {
) -> Self {
let providers = match mode {
ExecutionMode::PlatformDefault => platform_default_providers(model_path, validation),
+ ExecutionMode::GpuPreferred => platform_default_providers(model_path, validation)
+ .into_iter()
+ .filter(|provider| *provider != ExecutionProvider::Xnnpack)
+ .collect(),
ExecutionMode::CpuOnly => vec![ExecutionProvider::Cpu],
};
Self::from_providers(providers)
@@ -149,18 +150,23 @@ pub(super) fn provider_attempt(
provider: ExecutionProvider,
_model_path: &str,
_model_namespace: &str,
+ _gpu_options: Option<&GpuOptions>,
) -> ProviderAttempt {
match provider {
ExecutionProvider::Cpu => ProviderAttempt::cpu_only(),
#[cfg(any(target_os = "ios", target_os = "macos"))]
ExecutionProvider::CoreMl => {
let (coreml_provider, coreml_cache_dir) =
- coreml_provider(_model_path, _model_namespace);
+ coreml_provider(_model_path, _model_namespace, _gpu_options.is_some());
ProviderAttempt {
- providers: vec![
- coreml_provider,
- CPU::default().with_arena_allocator(true).build(),
- ],
+ providers: if _gpu_options.is_some() {
+ vec![coreml_provider]
+ } else {
+ vec![
+ coreml_provider,
+ CPU::default().with_arena_allocator(true).build(),
+ ]
+ },
disable_intra_op_spinning: false,
coreml_cache_dir,
execution_provider: ExecutionProvider::CoreMl,
@@ -168,7 +174,11 @@ pub(super) fn provider_attempt(
}
#[cfg(any(target_os = "android", target_os = "linux", target_os = "windows"))]
ExecutionProvider::WebGpu => ProviderAttempt {
- providers: webgpu_attempt_providers(),
+ providers: if let Some(options) = _gpu_options {
+ vec![webgpu_provider(options.prefer_nhwc)]
+ } else {
+ webgpu_attempt_providers()
+ },
disable_intra_op_spinning: true,
coreml_cache_dir: None,
execution_provider: ExecutionProvider::WebGpu,
@@ -179,12 +189,28 @@ pub(super) fn provider_attempt(
}
}
-pub(super) fn build_session(model_path: &str, attempt: ProviderAttempt) -> MlResult<Session> {
+pub(super) fn build_session(
+ model_path: &str,
+ attempt: ProviderAttempt,
+ gpu_options: Option<&GpuOptions>,
+) -> MlResult<Session> {
let mut builder = Session::builder()?
.with_optimization_level(GraphOptimizationLevel::All)?
.with_intra_threads(1)?
.with_inter_threads(1)?;
+ if let Some(options) = gpu_options
+ && matches!(
+ attempt.execution_provider,
+ ExecutionProvider::CoreMl | ExecutionProvider::WebGpu
+ )
+ {
+ builder = builder.with_disable_cpu_fallback()?;
+ for &(name, value) in &options.dimensions {
+ builder = builder.with_dimension_override(name, value)?;
+ }
+ }
+
if attempt.disable_intra_op_spinning {
builder = builder.with_intra_op_spinning(false)?;
}
@@ -284,12 +310,20 @@ fn accelerated_provider_allowed(
fn coreml_provider(
model_path: &str,
model_namespace: &str,
+ full_gpu: bool,
) -> (ExecutionProviderDispatch, Option<PathBuf>) {
let mut provider = CoreML::default()
.with_model_format(ModelFormat::MLProgram)
.with_compute_units(ComputeUnits::All)
.with_specialization_strategy(SpecializationStrategy::Default);
+ if full_gpu {
+ provider = provider
+ .with_compute_units(ComputeUnits::CPUAndGPU)
+ .with_low_precision_accumulation_on_gpu(false)
+ .with_static_input_shapes(true);
+ }
+
let mut prepared_cache_dir = None;
if ENABLE_PERSISTENT_COREML_CACHE {
match coreml_cache::prepare_directory(model_path, model_namespace) {
@@ -312,8 +346,13 @@ fn coreml_provider(
}
#[cfg(any(target_os = "android", target_os = "linux", target_os = "windows"))]
-fn webgpu_provider() -> ExecutionProviderDispatch {
- let provider = WebGPU::default().with_preferred_layout(PreferredLayout::NCHW);
+fn webgpu_provider(prefer_nhwc: bool) -> ExecutionProviderDispatch {
+ let layout = if prefer_nhwc {
+ PreferredLayout::NHWC
+ } else {
+ PreferredLayout::NCHW
+ };
+ let provider = WebGPU::default().with_preferred_layout(layout);
#[cfg(any(target_os = "android", target_os = "linux"))]
let provider = provider.with_dawn_backend_type(DawnBackendType::Vulkan);
#[cfg(target_os = "windows")]
@@ -324,7 +363,7 @@ fn webgpu_provider() -> ExecutionProviderDispatch {
#[cfg(target_os = "android")]
fn webgpu_attempt_providers() -> Vec<ExecutionProviderDispatch> {
vec![
- webgpu_provider(),
+ webgpu_provider(false),
xnnpack_provider().fail_silently(),
CPU::default().with_arena_allocator(true).build(),
]
@@ -333,7 +372,7 @@ fn webgpu_attempt_providers() -> Vec<ExecutionProviderDispatch> {
#[cfg(any(target_os = "linux", target_os = "windows"))]
fn webgpu_attempt_providers() -> Vec<ExecutionProviderDispatch> {
vec![
- webgpu_provider(),
+ webgpu_provider(false),
CPU::default().with_arena_allocator(true).build(),
]
}
@@ -366,6 +405,28 @@ mod tests {
AccelerationValidation, ExecutionMode, ExecutionProvider, ProviderPlan, run_provider_plan,
};
+ #[test]
+ fn gpu_preferred_falls_back_directly_to_cpu() {
+ let mut plan = ProviderPlan::new(
+ ExecutionMode::GpuPreferred,
+ "model.onnx",
+ AccelerationValidation::Unvalidated,
+ );
+ let mut attempted = Vec::new();
+ let selected = run_provider_plan(&mut plan, |provider| {
+ attempted.push(provider);
+ if provider == ExecutionProvider::Cpu {
+ Ok(provider)
+ } else {
+ Err(())
+ }
+ })
+ .unwrap();
+ assert_eq!(selected, (ExecutionProvider::Cpu, ExecutionProvider::Cpu));
+ assert!(!attempted.contains(&ExecutionProvider::Xnnpack));
+ assert!(!plan.has_fallback());
+ }
+
#[test]
fn construction_falls_through_and_selects_the_successful_provider() {
let mut plan = accelerated_provider_plan();