Skip to content

armgit / ente

Public Imported from GitHub
0

πŸ’š End-to-end encrypted cloud for everything.

[photos] Keep OCR testing on Android and simplify model docs

laurenspriemc2990ab4fa3a
diff --git a/infra/ml/playground/optimizations/README.md b/infra/ml/playground/optimizations/README.md
index 89c468eea8..f58323e0ba 100644
--- a/infra/ml/playground/optimizations/README.md
+++ b/infra/ml/playground/optimizations/README.md
@@ -17,84 +17,21 @@ uv run --no-project --with numpy==2.5.3 --with onnx==1.22.0 python \
--output-dir infra/ml/playground/optimizations/models/ocr
```
-Missing source files are downloaded from `https://models.ente.com/PP-OCRv5`.
-The script verifies every source hash and requires every generated model to
-match the SHA-256 of its qualified artifact. It writes `det.onnx`, `cls.onnx`,
-`rec.onnx`, the unchanged dictionary, and `ocr_model_manifest.json`. Both
-platforms download identical model bytes. Upload the ONNX files to new versioned
-URLs; the recognizer has a new output contract and must not replace the old URL.
-
-The selected transformations are:
-
-- Detection: fold the final transposed-convolution bias and batch normalization
- into the head weights. Preserve the original backbone affine expressions.
-- Classification: canonicalize scalar affine constants and express 18 expanded
- activations as `HardSigmoid(alpha=1/6, beta=0.5) * x`.
-- Recognition: canonicalize scalar affine constants, fold input affine operations
- into 12 unpadded convolutions, and replace four expanded normalization blocks
- with `LayerNormalization` using the original parameters and epsilon.
-- Detection and recognition: expand `HardSwish` into
- `HardSigmoid(alpha=1/6, beta=0.5) * x` for CoreML compatibility.
-- Recognition: pad the final weight matrix's 18,385 output columns to 18,432 for
- the existing WebGPU runtime's vectorized matrix multiplication, then slice
- back to 18,385 before the original bias and softmax.
-- Recognition: reduce the original probabilities to the first winning token
- index and its probability for each time step, using FP32 arithmetic supported
- by both GPU providers.
-
-The recognizer returns FP32 `[N,T,2]`, with index first and probability second.
-For probabilities `p`, maximum `m`, vocabulary size `V`, and token index `i`, the
-compact head computes `min(i + V * ceil(m - p[i]))`. Since probabilities are in
-`[0,1]`, the ceiling is zero for maxima and one for every lower value. The
-minimum therefore selects the first tied maximum without integer casts or
-boolean selection. The other output is the original maximum probability.
-See the [ONNX Ceil definition](https://onnx.ai/onnx/operators/onnx__Ceil.html).
-
-The shared models retain FP32 parameters, vocabulary, CTC blank/repeat handling,
-confidence averaging, and character-span rules. These transformations preserve
-the mathematical operations but can introduce small floating-point differences.
-
-The Rust OCR caller keeps its 960-pixel longest-side detector limit and existing
-recognition crop sizes. It specializes sessions for actual dimensions without
-changing input pixels and retains two detector shapes, six classifier batch
-sizes, and eight recently used recognizer shapes.
-CoreML uses MLProgram, CPUAndGPU, and disables low-precision GPU accumulation.
-WebGPU uses NHWC internally for detection and NCHW for recognition. GPU session
-construction requires full provider coverage; failures retry on a single CPU
-thread. OCR never selects XNNPACK. Android classification uses the single-thread
-CPU because it was faster than WebGPU on Pixel 8; it uses the same classifier
-file as CoreML. Internal iOS uses CoreML for all three stages.
-
-Qualification uses unchanged ONNX Runtime 1.28.1 on Pixel 8 with native
-Dawn/Vulkan WebGPU and Apple M4 with CoreML. The saved model comparison covers
-317 crop sources from 17 unique photos, 21,256 recognition time steps, 320
-classification inputs, and 9,383,936 detector probabilities. Neither provider
-changed any winning token, orientation decision, or detector threshold decision
-in those cases. Every CoreML case formed one complete partition using GPU
-compute devices; both providers completed with CPU fallback disabled. Separate
-compact output checks matched all 144 rows exactly, including ties, uniform
-probabilities, and adjacent FP32 values. These checks do not establish bit-identical outputs on
-all images or performance on an iPhone. Smaller detector inputs were excluded
-because they missed text in the supplied photos.
-
-The shared artifacts produced these warm native inference medians on 10
-September 2026. Each range contains two trial medians, with 40 calls per trial
-and the first 10 discarded. Runs use one CPU inference thread and include the
-normal CPU input/output transfer; model loading, compilation, image decoding,
-and Rust preprocessing/postprocessing are excluded. CoreML measurements are
-from an Apple M4 Mac, not an iPhone.
-
-| Stage and NCHW input | M4 CoreML GPU | Pixel 8 native WebGPU |
-| --- | ---: | ---: |
-| Empty-image detection, 1Γ—3Γ—960Γ—704 | 10.4–10.9 ms | 128.6–132.2 ms |
-| Empty-image detection, 1Γ—3Γ—960Γ—960 | 13.3–13.6 ms | 173.3–175.0 ms |
-| Classification, 6Γ—3Γ—48Γ—192 | 1.4–2.0 ms | 19.3–33.9 ms |
-| Recognition, 6Γ—3Γ—48Γ—320 | 9.5–9.6 ms | 104.1–104.8 ms |
-| Recognition, 6Γ—3Γ—48Γ—640 | 16.8–17.0 ms | 189.5–190.4 ms |
-
-The same classifier took 8.7–8.8 ms on the Pixel's single-thread CPU, which is
-why the Android caller keeps that stage on CPU. Both uniform-gray detection
-inputs produced no pixels at or above the 0.3 threshold.
+The script keeps FP32 weights and applies these transformations:
+
+- Detection: fold the transposed-convolution head's bias and batch normalization,
+ and expand `HardSwish` into `HardSigmoid * x` for CoreML compatibility.
+- Classification: canonicalize scalar constants and simplify expanded activations
+ to `HardSigmoid * x`.
+- Recognition: fold affine operations into unpadded convolutions, fuse layer
+ normalization, expand `HardSwish`, and align the final matrix multiplication
+ for WebGPU without changing the vocabulary. Return FP32 `[N,T,2]` containing
+ the first winning token index and its probability; Rust retains CTC decoding.
+
+Missing sources are downloaded from `https://models.ente.com/PP-OCRv5`. The script
+verifies source and output hashes and writes three ONNX files, the unchanged
+dictionary, and `ocr_model_manifest.json`. Publish the ONNX files at new URLs:
+the recognizer's output contract differs from the original model.
## Rebuilding the models
diff --git a/mobile/apps/photos/changes/ocr-gpu-internal.md b/mobile/apps/photos/changes/ocr-gpu-internal.md
deleted file mode 100644
index 83b2b3d571..0000000000
--- a/mobile/apps/photos/changes/ocr-gpu-internal.md
+++ /dev/null
@@ -1 +0,0 @@
-- Internal users can test GPU-accelerated text recognition on Android and iOS.
diff --git a/mobile/apps/photos/lib/services/machine_learning/ocr_service.dart b/mobile/apps/photos/lib/services/machine_learning/ocr_service.dart
index 82247bfd7d..fc0f05a1ec 100644
--- a/mobile/apps/photos/lib/services/machine_learning/ocr_service.dart
+++ b/mobile/apps/photos/lib/services/machine_learning/ocr_service.dart
@@ -52,9 +52,12 @@ class OcrService {
if (!_rustOcrEnabled()) {
return OcrBackendKind.legacy;
}
- if (_isAndroid || _isIOS) {
+ if (_isAndroid) {
return OcrBackendKind.rust;
}
+ if (_isIOS) {
+ return OcrBackendKind.vision;
+ }
return OcrBackendKind.legacy;
}