π End-to-end encrypted cloud for everything.
Dart 2faandroidauthye2eeencryptionend-to-end-encryptionfluttergolanggoogle-photosgoogle-photos-alternativeiosopensourcephotosprivacyprivacy-toolstypescriptzero-knowledge
[photos] Keep OCR testing on Android and simplify model docs
laurenspriemc2990ab4fa3a
diff --git a/infra/ml/playground/optimizations/README.md b/infra/ml/playground/optimizations/README.mdindex 89c468eea8..f58323e0ba 100644--- a/infra/ml/playground/optimizations/README.md+++ b/infra/ml/playground/optimizations/README.md@@ -17,84 +17,21 @@ uv run --no-project --with numpy==2.5.3 --with onnx==1.22.0 python \--output-dir infra/ml/playground/optimizations/models/ocr```-Missing source files are downloaded from `https://models.ente.com/PP-OCRv5`.-The script verifies every source hash and requires every generated model to-match the SHA-256 of its qualified artifact. It writes `det.onnx`, `cls.onnx`,-`rec.onnx`, the unchanged dictionary, and `ocr_model_manifest.json`. Both-platforms download identical model bytes. Upload the ONNX files to new versioned-URLs; the recognizer has a new output contract and must not replace the old URL.--The selected transformations are:--- Detection: fold the final transposed-convolution bias and batch normalization- into the head weights. Preserve the original backbone affine expressions.-- Classification: canonicalize scalar affine constants and express 18 expanded- activations as `HardSigmoid(alpha=1/6, beta=0.5) * x`.-- Recognition: canonicalize scalar affine constants, fold input affine operations- into 12 unpadded convolutions, and replace four expanded normalization blocks- with `LayerNormalization` using the original parameters and epsilon.-- Detection and recognition: expand `HardSwish` into- `HardSigmoid(alpha=1/6, beta=0.5) * x` for CoreML compatibility.-- Recognition: pad the final weight matrix's 18,385 output columns to 18,432 for- the existing WebGPU runtime's vectorized matrix multiplication, then slice- back to 18,385 before the original bias and softmax.-- Recognition: reduce the original probabilities to the first winning token- index and its probability for each time step, using FP32 arithmetic supported- by both GPU providers.--The recognizer returns FP32 `[N,T,2]`, with index first and probability second.-For probabilities `p`, maximum `m`, vocabulary size `V`, and token index `i`, the-compact head computes `min(i + V * ceil(m - p[i]))`. Since probabilities are in-`[0,1]`, the ceiling is zero for maxima and one for every lower value. The-minimum therefore selects the first tied maximum without integer casts or-boolean selection. The other output is the original maximum probability.-See the [ONNX Ceil definition](https://onnx.ai/onnx/operators/onnx__Ceil.html).--The shared models retain FP32 parameters, vocabulary, CTC blank/repeat handling,-confidence averaging, and character-span rules. These transformations preserve-the mathematical operations but can introduce small floating-point differences.--The Rust OCR caller keeps its 960-pixel longest-side detector limit and existing-recognition crop sizes. It specializes sessions for actual dimensions without-changing input pixels and retains two detector shapes, six classifier batch-sizes, and eight recently used recognizer shapes.-CoreML uses MLProgram, CPUAndGPU, and disables low-precision GPU accumulation.-WebGPU uses NHWC internally for detection and NCHW for recognition. GPU session-construction requires full provider coverage; failures retry on a single CPU-thread. OCR never selects XNNPACK. Android classification uses the single-thread-CPU because it was faster than WebGPU on Pixel 8; it uses the same classifier-file as CoreML. Internal iOS uses CoreML for all three stages.--Qualification uses unchanged ONNX Runtime 1.28.1 on Pixel 8 with native-Dawn/Vulkan WebGPU and Apple M4 with CoreML. The saved model comparison covers-317 crop sources from 17 unique photos, 21,256 recognition time steps, 320-classification inputs, and 9,383,936 detector probabilities. Neither provider-changed any winning token, orientation decision, or detector threshold decision-in those cases. Every CoreML case formed one complete partition using GPU-compute devices; both providers completed with CPU fallback disabled. Separate-compact output checks matched all 144 rows exactly, including ties, uniform-probabilities, and adjacent FP32 values. These checks do not establish bit-identical outputs on-all images or performance on an iPhone. Smaller detector inputs were excluded-because they missed text in the supplied photos.--The shared artifacts produced these warm native inference medians on 10-September 2026. Each range contains two trial medians, with 40 calls per trial-and the first 10 discarded. Runs use one CPU inference thread and include the-normal CPU input/output transfer; model loading, compilation, image decoding,-and Rust preprocessing/postprocessing are excluded. CoreML measurements are-from an Apple M4 Mac, not an iPhone.--| Stage and NCHW input | M4 CoreML GPU | Pixel 8 native WebGPU |-| --- | ---: | ---: |-| Empty-image detection, 1Γ3Γ960Γ704 | 10.4β10.9 ms | 128.6β132.2 ms |-| Empty-image detection, 1Γ3Γ960Γ960 | 13.3β13.6 ms | 173.3β175.0 ms |-| Classification, 6Γ3Γ48Γ192 | 1.4β2.0 ms | 19.3β33.9 ms |-| Recognition, 6Γ3Γ48Γ320 | 9.5β9.6 ms | 104.1β104.8 ms |-| Recognition, 6Γ3Γ48Γ640 | 16.8β17.0 ms | 189.5β190.4 ms |--The same classifier took 8.7β8.8 ms on the Pixel's single-thread CPU, which is-why the Android caller keeps that stage on CPU. Both uniform-gray detection-inputs produced no pixels at or above the 0.3 threshold.+The script keeps FP32 weights and applies these transformations:++- Detection: fold the transposed-convolution head's bias and batch normalization,+ and expand `HardSwish` into `HardSigmoid * x` for CoreML compatibility.+- Classification: canonicalize scalar constants and simplify expanded activations+ to `HardSigmoid * x`.+- Recognition: fold affine operations into unpadded convolutions, fuse layer+ normalization, expand `HardSwish`, and align the final matrix multiplication+ for WebGPU without changing the vocabulary. Return FP32 `[N,T,2]` containing+ the first winning token index and its probability; Rust retains CTC decoding.++Missing sources are downloaded from `https://models.ente.com/PP-OCRv5`. The script+verifies source and output hashes and writes three ONNX files, the unchanged+dictionary, and `ocr_model_manifest.json`. Publish the ONNX files at new URLs:+the recognizer's output contract differs from the original model.## Rebuilding the modelsdiff --git a/mobile/apps/photos/changes/ocr-gpu-internal.md b/mobile/apps/photos/changes/ocr-gpu-internal.mddeleted file mode 100644index 83b2b3d571..0000000000--- a/mobile/apps/photos/changes/ocr-gpu-internal.md+++ /dev/null@@ -1 +0,0 @@-- Internal users can test GPU-accelerated text recognition on Android and iOS.diff --git a/mobile/apps/photos/lib/services/machine_learning/ocr_service.dart b/mobile/apps/photos/lib/services/machine_learning/ocr_service.dartindex 82247bfd7d..fc0f05a1ec 100644--- a/mobile/apps/photos/lib/services/machine_learning/ocr_service.dart+++ b/mobile/apps/photos/lib/services/machine_learning/ocr_service.dart@@ -52,9 +52,12 @@ class OcrService {if (!_rustOcrEnabled()) {return OcrBackendKind.legacy;}- if (_isAndroid || _isIOS) {+ if (_isAndroid) {return OcrBackendKind.rust;}+ if (_isIOS) {+ return OcrBackendKind.vision;+ }return OcrBackendKind.legacy;}