Skip to content

armgit / ente

Public Imported from GitHub
0

đź’š End-to-end encrypted cloud for everything.

Further experiments

laurenspriem0ee9992b7a0c
diff --git a/infra/ml/playground/optimizations/benchmark_reports/20260717_ANDROID_WEBGPU_BENCHMARK_REPORT.md b/infra/ml/playground/optimizations/benchmark_reports/20260717_ANDROID_WEBGPU_BENCHMARK_REPORT.md
index b75f49056c..d9cc41a72d 100644
--- a/infra/ml/playground/optimizations/benchmark_reports/20260717_ANDROID_WEBGPU_BENCHMARK_REPORT.md
+++ b/infra/ml/playground/optimizations/benchmark_reports/20260717_ANDROID_WEBGPU_BENCHMARK_REPORT.md
@@ -1,4 +1,4 @@
-# Android WebGPU Release Smoke Test and Benchmark
+# Android WebGPU Release Smoke Test, MobileFaceNet Rewrite, and Tuning Benchmark
- **Date:** 17 July 2026
- **Device:** Pixel 8 (`shiba`), Google Tensor G3, Mali-G715, Android 17 / API 37
@@ -8,8 +8,266 @@
- **WebGPU configuration:** forced `true` in the parity runner for this investigation
- **Raw evidence:** [`infra/ml/test/out/android_webgpu_release_2026-07-17/`](../../../test/out/android_webgpu_release_2026-07-17/)
- **Machine-readable summary:** [`benchmark_summary.json`](../../../test/out/android_webgpu_release_2026-07-17/benchmark_summary.json)
+- **Follow-up revision:** `18e246ff6432`, plus the model rewrite and temporary tuning instrumentation described below
+- **Follow-up evidence:** [`infra/ml/test/out/android_webgpu_tuning_2026-07-17/`](../../../test/out/android_webgpu_tuning_2026-07-17/)
+- **Follow-up summary:** [`benchmark_summary.json`](../../../test/out/android_webgpu_tuning_2026-07-17/benchmark_summary.json)
-## Outcome
+## Follow-up outcome: MobileFaceNet fixed; provider tuning is effectively neutral
+
+The follow-up changes the most important conclusion from the initial smoke
+test. MobileFaceNet no longer needs a CPU-specific provider policy. Rewriting
+its 33 unsupported `PRelu` nodes as an exact, portable arithmetic expression
+makes the entire optimized graph execute on WebGPU and reduces median FP32
+MobileFaceNet inference from **99.5 ms to 14.4 ms per face**. That is an 85.5%
+latency reduction, or a **6.91x speedup**, with output-identical full-corpus
+results.
+
+Verbose ONNX Runtime placement diagnostics reported:
+
+| Model | Optimized nodes placed on WebGPU | CPU nodes | Result |
+|---|---:|---:|---|
+| Static YOLO | 277 | 0 | Fully WebGPU |
+| MobileCLIP | 660 | 0 | Fully WebGPU |
+| Portable MobileFaceNet | 231 | 0 | **Fully WebGPU** |
+
+The complete release suite then compared graph capture, preferred layout,
+FP16, all four bucket buffer caches, and preallocated GPU I/O. Each variant ran
+the full 14-fixture corpus with one warm-up and five measured passes per
+fixture. All runs began and ended at Android thermal status `0`.
+
+The actionable result is:
+
+- Keep **FP32**, following the requested accuracy-first final policy. FP16 was
+ much faster, but two very small faces in a screenshot produced face-embedding
+ cosines near `0.959` relative to FP32.
+- Keep **NCHW**. NHWC made every model's median inference 0.36–0.41% slower.
+- Keep normal ONNX Runtime-managed input/output. Explicit GPU I/O
+ preallocation merely moved synchronization from `Session::run` into the
+ device-to-host copy and did not improve total latency.
+- Graph capture and bucket caches produced small, parity-exact isolated gains,
+ so they were selected for the requested final combination. The combination
+ itself was effectively neutral: 0.38% faster end to end without CR2, while
+ YOLO and CLIP total-inference medians were 0.77% and 0.50% slower.
+- Do **not** enable graph capture or bucket caches in production based on this
+ single-device result. Their isolated changes are within run-to-run noise and
+ did not add up in the final controlled configuration.
+
+The recommended Android policy after this follow-up is therefore the simple
+one: **portable FP32 models, NCHW, WebGPU -> XNNPACK -> CPU fallback, default
+buffer caches, and normal I/O**. The model rewrite is a clear win; the tested
+provider knobs are not.
+
+The reproducible generator and manifest now describe
+`mobilefacenet_portable_static_b1.onnx`. The app's production model URL/SHA was
+not changed because this run did not upload a new CDN artifact. Deployment
+requires uploading the generated file and then updating `FaceEmbeddingModel`
+to its final remote filename and SHA-256.
+
+## Portable MobileFaceNet operator rewrite
+
+Replacing PReLU with one plain `Relu` would be incorrect whenever the learned
+slope is nonzero. The exact identity is:
+
+```text
+PReLU(x, alpha) = Relu(x) - alpha * Relu(x * -1)
+```
+
+The rewrite deliberately uses `Mul(x, -1)` instead of `Neg(x)`. `Relu`, `Mul`,
+and `Sub` are all listed by both the
+[CoreML MLProgram operator table](https://onnxruntime.ai/docs/execution-providers/CoreML-ExecutionProvider.html#mlprogram)
+and the
+[WebGPU operator table](https://github.com/microsoft/onnxruntime/blob/main/js/web/docs/webgpu-operators.md),
+whereas `Neg` is not listed for CoreML MLProgram. A scalar `LeakyRelu` is not
+equivalent because MobileFaceNet has learned per-channel slopes. `Where` also
+lacks documented CoreML MLProgram coverage. The selected five-node expression
+is therefore the smallest exact common representation found in the two
+published support lists.
+
+Each original six-node `Relu`/`Abs`/`Sub`/`Mul`/`Mul`/`Add` activation becomes
+five nodes: positive `Relu`, multiply by `-1`, negative `Relu`, slope `Mul`, and
+`Sub`. The `[1, C, 1, 1]` slope tensors are preserved exactly.
+
+| Property | Previous WebGPU model | Portable model |
+|---|---:|---:|
+| Serialized ONNX nodes | 101 | 233 |
+| `PRelu` nodes | 33 | 0 |
+| Portable activation nodes per channel block | 1 unsupported `PRelu` | 5 supported arithmetic nodes |
+| Model bytes | 5,238,749 | 5,278,803 |
+| SHA-256 | `f525daf3…ccfd` | `0763fc33…a36` |
+| ORT execution placement | 132 WebGPU / 33 CPU | **231 WebGPU / 0 CPU** |
+| FP32 steady total-inference p50 | 99.5 ms | **14.4 ms** |
+
+Serialized node counts and ORT optimized-plan node counts are different
+quantities. The placement result is the relevant runtime comparison: 33 CPU
+nodes before the rewrite and zero after it.
+
+Validation was performed at three levels:
+
+1. `onnx.checker.check_model(..., full_check=True)` passes.
+2. Twenty deterministic random inputs compared against the prior optimized
+ PReLU model had maximum absolute difference `0.0`; minimum cosine was within
+ floating-point rounding of 1 (maximum cosine distance `5.960e-8`).
+3. The 14-fixture Android corpus produced identical face counts, CLIP outputs,
+ and all 25 face embeddings relative to the earlier FP32 Android result. The
+ minimum reported cosine was `0.9999999999999998`.
+
+CoreML compatibility in this follow-up is based on the published MLProgram
+operator list and ONNX validation. The new graph was benchmarked and its full
+GPU placement proven on Android WebGPU; it was not re-profiled on an iPhone in
+this run.
+
+## Follow-up benchmark method
+
+All tuning APKs were Flutter AOT release builds with release Rust. The runner
+asserted `dart.vm.product == true` and `dart.vm.profile == false`; the final APK
+was non-debuggable according to `apkanalyzer`. The final FP32-winners APK was
+243,478,077 bytes with SHA-256
+`e2a805408ccf71e2d48822117cd2a54feaccabdde699c41cb16a100b8974b5af`.
+
+For every variant:
+
+- WebGPU was temporarily forced on in the integration runner.
+- App data was cleared, giving process-cold sessions and model loads.
+- All 14 fixtures ran six times: one unmeasured warm-up and five measured
+ passes.
+- Exact native timers covered input tensor preparation, synchronous inference,
+ output access/copy, and their combined total for every model call.
+- The measured corpus contains 70 YOLO calls, 70 MobileCLIP calls, and 125
+ MobileFaceNet calls after warm-up removal.
+- Results, complete logcat, and before/after thermal snapshots were retained
+ for each run.
+
+The end-to-end column below sums one like-numbered measured pass for every
+fixture and reports the median of the five corpus rounds. CR2 is excluded from
+that column because the failed native RAW path contributes a variable external
+timeout of roughly 6–16 seconds before the successful JPEG fallback. All model
+inference values and correctness checks still include the CR2 fallback.
+
+## Isolated tuning results
+
+“Inference” is the complete native inference timer, not `Session::run` alone.
+Percentages are versus the portable FP32/NCHW/default-cache/normal-I/O baseline.
+
+| Variant | YOLO p50 | Change | MobileCLIP p50 | Change | MobileFaceNet p50 | Change | Corpus p50, no CR2 | Change | Output parity vs FP32 baseline |
+|---|---:|---:|---:|---:|---:|---:|---:|---:|---|
+| **FP32 baseline** | 108.111 ms | — | 136.637 ms | — | 14.397 ms | — | 9,676.216 ms | — | Exact |
+| Graph capture | 108.161 ms | +0.05% | **134.893 ms** | **-1.28%** | 14.373 ms | -0.16% | **9,597.319 ms** | **-0.82%** | Exact |
+| Preferred NHWC | 108.540 ms | +0.40% | 137.198 ms | +0.41% | 14.449 ms | +0.36% | 9,704.058 ms | +0.29% | Exact |
+| All four bucket caches | 108.166 ms | +0.05% | 137.276 ms | +0.47% | **13.738 ms** | **-4.58%** | 9,613.981 ms | -0.64% | Exact |
+| Preallocated GPU I/O | 108.465 ms | +0.33% | 137.751 ms | +0.81% | 13.838 ms | -3.88% | 9,655.368 ms | -0.22% | Exact |
+| Full FP16 models | **78.299 ms** | **-27.58%** | **102.451 ms** | **-25.02%** | **12.427 ms** | **-13.68%** | **8,727.863 ms** | **-9.80%** | Face accuracy concern |
+| **Final FP32 combination**: graph capture + bucket caches + NCHW + normal I/O | 108.946 ms | +0.77% | 137.317 ms | +0.50% | 14.373 ms | -0.17% | 9,638.985 ms | -0.38% | Exact |
+
+The baseline's per-model p95 total-inference values were 114.050 ms for YOLO,
+145.308 ms for MobileCLIP, and 19.474 ms for MobileFaceNet. The final FP32
+combination's corresponding values were 114.048, 146.538, and 18.956 ms. There
+is no hidden tail-latency win large enough to change the recommendation.
+
+Using the median model totals and this corpus's 14 YOLO, 14 CLIP, and 25 face
+calls, the portable FP32 baseline needs about **3.786 seconds** of model
+inference per corpus. The initial split-MobileFaceNet run needed 5.862 seconds.
+The operator rewrite therefore reduces aggregate model time by approximately
+35.4% on this face distribution.
+
+### Graph capture
+
+Graph capture was parity-exact. It improved MobileCLIP by 1.28% and the
+no-CR2 corpus by 0.82%, while YOLO and MobileFaceNet were flat. Those changes
+were small enough to be run-order noise, and the final combination did not
+retain the CLIP improvement. It should remain off until repeated across devices
+or supported by production telemetry.
+
+### NCHW versus NHWC
+
+NCHW won every pure-inference comparison. NHWC increased median total
+inference by 0.40% for YOLO, 0.41% for MobileCLIP, and 0.36% for MobileFaceNet.
+The Pixel 8's WebGPU/Vulkan implementation should keep NCHW for these exported
+graphs.
+
+### Bucket buffer caches
+
+Setting storage, uniform, query-resolve, and default buffer caches to `bucket`
+was parity-exact. MobileFaceNet improved 4.58% in isolation, but YOLO was flat
+and CLIP regressed 0.47%. The improvement did not survive the final combined
+run, where MobileFaceNet was only 0.17% faster than baseline. The added cache
+policy is not justified by this evidence.
+
+### Preallocated WebGPU I/O
+
+The preallocation experiment used persistent CPU and `WebGPU_Buffer` tensors,
+`IoBinding`, and synchronous `CopyTensors` calls around each inference. It
+worked and was output-identical, but the timing boundary exposed why it did not
+help:
+
+| Model | Policy | Input p50 | `Session::run` p50 | Output/sync p50 | Complete inference p50 |
+|---|---|---:|---:|---:|---:|
+| YOLO | Normal ORT I/O | 0.031 ms | 107.385 ms | 0.800 ms | **108.111 ms** |
+| YOLO | Preallocated GPU I/O | 2.212 ms | 13.610 ms | 92.544 ms | 108.465 ms |
+| MobileCLIP | Normal ORT I/O | 0.024 ms | 136.603 ms | 0.008 ms | **136.637 ms** |
+| MobileCLIP | Preallocated GPU I/O | 0.679 ms | 30.162 ms | 105.752 ms | 137.751 ms |
+| MobileFaceNet | Normal ORT I/O | 0.018 ms | 14.387 ms | 0.003 ms | 14.397 ms |
+| MobileFaceNet | Preallocated GPU I/O | 0.405 ms | 7.406 ms | 5.619 ms | **13.838 ms** |
+
+With explicit GPU output tensors, `Session::run` returns before the final
+device work and readback are synchronized. The output copy then pays that cost.
+Reporting only the run column would falsely claim an approximately 8x YOLO
+speedup; the complete timer correctly shows a 0.33% regression. Keeping normal
+ORT-managed I/O is simpler and faster overall for the current CPU
+preprocessing/postprocessing pipeline.
+
+### FP32 versus FP16
+
+All three models were converted to genuine FP16 graphs with FP16 inputs,
+outputs, weights, and intermediates; this was not merely an input cast. Each
+passed full ONNX checking and executed the complete corpus.
+
+| Model | FP32 bytes | FP16 bytes | FP32 p50 | FP16 p50 | Latency change |
+|---|---:|---:|---:|---:|---:|
+| YOLO | 32,355,091 | 16,229,361 | 108.111 ms | **78.299 ms** | **-27.58%** |
+| MobileCLIP | 143,061,211 | 71,753,862 | 136.637 ms | **102.451 ms** | **-25.02%** |
+| MobileFaceNet | 5,278,803 | 2,723,399 | 14.397 ms | **12.427 ms** | **-13.68%** |
+
+FP16 preserved all 14 face counts. MobileCLIP parity was strong: median cosine
+`0.9998346`, minimum `0.9997101`. Face embeddings had median cosine
+`0.9991900`, but the minimum was `0.9590737`; the two worst cases were very
+small faces in `ui_app.webp`. Detector rounding slightly changes their boxes
+and landmarks, and that crop difference is amplified by the embedding model.
+
+Following the requested final policy, FP16 was therefore **not** included in
+the combined run. It remains the only tested setting with a large speedup and
+is worth a future mixed-precision accuracy study—for example FP16 YOLO/CLIP
+with FP32 MobileFaceNet and explicit face-quality thresholds—but it should not
+replace the current FP32 face pipeline without a broader retrieval/identity
+evaluation.
+
+## Final FP32 combination
+
+The full final test used FP32 models, graph capture, all bucket caches, NCHW,
+and normal ORT I/O. It completed all 14 fixtures and all expected model calls,
+remained at thermal status `0`, and produced exact FP32 baseline parity:
+
+| Correctness check | Result |
+|---|---:|
+| Fixtures with identical face counts | 14/14 |
+| CLIP cosine, minimum | `0.9999999999999998` |
+| Face-embedding cosine, minimum | `0.9999999999999998` |
+| Fatal/native/ORT failures | 0 |
+
+The combination's no-CR2 corpus median was 9,638.985 ms versus 9,676.216 ms
+for baseline, only 0.38% faster. Its aggregate median model time was 3.807
+seconds versus 3.786 seconds for baseline. In other words, the tiny pipeline
+change came from non-model variation; the combined model configuration was
+slightly slower overall. No runtime tuning knob tested here closes the gap to
+the approximately 20 ms CoreML results on A17 Pro.
+
+The remaining Android performance gap is more likely architectural: CoreML
+can fuse and compile these graphs for Apple's tightly integrated GPU/ANE stack,
+while ORT WebGPU emits portable Dawn/Vulkan workloads with different fusion,
+dispatch, and memory behavior. The next high-value work is model-level
+optimization or quantization and cross-vendor measurement, not additional
+session flags from this set.
+
+## Initial smoke-test outcome (superseded by the follow-up above)
The custom ONNX Runtime build works in a real, non-debuggable release APK, and
the indexing models genuinely execute through Dawn's Vulkan WebGPU backend on
@@ -290,7 +548,7 @@ There was no thermal-throttling status transition. The phone was attached over
USB, so this run is suitable for latency and functional validation, not an
energy-efficiency claim.
-## Recommendation
+## Initial recommendation (superseded by the follow-up above)
The custom AAR and WebGPU integration are technically ready for a broader
pilot: release loading, Dawn/Vulkan initialization, GPU node placement,
@@ -354,6 +612,25 @@ The evidence is kept under the gitignored
- `python/`: generated Python goldens for the exact fixture manifest and model
hashes.
-The forced WebGPU flag and all benchmark-only Dart/Rust instrumentation were
-removed after collecting and validating these artifacts. No production
-provider-selection behavior was changed as part of this benchmark report.
+The follow-up evidence is kept under the gitignored
+`infra/ml/test/out/android_webgpu_tuning_2026-07-17/` tree:
+
+- `runs/<variant>/results.json`: application-visible outputs and five measured
+ end-to-end samples per fixture.
+- `runs/<variant>/logcat.txt`: exact per-model input, inference, output, and
+ total timings for all warm-up and measured calls.
+- `runs/<variant>/thermal_before.txt` and `thermal_after.txt`: Android thermal
+ service snapshots.
+- `benchmark_summary.json`: parsed distributions and FP32-baseline parity for
+ every isolated and final run.
+- `placement_evidence.txt`: the filtered verbose ORT placement proof for all
+ three rewritten-model sessions.
+- `models/fp16/`: the checked FP16 experiment artifacts.
+- `apks/`: the exact release APKs used for FP32, FP16, and the final FP32
+ combination.
+
+The forced WebGPU flag, system-property variants, I/O-binding pilot, and all
+benchmark-only Dart/Rust instrumentation were removed after collecting and
+validating these artifacts. No production provider-selection behavior was
+changed. The reproducible portable MobileFaceNet transformation, its manifest,
+and this report are the durable changes.