π End-to-end encrypted cloud for everything.
ML Indexing Parity Test Suite
This directory contains the ML indexing parity framework for desktop, Android, iOS, and Python ground truth.
The desktop runner smoke-tests the desktop-specific N-API bindings and pinned ONNX Runtime library. Deep validation of the shared Rust pipeline lives in the ente-ml crate's ml_indexing integration test.
Layout
ground_truth/: schema, manifest, and ONNX-backed Python pipeline.comparator/: parity comparison engine and threshold checks.tools/: suite orchestration and CLI entrypoints.ml_indexing/: asset lock for the Cargo-based Rust ML indexing test.- Runtime-only artifacts (gitignored):
test_data/,out/,.cache/.
Prerequisites
- Python +
uv - Node + the desktop build artifacts (
npm ci && npm run postinstall && node scripts/napi.js buildindesktop/) when running desktop parity - Flutter SDK (Android/iOS parity)
- Android emulator/device and iOS simulator/device when running mobile parity
Local Run (One Command)
bash infra/ml/test/run_ml_parity_tests.sh
Common flags:
--platforms all|desktop|android|ios--output-dir <path>--verbose(stream full runner/comparator logs to terminal)--render-detection-overlays(generate annotated detection images underout/parity/detections/<platform>/; includes selected platforms pluspythonground truth)--reuse-mobile-application-binary(reuse an existing built mobile binary when available; useful for repeated local parity runs without code changes)--no-parallel-mobile-runners(force sequential android/ios runner execution)
Outputs go to infra/ml/test/out/parity/ by default, including:
comparison_report.json(machine-readable comparison output)parity_report.html(readable HTML report with per-file metrics for both pass and fail files)
run_ml_parity_tests.sh compares each available platform against Python ground truth (python -> <platform>).
Optional mobile reuse env vars:
ML_PARITY_ANDROID_BUILD_MODE(profileby default; setdebugorreleaseexplicitly if needed)ML_PARITY_ANDROID_EXISTING_APP_URL,ML_PARITY_IOS_EXISTING_APP_URL(reuse a currently running app via VM service URL)ML_PARITY_ANDROID_APPLICATION_BINARY,ML_PARITY_IOS_APPLICATION_BINARY(explicit prebuilt binary path forflutter drive --use-application-binary)
Rust ML Indexing Cargo Test
The Rust-only ML indexing test lives at rust/crates/ml/tests/ml_indexing.rs.
It is gated by the ente-ml/ml-assets Cargo feature so ordinary local
cargo test --workspace runs do not download ML assets or execute indexing.
From rust/:
cargo test -p ente-ml --test ml_indexing --features ml-assets -- --nocapture
By default the test downloads the external fixture manifest, fixture images,
Python golden JSON, ONNX Runtime dynamic library, and ONNX models into
rust/.cache/, with SHA-256 verification for every asset.
Optional env vars:
ENTE_ML_INDEXING_PRINT_STATS=1: print a per-metric comparison summary (count, max observed value, threshold) even when the test passes; on failure the summary is always appended to the failure message.
infra/ml/test/ml_indexing/assets.json locks the external manifest, Python
golden, model URLs, expected unsupported decode files, and thresholds. The
current RAW/CR2 fixture is expected to fail Rust decode; the test verifies that
failure path and skips golden comparison for that file until RAW support lands.
Detection Overlay Visualizer
Render face detection overlays (box + score + landmarks) for each platform output:
uv run --project infra/ml \
python infra/ml/test/tools/render_face_detection_overlays.py \
--parity-dir infra/ml/test/out/parity \
--platform ios \
--platform android
By default this writes annotated images to:
infra/ml/test/out/parity/detections/<platform>/*.png
Useful optional filters:
--file-id <fixture_name>(repeatable)--output-dir <custom_dir>
Ground-Truth Decode Visualizer
Render the exact Python ground-truth decode output (including EXIF orientation handling) for every manifest fixture, then review the generated gallery.
uv run --project infra/ml \
python infra/ml/test/tools/visualize_ground_truth_decodes.py \
--manifest infra/ml/test/ground_truth/manifest.json \
--output-dir infra/ml/test/out/parity/python_decode_preview \
--open
This script calls ground_truth._runtime.decode_image_rgb directly, so it tracks decode/orientation behavior changes in the Python reference pipeline.
Outputs:
infra/ml/test/out/parity/python_decode_preview/index.htmlinfra/ml/test/out/parity/python_decode_preview/decoded/*.png
Golden Update / Maintenance
Goldens under this monorepo's out/, test_data/, and cache directories are
runtime artifacts and are not committed. The Rust ML indexing golden JSON is
different: it belongs in the external test-fixtures repo next to the image
corpus, because it describes that corpus.
Use this process when corpus/threshold/model behavior changes intentionally:
- Update corpus metadata in the external fixture repo. Rust ML indexing thresholds live in
infra/ml/test/ml_indexing/assets.json; shell-suite thresholds live inThresholdConfigininfra/ml/test/comparator/compare.py. - Regenerate and compare with a real run:
bash infra/ml/test/run_ml_parity_tests.sh --platforms all --output-dir infra/ml/test/out/parity
- Regenerate the Rust ML indexing golden in the external fixture repo:
uv run --project infra/ml \
python infra/ml/test/tools/generate_ml_indexing_golden.py \
--assets-repo-dir /path/to/test-fixtures/ml/indexing/v1
- Review
infra/ml/test/out/parity/parity_report.html,comparison_report.json, per-platformresults.jsonfiles, and the generated externalgoldens/python/results.json. - If drift is expected, update thresholds/known exceptions and the Python golden SHA in
infra/ml/test/ml_indexing/assets.json, then rerun until stable. - Commit code/config/docs updates in this repo; commit the corpus manifest and Python golden JSON in the external fixture repo. Do not commit generated
out/,test_data/, or cache artifacts in this repo.