Python tool for aircraft Doppler audio geolocation using historical ADS-B flight data (Source-available)
Python acoustic-geolocationads-baircraft-trackingairplaneaudio-forensicsdoppler-effectdoppler-shiftgeolocationosintsignal-processing
Document measured real-pipeline speedups and reproducible benchmarks
Cinna Davis19d82aa987bd
diff --git a/README.md b/README.mdindex ff1a360..827aa2f 100644--- a/README.md+++ b/README.md@@ -88,7 +88,11 @@ Recording-time evidence matters: audio-relative seconds do not identify which daThe automatic search uses conservative score ceilings before completing every fit: incompatible residuals and an established Top-K cutoff can stop further group fitting at a shared clock. It batches eligible clock fits, reuses exact acoustic calculations in dense tiles, and divides overloaded areas or single-cell clock ranges into checkpointed pieces. [Search screening and real-data timing](docs/DATA_AND_SCALE.md#search) describe the bounds and their measured limits.-At one source-informed receiver point, median scoring time fell from **68.92 to 25.59 CPU seconds**, a **2.69×** speedup. All 3,905 associations across 16 event groups matched the scalar-corrected baseline in three alternating pairs. The [performance report](docs/PERFORMANCE.md) describes this bounded test, which does not establish nationwide runtime or independent location accuracy.+The latest changes reduced median scoring time at one source-informed receiver point from **28.80 to 15.57 CPU seconds**, a **1.85×** incremental speedup. A narrower clock workload improved from **4.39 to 2.21 CPU seconds**, or **1.98×**. Each comparison used three alternating pairs on actual recorded ridges and observed flight trajectories. All 3,905 broad-window associations and 1,055 narrow-window associations matched exactly across 16 independent groups, including diagnostics and retention records.++The implementation reuses exact fits within a bounded 32 MiB cache and avoids decoding unrelated finalist trajectories. Separate measurements cover refinement, optional plots, and compact checkpoint encoding. The [performance report](docs/PERFORMANCE.md) gives the methodology, reproduction commands, memory tradeoff, and earlier historical results. Point timings use NumPy 2.5.4, while complete HTTP checks use NumPy 2.3.5. The full CONUS baseline reached its runtime limit, so these measurements do not establish nationwide completion time or independent location accuracy.++A separate bounded HTTP pair processed the complete original recording in **264.81 → 188.32 wall seconds**, an observed **1.41×** speedup. Both runs passed the full-pipeline checks and returned 23 candidates with `inconclusive / search_incomplete`. Scientific outputs, the retention ledger, decoded audio, and diagnostics matched without numerical tolerance after the declared run-metadata normalization. This is one sequential pair on a shared host, with source-informed regional bounds and a fitted recording clock. The [HTTP measurement record](benchmarks/results/bounded-http-20261103.json) preserves the settings and equality checks.Recent search work makes omitted alternatives and failed model evaluations visible. For more detail see [finite-search evidence](docs/FINITE_SEARCH_PRODUCER.md) and [clock-trial accounting](docs/SCORING_ASSESSMENT.md).diff --git a/benchmarks/results/bounded-http-20261103.json b/benchmarks/results/bounded-http-20261103.jsonnew file mode 100644index 0000000..0d6cdbe--- /dev/null+++ b/benchmarks/results/bounded-http-20261103.json@@ -0,0 +1,165 @@+{+ "scope": "Fresh actual HTTP upload through intake, discovery, search, refinement and decision on an archived four-point real workload. Source-informed region/time and supplied cached inputs; no independent location-accuracy or nationwide-runtime claim.",+ "baseline_revision": "aa3dff686c989524daa20c081a816b792d9c8890",+ "optimized_production_revision": "5699a657d57dc4f66b74ef000e20d3ba88f84704",+ "local_verified_revision": "5952603bbe8f7aaff25c6f8c5f16d6f58da02ad9",+ "revision_note": "Root verified the published production source is equivalent to the frozen local source used for this run.",+ "runtime": {+ "python": "3.12.14",+ "numpy": "2.3.5",+ "scipy": "1.17.0",+ "sqlite": "3.53.1",+ "OPENBLAS_NUM_THREADS": 1,+ "OMP_NUM_THREADS": 1,+ "MKL_NUM_THREADS": 1+ },+ "method": "One fresh baseline followed by one fresh optimized HTTP run, identical original MP3 bytes, request, operator configuration and interpreter. Shared-host wall observations; no whole-workflow process CPU measurement was collected. HTTP receipt elapsed time includes verification overhead.",+ "request": {+ "schema_version": 1,+ "search": {+ "coverage": "conus",+ "bounds_wsen": [+ -73.545,+ 40.865,+ -73.505,+ 40.895+ ]+ },+ "time_window": {+ "start_utc": "2026-09-08T20:17:20Z",+ "end_utc": "2026-09-08T20:17:40Z",+ "source": "user",+ "source_detail": "Source-informed integration replay around the historical fitted recording start and benchmark receiver region; not independently measured recording time or a location-accuracy evaluation."+ }+ },+ "operator_settings": {+ "offline": true,+ "spacing_m": 2000,+ "tile_cells": 16,+ "top_k": 20,+ "refine_spacing_m": 500,+ "refine_finalists": 3,+ "search_workers": 1+ },+ "source": {+ "sha256": "e1c13dafd8d1634bbf84df843c31771318e15a31ebb86dd165af7ee312ab80a0",+ "bytes": 42085293+ },+ "http_wall_s": {+ "baseline": 264.80960926900025,+ "current": 188.32315037399985+ },+ "http_wall_speedup": 1.4061447503565139,+ "http_elapsed_reduction_percent": 28.883566236942528,+ "search_wall_s": {+ "baseline": 154.04968578999978,+ "current": 89.74133267199977+ },+ "search_process_peak_rss_mib": {+ "baseline": 399.6484375,+ "current": 364.00390625+ },+ "observed_stage_seconds": {+ "baseline": {+ "queued": 1.0047268867492676,+ "intake": 11.024419069290161,+ "discovery": 8.0289888381958,+ "search": 153.97183418273926,+ "refinement": 82.41287589073181+ },+ "current": {+ "queued": 1.0018911361694336,+ "intake": 12.031001806259155,+ "discovery": 7.043560028076172,+ "search": 90.870934009552,+ "refinement": 70.44122910499573+ }+ },+ "stage_timing_note": "Stages use first observed HTTP state transitions with one-second polling uncertainty; unobserved short stages have no inferred duration. Process peak RSS is a worker-process high-water mark, not stage-only memory.",+ "search_work": {+ "baseline": {+ "status": "partial",+ "tiles_total": 2,+ "tiles_completed": 2,+ "grid_points_searched": 4,+ "bounded_concurrency": 1,+ "refinement_points_searched": 40,+ "refinement_points_skipped": 80,+ "tile_failures": [],+ "refinement_failure": null+ },+ "current": {+ "status": "partial",+ "tiles_total": 2,+ "tiles_completed": 2,+ "grid_points_searched": 4,+ "bounded_concurrency": 1,+ "refinement_points_searched": 40,+ "refinement_points_skipped": 80,+ "tile_failures": [],+ "refinement_failure": null+ }+ },+ "outcomes": {+ "baseline": {+ "status": "inconclusive",+ "reason_code": "search_incomplete",+ "candidate_count": 23,+ "event_count": 18+ },+ "current": {+ "status": "inconclusive",+ "reason_code": "search_incomplete",+ "candidate_count": 23,+ "event_count": 18+ }+ },+ "full_pipeline_verified": {+ "baseline": true,+ "current": true+ },+ "scientific_json_sqlite_parity": {+ "passed": true,+ "semantic_differences": 0,+ "checked_artifacts": {+ "json": 23,+ "sqlite": 1,+ "root_json": 3+ },+ "retention_ledgers": {+ "search/retention.sqlite": {+ "baseline_rows": {+ "bounds": 48226,+ "records": 958,+ "run": 1,+ "stages": 18+ },+ "current_rows": {+ "bounds": 48226,+ "records": 958,+ "run": 1,+ "stages": 18+ }+ }+ }+ },+ "intake_array_pixel_parity": {+ "passed": true,+ "differences": 0,+ "checked": {+ "intake_diagnostics": 1,+ "decoded_pcm": 1,+ "csv": 18,+ "npz": 0,+ "npz_arrays": 0,+ "png": 1+ },+ "manifest_verified_artifacts": {+ "baseline": 21,+ "current": 21+ },+ "normalization": "Only the exact input.mp3 path in intake.decode.command is replaced.",+ "array_rule": "Equal member names, dtype, shape, and element bits; no numerical tolerance. Memory layout does not affect equality."+ },+ "excluded_attempts_note": "A dependency-failed startup attempt and interrupted broad-area runs are preserved separately and are not timing or parity arms."+}diff --git a/benchmarks/results/checkpoint-encoding-20261103.json b/benchmarks/results/checkpoint-encoding-20261103.jsonnew file mode 100644index 0000000..50f3f7d--- /dev/null+++ b/benchmarks/results/checkpoint-encoding-20261103.json@@ -0,0 +1,126 @@+{+ "scope": "JSON encoding only for a completed real tile; no scoring or pipeline speed claim",+ "input": "/workspace/scratch/be56be4080ef/measurements/upload-baseline/jobs/ca6030889b74472e983fd2cdf3c81a59/search/tile-58_68~2~3.json",+ "input_bytes": 39363929,+ "candidate_associations": 19977,+ "all_decoded_outputs_exact": true,+ "payload_hash_unchanged": true,+ "retention_checksum_unchanged": true,+ "runs": [+ {+ "repetition": 1,+ "encoding": "pretty",+ "wall_seconds": 1.363638642999831,+ "cpu_seconds": 1.3627130250000001,+ "bytes": 39363929+ },+ {+ "repetition": 1,+ "encoding": "compact",+ "wall_seconds": 0.2668106649998663,+ "cpu_seconds": 0.2668126719999999,+ "bytes": 23490375+ },+ {+ "repetition": 2,+ "encoding": "compact",+ "wall_seconds": 0.28472605499973724,+ "cpu_seconds": 0.28471583099999975,+ "bytes": 23490375+ },+ {+ "repetition": 2,+ "encoding": "pretty",+ "wall_seconds": 1.1696782859999075,+ "cpu_seconds": 1.1692302669999997,+ "bytes": 39363929+ },+ {+ "repetition": 3,+ "encoding": "pretty",+ "wall_seconds": 1.1917218049993608,+ "cpu_seconds": 1.1912622210000006,+ "bytes": 39363929+ },+ {+ "repetition": 3,+ "encoding": "compact",+ "wall_seconds": 0.2871776769998178,+ "cpu_seconds": 0.28717892699999936,+ "bytes": 23490375+ },+ {+ "repetition": 4,+ "encoding": "compact",+ "wall_seconds": 0.28859559099964827,+ "cpu_seconds": 0.28853438500000017,+ "bytes": 23490375+ },+ {+ "repetition": 4,+ "encoding": "pretty",+ "wall_seconds": 1.320897430999139,+ "cpu_seconds": 1.3204166070000003,+ "bytes": 39363929+ },+ {+ "repetition": 5,+ "encoding": "pretty",+ "wall_seconds": 1.5711264759993355,+ "cpu_seconds": 1.570734088,+ "bytes": 39363929+ },+ {+ "repetition": 5,+ "encoding": "compact",+ "wall_seconds": 0.2919367089998559,+ "cpu_seconds": 0.29180276199999966,+ "bytes": 23490375+ }+ ],+ "pretty": {+ "cpu_seconds": 1.3204166070000003,+ "wall_seconds": 1.320897430999139,+ "bytes": 39363929+ },+ "compact": {+ "cpu_seconds": 0.28717892699999936,+ "wall_seconds": 0.2871776769998178,+ "bytes": 23490375+ },+ "cpu_speedup": 4.5978882252735875,+ "size_reduction_fraction": 0.403251260817994,+ "schema_version": 1,+ "date": "2026-11-03",+ "python": "3.12.14",+ "method": {+ "repeats_per_encoding": 5,+ "order": "Alternating pretty/compact, then compact/pretty; raw runs below preserve actual order.",+ "timing_scope": "json.dumps plus trailing newline only. Excludes loading, decoding, equality checks, file writes, scoring and the pipeline.",+ "comparison": "Complete canonical decoded JSON, every field and array order; no numerical tolerance. Also checked original pretty bytes and unchanged payload hash and retention checksum.",+ "baseline_encoding": {+ "indent": 2,+ "separators": null,+ "allow_nan": false+ },+ "optimized_encoding": {+ "indent": null,+ "separators": [+ ",",+ ":"+ ],+ "allow_nan": false+ }+ },+ "provenance": {+ "raw_measurement": "pipeline-work/compact-checkpoints/summary.json",+ "original_harness": "pipeline-work/compact-checkpoints/benchmark.py",+ "reproduction_harness": "scripts/benchmark_real_checkpoint_encoding.py",+ "input_origin": "Completed tile from the real MP3 upload baseline search, not a generated fixture.",+ "baseline_commit": "aa3dff686c989524daa20c081a816b792d9c8890",+ "local_implementation_commit": "5952603",+ "public_implementation_commit": "5699a657d57dc4f66b74ef000e20d3ba88f84704",+ "measurement_before_edit": true,+ "source_limit": "The timed expressions were tested before the equivalent writer hook was applied. These are encoding measurements, not timings of the complete atomic writer."+ }+}diff --git a/benchmarks/results/decision-refinement-20261103.json b/benchmarks/results/decision-refinement-20261103.jsonnew file mode 100644index 0000000..d448302--- /dev/null+++ b/benchmarks/results/decision-refinement-20261103.json@@ -0,0 +1,86 @@+{+ "schema_version": 1,+ "date": "2026-11-03",+ "scope": "One paired refinement/decision replay from frozen real-recording finalists at a source-informed location and clock. No search, nationwide runtime, or independent accuracy claim.",+ "sources": {+ "baseline_commit": "aa3dff686c989524daa20c081a816b792d9c8890",+ "optimized_implementation_subsequently_committed_locally_in": "5ca480f8678ca1ed31a765768324f623a44f0da8",+ "public_implementation_commit": "90d95d7ea793a7dff5388ec9c7b5caf569b0541d",+ "final_public_production_commit": "5699a657d57dc4f66b74ef000e20d3ba88f84704",+ "source_capture_limit": "The current arm ran the uncommitted automatic.py and scoped FlightIndex lookup edits. The original harness did not capture a source snapshot or revision with its output. Later main integration and the 32 MiB batch cache/checkpoint edits are not measured by this pair.",+ "recorded_job": "/workspace/scratch/1e88d5469ed3/DOPPLERGEIST/results/real-performance-e2e-bounded-20261103/jobs/b2bd1a92723545b89d393c7d163148a8",+ "recording_sha256": "e1c13dafd8d1634bbf84df843c31771318e15a31ebb86dd165af7ee312ab80a0",+ "raw_measurements": [+ "pipeline-work/decision-baseline-plain/summary.json",+ "pipeline-work/decision-current-plain/summary.json"+ ],+ "original_harness": "pipeline-work/replay_real_decision.py",+ "reproduction_harness": "scripts/benchmark_real_decision.py"+ },+ "method": {+ "repeats_per_source": 1,+ "order": [+ "baseline",+ "current"+ ],+ "profiled": false,+ "runtime": "/workspace/scratch/1e88d5469ed3/perf-venv/bin/python",+ "numerical_library_threads": 1,+ "timing_scope": "The complete _decide call, including native trajectory loading, refinement, sensitivity profiles and decision/refined-candidate file writes. Excludes imports, loading saved search/discovery metadata, preparing measurements/index handles/terrain, all prior pipeline phases, comparison, and harness result.json serialization.",+ "instrumentation": "The original harness counted _frame_track decodes. Both arms used the same counter. The raw ridge_cache=none flag means no experimental wrapper override; the current production _decide already used scoped refinement/profile caches.",+ "memory_scope": "Linux process VmHWM via process_peak_rss_kib, including setup and imports; not isolated cache memory.",+ "comparison": "Exact complete canonical JSON for every refined candidate, sensitivity result, decision and return result. Preserves scalar representation and array order; no numerical tolerance.",+ "limitation": "One pair cannot establish run-to-run variation. The clean reproduction helper removes the decode-counter instrumentation."+ },+ "workload": {+ "duration_seconds": 3600.1845,+ "ridge_records": 18,+ "independent_groups": 16,+ "search_finalists": 20,+ "refinement_limit": 3,+ "returned_candidates": 23,+ "status": "inconclusive",+ "reason_code": "search_incomplete"+ },+ "comparison": {+ "equal": true,+ "artifacts": [+ "refined-candidates.json",+ "decision.json",+ "result.json"+ ]+ },+ "runs": {+ "baseline": {+ "cpu_seconds": 83.70841713499999,+ "operations": {+ "decoded_samples": 233165,+ "decoded_trajectories": 881+ },+ "process_peak_rss_mib": 219.16796875,+ "reason_code": "search_incomplete",+ "returned_candidates": 23,+ "ridge_cache": "none",+ "scope": "Refinement/decision replay of recorded-audio finalist hypotheses; source-informed workload, no accuracy claim",+ "status": "inconclusive",+ "wall_seconds": 83.8421902730006+ },+ "current": {+ "cpu_seconds": 77.487010169,+ "operations": {+ "decoded_samples": 5303,+ "decoded_trajectories": 18+ },+ "process_peak_rss_mib": 215.890625,+ "reason_code": "search_incomplete",+ "returned_candidates": 23,+ "ridge_cache": "none",+ "scope": "Refinement/decision replay of recorded-audio finalist hypotheses; source-informed workload, no accuracy claim",+ "status": "inconclusive",+ "wall_seconds": 77.56249172400021+ }+ },+ "cpu_speedup": 1.0802896763267937,+ "wall_speedup": 1.080963084210164,+ "cpu_reduction_percent": 7.432235823987055+}diff --git a/benchmarks/results/exact-fit-reuse-20261103.json b/benchmarks/results/exact-fit-reuse-20261103.jsonnew file mode 100644index 0000000..500a143--- /dev/null+++ b/benchmarks/results/exact-fit-reuse-20261103.json@@ -0,0 +1,412 @@+{+ "schema_version": 1,+ "date": "2026-11-03",+ "scope": "Actual recorded-ridge and observed-trajectory point scoring at a source-informed receiver and clock. No independent accuracy, national-runtime or complete-pipeline speed claim.",+ "sources": {+ "baseline_commit": "aa3dff686c989524daa20c081a816b792d9c8890",+ "optimized_scoring_base_commit": "5ca480f8678ca1ed31a765768324f623a44f0da8",+ "optimized_reproducible_commit": "5699a657d57dc4f66b74ef000e20d3ba88f84704",+ "published_source_equivalence": "The measured base is a local offline commit plus the stated worktree patch. The public reproducible commit contains identical exercised scoring paths and the final 32 MiB limit. Its additional compact-checkpoint changes are outside the point-scoring timing scope.",+ "optimized_worktree_change": "src/dopplergeist/matching.py: _RidgeFitCache.batch_limit changed from 16*1024*1024 to 32*1024*1024, plus the cache-limit docstring.",+ "source_recording_limit": "The harness did not record HEAD per worker. The exercised scoring implementation is the stated base commit plus the stated worktree change. Concurrent terrain/ENF integration 5420163 did not change the exercised FlightIndex or scoring paths.",+ "discovery": "evidence/audio/w1-sensitivity-20261102/reference/discovery.json",+ "receiver_reference": "evidence/results/reference-local/report.json",+ "harness": "scripts/benchmark_prepared_measurements.py",+ "concurrent_integration_commit": "5420163f026b6c2621226add88cfa05fda326b46"+ },+ "method": {+ "repeats_per_source_per_workload": 3,+ "order_each_workload": [+ "baseline",+ "current",+ "current",+ "baseline",+ "baseline",+ "current"+ ],+ "workload_order": [+ "narrow",+ "broad"+ ],+ "batch_cache_limit_bytes": 33554432,+ "batch_cache_max_entries": 4096,+ "comparison": "Exact canonical JSON equality of every scored association, diagnostics, support counts, ranked rows, and retention records/bounds/stages; no numerical tolerance.",+ "timing_scope": "The complete point-scoring call, including retention writes. Excludes imports, input loading, fixture preparation, acquisition, discovery, nationwide search, refinement and result JSON serialization.",+ "primary_metric": "process CPU seconds",+ "concurrency_limit": "A full upload ran on the shared host during these measurements. Wall times are recorded but are not exclusive-host measurements.",+ "memory_scope": "Reported getrusage(RUSAGE_SELF).ru_maxrss through scoring, including imports, input loading and captured candidate diagnostics. VmHWM was not recorded; ru_maxrss can retain an inherited pre-exec peak. This is not isolated cache-heap measurement."+ },+ "timing_environment": {+ "os": "Linux 6.18.57 x86_64",+ "processor": "AMD EPYC 9V74 80-Core Processor",+ "python": "3.12.14",+ "numpy": "2.5.4",+ "scipy": "1.17.0",+ "pandas": "2.2.3",+ "pyarrow": "26.0.1",+ "pyproj": "3.8.0",+ "additional_import_directory": "/workspace/scratch/ef866e3c48e0/python-deps",+ "thread_limits": {+ "OPENBLAS_NUM_THREADS": 1,+ "OMP_NUM_THREADS": 1,+ "MKL_NUM_THREADS": 1,+ "NUMEXPR_NUM_THREADS": 1+ },+ "runtime_distinction": "These point timings use NumPy 2.5.4. Full HTTP experiments and the separate correctness check below use NumPy 2.3.5. Do not combine their timings as one runtime."+ },+ "workloads": {+ "broad": {+ "fixture_name": "point-120s.json",+ "clock_half_width_seconds": 120,+ "fixture_counts": {+ "trajectories": 452,+ "trajectory_samples": 91889,+ "ridge_records": 16,+ "independent_groups": 16,+ "ridge_samples": 724,+ "cpa_rows": 100,+ "unsupported_cpa_tracks": 4+ },+ "index_input_hash": "ed0ab12af801d1451edea17f1212740758569338dd601f9eb4a82c763b291955",+ "receiver_point": {+ "point_id": 41,+ "lat": 40.87693175890468,+ "lon": -73.52498229500986,+ "elev_m": -31.25569725036621,+ "x": 1861200.0,+ "y": 2206325.0+ },+ "query_bounds": [+ -74.299988,+ 40.100005,+ -72.700011,+ 41.599964+ ],+ "query_time_s": [+ 1788898409.6620934,+ 1788902359.1620934+ ],+ "recording_start_window_s": [+ 1788898529.6620934,+ 1788898769.6620934+ ],+ "budget": {+ "spacing_m": 2000.0,+ "tile_cells": 32,+ "max_tracks_per_tile": 5000,+ "max_track_samples_per_tile": 1000000,+ "max_cpa_rows": 100000,+ "max_candidates_per_tile": 20000,+ "top_k": 100,+ "refine_spacing_m": 100.0,+ "refine_finalists": 10,+ "cpa_step_s": 2.0,+ "max_tiles": null+ },+ "scoring": {+ "mode": "robust_ridge"+ },+ "identical_results": true,+ "operations_each_run": {+ "scalar_fit_calls": 72,+ "acoustic_calls": 850,+ "scored_candidates": 3905+ },+ "retained_each_run": 100,+ "medians": {+ "baseline": {+ "wall_seconds": 28.81658592599979,+ "cpu_seconds": 28.803255077,+ "peak_rss_mib": 260.4609375+ },+ "current": {+ "wall_seconds": 15.57937161100017,+ "cpu_seconds": 15.571095273,+ "peak_rss_mib": 293.25390625+ }+ },+ "speedup": {+ "wall_seconds": 1.8496629161636522,+ "cpu_seconds": 1.8497899198487553+ },+ "cpu_reduction_percent": 45.93980704134428,+ "runs": [+ {+ "source": "baseline",+ "repetition": 1,+ "cpu_seconds": 26.394216981,+ "wall_seconds": 26.41746890900049,+ "load_seconds_excluded": 1.1986803300005704,+ "peak_rss_mib": 259.3203125+ },+ {+ "source": "current",+ "repetition": 1,+ "cpu_seconds": 14.077091585999998,+ "wall_seconds": 14.12715121400015,+ "load_seconds_excluded": 1.4733468790000188,+ "peak_rss_mib": 293.578125+ },+ {+ "source": "current",+ "repetition": 2,+ "cpu_seconds": 15.571095273,+ "wall_seconds": 15.57937161100017,+ "load_seconds_excluded": 1.1838955880002686,+ "peak_rss_mib": 292.18359375+ },+ {+ "source": "baseline",+ "repetition": 2,+ "cpu_seconds": 28.803255077,+ "wall_seconds": 28.81658592599979,+ "load_seconds_excluded": 1.3216495369997574,+ "peak_rss_mib": 260.4609375+ },+ {+ "source": "baseline",+ "repetition": 3,+ "cpu_seconds": 29.232790216999998,+ "wall_seconds": 29.298039747000075,+ "load_seconds_excluded": 1.2159543139996458,+ "peak_rss_mib": 261.0625+ },+ {+ "source": "current",+ "repetition": 3,+ "cpu_seconds": 15.910992931,+ "wall_seconds": 15.964019279999775,+ "load_seconds_excluded": 1.3493243950006217,+ "peak_rss_mib": 293.25390625+ }+ ]+ },+ "narrow": {+ "fixture_name": "point-10s.json",+ "clock_half_width_seconds": 10,+ "fixture_counts": {+ "trajectories": 426,+ "trajectory_samples": 87206,+ "ridge_records": 16,+ "independent_groups": 16,+ "ridge_samples": 724,+ "cpa_rows": 97,+ "unsupported_cpa_tracks": 4+ },+ "index_input_hash": "ed0ab12af801d1451edea17f1212740758569338dd601f9eb4a82c763b291955",+ "receiver_point": {+ "point_id": 41,+ "lat": 40.87693175890468,+ "lon": -73.52498229500986,+ "elev_m": -31.25569725036621,+ "x": 1861200.0,+ "y": 2206325.0+ },+ "query_bounds": [+ -74.299988,+ 40.100005,+ -72.700011,+ 41.599964+ ],+ "query_time_s": [+ 1788898519.6620934,+ 1788902249.1620934+ ],+ "recording_start_window_s": [+ 1788898639.6620934,+ 1788898659.6620934+ ],+ "budget": {+ "spacing_m": 2000.0,+ "tile_cells": 32,+ "max_tracks_per_tile": 5000,+ "max_track_samples_per_tile": 1000000,+ "max_cpa_rows": 100000,+ "max_candidates_per_tile": 20000,+ "top_k": 100,+ "refine_spacing_m": 100.0,+ "refine_finalists": 10,+ "cpa_step_s": 2.0,+ "max_tiles": null+ },+ "scoring": {+ "mode": "robust_ridge"+ },+ "identical_results": true,+ "operations_each_run": {+ "scalar_fit_calls": 31,+ "acoustic_calls": 53,+ "scored_candidates": 1055+ },+ "retained_each_run": 100,+ "medians": {+ "baseline": {+ "wall_seconds": 4.393366551999861,+ "cpu_seconds": 4.386338558,+ "peak_rss_mib": 210.37109375+ },+ "current": {+ "wall_seconds": 2.215672813999845,+ "cpu_seconds": 2.21248752,+ "peak_rss_mib": 211.93359375+ }+ },+ "speedup": {+ "wall_seconds": 1.9828588969635517,+ "cpu_seconds": 1.9825370847741552+ },+ "cpu_reduction_percent": 49.55958162498044,+ "runs": [+ {+ "source": "baseline",+ "repetition": 1,+ "cpu_seconds": 4.095521208999999,+ "wall_seconds": 4.10293381000065,+ "load_seconds_excluded": 1.188167754999995,+ "peak_rss_mib": 209.8984375+ },+ {+ "source": "current",+ "repetition": 1,+ "cpu_seconds": 2.21248752,+ "wall_seconds": 2.215672813999845,+ "load_seconds_excluded": 1.1429735729998356,+ "peak_rss_mib": 211.1875+ },+ {+ "source": "current",+ "repetition": 2,+ "cpu_seconds": 2.465027842,+ "wall_seconds": 2.475018588999774,+ "load_seconds_excluded": 1.2079138489998513,+ "peak_rss_mib": 211.953125+ },+ {+ "source": "baseline",+ "repetition": 2,+ "cpu_seconds": 4.386338558,+ "wall_seconds": 4.393366551999861,+ "load_seconds_excluded": 1.1909067930000674,+ "peak_rss_mib": 210.53515625+ },+ {+ "source": "baseline",+ "repetition": 3,+ "cpu_seconds": 4.829120257,+ "wall_seconds": 4.8339807339998515,+ "load_seconds_excluded": 1.2563498580002488,+ "peak_rss_mib": 210.37109375+ },+ {+ "source": "current",+ "repetition": 3,+ "cpu_seconds": 2.202057387,+ "wall_seconds": 2.2054391869996834,+ "load_seconds_excluded": 1.1738416859998324,+ "peak_rss_mib": 211.93359375+ }+ ]+ }+ },+ "capacity_tradeoff": {+ "scope": "Same broad real-input fixture, with an instrumented OrderedDict counting exact cache lookups and evictions. Outputs equal the unmodified baseline.",+ "order_mib": [+ 16,+ 32,+ 32,+ 16+ ],+ "selected_limit_mib": 32,+ "median_cpu_reduction_percent": 8.467267747039608,+ "decision": "The larger limit repeatedly reduced CPU by about 8.5% and remaining batch executions by about 45%, for about 16 MiB more reported worker RSS. Allocation remains demand-driven and bounded.",+ "capacities": {+ "16": {+ "limit_mib": 16,+ "identical_results": true,+ "counters_each_run": {+ "lookups": 7351,+ "hits": 5403,+ "misses": 1948,+ "evictions": 1443,+ "max_retained_bytes": 16777214,+ "max_retained_batches": 554,+ "limit_bytes": 16777216,+ "exact_outputs": true+ },+ "median_cpu_seconds": 18.937758264,+ "median_reported_peak_rss_mib": 276.974609375,+ "runs": [+ {+ "cpu_seconds": 19.183687540999998,+ "wall_seconds": 19.271691674000067,+ "peak_rss_mib": 277.24609375+ },+ {+ "cpu_seconds": 18.691828987,+ "wall_seconds": 18.70969998199962,+ "peak_rss_mib": 276.703125+ }+ ]+ },+ "32": {+ "limit_mib": 32,+ "identical_results": true,+ "counters_each_run": {+ "lookups": 7351,+ "hits": 6281,+ "misses": 1070,+ "evictions": 17,+ "max_retained_bytes": 33553175,+ "max_retained_batches": 1057,+ "limit_bytes": 33554432,+ "exact_outputs": true+ },+ "median_cpu_seconds": 17.3342475665,+ "median_reported_peak_rss_mib": 293.1953125,+ "runs": [+ {+ "cpu_seconds": 17.593079808,+ "wall_seconds": 17.629105033999622,+ "peak_rss_mib": 293.48828125+ },+ {+ "cpu_seconds": 17.075415325,+ "wall_seconds": 17.10798401699958,+ "peak_rss_mib": 292.90234375+ }+ ]+ }+ }+ },+ "second_runtime_correctness": {+ "purpose": "Check parity in the complete runtime used for full HTTP verification; these are correctness checks, not a new timing comparison.",+ "python_executable": "/workspace/scratch/1e88d5469ed3/perf-venv/bin/python",+ "pythonpath": "src",+ "environment": {+ "python": "3.12.14",+ "numpy": "2.3.5",+ "scipy": "1.17.0",+ "pandas": "2.2.3",+ "pyarrow": "26.0.1",+ "pyproj": "3.8.0",+ "pytest": "9.2.0"+ },+ "source_at_check": {+ "base_commit": "5420163f026b6c2621226add88cfa05fda326b46",+ "worktree_change": "src/dopplergeist/matching.py: _RidgeFitCache.batch_limit changed from 16*1024*1024 to 32*1024*1024, plus the cache-limit docstring."+ },+ "real_fixture": "point-120s.json",+ "pytest_files": [+ "tests/test_real_batch_scalar_parity.py",+ "tests/test_real_batch_ridge.py",+ "tests/test_real_prepared_measurements.py"+ ],+ "pytest_passed": 7,+ "pytest_failed": 0,+ "cache_check_script": "scripts/check_real_batch_cache.py",+ "cache_checks_passed": 50,+ "cache_checks_failed": 0,+ "cache_check_event": "event-0011",+ "cache_check_clock_rows": 21+ }+}diff --git a/benchmarks/results/real-discovery-plots-20261103.json b/benchmarks/results/real-discovery-plots-20261103.jsonnew file mode 100644index 0000000..f44f2d0--- /dev/null+++ b/benchmarks/results/real-discovery-plots-20261103.json@@ -0,0 +1,63 @@+{+ "baseline": {+ "trials": [+ {+ "wall_s": 4.463220791999447,+ "cpu_s": 4.383185833000001+ },+ {+ "wall_s": 4.403667614000369,+ "cpu_s": 4.401537119999999+ },+ {+ "wall_s": 4.081281009000122,+ "cpu_s": 4.070385219000002+ }+ ],+ "median_wall_s": 4.403667614000369,+ "median_cpu_s": 4.383185833000001+ },+ "current": {+ "trials": [+ {+ "wall_s": 3.3099670270003116,+ "cpu_s": 3.3087660770000005+ },+ {+ "wall_s": 3.27719621200049,+ "cpu_s": 3.2770203450000004+ },+ {+ "wall_s": 3.101882871000271,+ "cpu_s": 3.100890203999999+ }+ ],+ "median_wall_s": 3.27719621200049,+ "median_cpu_s": 3.2770203450000004+ },+ "source": {+ "source_sha256": "262d9c4eba1c80657813c604140ba2afbbaabed2b3aebc833cba7cafdc005e9f",+ "duration_s": 3600.1845,+ "sample_rate_hz": 4000,+ "channels": 1,+ "source_format": "wav",+ "time_window": null+ },+ "ridge_count": 18,+ "group_count": 16,+ "exact_png_pixels": true,+ "wall_speedup": 1.3437302282588233,+ "cpu_speedup": 1.3375522186451363,+ "baseline_commit": "aa3dff686c989524daa20c081a816b792d9c8890",+ "scope": "All event PNGs from real measured ridges; excludes intake, discovery, and equality checks",+ "runtime": {+ "python": "3.12.14",+ "numpy": "2.3.5",+ "scipy": "1.17.0",+ "matplotlib": "3.10.8"+ },+ "execution_note": "Three alternating baseline/current pairs on a shared host during full-upload work. Wall times are observations, not isolated-system estimates.",+ "default_pipeline_affected": false,+ "optimized_component": "Clip optional event diagnostic mesh to visible frequency cells with boundary bins; full spectral normalization and measured ridges unchanged.",+ "optimized_revision_note": "Timed the working-tree plotting change before final integration; the same plotting change is included in the final performance implementation."+}diff --git a/docs/PERFORMANCE.md b/docs/PERFORMANCE.mdindex f137283..4c114b4 100644--- a/docs/PERFORMANCE.md+++ b/docs/PERFORMANCE.md@@ -1,6 +1,423 @@-# Measured scoring performance+# Measured pipeline performance-The 2026-11-03 implementation reduces median CPU time by **62.87%** on the broad-clock workload described here.+The latest changes reduce median CPU time for complete point scoring by **45.94%** on the broad-clock workload.+The narrow-clock workload uses **49.56%** less CPU time.+Both comparisons preserve every scored association and its diagnostics exactly.+The changes reuse calculations and reduce data decoding without reducing the search domain, event coverage, solver iterations, or numerical precision.++These are incremental improvements over an already optimized baseline.+The earlier **2.69×** scorer result appears in the [historical comparison](#historical-scorer-comparison-2026-11-03).+The measurements cover different stages and environments, so their speedups do not combine into a complete-pipeline ratio.++## Exact fit reuse: current point-scoring comparison++Each workload used three alternating baseline/current pairs in separate Python processes.+The order was baseline/current, current/baseline, then baseline/current.+Each source used the same frozen recorded ridges, observed trajectories, candidate budgets, and receiver point.++| Recording-start window | Baseline median CPU | Current median CPU | CPU speedup | Scored associations |+| --- | ---: | ---: | ---: | ---: |+| ±120 seconds | 28.803255077 s | 15.571095273 s | **1.84979×** | 3,905 |+| ±10 seconds | 4.386338558 s | 2.212487520 s | **1.98254×** | 1,055 |++The timed operation includes complete point scoring and retention writes.+It excludes imports, input loading, fixture preparation, acquisition, discovery, nationwide search, refinement, and result JSON serialization.+The receiver point and recording clock come from a historical fit.+This experiment measures computation on actual observations, without an independent location-accuracy claim.++All six outputs within each workload match exactly, with no numerical tolerance.+The comparison includes every scored association, fitted diagnostic, support count, ranked row, and retention record, bound, and stage.+Both workloads supply all 16 independent groups, and both sources retain the same 100 associations.+The narrow workload checks that the improvement also applies to fewer clock trials.+Changing the clock window itself is not a code speedup and requires independent time evidence.++### Workload and runtime++Both fixtures contain 16 recorded ridge records and 724 ridge samples.+The broad fixture contains 452 observed trajectories, 91,889 trajectory samples, and 100 CPA rows.+The narrow fixture contains 426 observed trajectories, 87,206 trajectory samples, and 97 CPA rows.+Both retain four unsupported CPA tracks in their accounting.+The preparation command rejects a budget excess instead of silently truncating inputs.++The point timings used Python 3.12.14, **NumPy 2.5.4**, SciPy 1.17.0, pandas 2.2.3, pyarrow 26.0.1, and pyproj 3.8.0.+The host used Linux 6.18.57 and an AMD EPYC 9V74 processor.+OpenBLAS, OpenMP, MKL, and NumExpr each used one thread.+A full upload ran on the shared host during these measurements, so process CPU time is the primary metric.++| Window | Baseline median wall time | Current median wall time | Reported peak RSS, baseline → current |+| --- | ---: | ---: | ---: |+| ±120 seconds | 28.816586 s | 15.579372 s | 260.46 → 293.25 MiB |+| ±10 seconds | 4.393367 s | 2.215673 s | 210.37 → 211.93 MiB |++These wall times describe a shared host, without exclusive CPU access.+Reported RSS includes imports, input loading, and captured diagnostics through scoring.+The measurement uses `getrusage(RUSAGE_SELF).ru_maxrss`, which can retain an inherited peak across execution.+Separate `VmHWM` readings were not recorded.+These figures describe process memory, without an isolated measurement of the cache heap.++The baseline commit is `aa3dff686c989524daa20c081a816b792d9c8890`.+The measured scoring implementation is local commit `5ca480f8678ca1ed31a765768324f623a44f0da8` plus the 32 MiB batch-cache patch.+That worktree patch changed `_RidgeFitCache.batch_limit` from 16 MiB to 32 MiB and clarified the cache-limit docstring.+The harness did not record HEAD separately for each worker.+Concurrent terrain/ENF integration at local commit `5420163f026b6c2621226add88cfa05fda326b46` did not change the exercised FlightIndex or scoring paths.+Published commit `5699a657d57dc4f66b74ef000e20d3ba88f84704` contains the same exercised scoring paths and the final 32 MiB limit.+Its additional compact-checkpoint change does not affect this point-scoring measurement.+The [exact-fit measurement record](../benchmarks/results/exact-fit-reuse-20261103.json) contains all twelve final runs, source details, input hashes, and workload counts.++The complete HTTP runtime uses **NumPy 2.3.5**.+The seven selected real-input tests and all 50 cache checks also passed in that runtime, with `PYTHONPATH=src` only.+Those checks cover scalar parity, prepared measurements, cache mutation, eviction, layout, and nested scope restoration.+They provide a separate correctness check, without another point-timing measurement.+The two NumPy environments remain separate in the recorded results.++### Why exact reuse reduces work++Dense candidate traversal repeats complete batches of identical ridge fits.+The new scoped cache reuses successful batches with the same fitter, parameters, and array contents.+Its keys preserve input order, shape, dtype, strides, and alignment.+They also preserve the frequency bounds, frequency scale, and drift fraction without rounding.+Memory layout matters because existing NumPy reductions can differ by one ULP between contiguous layouts.++Each cache hit returns separate diagnostic dictionaries.+Mutations cannot alter a later cached result.+Unsupported representations use the original path, and the cache does not retain failed or unsupported results.+The cache lifetime follows the existing scoring context, including nested-scope restoration.+The scorer also reuses validated event ceilings with their complete diagnostic dependencies.+The fitting equations, six IRLS iterations, conservative screens, tie behavior, and candidate accounting remain unchanged.++### Why the batch-cache limit is 32 MiB++The batch cache holds at most 4,096 entries and 32 MiB per active scoring context.+The byte limit accounts for keys, diagnostic values, and dictionary storage.+This allocation grows with demand and is separate from the acoustic and prepared-measurement caches.++An instrumented capacity comparison used the same broad fixture in four sequential runs: 16 MiB, 32 MiB, 32 MiB, then 16 MiB.+All outputs matched exactly.+Each run made 7,351 batch lookups.++| Batch-cache limit | Median CPU, two runs | Hits per run | Fits executed per run | Evictions per run | Maximum retained bytes |+| --- | ---: | ---: | ---: | ---: | ---: |+| 16 MiB | 18.937758264 s | 5,403 | 1,948 | 1,443 | 16,777,214 |+| 32 MiB | 17.334247567 s | 6,281 | 1,070 | 17 | 33,553,175 |++The 32 MiB limit reduced CPU time by **8.47%** relative to 16 MiB in this capacity comparison.+Median reported process RSS increased from 276.97 MiB to 293.20 MiB.+The larger limit retains more repeated fits and avoids repeated eviction on this workload.+The final point-scoring table uses fresh runs with this selected limit.+The capacity runs include instrumentation and remain separate from the final speedup calculation.+The exact-fit measurement record preserves both capacities and every counter.++## Other measured stages++These comparisons use actual recorded-audio jobs and observed flight data.+Each row has its own scope and repetition count.++| Stage | Baseline CPU | Current CPU | Repetitions | Equality check |+| --- | ---: | ---: | --- | --- |+| Refinement and decision | 83.7084 s | 77.4870 s | One unprofiled pair | Complete refined candidates and final JSON |+| Optional event plots | 4.383 s | 3.277 s | Three alternating pairs | Every PNG pixel |+| Checkpoint JSON encoding | 1.32042 s | 0.28718 s | Five alternating pairs | Complete decoded payload and identifiers |++### Refinement and decision++The replay starts from fixed archived finalists, without repeating discovery or search.+The trajectory lookup now selects the named finalist IDs within the same space/time envelope before decompressing tracks.+Decoded tracks decreased from **881 to 18**, and decoded samples decreased from 233,165 to 5,303.+The fitter still receives the same required complete tracks, with the original validation and sample-budget checks.+The single unprofiled pair used **7.43%** less CPU time.+Both timed arms included the same decode counter.+It returned the same 23 candidates and the same `inconclusive / search_incomplete` result.+This is a phase observation, without a repeated-run estimate of total pipeline speed.+The [refinement measurement record](../benchmarks/results/decision-refinement-20261103.json) preserves the observations and their scope.++### Optional event plots++The plot comparison rendered all 18 event images from a complete 3,600.1845-second real recording with 16 independent groups.+The renderer computes the same full spectral normalization and passes only visible frequency bins to the plotting library.+The median CPU speedup was **1.34×**, with identical pixels in every image.+Diagnostic plots are off by default, so this gain applies only when plots are requested.+It excludes audio intake and acoustic discovery.+The [plot measurement record](../benchmarks/results/real-discovery-plots-20261103.json) preserves all three pairs, the recorded input, and the NumPy 2.3.5 environment.++### Compact internal checkpoints++The encoding comparison used one completed real tile with 19,977 candidate associations.+Compact JSON reduced its encoded size from **39,363,929 to 23,490,375 bytes**, a **40.33%** reduction.+Median encoding CPU time improved by **4.60×** across five alternating pairs.+Every decoded value, payload hash, and retention checksum remained identical.+The production change preserves checkpoint filenames, atomic writes, and resume identifiers.+Human-facing reports retain their readable formatting.++The timer covers JSON encoding only.+It excludes file writes, reads, scoring, and complete checkpoint or resume operations.+The [checkpoint measurement record](../benchmarks/results/checkpoint-encoding-20261103.json) preserves all ten runs and the decoded-output checks.++## Current complete-recording HTTP comparison++One sequential baseline/current pair completed the entire bounded HTTP workflow on the original 3,600.1845-second MP3.+It used the separate **NumPy 2.3.5** runtime on a shared host.++| Measurement | Baseline | Current |+| --- | ---: | ---: |+| HTTP verification elapsed wall time | 264.809609269 s | 188.323150374 s |+| Event records | 18 | 18 |+| Returned candidates | 23 | 23 |+| Full pipeline requirements met | Yes | Yes |++The observed elapsed-time speedup is **1.40614×**, or **28.88%** less wall time.+This is one pair, without a repeated-run estimate or a process CPU comparison.+Both runs passed the HTTP and full-pipeline verification checks.+Both returned `inconclusive / search_incomplete`.+The 18 event records form 16 independent groups.+Optional event plots remained disabled during both HTTP runs.++The request used the source-informed bounds `[-73.545, 40.865, -73.505, 40.895]` and recording-start window `2026-09-08T20:17:20Z` through `20:17:40Z`.+Both sources used the same observed flight index, terrain, one search worker, 2 km coarse spacing, and 500 m refinement spacing.+The top-K limit remained 20, with three refinement finalists.+Both searches processed four coarse points across two tiles, plus the same 40 existing refinement points.+Both skipped 80 refinement points under the existing rules.+These explicit regional bounds define the comparison workload and do not establish national completion or independent location accuracy.++The scientific comparison found zero differences across 23 job JSON artifacts, three root JSON artifacts, and the logical SQLite ledger.+It normalizes declared implementation hashes, run and job identities, performance fields, and local paths.+It excludes declared resume and worker metadata.+The performance fields are elapsed time and reported process peak RSS.+It compares the full operator configuration separately from its digest.+SQLite comparison covers all logical tables, columns, and rows, except `run.identity` and physical storage layout.+Decoded scientific values, numeric types, and signed zeros compare exactly, with no numerical tolerance.+The [pinned comparison script](https://github.com/1337isnot1337/DOPPLERGEIST/blob/a3ba35342e491556973be34022f3e0a10ac1f6f4/scripts/compare_real_upload_performance.py) defines the complete normalization rules.+The ledger preserves 48,226 bounds, 958 records, and 18 stages.+Supplementary checks also matched one intake diagnostic, one decoded PCM file, 18 CSV files, and the decoded pixels of one PNG exactly.+This workload contains no NPZ files.+The intake comparison replaces only the exact input path in the recorded decoder command.++The [bounded HTTP measurement record](../benchmarks/results/bounded-http-20261103.json) preserves the timings, settings, source identities, and equality results.+Published commit `5699a657d57dc4f66b74ef000e20d3ba88f84704` matches all 34 production source files in the frozen HTTP checkout.++A full CONUS baseline reached its declared 1,800-second limit before completion (1,800.23 seconds of wall time).+That unfinished search provides no nationwide speed ratio.+Regional source data do not establish national flight coverage or independent location accuracy.+The historical HTTP checks below describe their own revisions and workloads.++## Reproduce the current comparisons++Use Python 3.12.14 to match the recorded point-timing runtime.+Use Git and the GitHub CLI `gh` with access to the public checkpoint assets.+Run these commands from the repository root.+Use a fresh output directory for each comparison.++Install the point-timing environment and restore the observed inputs:++```bash+python3.12 -m venv .venv-point+.venv-point/bin/python -m pip install '.[test]' \+ 'numpy==2.5.4' 'scipy==1.17.0' 'pandas==2.2.3' \+ 'pyarrow==26.0.1' 'pyproj==3.8.0'+.venv-point/bin/python scripts/restore_artifacts.py \+ --bundle data-adsb-derived --bundle data-terrain+PYTHONPATH=src .venv-point/bin/python scripts/prepare_reference_upload.py \+ --data-root data --output results/incremental-inputs+```++Create separate source trees from the public baseline and optimized commits:++```bash+git fetch origin+git worktree add --detach ../dopplergeist-incremental-baseline \+ aa3dff686c989524daa20c081a816b792d9c8890+git worktree add --detach ../dopplergeist-timing-current \+ 5699a657d57dc4f66b74ef000e20d3ba88f84704+```++The public optimized commit contains the final 32 MiB limit.+The source disclosure above identifies the local commit and worktree patch used during measurement.++Freeze and compare both workloads:++```bash+export OPENBLAS_NUM_THREADS=1 OMP_NUM_THREADS=1 MKL_NUM_THREADS=1 NUMEXPR_NUM_THREADS=1+for width in 120 10; do+ PYTHONPATH=src .venv-point/bin/python scripts/benchmark_prepared_measurements.py prepare \+ --discovery evidence/audio/w1-sensitivity-20261102/reference/discovery.json \+ --index results/incremental-inputs/reference-region.sqlite \+ --clock-half-width-s "$width" \+ --output "results/incremental-inputs/point-${width}s.json"++ PYTHONPATH=src .venv-point/bin/python scripts/benchmark_prepared_measurements.py compare \+ --fixture "results/incremental-inputs/point-${width}s.json" \+ --baseline-source ../dopplergeist-incremental-baseline/src \+ --current-source ../dopplergeist-timing-current/src \+ --repeats 3 --timeout-s 600 \+ --output "results/incremental-${width}s"+done+```++The comparison rejects changed inputs, missing trajectories, invalid retention ledgers, or any complete-output mismatch.+Both arms use the same interpreter and library versions.++Run the seven selected real-input tests and the cache checks:++```bash+DOPPLERGEIST_REAL_CPA_FIXTURE=results/incremental-inputs/point-120s.json \+PYTHONPATH=src .venv-point/bin/python -m pytest -q \+ tests/test_real_batch_scalar_parity.py \+ tests/test_real_batch_ridge.py \+ tests/test_real_prepared_measurements.py++PYTHONPATH=src .venv-point/bin/python scripts/check_real_batch_cache.py \+ --tracks data/adsb/2026-09-08/reference-region.parquet \+ --discovery evidence/audio/w1-sensitivity-20261102/reference/discovery.json \+ --candidate evidence/results/reference-local/report.json+```++The separate HTTP-runtime correctness check used the same commands with NumPy 2.3.5 and no additional dependency directory on `PYTHONPATH`.+It passed all seven tests and all 50 cache checks.++For the other stage comparisons, use a separate NumPy 2.3.5 environment:++```bash+python3.12 -m venv .venv-stages+.venv-stages/bin/python -m pip install '.[test]' \+ 'numpy==2.3.5' 'scipy==1.17.0' 'pandas==2.2.3' \+ 'pyarrow==26.0.1' 'pyproj==3.8.0' 'matplotlib==3.10.8'+```++The original refinement and plot comparisons used NumPy 2.3.5.+The checkpoint encoding helper uses the Python standard library.+To repeat the separate runtime checks, use `.venv-stages/bin/python` in the previous test and cache commands.++For refinement, use a completed real job with saved finalists, flight indexes, and terrain:++```bash+PYTHONPATH=src .venv-stages/bin/python scripts/benchmark_real_decision.py \+ --source ../dopplergeist-incremental-baseline/src \+ --job /path/to/real-job --output results/decision-baseline+PYTHONPATH=src .venv-stages/bin/python scripts/benchmark_real_decision.py \+ --source ../dopplergeist-timing-current/src \+ --job /path/to/real-job --output results/decision-current \+ --compare-to results/decision-baseline+```++The script compares every refined candidate, sensitivity result, and decision after the timed operation.+It omits the exploratory decode counter from the original pair.+For repeated measurements, alternate the source order and use new output directories.++For optional plots, use saved intake and discovery outputs from a real recording:++```bash+PYTHONPATH=../dopplergeist-timing-current/src \+.venv-stages/bin/python scripts/benchmark_real_discovery_plots.py \+ /path/to/real-job --baseline ../dopplergeist-incremental-baseline \+ --repeats 3 --output results/plot-comparison+```++For checkpoint encoding and compatibility, use the original filename of a completed real tile:++```bash+.venv-stages/bin/python scripts/benchmark_real_checkpoint_encoding.py \+ --checkpoint /path/to/real-job/search/tile-ID.json --repeats 5 \+ --output results/checkpoint-encoding.json++DOPPLERGEIST_REAL_TILE_CHECKPOINT=/path/to/real-job/search/tile-ID.json \+PYTHONPATH=src .venv-stages/bin/python -m pytest -q \+ tests/test_real_compact_checkpoints.py+```++The encoding helper compares readable and compact JSON, including one trailing newline in both timed operations.+Input decoding, output decoding, file writes, and equality checks remain outside the timer.+The compatibility tests also exercise atomic replacement and saved support-accounting parity.++### Complete bounded HTTP replay++Use the NumPy 2.3.5 environment and public source worktrees from the preceding commands.+Install FFmpeg so that `ffmpeg` and `ffprobe` are on `PATH`.+The [bounded upload helper](../scripts/benchmark_real_bounded_upload.py) starts each source in a fresh HTTP service.+It uses the same input file, request, operator configuration, and interpreter for both arms.++Restore the recorded inputs and prepare the shared index:++```bash+.venv-stages/bin/python scripts/restore_artifacts.py \+ --bundle data-source --bundle data-adsb-derived --bundle data-terrain+PYTHONPATH=../dopplergeist-timing-current/src \+.venv-stages/bin/python scripts/prepare_reference_upload.py \+ --data-root data --output results/bounded-inputs++.venv-stages/bin/python - <<'PY'+import json+from pathlib import Path+root = Path('results/bounded-inputs')+request = json.loads((root / 'request.json').read_text())+request['search']['bounds_wsen'] = [-73.545, 40.865, -73.505, 40.895]+request['time_window']['source_detail'] = (+ 'Source-informed integration replay around the historical fitted recording '+ 'start and benchmark receiver region; not independently measured recording '+ 'time or a location-accuracy evaluation.'+)+operator = json.loads((root / 'operator.json').read_text())+operator['search_workers'] = 1+for name, value in (('request.json', request), ('operator.json', operator)):+ (root / name).write_text(json.dumps(value, indent=2) + '\n')+PY+```++The helper preserves the recorded 20:17:20–20:17:40 UTC window, search budgets, and refinement settings.+The public `data-source` bundle contains `reference-recording.wav`, not the original timed MP3.+For the original workload, supply the archived complete MP3:++```bash+cp /path/to/archived/reference-complete.mp3 results/bounded-inputs/recording.mp3+sha256sum results/bounded-inputs/recording.mp3+```++The measured MP3 contains 42,085,293 bytes and has SHA-256 `e1c13dafd8d1634bbf84df843c31771318e15a31ebb86dd165af7ee312ab80a0`.+If only the published WAV is available, use this command instead of the copy command:++```bash+ffmpeg -n -nostdin -i data/source/reference-recording.wav \+ -map 0:a:0 -vn -map_metadata -1 -ac 1 -ar 48000 \+ -c:a libmp3lame -b:a 96k results/bounded-inputs/recording.mp3+```++This conversion creates a new encoding of the real recording.+It defines a new comparison workload and requires fresh results.+The original hash, event counts, candidate counts, and timings do not apply automatically.+Both arms must use the same resulting MP3 file.++Run the two fresh HTTP arms sequentially:++```bash+export OPENBLAS_NUM_THREADS=1 OMP_NUM_THREADS=1 MKL_NUM_THREADS=1 NUMEXPR_NUM_THREADS=1+.venv-stages/bin/python scripts/benchmark_real_bounded_upload.py \+ --source ../dopplergeist-incremental-baseline/src \+ --inputs results/bounded-inputs --output results/http-baseline --timeout-s 900+.venv-stages/bin/python scripts/benchmark_real_bounded_upload.py \+ --source ../dopplergeist-timing-current/src \+ --inputs results/bounded-inputs --output results/http-current --timeout-s 900+```++Fetch the pinned comparison tools and run both equality checks:++```bash+git fetch origin a3ba35342e491556973be34022f3e0a10ac1f6f4+git worktree add --detach ../dopplergeist-comparison-tools \+ a3ba35342e491556973be34022f3e0a10ac1f6f4+.venv-stages/bin/python \+ ../dopplergeist-comparison-tools/scripts/compare_real_upload_performance.py \+ --baseline results/http-baseline --current results/http-current \+ --output results/http-science-comparison.json+.venv-stages/bin/python scripts/check_real_upload_arrays.py \+ --baseline results/http-baseline --current results/http-current \+ --output results/http-array-comparison.json+```++The comparator pin contains the exact comparison implementation used for the recorded pair.+The separate tools worktree preserves its source identity without changing either measured production source.+The array check covers the saved artifacts that actually exist in each workload.++## Historical scorer comparison (2026-11-03)++The earlier scorer implementation reduced median CPU time by **62.87%** on its broad-clock workload.The same workload takes **68.92 → 25.59 CPU seconds**, a **2.69× speedup**.Every scored candidate, diagnostic, support count, and retention record matches the scalar-consistent baseline exactly.@@ -9,7 +426,7 @@ They exclude discovery, downloads, input loading, nationwide search, and refinemThe receiver point and recording clock come from a historical fit.This experiment does not establish independent location accuracy or a nationwide completion time.-## Measurements+### MeasurementsEach result is the median of three runs per source in separate Python processes.The harness alternates baseline/current, current/baseline, then baseline/current.@@ -33,7 +450,7 @@ The machine used an AMD EPYC 9V74 processor, Linux 6.18.57, and Python 3.12.14.Package versions were NumPy 2.3.5, SciPy 1.17.0, and pandas 2.2.3.[The measurement record](../benchmarks/results/performance-20261103.json) contains every run, the input hash, source commits, and workload counts.-### Fixed workloads+#### Fixed workloadsBoth workloads use all 16 recorded ridge records, 16 independent groups, and 724 ridge samples from the archived discovery result.The trajectory query retains complete recorded tracks within the declared query.@@ -55,7 +472,7 @@ The two clock windows define different workloads.The difference between those workloads is not a code speedup.A narrower clock window requires independent recording-time evidence.-## Changes that reduce work+### Changes that reduce work**Batch surviving clocks after each group.**The scorer preserves the event order within each clock trial.@@ -91,7 +508,7 @@ The staged table has a limit of 4,096 event-clock slots.Larger products use the existing scalar traversal.This limit changes the execution path without reducing the search domain.-## Numerical baseline+### Numerical baselineThe untouched starting revision is `7a8dd27979bd9a984c8675c38a00dfbf98a5b01e`.Its batch fitter uses NumPy array squaring where the scalar fitter uses Python scalar powers.@@ -114,7 +531,7 @@ One discarded candidate retains the legacy rounding difference in its old diagnoExact equality with untouched main is therefore not the correctness claim.An earlier single observation took 77.43 CPU seconds on untouched main, but the table uses the repeated scalar-consistent baseline.-## Reproduce the comparison+### Reproduce the historical comparisonUse Python 3.11 or later, Git, and the GitHub CLI `gh` configured for release downloads.Run these POSIX-shell commands from the repository root.@@ -178,7 +595,7 @@ It uses two pairs per workload and saves the raw timing summaries.The workflow also runs the focused real-input tests.It does not run the historical synthetic suite.-## Method for further optimization+### Method for further optimization1. Freeze the source, recording, trajectory index, clock window, and search budgets.2. Profile a complete bounded workload with all independent groups.@@ -200,7 +617,7 @@ An existing process continues to use its loaded source.New measurements require a new run with the new source identity.Checkpoint reuse must pass the existing source and input identity rules.-## Independent runner and integration validation+### Independent runner and integration validationThe [independent CI run](https://github.com/1337isnot1337/DOPPLERGEIST/actions/runs/41656583534) passed all eight focused real-input tests.Both complete-output comparisons also passed, with two alternating pairs per workload.@@ -215,7 +632,7 @@ The GitHub runner used Python 3.12, NumPy 2.5.4, SciPy 1.18.1, and pandas 3.1.0[The CI measurement record](../benchmarks/results/performance-ci-20261103.json) preserves all eight runs and their source revisions.These ratios support the local result on a second environment without pooling timings from different machines.-### Integration with current main+#### Integration with main at d6a0c860The tested integration commit is `d6a0c860bd5a64797b3bde29bb1f74c689d0ed52`.It includes main `6d3e3f7`, with PR #210's configured workers and PR #213's association traversal optimization.@@ -228,7 +645,7 @@ Narrow-window median CPU time was **5.6747 → 4.0153 seconds (1.4133×)**.The CI measurement record includes all eight integration runs separately from the earlier measurements.Python 3.11 and 3.14 installation, compilation, CLI, and packaged-asset checks also passed.-### Complete-recording HTTP checks+#### Complete-recording HTTP checksThe full-domain HTTP replay used the entire 3,600.1845-second reference recording and an explicit source-informed clock.It completed intake, discovery, and acquisition, then reached its 900-second limit during adaptive geographic search.diff --git a/scripts/benchmark_real_bounded_upload.py b/scripts/benchmark_real_bounded_upload.pynew file mode 100644index 0000000..7fa206d--- /dev/null+++ b/scripts/benchmark_real_bounded_upload.py@@ -0,0 +1,38 @@+"""Run a fresh bounded HTTP workflow with explicit real-recording inputs.++The input directory must contain recording.mp3, request.json, and operator.json.+Pass the src directory of a pinned source checkout and a new output directory+for each arm. The fixed archived receiver bounds are source-informed and do not+establish independent location accuracy. No observations or fixtures are generated.+"""+import argparse+import json+from pathlib import Path+import sys++parser = argparse.ArgumentParser(description=__doc__)+parser.add_argument("--source", type=Path, required=True)+parser.add_argument("--inputs", type=Path, required=True)+parser.add_argument("--output", type=Path, required=True)+parser.add_argument("--timeout-s", type=float, default=900.0)+args = parser.parse_args()+source = args.source.resolve()+sys.path[:0] = [str(source), str(source.parent / "scripts")]++import dopplergeist+from verify_upload_http import verify_http++if Path(dopplergeist.__file__).resolve().parent != source / "dopplergeist":+ raise RuntimeError("Verifier imported a different source tree")+request = json.loads((args.inputs / "request.json").read_text())+operator = json.loads((args.inputs / "operator.json").read_text())+if request.get("search", {}).get("bounds_wsen") != [-73.545, 40.865, -73.505, 40.895]:+ raise ValueError("This benchmark requires the archived four-point receiver bounds")+receipt = verify_http(args.inputs / "recording.mp3", args.output,+ request=request, operator=operator, expect_status="inconclusive",+ expect_reason="search_incomplete", timeout_s=args.timeout_s,+ poll_interval_s=1.0, require_full_pipeline=True)+print(json.dumps({key: receipt.get(key) for key in+ ("verification_passed", "status", "reason_code", "full_pipeline_observed",+ "full_pipeline_requirements_met", "elapsed_s", "job_id")}))+raise SystemExit(0 if receipt["verification_passed"] else 1)diff --git a/scripts/benchmark_real_checkpoint_encoding.py b/scripts/benchmark_real_checkpoint_encoding.pynew file mode 100644index 0000000..c1e05a7--- /dev/null+++ b/scripts/benchmark_real_checkpoint_encoding.py@@ -0,0 +1,93 @@+"""Measure JSON encoding of an existing completed real-search checkpoint.++Supply an actual tile file with its original payload hash and retention checksum.+The source is never modified. Both encodings must preserve the complete decoded+record exactly. Timing covers json.dumps and its trailing newline only; loading,+decoding, comparison, file writes, and search/scoring are excluded.+"""+from __future__ import annotations++import argparse+import hashlib+import json+from pathlib import Path+import platform+import statistics+import time+++def canonical(value):+ """Preserve scalar representation and list order; ignore object-key order."""+ return json.dumps(value, sort_keys=True, allow_nan=False)+++def main(argv=None):+ parser = argparse.ArgumentParser(description=__doc__)+ parser.add_argument("--checkpoint", type=Path, required=True,+ help="Original completed tile-*.json from a real search")+ parser.add_argument("--output", type=Path, required=True, help="New summary JSON file")+ parser.add_argument("--repeats", type=int, default=5)+ args = parser.parse_args(argv)+ if args.repeats < 1:+ parser.error("--repeats must be positive")+ if args.output.exists():+ raise FileExistsError(f"Choose a new output file: {args.output}")+ source = args.checkpoint.resolve()+ if not source.name.startswith("tile-") or source.suffix != ".json":+ parser.error("Supply the original completed tile-*.json filename")+ original = source.read_text()+ value = json.loads(original)+ payload = value["payload"]+ candidates = payload["counts"]["candidates"]+ if type(candidates) is not int or candidates < 1:+ raise ValueError("The recorded tile must contain actual candidate associations")+ if not value["run_hash"] or not payload["retention_checksum"]:+ raise ValueError("The tile is missing its run or retention identity")+ payload_hash = hashlib.sha256(canonical(payload).encode()).hexdigest()+ if payload_hash != value["payload_hash"]:+ raise ValueError("The original tile payload hash does not match")+ expected = canonical(value)+ runs = []+ for repetition in range(args.repeats):+ order = ("pretty", "compact") if repetition % 2 == 0 else ("compact", "pretty")+ for name in order:+ compact = name == "compact"+ started_wall, started_cpu = time.perf_counter(), time.process_time()+ encoded = json.dumps(value, indent=None if compact else 2,+ separators=(",", ":") if compact else None,+ allow_nan=False) + "\n"+ wall, cpu = time.perf_counter() - started_wall, time.process_time() - started_cpu+ decoded = json.loads(encoded)+ if canonical(decoded) != expected:+ raise AssertionError(f"Complete decoded record changed: {name}")+ if (hashlib.sha256(canonical(decoded["payload"]).encode()).hexdigest() != payload_hash+ or decoded["payload"]["retention_checksum"] != payload["retention_checksum"]):+ raise AssertionError(f"Checkpoint identity changed: {name}")+ runs.append({"repetition": repetition + 1, "encoding": name,+ "cpu_seconds": cpu, "wall_seconds": wall,+ "bytes": len(encoded.encode())})+ del decoded, encoded+ summary = {+ "schema_version": 1,+ "scope": "JSON encoding only for a completed real tile; no scoring or pipeline speed claim",+ "input": str(source), "input_bytes": source.stat().st_size,+ "candidate_associations": candidates, "python": platform.python_version(),+ "repeats_per_encoding": args.repeats,+ "all_decoded_outputs_exact": True, "payload_hash_unchanged": True,+ "retention_checksum_unchanged": True, "runs": runs,+ }+ for name in ("pretty", "compact"):+ selected = [row for row in runs if row["encoding"] == name]+ summary[name] = {key: statistics.median(row[key] for row in selected)+ for key in ("cpu_seconds", "wall_seconds", "bytes")}+ summary["cpu_speedup"] = summary["pretty"]["cpu_seconds"] / summary["compact"]["cpu_seconds"]+ summary["size_reduction_fraction"] = 1 - summary["compact"]["bytes"] / summary["pretty"]["bytes"]+ args.output.parent.mkdir(parents=True, exist_ok=True)+ with args.output.open("x") as output:+ json.dump(summary, output, indent=2, allow_nan=False)+ output.write("\n")+ print(json.dumps({key: value for key, value in summary.items() if key != "runs"}, allow_nan=False))+++if __name__ == "__main__":+ main()diff --git a/scripts/benchmark_real_decision.py b/scripts/benchmark_real_decision.pynew file mode 100644index 0000000..adb8cfe--- /dev/null+++ b/scripts/benchmark_real_decision.py@@ -0,0 +1,130 @@+"""Replay refinement and decision from an existing recorded-audio job.++The job must have a saved search result, prebuilt flight indexes, and local+terrain files. All referenced inputs must remain available. No audio or aircraft+observations are generated, and no data are downloaded. Timing excludes imports,+input preparation, and output comparison. This is a phase benchmark, not a new+search or an independent location-accuracy evaluation.++Run once with a baseline source tree, then use --compare-to on the second run+to compare every refined candidate, sensitivity result, and decision exactly.+Use a new output directory for each run and alternate source order for repeats.+"""+from __future__ import annotations++import argparse+import json+import os+from pathlib import Path+import sys+import time+++def read_json(path):+ return json.loads(Path(path).read_text())+++def write_new(path, value):+ with Path(path).open("x") as output:+ json.dump(value, output, sort_keys=True, indent=2)+ output.write("\n")+++def compare_outputs(actual, reference):+ """Ignore object-key order; require exact scalar values and array order."""+ names = ["refined-candidates.json", "decision.json", "result.json"]+ if any((Path(folder) / "spectral-comparison.json").exists() for folder in (actual, reference)):+ names.append("spectral-comparison.json")+ mismatches = []+ for name in names:+ paths = [Path(folder) / name for folder in (actual, reference)]+ if not all(path.is_file() for path in paths):+ mismatches.append(name)+ continue+ values = [json.dumps(read_json(path), sort_keys=True, separators=(",", ":")) for path in paths]+ if values[0] != values[1]:+ mismatches.append(name)+ return {"reference": str(Path(reference).resolve()), "artifacts": names,+ "equal": not mismatches, "mismatches": mismatches}+++def main(argv=None):+ parser = argparse.ArgumentParser(description=__doc__)+ parser.add_argument("--source", type=Path, required=True,+ help="Source directory containing the dopplergeist package")+ parser.add_argument("--job", type=Path, required=True,+ help="Existing real job directory with saved search and input metadata")+ parser.add_argument("--output", type=Path, required=True, help="New output directory")+ parser.add_argument("--compare-to", type=Path,+ help="Previous replay directory; fail if any result artifact differs")+ args = parser.parse_args(argv)+ if args.output.exists():+ raise FileExistsError(f"Choose a new output directory: {args.output}")+ for variable in ("OPENBLAS_NUM_THREADS", "OMP_NUM_THREADS", "MKL_NUM_THREADS"):+ os.environ[variable] = "1"+ source = args.source.resolve()+ sys.path.insert(0, str(source))++ import dopplergeist+ from dopplergeist import automatic+ from dopplergeist.contracts import validate_event+ from dopplergeist.data_access import FlightIndex+ from dopplergeist.intake import IntakeRecording+ from dopplergeist.matching import prepare_events+ from dopplergeist.resource_usage import process_peak_rss_kib+ from dopplergeist.terrain import Terrain++ if Path(dopplergeist.__file__).resolve().parent != source / "dopplergeist":+ raise RuntimeError("Replay imported a different source tree")+ config = read_json(args.job / "operator.json")+ search = read_json(args.job / "search/search-report.json")+ ground = config.get("terrain", {})+ if not config.get("flight_indexes") or not (ground.get("rasters") or "constant_m" in ground):+ raise ValueError("Replay requires explicit flight_indexes and local terrain in operator.json")+ if not search.get("ranked"):+ raise ValueError("The recorded search result contains no finalists to refine")+ events = [validate_event(event) for event in read_json(args.job / "discovery/discovery.json")["events"]]+ prepared, measurements = prepare_events(+ events, c=config.get("sound_speed_mps", 343.0),+ max_distance_m=config.get("max_distance_m", 20000.0))+ indexes = [FlightIndex(path) for path in config["flight_indexes"]]+ terrain = Terrain(ground.get("rasters"), constant_m=ground.get("constant_m"),+ geoid_path=ground.get("geoid_raster"))+ recording = IntakeRecording(args.job / "intake/recording.f32",+ read_json(args.job / "intake/recording.json"),+ read_json(args.job / "intake/intake.json"))+ state = {"recording": recording.metadata,+ "coverage": automatic._coverage(read_json(args.job / "request.json"), search, config)}+ args.output.mkdir(parents=True, exist_ok=False)++ def finish(status, reason, message, **fields):+ return dict(status=status, reason_code=reason, message=message, **fields)++ start, cpu_start = time.perf_counter(), time.process_time()+ result = automatic._decide(+ search, events, prepared, measurements, indexes, terrain, state,+ args.output, config, finish, lambda: None, recording=recording)+ elapsed, cpu = time.perf_counter() - start, time.process_time() - cpu_start+ summary = {+ "wall_seconds": elapsed, "cpu_seconds": cpu,+ "process_peak_rss_mib": process_peak_rss_kib() / 1024,+ "source": str(source), "job": str(args.job.resolve()),+ "numerical_library_threads": 1,+ "ridge_records": len(events), "independent_groups": len(prepared),+ "search_finalists": len(search["ranked"]),+ "refinement_limit": config.get("refine_finalists", 5),+ "returned_candidates": len(result.get("candidates", [])),+ "status": result["status"], "reason_code": result["reason_code"],+ "scope": "Refinement/decision from frozen recorded-audio finalists; no search or accuracy measurement",+ }+ write_new(args.output / "result.json", result)+ if args.compare_to is not None:+ summary["comparison"] = compare_outputs(args.output, args.compare_to)+ write_new(args.output / "summary.json", summary)+ print(json.dumps(summary), flush=True)+ if args.compare_to is not None and not summary["comparison"]["equal"]:+ raise SystemExit("Decision replay differs: " + ", ".join(summary["comparison"]["mismatches"]))+++if __name__ == "__main__":+ main()diff --git a/scripts/check_real_upload_arrays.py b/scripts/check_real_upload_arrays.pynew file mode 100644index 0000000..605dd8b--- /dev/null+++ b/scripts/check_real_upload_arrays.py@@ -0,0 +1,209 @@+#!/usr/bin/env python3+"""Supplement JSON/SQLite parity with the actual arrays from fresh HTTP runs.++Both runs must have completed full-workflow verification and stopped their+services. Compare complete intake diagnostics, decoded PCM and CSV bytes, NPZ+member arrays, and decoded PNG pixels. Verify the selected files against each+run's finalized job manifest before reading their contents.+"""+from __future__ import annotations++import argparse+from collections import Counter+import hashlib+import json+from pathlib import Path++import numpy as np+++ARRAY_SUFFIXES = {".f32", ".csv", ".npz", ".png"}+++def read_json(path):+ def reject(value):+ raise ValueError(f"Nonfinite JSON constant in {path}: {value}")+ return json.loads(path.read_text(), parse_constant=reject)+++def same_file_bytes(left, right):+ """Compare exact bytes with bounded memory, including signed-zero bits."""+ with left.open("rb") as first, right.open("rb") as second:+ while True:+ a, b = first.read(1024 * 1024), second.read(1024 * 1024)+ if a != b:+ return False+ if not a:+ return True+++def same_array_bits(left, right):+ """Require equal dtype, shape, and element bits, independent of layout."""+ if left.dtype != right.dtype or left.shape != right.shape:+ return False+ if left.dtype.hasobject:+ raise ValueError("Object arrays are not supported by the scientific array comparator")+ with np.nditer([left, right], flags=["external_loop", "buffered", "zerosize_ok"],+ op_flags=[["readonly"], ["readonly"]], order="C", buffersize=65536) as iterator:+ return all(a.tobytes() == b.tobytes() for a, b in iterator)+++class Run:+ def __init__(self, directory):+ self.directory = Path(directory).resolve()+ receipt_path = self.directory / "http-verification.json"+ if not receipt_path.is_file():+ raise ValueError(f"Run has no finalized HTTP verification receipt: {self.directory}")+ self.receipt = read_json(receipt_path)+ if (not self.receipt.get("finished_utc")+ or self.receipt.get("service", {}).get("returncode") != 0+ or self.receipt.get("service", {}).get("forced_kill") is True+ or not all(self.receipt.get(key) is True for key in+ ("verification_passed", "implementation_unchanged",+ "full_pipeline_observed", "full_pipeline_requirements_met"))):+ raise ValueError(f"Run is unfinished or lacks successful full-workflow verification: {self.directory}")+ digest, source_bytes = (self.receipt.get(key) for key in ("source_sha256", "source_bytes"))+ if (not isinstance(digest, str) or len(digest) != 64+ or any(value not in "0123456789abcdef" for value in digest)+ or type(source_bytes) is not int or source_bytes <= 0):+ raise ValueError("HTTP receipt has no valid source identity and byte count")+ job_id = self.receipt.get("job_id")+ if not isinstance(job_id, str) or not job_id or Path(job_id).name != job_id or job_id in {".", ".."}:+ raise ValueError("HTTP receipt contains an invalid job identifier")+ self.job = (self.directory / "jobs" / job_id).resolve()+ if not self.job.is_relative_to(self.directory / "jobs"):+ raise ValueError("Job directory leaves the verified run")+ final = read_json(self.job / "result.json")+ if (final.get("status") in {"queued", "running"}+ or any(final.get(key) != self.receipt.get(key) for key in ("status", "reason_code"))+ or len(final.get("events", [])) != self.receipt.get("event_count")+ or len(final.get("candidates", [])) != self.receipt.get("candidate_count")):+ raise ValueError("Final job result does not match its completed HTTP receipt")+ provenance = read_json(self.job / "provenance.json")+ if ((self.directory / "continuation-start.json").exists() or "resumed_from" in provenance):+ raise ValueError("This comparator requires fresh HTTP runs, without continuation or resume")+ manifest = read_json(self.directory / "job-manifest.json")+ if manifest.get("job_id") != job_id or not isinstance(manifest.get("files"), dict):+ raise ValueError("Finalized job manifest does not identify this HTTP job")+ self.manifest = manifest["files"]+ self.inventory = {path.relative_to(self.job).as_posix(): path+ for path in self.job.rglob("*")+ if path.is_file() and path.suffix.lower() in ARRAY_SUFFIXES}+ expected = {name for name in self.manifest if Path(name).suffix.lower() in ARRAY_SUFFIXES}+ if set(self.inventory) != expected:+ raise ValueError("Actual array inventory differs from the finalized job manifest: "+ f"missing={sorted(expected - self.inventory.keys())}, "+ f"extra={sorted(self.inventory.keys() - expected)}")+ if "intake/recording.f32" not in self.inventory:+ raise ValueError("Verified full workflow has no decoded PCM artifact")+ self.verified = 0++ def verify_file(self, relative):+ path = self.job / relative+ if path.is_symlink() or not path.resolve().is_relative_to(self.job):+ raise ValueError(f"Artifact leaves its verified job: {relative}")+ with path.open("rb") as stream:+ actual = {"bytes": path.stat().st_size,+ "sha256": hashlib.file_digest(stream, "sha256").hexdigest()}+ if self.manifest.get(relative) != actual:+ raise ValueError(f"Artifact no longer matches the finalized job manifest: {relative}")+ self.verified += 1+ return actual++ def intake(self):+ self.verify_file("intake/intake.json")+ value = read_json(self.job / "intake/intake.json")+ # This command argument is the sole run-specific field in intake.+ # Preserve every other field, including downmix and sample-zero policy.+ command = value.get("decode", {}).get("command")+ if isinstance(command, list):+ source = str(self.job / "input.mp3")+ value["decode"]["command"] = ["<job>/input.mp3" if item == source else item for item in command]+ return value+++def compare_runs(baseline, current):+ left, right = Run(baseline), Run(current)+ differences, checked, pcm = [], Counter(), {}++ def different(artifact, reason, **details):+ differences.append({"artifact": artifact, "reason": reason, **details})++ sources = [{key: run.receipt.get(key) for key in ("source_sha256", "source_bytes")}+ for run in (left, right)]+ if sources[0] != sources[1]:+ different("HTTP source", "source_bytes_or_identity_changed", baseline=sources[0], current=sources[1])+ intake = [run.intake() for run in (left, right)]+ encoded = [json.dumps(value, sort_keys=True, allow_nan=False) for value in intake]+ if encoded[0] != encoded[1]:+ different("intake/intake.json", "diagnostics_changed", baseline=intake[0], current=intake[1])+ checked["intake_diagnostics"] += 1+ inventories = [set(run.inventory) for run in (left, right)]+ if inventories[0] != inventories[1]:+ different("array inventory", "missing_or_extra_artifacts",+ baseline_only=sorted(inventories[0] - inventories[1]),+ current_only=sorted(inventories[1] - inventories[0]))+ for name in sorted(inventories[0] & inventories[1]):+ paths = [run.inventory[name] for run in (left, right)]+ records = [run.verify_file(name) for run in (left, right)]+ suffix = paths[0].suffix.lower()+ if suffix in {".f32", ".csv"}:+ if not same_file_bytes(*paths):+ different(name, "file_bytes_changed", baseline=records[0], current=records[1])+ checked["decoded_pcm" if suffix == ".f32" else "csv"] += 1+ if suffix == ".f32":+ pcm[name] = {"baseline": records[0], "current": records[1]}+ elif suffix == ".png":+ from matplotlib.image import imread+ arrays = [imread(path) for path in paths]+ if not same_array_bits(*arrays):+ different(name, "decoded_pixels_changed", baseline_shape=list(arrays[0].shape),+ current_shape=list(arrays[1].shape))+ checked["png"] += 1+ elif suffix == ".npz":+ with np.load(paths[0], allow_pickle=False) as first, np.load(paths[1], allow_pickle=False) as second:+ members = [archive.files for archive in (first, second)]+ if any(len(names) != len(set(names)) for names in members):+ raise ValueError(f"NPZ contains duplicate array names: {name}")+ if set(members[0]) != set(members[1]):+ different(name, "array_members_changed", baseline=sorted(members[0]), current=sorted(members[1]))+ for member in sorted(set(members[0]) & set(members[1])):+ arrays = [archive[member] for archive in (first, second)]+ if not same_array_bits(*arrays):+ different(name, "array_dtype_shape_or_bits_changed", member=member,+ baseline={"dtype": str(arrays[0].dtype), "shape": list(arrays[0].shape)},+ current={"dtype": str(arrays[1].dtype), "shape": list(arrays[1].shape)})+ checked["npz_arrays"] += 1+ checked["npz"] += 1+ return {"passed": not differences, "baseline": str(left.directory), "current": str(right.directory),+ "scope": "Supplementary exact intake/array/pixel comparison of two verified fresh HTTP workflows",+ "checked": {key: checked[key] for key in+ ("intake_diagnostics", "decoded_pcm", "csv", "npz", "npz_arrays", "png")},+ "manifest_verified_artifacts": {"baseline": left.verified, "current": right.verified},+ "source": sources[0], "decoded_pcm": pcm,+ "difference_count": len(differences), "differences": differences,+ "normalization": "Only the exact input.mp3 path in intake.decode.command is replaced.",+ "array_rule": "Equal member names, dtype, shape, and element bits; no numerical tolerance. Memory layout does not affect equality.",+ "limitations": ["PNG encoding metadata and NPZ ZIP packaging are not compared.",+ "The separate JSON/SQLite comparator remains required.",+ "Equal outputs establish baseline consistency, not location accuracy."]}+++def main():+ parser = argparse.ArgumentParser(description=__doc__)+ parser.add_argument("--baseline", type=Path, required=True)+ parser.add_argument("--current", type=Path, required=True)+ parser.add_argument("--output", type=Path, required=True, help="New JSON report path")+ args = parser.parse_args()+ report = compare_runs(args.baseline, args.current)+ args.output.parent.mkdir(parents=True, exist_ok=True)+ with args.output.open("x") as stream:+ json.dump(report, stream, indent=2, allow_nan=False)+ stream.write("\n")+ print(json.dumps({"report": str(args.output), "passed": report["passed"],+ "checked": report["checked"], "difference_count": report["difference_count"]}))+ return 0 if report["passed"] else 1+++if __name__ == "__main__":+ raise SystemExit(main())