Skip to content

Python tool for aircraft Doppler audio geolocation using historical ADS-B flight data (Source-available)

9.4 KB
Raw Blame

Flight checking and sieve performance

Measured result

The final integrated version combines these changes with the prepared-input and prefix-cache work from PR #256, pinned at 5eaee3a43120af13c9bc16ef7f03b782251cdac0. Three alternating recorded-input pairs measured 1.70x faster query-through-retention processing and 41% less CPU time. These are direct measurements of the combined implementation; the individual speedups below are not multiplied.

The timed source combined PR #256 at 4c213fa36777806bb24f4541eff3bc302fa24f6d with these changes. The subsequent 5eaee3a update preserves exact immutable NumPy float64 metadata from live discovery, allowing prepared-input and prefix caches to remain eligible. Recorded JSON inputs do not retain that scalar type. Its recorded-input tests verify the live metadata behavior; the timing series was not rerun for this update.

Final integrated scope Baseline Integrated version Improvement
Complete measured CPU time 33.451 s 19.724 s 1.70x
Complete measured wall time 33.505 s 19.760 s 1.70x
Query and distance-sieve CPU time 6.276 s 0.345 s 18.20x
CPA, association scoring, and retention CPU time 27.215 s 19.379 s 1.40x
Worker peak resident memory 498.375 MiB 374.262 MiB 24.9% less

Worker memory uses Linux /proc/self/status VmHWM after exec. It includes imports and input loading. It excludes the comparison parent's memory, which can otherwise contaminate ru_maxrss across fork/exec.

The integrated code retains the existing scoring-cache implementation and adds native-batch identity to prepared-input, fit-snapshot, and prefix-cache guards. Changed custom hooks therefore invalidate reused fits and eliminated prefixes. The recorded-input integration suite passed 143 checks plus four subtests; three additional recorded hook regressions also passed.

Isolated contribution of this change

The clock-partition query bound removes impossible whole flight segments before JSON decoding, spline construction, distance checks, and closest-approach work. The dense recorded replay queried 197 segments instead of 4,694. It preserved all 16,101 scored candidate payloads, the 100 retained results, and the complete retention records and trial evidence.

Three alternating baseline/current pairs gave these median CPU times:

Scope Baseline This change Speedup
Index query and conservative distance sieve 6.413 s 0.398 s 16.11x
CPA, association scoring, and retention 26.500 s 24.964 s 1.06x
Complete measured hot path 34.073 s 25.356 s 1.34x

Each row has its own median; stage medians need not sum to the total median. Total wall-time medians were 34.146 and 25.400 seconds. The baseline is b5ff48fe0e9d6e6feac7a4faf5268630f0c85d8b, which already includes native ridge and acoustic solvers and numeric flight decoding. Earlier gains are not credited here.

This replay uses the original recording's 16 prepared event groups and 724 ridge samples, one source-informed receiver, a full-day regional observation index, and one existing 348-second association ownership partition. It is a computation benchmark, not an independent location-accuracy or nationwide-throughput result.

Timing includes index traversal, track decoding, the distance sieve, CPA processing, association scoring, and retention. It excludes imports, index metadata, event preparation, downloads, terrain sampling, audio discovery, and refinement. Both sources used the same Python 3.12 runtime, dependencies, single-thread numerical settings, CPU affinity, recorded inputs, and GCC flags. The measured numerical functions were not replaced by counting wrappers.

What changed

Select possible source times before decoding tracks

Let the first matched passage imply recording start A inside the existing half-open partition [L, U). Its clock interval has tolerance t_first. Each effective tolerance is quality.cpa_tolerance_s + offset_tolerance_s / 2. A later passage with implied start B and tolerance t_event can participate only if its interval intersects the first interval. Thus it lies within the partition expanded by both tolerances. Any event may be first, so the bound uses the maximum effective tolerance across every event.

Add each event's audio-relative CPA time to bound receive time. Subtract at most maximum_CPA_radius / sound_speed to bound emission time. Every inverse operation rounds outward, including subtraction at Unix timestamp precision.

The query selects complete segments touching this envelope. It never clips a trajectory, removes an interior sample, or changes its spline. Serial and worker paths derive the same anchor once and use it for both querying and scoring. Silence filters, noncanonical settings, invalid tolerances, and disjoint query windows keep the original broad query.

On the dense replay:

Work Baseline This change
Decoded flight segments 4,694 197
Decoded source samples 713,100 40,690
CPA rows 944 95
Association work units 302,818 147,985
Unsupported CPA track evaluations 62 4

Every removed unsupported track's complete emission interval misses the proven query envelope. All four retained unsupported reports remain identical. The 590,900 unknown trials and 15,777 unclassified candidate assessments also remain unchanged. The replay remains incomplete where its original evidence was incomplete.

Fuse closest-approach neighbor calculations

An optional C extension traverses the existing bounded spatial-neighbor graph. It combines squared distances, local-minimum selection, and adjacent-segment projection without creating expanded observer-index and coordinate arrays. It preserves arithmetic order, asymmetric minima, first-edge ties, and row order. Python still performs endpoint spline optimization and final row construction.

Five alternating trials of ten iterations on eight archived ADS-B segments gave these full CPA-stage results, including preparation:

Receiver points Baseline CPU per iteration Native CPA Speedup
1 1.411 ms 1.091 ms 1.29x
32 3.598 ms 2.767 ms 1.30x
128 4.977 ms 3.572 ms 1.39x
1,024 15.017 ms 10.183 ms 1.47x
4,096 48.600 ms 38.321 ms 1.27x

All 205 baseline comparisons matched exactly. This small real-track workload measures the CPA implementation, not complete search throughput.

The extension is optional. Unsupported layouts, floating-point policies, or exceptional arithmetic retain the Python path. Set DOPPLERGEIST_CPA_BACKEND=python before starting the process to select it explicitly. Checkpoints record the active backend. Existing implementation-mismatch checks require a new output directory for ordinary checkpoints from older code. The separately supported legacy continuation requires Python ridge, acoustic, and CPA backends.

Batch the unchanged native ridge fit

The native ridge extension now accepts a matrix of clock trials. It reuses scratch storage and releases the GIL once per batch. Each row keeps the existing six IRLS iterations, pairwise reductions, box constraints, and output arithmetic. Custom scalar hooks, old extension binaries, unusual inputs, and failed probes retain the established fallback. Native output may not overlap its inputs.

The dense replay uses 5,421 native batches for 227,378 trial rows, plus 862 scalar fits. Batching alone measured a modest 1.06x dense improvement; its sparse replay was approximately neutral. The main sieve gain comes from doing less impossible work before decoding.

Reproduce and validate

Build the optional extensions with CC=gcc python setup.py build_ext --inplace. Use the recorded-input preparation in .github/workflows/real-performance.yml; it restores public observations and creates benchmark-inputs/dense-query.json. Build the same existing native extensions in the pinned baseline checkout.

OPENBLAS_NUM_THREADS=1 OMP_NUM_THREADS=1 MKL_NUM_THREADS=1 \
python scripts/benchmark_prepared_measurements.py compare \
  --baseline-source /path/to/baseline/src --current-source src \
  --fixture benchmark-inputs/dense-query.json \
  --index benchmark-inputs/reference-region.sqlite \
  --include-track-query --clock-partition-query \
  --require-current-native --require-current-native-acoustic \
  --require-current-native-cpa --repeats 3 --output results/flight-sieve

python scripts/benchmark_real_cpa_rows.py \
  --baseline-source /path/to/baseline/src --points 1 32 128 1024 4096 \
  --repeats 5 --iterations 10 --skip-reuse-comparison \
  --output results/flight-cpa.json

Use one available CPU affinity for both arms when comparing elapsed time. The normal comparison requires exact complete output equality. Partition-query mode also requires exact scored outcomes, rankings, trial evidence, and retention. Only declared early-work counts may decrease; each removed unsupported interval must be outside the query. Repeated outputs must be deterministic within each arm. Profiler runs and numerical wrapper instrumentation cannot produce speed claims.

The added recorded-input tests cover native arithmetic, buffer overlap, fallback behavior, outward clock boundaries, complete-track preservation, serial/worker query integration, and checkpoint backend identity. No generated recordings or aircraft tracks are used. The remaining dominant cost is association scoring.