Skip to content

Python tool for aircraft Doppler audio geolocation using historical ADS-B flight data (Source-available)

14.1 KB
Raw Blame

Implementation handoff

Optimization priority: merged improvements (November 3)

The requested cheap-to-expensive screening work is now on main: PR #158 added the conservative broad-window measured-ridge screen; PR #169 added a fixed-clock event/flight precheck; PR #170 batches compatible clock fits; PRs #173–174 advance fit pruning and adaptive tile handling. On one real regional development point, #158 preserved the same 92 retained rows while reducing full nonlinear fits from 30,361 to 17,202 and process CPU from 48.27 to 30.04 seconds. That is a measured local improvement, not a nationwide runtime or localization-accuracy result. The manually estimated narrow speed/range bands from the reference video were not copied into automatic rejection rules.

PR #175 corrects acoustic-bound provenance, #180 screens later event clocks, #181 reports completed parallel checkpoints, and #182 retries overloaded track queries with the existing conservative distance sieve. A real overloaded cell retained 614 of 1,929 queried tracks, including all 200 unsupported tracks; full and filtered scoring had identical results and retention checksums. The unsupported motion remains a limitation, not a verified absence of aircraft. PR #188 adds resumable clock partitions for single cells that exceed the association-count limit. The full-day overloaded cell's 23,136 associations were preserved across eight partitions; 142 tracks remained unsupported. See DATA_AND_SCALE for the scope of these checks. PR #191 excludes unsupported trajectories only when their entire spline is provably outside the search radius. PR #193 adds bounded exact-input acoustic reuse for dense tiles and an explicit upload-search worker setting. A complete 3,795-association partition preserved its full payload and retention checksum; its CPU time changed from 76.05 to 74.60 seconds. A complete 19,341-association partition also preserved its payload and checksum, with CPU time changing from 276.99 to 250.88 seconds. Neither measurement predicts nationwide runtime or establishes a location. PR #196 removes the automatic CPA filter's unsupported minimum speed and near-sonic maximum. On one real cell, five passes below 1 m/s admitted 1,166 additional associations while retaining all 23,136 earlier associations. The finite emitted-frequency range remains a separate modeling prior. PR #205 adds a conservative acoustic score ceiling against the current Top-K floor. Retained candidates matched exactly on a 468-association real point (37.77 → 28.23 CPU seconds) and a 2,443-association real clock partition (158.01 → 149.56 CPU seconds). These single paired measurements are not a nationwide speed or accuracy estimate. PR #213 skips empty event/clock traversal frames during association construction. On one recorded-input fixture with 15 independent groups, 35 CPA rows and 10,549 candidates, all candidates, ordering and work accounting matched the baseline across 15 boundary and budget cases. Five alternating runs reduced median CPU for association construction from 0.578 to 0.336 seconds (1.72×). Reproduce with benchmark_real_association.py and a saved real fixture in the benchmark_real_clock_batch.py format:

PYTHONPATH=src python scripts/benchmark_real_association.py \
  REAL_FIXTURE.json NEW_RESULT.json --baseline-ref adbfb3e --repetitions 5

This is a regional stage measurement, not nationwide throughput or localization accuracy.

The full-day reference upload and independent real-world evidence remain open. The real-audio collection now includes a 314-video scanner survey, a 155-clip licensed AeroSonicDB scanner-only sample, and a 25-row reviewed candidate index. In the AeroSonicDB sample, 51/125 publisher-labeled aircraft clips and 0/30 ambient clips produced scanner proposals; publisher coordinates are nearby-place references, not exact microphone locations. The requested set of 20 known-location usable aircraft recordings plus five unknown-location usable aircraft recordings is still unmet: publisher site coordinates do not establish microphone accuracy, and moving-tone review does not confirm aircraft identity. No new corpus recordings were matched to flights or localized.

The swarm task plan provides 12 actionable GitHub issues, dependencies, suggested agent allocation and a copyable assignment prompt.

Full-day reference upload (November 3)

The public source is the original audio/video; the separate method video explains the analysis. On November 3, the source upload was titled “Unknown location - raw audio” and its video track was 128×128 pixels. Frames inspected at 0 and 3 seconds were identical and mostly black with a tiny landscape; they did not establish a precise recording location. The remainder of the video was not visually inspected. This check is evaluation-only and supplies no search inputs. The existing source packet also contains the creator's companion-video Street View comparison (data/source/true_camera.png), which identifies an apparent camera location. The public reference remains approximate rather than a surveyed microphone coordinate. See the Dopplerprint comparison for another implementation's use of the same reference. The user approved the nearby OspreyWatch Oyster Bay camera listing with a 1 km radius as a rough evaluation band for this recording. This assumes the microphone is near the camera; that offset is unmeasured. Compare the final location to the listing only after search, label the distance approximate, and do not use the reference or radius as a search bound, candidate filter, or tuning target. A location inside the band still needs a supported pipeline decision. This does not establish surveyed microphone accuracy or independent field or nationwide validation. See the user's evaluation-only clarification. The uploaded MP3 has SHA-256 674df668c25608ef3e9199bf3b48d760642456d679729c9661ac5cb2a22e179c. The automatic test uses the whole recording, no supplied flight IDs or selected location, CONUS coverage, and a broad 2026-09-08 00:00 to 2026-09-09 12:00 UTC window covering the stated local calendar day without a start time. Audio discovery produced 17 ridges; the daily ADSB.lol indexes retain all aircraft types. A video image can help check a result, but is not by itself a precise recorder GPS measurement.

Snapshot: 2026-11-03 16:30 UTC. No coordinate has been accepted. The original job a90452172daf453986cf9bed846b0e2d remains preserved with 853 tile files, but its current result is failed / processing_error: its audio checkpoint is not readable by the resume path. Do not treat its tile count as a completed search or use its result as a location.

The same MP3 is being searched in optimized job 7ae679c304b849a0a2b1f1085b9312d5, under /home/cinna/.local/state/dopplergeist-optimized-upload-20261103/jobs/. This job has the completed whole-recording audio checkpoint and is running/search with 12 workers. It uses the broad CONUS window above; no coordinates, flight IDs or truth-derived bounds were supplied. Its current source is cb996dcb49b9d31a24c10b9b5418c35423f7bbcd (Combine clustered tile scheduling with faster scoring). The original decoded audio and 17-event discovery were reused; the updated detector reproduced the saved 17-event discovery exactly.

At the snapshot, the active search directory had 838 tile receipts, all keyed to its current run hash. The previous 851-receipt search directory is preserved at search-before-combined-cb996dc/; 13 of those receipts were not present in the new active directory and remain available in the archive. Progress 0.55 in result.json is stale. See issue #13 for the latest process and checkpoint status. Older remote/homelab staging artifacts belong to different run identities and have not been imported into this job.

Before accepting any candidate, check current search completeness, omitted coverage, unsupported-motion accounting, retained alternatives and uncertainty. The 17 discovered ridges are not proof of 17 independent aircraft passages; the previous aligned-sample check did not establish independence. No final search result or location has been produced.

Read first

The user wants a full MP3-upload-to-location pipeline with automatic Doppler-event detection. Up to 1,000 m error is acceptable; 100–500 m or better is preferred. Report all three thresholds. Whole-recording intake, automatic discovery and bounded tiled search are connected to the local upload application on main; the configured research CLI also remains available with manually selected events. Independent real-recording acceptance remains open. Check the README, pinned verification note and current PR state before making a present-tense feature assessment. Do not treat a regional reference case or a synthetic result as nationwide real-audio validation.

The original implementation was not released. This repository contains our independent implementation and the accumulated evidence, including unsuccessful cases. PROGRESS.md lists the verified results and limits.

Original work-area split

This table preserves the initial planning split. It is not an inventory of missing features or a new assignment; use current issues and the discussion board for those.

Area Main files Concrete next result
Event discovery audio.py, new discovery module Full-file event proposals without supplied windows/bands; harmonic deduplication; noise-only rejection; saved diagnostic evidence
Upload/orchestration cli.py, pipeline.py, new interface MP3 input, explicit recording date/time uncertainty, detection-to-search orchestration, progress and inconclusive results
Search scale search.py, terrain.py, tracks.py Bounded-memory tiled national search with preserved global timing and comparable ranking
Validation scripts/, tests/, separate output directories Independent real cases with withheld location, known audio clock, and reported misses/false matches

These are suggested boundaries, not assignments. Coordinate before changing shared files. Preserve existing evidence; write new runs to new directories and record the implementation commit, input hashes, configuration and evaluation scope.

Replay

Install as in the README and restore release assets using ARTIFACTS.md. The real regional replay requires source, ADS-B-derived and terrain bundles:

.venv/bin/dopplergeist run data/source/reproduce-local.json --output results/replay-reference
.venv/bin/dopplergeist run data/source/reproduce-faint-alltypes-local.json --output results/replay-faint
.venv/bin/python scripts/benchmark_real_features.py

The benchmark script currently uses its fixed results/real-feature-benchmark output path. Preserve/move that restored directory before rerunning the benchmark. The full raw ADS-B archive is optional for these replays; the derived Parquet files are sufficient. The original source MP4 is not uploaded, at the user's request.

Important pitfalls

  • The high-quality two-event case uses source-informed speed, slant and frequency bounds. Ridge extraction within selected bands is not event discovery.
  • The faint event's best matching track is a CRJ9 jet. Removing every jet prevents this case from succeeding. Current policy excludes unknown aircraft types along with jets; this can exclude non-jet flights.
  • Its straight-pass fit suggests about 1 km slant, while the matching real track passes at about 4.9 km from the public reference. Tight filters derived from a straight-flight fit can exclude curved real trajectories.
  • The faint-event result remains 357.47 m off. Its clock comes from the other two events; it is not an independent known-time success.
  • The prominent raw-audio events are near 2156 s and 3076 s; some video interfaces display times offset by 3600 s. The faint raw-audio window is 1250–1310 s.
  • Readsb traces in the daily tar are gzip compressed even when named .json. The type dictionary uses dictionaries such as {"desc":"L2J","wtc":"M"}.
  • Geometric altitude and barometric fallback are labelled separately. Ground DEM orthometric height needs the configured geoid conversion for GNSS comparisons.
  • search.cpa_table_dense is a retained test oracle. The production sparse CPA path must retain multiple local approaches and match the reference on small cases.
  • Event clocks remain locked together. Independent shifts can manufacture a match.
  • The configured research pipeline preserves supported rankings across typed acoustic-support failures and reports excluded work. An empty candidate set produces a report without a best hypothesis. This research report does not establish an accepted automatic location; inspect its explicit coverage limits.
  • The 40-passage dataset supplies helpful frequency annotations but lacks the explicit recorder-coordinate/audio-clock evidence needed here for location tests.

Data inventory

  • data/source/: source captions, transcript, still frames, reference WAV, configs, downloaded documentation, metadata and logs. Failed exploratory outputs are kept.
  • data/adsb/2026-09-08/: raw daily archive parts, provenance, type database, 27,840,403-row CONUS prop/unknown Parquet, regional Parquet, and a 15-minute all-type CONUS Parquet for the faint case.
  • data/terrain/: geoid, GLO-30 regional tile and cached GLO-90 tiles/catalogue.
  • data/research/: checksum-verified Zenodo dataset, extracted recordings, tracks, annotations and papers.
  • data/synthetic/: deterministic audio, tracks, configurations and separate truth.
  • results/: complete successful, failed and exploratory output directories.

The release manifest records every exported file and its hash. Historical reports retain original local paths; executable configs use relative input paths.