Python tool for aircraft Doppler audio geolocation using historical ADS-B flight data (Source-available)
Python acoustic-geolocationads-baircraft-trackingairplaneaudio-forensicsdoppler-effectdoppler-shiftgeolocationosintsignal-processing
Document canonical episode contracts, coverage and acoustic limits
Cinna Davisd1bfe6d412e1
diff --git a/README.md b/README.mdindex e1a58e6..9a9b60e 100644--- a/README.md+++ b/README.md@@ -35,7 +35,7 @@ See the [Quickstart](#quickstart) to get started.| Area | Available now | Entry point or remaining work || --- | --- | --- || Configured localization | Event measurement, flight ranking and physical refinement | `dopplergeist run` and `dopplergeist measure` on `main` |-| Automatic acoustic discovery | Whole-recording decoding, frequency-track discovery and harmonic grouping without supplied event windows or bands | `scripts/discover_audio.py`; produces event proposals and diagnostics |+| Automatic acoustic discovery | Whole-recording decoding and canonical acoustic episodes with measured children, explicit coverage and bounded diagnostics | `scripts/discover_audio.py`; produces event proposals and diagnostics || Historical inputs | Reusable flight indexes, gap-aware trajectory queries and cached terrain/geoid data | [Data acquisition and search APIs](docs/DATA_AND_SCALE.md) on `main` || Geographic search | US/CONUS and Germany profiles, parallel resumable tiles, joint matching across independent flyovers and disk-backed candidate accounting | [Country profiles](docs/COUNTRIES.md), [search](docs/DATA_AND_SCALE.md#search) and [finite-search evidence](docs/FINITE_SEARCH_PRODUCER.md) || Local upload application | CLI/API orchestration, persistent jobs and inspectable browser results on `main` | `dopplergeist analyze`, `dopplergeist serve` and `dopplergeist job`; see [Local upload workflow](docs/PRODUCT.md) |@@ -118,7 +118,7 @@ Automatic discovery scans an entire recording for moving acoustic frequency patt--output results/my-discovery```-Read `results/my-discovery/summary.json` and the diagnostics under `discovery/`. See [Audio discovery](docs/AUDIO_DISCOVERY.md) for settings, evidence and failure modes.+Read `results/my-discovery/summary.json` and the schema-2 episode catalog under `discovery/`. Proposal counts refer to canonical episodes; measured child tracks are counted separately. Acoustic episodes do not establish aircraft identity or physical source motion. See [Audio discovery](docs/AUDIO_DISCOVERY.md) for settings, evidence and failure modes.### Run the local upload applicationdiff --git a/docs/AUDIO_DISCOVERY.md b/docs/AUDIO_DISCOVERY.mdindex 62f090c..078edd3 100644--- a/docs/AUDIO_DISCOVERY.md+++ b/docs/AUDIO_DISCOVERY.md@@ -1,126 +1,172 @@-# Automatic audio-event discovery--`discover_events(recording, output_dir=None, settings=None)` consumes the-`IntakeRecording` from intake and returns shared-contract `DetectedEvent`-dictionaries. The normal call supplies no time windows, frequency bands, aircraft-IDs, search locations or truth. It scans all decoded samples with fixed 120 s-chunks / 60 s overlap, a 2 s Hann spectrum at 0.5 s hops and 40–1600 Hz coverage-(clamped below decoded Nyquist). Working memory is bounded by chunk size; a disk-cache holds the complete recording. Inputs under one spectral window return [].--Time-whitened spectral contrast suppresses persistent stationary tones. Local-maxima are linked with bounded motion, short missing spans and local slope-continuity. Minimum duration/support, resolved downward frequency change,-distributed motion and jump checks reject noise paths and switching stationary-tones. Duplicate overlapping chunk ridges merge before grouping harmonics. Both-integer ratios and rational ratios (e.g. measured 2f and 3f with an absent-fundamental) count as one independent event. The lowest measured harmonic is the-representative ridge; no fundamental is invented. Harmonic count denotes distinct-measured ratios, including fragmented traces of the same harmonic only once.-Harmonic grouping is optional consolidation: a ridge without a compatible-partner remains a single-ridge event candidate. The detector does not require a-second harmonic.--The output preserves recording-relative ridge samples and weights, measured-window/band, harmonic-group identity, contrast/support, spectral resolution and-uncertainty diagnostics. Resolution is not a calibrated error interval; no raw-quality score is a probability. No compulsory straight-flight speed, slant or-source-frequency estimate constrains downstream search. These are **acoustic-moving-tone proposals**, with aircraft identity unconfirmed. Association must-reject unsupported proposals and count harmonic groups once.--In the current real one-hour upload, automatic discovery produced 15 independent-groups: 14 have one observed ridge and one has three. This is a useful reminder-that harmonic grouping can consolidate evidence when multiple bands are-recorded, but a single-ridge candidate must remain eligible for downstream-assessment. The audio-only discovery result does not confirm aircraft identity-or recording location.--With output_dir, discovery saves `discovery.json` (all events, settings, chunk-counts, original harmonic ridges and input hash), `discovery.png` (overview), and-per-event CSV plus spectrogram PNG. Each spectrogram image is limited to a chunk's-span; the full measured ridge remains in JSON/CSV. Existing discovery outputs-are refused. A reproducible whole-file entry point is:+# Automatic acoustic episode discovery++Discovery scans the whole recording without event windows, aircraft IDs, flight+observations, coordinates or a flight-geometry template. Its unit is a **canonical+acoustic episode**, containing measured tonal child tracks. Common frequency+motion does not establish a single physical source, physical source motion, or+aircraft identity. Simultaneous trajectories can remain separate, and physical+source multiplicity is explicitly unresolved, including within an episode.++The Python API still returns `list[DetectedEvent]`: one record per measured child+track, with the canonical episode ID in `harmonic_group_id`. Count distinct group+IDs to count proposals. `discovery.json` is now **schema_version 2** and is the+explicit episode catalog. The shared upload/result and DetectedEvent contracts+remain version 1. See [interfaces](INTERFACES.md#detected-events).++## Processing and acceptance++1. Decode to a mono, disk-backed f32 cache. The automatic adapter selects 4, 8,+ 16 or 32 kHz according to requested global frequency coverage. FFmpeg applies+ channel averaging and explicit swresample low-pass filtering (filter_size 64,+ cutoff 0.95). Sample zero remains the first decoded sample after codec skips;+ resampling can round the final length by a sample. UTC requires separate+ recording-clock evidence.+2. Scan overlapping chunks (120/60 seconds by default). The default spectrum is+ a 2-second Hann window, 0.5-second hop, 40–1600 Hz. Temporal whitening and+ frequency-wise robust contrast retain the existing measured-peak method.+ Peaks carry frequency, contrast, and measured spectral amplitude in dB.+3. Retain provisional paths of at least four observations **before** duration or+ motion rejection. Reconcile duplicate measurements across chunks and then+ reciprocal-unique continuations. Distinct peaks in the same chunk are not+ deleted as near-unison duplicates. A continuation requires compatible endpoint+ slopes in either direction, forward and backward prediction, uncertainty,+ observed support and a gap penalty. A turn hidden in a gap is unresolved.+4. Assess resolved excursion, distributed motion, local prediction residual,+ sustained turns, and total variation. Differences and slopes use timestamps+ in seconds; missing intervals do not count as observed motion. Rising,+ falling and one sustained turning trajectory are eligible. Stationary tones,+ isolated tone switches, repeated modulation and wandering fail explicit gates.+5. Fit constant frequency scales between components with substantial shared+ support, resolved motion, high curve correlation and uncertainty-aware+ residuals. A bounded integer-family model (orders 1–12, smallest total order)+ supplies secondary harmonic evidence, including missing fundamentals.+ Noninteger relationships require correlated measured amplitude envelopes.+ A ratio match or temporal overlap alone is insufficient.+6. Reconsider groups jointly. For three or more components each needs two strong+ motion witnesses; every supported pair must agree with one joint scale fit,+ including short overlaps that cannot establish motion independently. A weak+ bridge cannot transitively collect unrelated groups. Sorting observations and+ associations makes grouping invariant to input order. Unresolved strong links+ are counted and traced separately; separate proposals do not certify separate+ physical sources. The model has one scale per child, at most 32 children per+ episode, and no free time warps or high-order rational search.+7. Apply episode acceptance using the union of **observed hop-sized cells**, not+ interval span or spectrogram-window footprints. Sustained acceptance needs+ at least 12 observed seconds and 65% support. Brief acceptance needs at least+ 3 observed seconds, 85% support, and a child with median contrast >=4.5,+ excursion >=6 frequency-uncertainty units, distributed motion >=60%, and+ prediction-error q95 <=0.75. These are uncalibrated engineering criteria,+ not a blanket reduction of the duration/contrast thresholds.++Gaps are never filled with invented ridge observations. Stitch gaps are at most+20 seconds and at most 35% of the pair's observed support; aggregate acceptance+also penalizes missing support. The output distinguishes the observed cells+from the wider spectral-window timing uncertainty. Very brief accepted episodes+with fewer than ten total samples are acoustic-review evidence only;+`quality.matching_eligible=0` prevents the matching adapter from failing on them.+The application returns `brief_acoustic_evidence_only` if none are matchable.+Other proposals need not match any flight.++## Coverage and resources++The default coverage remains 40–1600 Hz for compatibility. Higher frequencies+require decoding the original source at a sufficient rate; changing a scanner+setting cannot restore frequencies discarded in an old cache. The automatic+adapter and scanner CLI choose the rate before intake. Direct Python callers+must use `required_sample_rate(settings)` when invoking `intake_recording`.+Requested upper coverage must be <=45% of both source and decoded rates. A+request outside this usable range fails with `unsupported_frequency_coverage`;+it is not reported as an exhaustive negative. The lower limit requires at least+two cycles per window. Longer windows improve low-frequency resolution; shorter+windows improve temporal resolution while increasing frequency uncertainty.++The `coverage` object records requested and analyzed bin ranges, eligible peak+centers, source/decoded rates, resolution, hop, window, transition guard and edge+policy. Outermost bins support interpolation and cannot be peak centers. Inputs+shorter than a complete window have no spectral observations. The high-rate path+uses a single spectrum per chunk, so there are no overlapping frequency bands+that could duplicate proposals. Chunk grids must align to whole sample hops;+chunk provenance is retained through deduplication and stitching.++Default explicit limits: 60 peaks/frame, 4,096 provisional paths, 500,000 retained+observations, 200,000 association work units, 8,000,000 full-spectrum cells per+chunk and 100 diagnostic trace entries. Association work includes repeated joint+fit checks, not only unique pairs. Joint fit matrices have at most 32 columns.+Exceeding a processing budget raises a typed failure and records the limit;+it does not silently drop candidates and claim a complete scan. There is no cap+of two, three or four canonical proposals per recording. Increasing a work limit+in the Python settings is an explicit resource choice.++The decoded cache uses `4 * rate * seconds` bytes: about 55 MiB/hour at 4 kHz,+220 MiB/hour at 16 kHz. Working spectra stay chunk-bounded; raising rate increases+FFT memory and CPU, and reducing the hop increases frame count. Spectrum cells+are checked before allocation. The [engineering report](ACOUSTIC_EPISODES_REPORT.md)+records measured RSS, time and output volume, including regressions.++## Outputs and compatibility++`discovery.json` contains:++| Field | Meaning |+| --- | --- |+| `status`, `failure` | `complete` with zero proposals is a valid in-band result; `failed` is an execution, coverage or budget failure |+| `source_sha256`, `code`, `settings`, `runtime_s`, `coverage` | Input identity, git/implementation identity, library versions, effective configuration and runtime |+| `events` | Compatible measured DetectedEvent children; ridge points are stored once here |+| `episodes` | Canonical IDs, child IDs, raw track/chunk membership, fitted frequency scales, integer orders, observed intervals/gaps, confidence category and limitations |+| `counts` | Separate raw per-chunk paths/observations, provisional/deduplicated/stitched candidates, accepted children and canonical proposals |+| `ridge_count` | Accepted measured child count, retained for older count consumers |+| `rejections` | Counts and measured/threshold ranges per rejection reason; optional bounded candidate traces |+| `reconciliation_counts`, `reconciliation_decisions` | Counted and bounded duplicate, stitch, joint-group and unresolved-relation decisions; truncation is explicit |++IDs hash source identity and ordered measured content, and do not depend on input+ordering, proposal enumeration or unrelated accepted proposals. They are stable+for the same measurements and effective configuration, not guaranteed invariant+to codecs, alternate chunk whitening, or resampling that changes measurements.+Each child retains original raw-track identifiers and amplitude measurements;+observed frequency scales replace the old ratio dictionary in the matching+adapter. Legacy event groups without these scales still use their old adapter.+Equal aggregate weight per child preserves the existing consumer convention;+additional components do not become additional independent time constraints.++Confidence is categorical and uncalibrated: `uncalibrated_supported` or+`uncalibrated_brief_limited`. Frequency uncertainty uses the spectral resolution+and robust deviation, not a calibrated interval. `frequency_sigma_hz` and+`frequency_uncertainty_hz` retain the same diagnostic; matching retains its+>=1 Hz loss-scale floor. Every episode says `physical_motion_confirmed=false`+and `physical_source_multiplicity=unresolved`.++Schema 2 removes the duplicate `harmonic_ridges` array, legacy `fragment_links`+and automatic per-child CSV files. Read measurements from `events` and decisions+from the reconciliation fields. The plot benchmark reads both artifact versions.+Old evidence is preserved; [the starting-version documentation](https://github.com/1337isnot1337/DOPPLERGEIST/blob/451b67f01dcd1dfe44d47c42407f404706a5a9b7/docs/AUDIO_DISCOVERY.md)+contains historical counts and methodology, including cancelled synthetic work.++## Commands```bash-PYTHONPATH=src .venv/bin/python scripts/discover_audio.py input.mp3 --output results/new-audio-scan+# Default acoustic-only scan. A fresh output directory is required.+PYTHONPATH=src .venv/bin/python scripts/discover_audio.py input.mp3 \+ --output results/discovery --plot-summary++# Global wider coverage; source must actually retain those frequencies.+PYTHONPATH=src .venv/bin/python scripts/discover_audio.py input.wav \+ --min-frequency-hz 20 --max-frequency-hz 6000 \+ --output results/discovery-wide --plot-summary++# Optional higher temporal resolution and bounded rejected-candidate traces.+PYTHONPATH=src .venv/bin/python scripts/discover_audio.py input.wav \+ --window-s .5 --hop-s .125 --max-frequency-hz 6000 \+ --output results/discovery-fast --detailed-candidates --plot-event-details++# Software properties and existing genuine recorded evidence; no audio synthesis.+PYTHONPATH=src .venv/bin/python -m pytest -q tests/test_episodes.py \+ tests/test_real_discovery_normalization.py tests/test_real_upload_verification.py \+ tests/test_contracts.py tests/test_job_resume.py```-The script records source/code hashes, git state, settings, runtime and peak RSS.-It does not consume evaluation labels. Run post-hoc evaluation in a separate-process. Use a new output directory for every run.--Development evidence before DG-02 controls: full synthetic WAV (1180 s), 2-proposals spanning both planted passes, 7.24 s / 238044 KiB peak RSS; full public-reference WAV (3600.1845 s after decode), 8 proposals, 31.05 s / 249528 KiB peak-RSS. The two prominent known passes are covered, but are fragmented into multiple-proposals; the documented faint pass is missed. Six proposals do not cover either documented pass center; some appear to be-fragments of the same pass. Their identities remain unverified. They are not-labelled aircraft or certified false alarms. No-complete real event annotation exists to calculate reference precision/recall.-Earlier v1/v2 diagnostics preserve a missed synthetic event and excess proposals.-These are development checks, not independent location/nationwide validation.--Known limits: fixed spectral scale and minimum duration can miss steep/short,-faint, low-frequency or high-frequency events. A single wide ridge can fragment-across missing spans, and acoustically similar harmonic sources can be grouped-ambiguously. No algorithm can establish aircraft identity solely from an arbitrary-matching synthetic chirp. The existing real recording has no independently-verified UTC clock. This implementation does not establish 100 m localization.--Focused tests: `PYTHONPATH=src .venv/bin/python -m pytest -q tests/test_discovery.py`.-They cover complete-file seams, harmonic grouping including an absent fundamental,-simultaneous unrelated events, noise/stationary/upward-sweep rejection, short-unsupported clips, and crossfading stationary tones. Fixed-setting DG-02 corpus-results are recorded separately in the audio evidence handoff.--Fixed-code DG-02 control check at discovery commit `4a49646`: all 14 opaque-input-cases processed, 8 planted events found, 0 misses and 0 false alarms at event-IoU >= 0.3. Both the development and public deterministic held_out split find-4/4 events. Noise, stationary, upward electronic sweep and short clips produce-no event; harmonic, overlapping and MP3 positives are retained. Detection ran-through the label-free subprocess runner; a later process read truth for scoring.-No detector/settings changes followed the control run. These controls have no-eligible location cases and do not establish real-aircraft identity or accuracy.--Additional unchanged-code public-audio check: all 40 complete passage WAVs process-without errors; 28 have at least one proposal, 12 have none, 102 proposals total.-The run never supplied their frequency annotations. These counts do not establish-true aircraft-event recall/precision: fragments and incidental sources remain-unverified. Every prediction is saved under evidence/audio/w1-20261102/public40-4a49646.--Fragment follow-up: compatible disjoint ridge fragments can join across at most-20 s when the gap is at most 35% of the combined observed span, both endpoint-slopes descend, the observed frequency drop/rate is compatible with those slopes,-and each endpoint has only one compatible continuation. Competing continuations-remain separate. No frequency samples are invented across missing intervals;-harmonic comparison excludes unsupported spans. `fragment_count` and-`max_observed_gap_s` retain missing-support evidence, and `fragment_links` records-accepted/ambiguous continuations with reasons. This is a tonal continuity-hypothesis; it does not prove a flight identity.--`quality.frequency_sigma_hz` is the integration consumer's canonical robust-loss-scale, with `frequency_uncertainty_hz` retained for compatibility. Both currently-carry the same resolution/robust-deviation diagnostic, with no statistical or-probability calibration implied by the field name. Matching applies its >=1 Hz-loss-scale floor. Later tests against the already public DG-02 controls are-regressions, not newly held-out accuracy evidence.--Sensitivity follow-up lowers the global seed contrast threshold from3.0 to2.5.-At this setting the full synthetic recording still yields2 groups, with first-ridge support extended from143–180.5s to119–196s. The reference emits16 proposals,-including a short1265.5–1282s ridge in the documented faint passage; post-run-comparison with the archived manual ridge gives median absolute difference-0.692Hz over23 points (90th percentile2.139Hz). This is acoustic overlap with an-existing development measurement, not independently known aircraft or location-truth. Fourteen known synthetic controls retain8 hits/0 misses/0 false alarms.-The extra unverified reference proposals remain a matching/search cost. Earlier-3.0-threshold evidence and the faint-pass miss stay preserved; no held-out or-100m localization claim follows from this development tuning.--Discovery returns all accepted independent groups; it has no global two- or-three-event maximum. Four- and six-episode generated audio regressions check that-later events survive and harmonics count once. These audio-only controls test-retention, not geographic accuracy. The per-frame peak budget remains an explicit-bounded-analysis setting, and unsupported or missed tones are not a completeness-guarantee. The contrast 2.5 settings are frozen for the new public-camera batch.+Summary plots use canonical episodes. Optional detail plots render one bounded+spectrogram per episode, with all its child tracks, once. They do not render the+same group repeatedly for each harmonic. Normal operation records aggregated+noise rejection rather than writing a file per rejected fragment. Input/decoder+failures exit unsuccessfully; discovery failures additionally write a failed+artifact. No command above acquires flight data or estimates a location.diff --git a/docs/INTERFACES.md b/docs/INTERFACES.mdindex 17671d9..2245316 100644--- a/docs/INTERFACES.md+++ b/docs/INTERFACES.md@@ -67,8 +67,10 @@ points. Each point has `time_s`, `frequency_hz` and an optional nonnegative `weilie within its band. Times are relative to the beginning of the decoded recording,not to the beginning of an analysis chunk. Chunked discovery must add its offset.-Optional `harmonic_group_id` identifies ridges believed to come from the same-flyover. Harmonics must not become independent time constraints. Optional+Optional `harmonic_group_id` is the canonical acoustic proposal key. Related+measured components contribute once as an episode, with physical source+multiplicity unresolved; a group does not establish a flyover or source motion.+Harmonics must not become independent time constraints. Optional`cpa_receive_s` is an estimated received CPA time, in audio-relative seconds, andmay lie outside a truncated observed event. Optional positive increasing ranges`f0_range_hz`, `speed_range_mps` and `slant_range_m` are measured proposal bounds.@@ -78,6 +80,16 @@ They are not guaranteed true geometry; the matcher must account for uncertainty.(e.g. `ridge_rmse_hz`). A diagnostic score is not a calibrated probability.There is no blanket `confidence` float whose semantics different workers can guess.+Automatic discovery's separate artifact is now `schema_version: 2`; its `episodes`+catalog references compatible `events` children. See [the artifact contract and+migration](AUDIO_DISCOVERY.md#outputs-and-compatibility). Modern children carry+`quality.episode_model_version=2`, positive `episode_frequency_scale`, and numeric+`matching_eligible`. Matching normalizes these measured scales within each+canonical group, with legacy rational-ratio behavior for old records. Accepted+brief episodes with insufficient samples remain acoustic-review evidence. This+additive numeric quality metadata does not change DetectedEvent's strict keys or+the upload/result schema version.+## Location candidatesA `LocationCandidate` contains `lat`, `lon`, finite `score`, a descriptivediff --git a/docs/PRODUCT.md b/docs/PRODUCT.mdindex 737d40b..7957486 100644--- a/docs/PRODUCT.md+++ b/docs/PRODUCT.md@@ -290,9 +290,12 @@ retained approaches prevent an affirmative confidence decision.For chunk-boundary checks on real recordings, an operator can set`"discovery": {"chunk_s": 120.0, "overlap_s": 60.0}` in this private configuration.-These are the only discovery overrides supported by the product adapter. The-existing detector validates the overlap and chunk constraints; spectral bands,-acceptance thresholds and other scientific settings keep their defaults. The+The operator may also set global `min_frequency_hz`, `max_frequency_hz`,+`window_s` and `hop_s`. The adapter selects a sufficient decode rate before+intake and rejects unsupported source coverage explicitly. These configure the+whole-recording analysis, not individual event windows or bands; they remain+outside the upload request. Acceptance thresholds keep their defaults. See+[acoustic discovery](AUDIO_DISCOVERY.md) for coverage, time-grid and work limits. Theactual settings and analyzed chunks are saved in `discovery/discovery.json`, andthe operator configuration hash is retained in provenance. This configuration isnot an upload-request field. A chunk-size comparison must rerun detection and the