Skip to content

Python tool for aircraft Doppler audio geolocation using historical ADS-B flight data (Source-available)

Document canonical episode contracts, coverage and acoustic limits

Cinna Davisd1bfe6d412e1
diff --git a/README.md b/README.md
index e1a58e6..9a9b60e 100644
--- a/README.md
+++ b/README.md
@@ -35,7 +35,7 @@ See the [Quickstart](#quickstart) to get started.
| Area | Available now | Entry point or remaining work |
| --- | --- | --- |
| Configured localization | Event measurement, flight ranking and physical refinement | `dopplergeist run` and `dopplergeist measure` on `main` |
-| Automatic acoustic discovery | Whole-recording decoding, frequency-track discovery and harmonic grouping without supplied event windows or bands | `scripts/discover_audio.py`; produces event proposals and diagnostics |
+| Automatic acoustic discovery | Whole-recording decoding and canonical acoustic episodes with measured children, explicit coverage and bounded diagnostics | `scripts/discover_audio.py`; produces event proposals and diagnostics |
| Historical inputs | Reusable flight indexes, gap-aware trajectory queries and cached terrain/geoid data | [Data acquisition and search APIs](docs/DATA_AND_SCALE.md) on `main` |
| Geographic search | US/CONUS and Germany profiles, parallel resumable tiles, joint matching across independent flyovers and disk-backed candidate accounting | [Country profiles](docs/COUNTRIES.md), [search](docs/DATA_AND_SCALE.md#search) and [finite-search evidence](docs/FINITE_SEARCH_PRODUCER.md) |
| Local upload application | CLI/API orchestration, persistent jobs and inspectable browser results on `main` | `dopplergeist analyze`, `dopplergeist serve` and `dopplergeist job`; see [Local upload workflow](docs/PRODUCT.md) |
@@ -118,7 +118,7 @@ Automatic discovery scans an entire recording for moving acoustic frequency patt
--output results/my-discovery
```
-Read `results/my-discovery/summary.json` and the diagnostics under `discovery/`. See [Audio discovery](docs/AUDIO_DISCOVERY.md) for settings, evidence and failure modes.
+Read `results/my-discovery/summary.json` and the schema-2 episode catalog under `discovery/`. Proposal counts refer to canonical episodes; measured child tracks are counted separately. Acoustic episodes do not establish aircraft identity or physical source motion. See [Audio discovery](docs/AUDIO_DISCOVERY.md) for settings, evidence and failure modes.
### Run the local upload application
diff --git a/docs/AUDIO_DISCOVERY.md b/docs/AUDIO_DISCOVERY.md
index 62f090c..078edd3 100644
--- a/docs/AUDIO_DISCOVERY.md
+++ b/docs/AUDIO_DISCOVERY.md
@@ -1,126 +1,172 @@
-# Automatic audio-event discovery
-
-`discover_events(recording, output_dir=None, settings=None)` consumes the
-`IntakeRecording` from intake and returns shared-contract `DetectedEvent`
-dictionaries. The normal call supplies no time windows, frequency bands, aircraft
-IDs, search locations or truth. It scans all decoded samples with fixed 120 s
-chunks / 60 s overlap, a 2 s Hann spectrum at 0.5 s hops and 40–1600 Hz coverage
-(clamped below decoded Nyquist). Working memory is bounded by chunk size; a disk
-cache holds the complete recording. Inputs under one spectral window return [].
-
-Time-whitened spectral contrast suppresses persistent stationary tones. Local
-maxima are linked with bounded motion, short missing spans and local slope
-continuity. Minimum duration/support, resolved downward frequency change,
-distributed motion and jump checks reject noise paths and switching stationary
-tones. Duplicate overlapping chunk ridges merge before grouping harmonics. Both
-integer ratios and rational ratios (e.g. measured 2f and 3f with an absent
-fundamental) count as one independent event. The lowest measured harmonic is the
-representative ridge; no fundamental is invented. Harmonic count denotes distinct
-measured ratios, including fragmented traces of the same harmonic only once.
-Harmonic grouping is optional consolidation: a ridge without a compatible
-partner remains a single-ridge event candidate. The detector does not require a
-second harmonic.
-
-The output preserves recording-relative ridge samples and weights, measured
-window/band, harmonic-group identity, contrast/support, spectral resolution and
-uncertainty diagnostics. Resolution is not a calibrated error interval; no raw
-quality score is a probability. No compulsory straight-flight speed, slant or
-source-frequency estimate constrains downstream search. These are **acoustic
-moving-tone proposals**, with aircraft identity unconfirmed. Association must
-reject unsupported proposals and count harmonic groups once.
-
-In the current real one-hour upload, automatic discovery produced 15 independent
-groups: 14 have one observed ridge and one has three. This is a useful reminder
-that harmonic grouping can consolidate evidence when multiple bands are
-recorded, but a single-ridge candidate must remain eligible for downstream
-assessment. The audio-only discovery result does not confirm aircraft identity
-or recording location.
-
-With output_dir, discovery saves `discovery.json` (all events, settings, chunk
-counts, original harmonic ridges and input hash), `discovery.png` (overview), and
-per-event CSV plus spectrogram PNG. Each spectrogram image is limited to a chunk's
-span; the full measured ridge remains in JSON/CSV. Existing discovery outputs
-are refused. A reproducible whole-file entry point is:
+# Automatic acoustic episode discovery
+
+Discovery scans the whole recording without event windows, aircraft IDs, flight
+observations, coordinates or a flight-geometry template. Its unit is a **canonical
+acoustic episode**, containing measured tonal child tracks. Common frequency
+motion does not establish a single physical source, physical source motion, or
+aircraft identity. Simultaneous trajectories can remain separate, and physical
+source multiplicity is explicitly unresolved, including within an episode.
+
+The Python API still returns `list[DetectedEvent]`: one record per measured child
+track, with the canonical episode ID in `harmonic_group_id`. Count distinct group
+IDs to count proposals. `discovery.json` is now **schema_version 2** and is the
+explicit episode catalog. The shared upload/result and DetectedEvent contracts
+remain version 1. See [interfaces](INTERFACES.md#detected-events).
+
+## Processing and acceptance
+
+1. Decode to a mono, disk-backed f32 cache. The automatic adapter selects 4, 8,
+ 16 or 32 kHz according to requested global frequency coverage. FFmpeg applies
+ channel averaging and explicit swresample low-pass filtering (filter_size 64,
+ cutoff 0.95). Sample zero remains the first decoded sample after codec skips;
+ resampling can round the final length by a sample. UTC requires separate
+ recording-clock evidence.
+2. Scan overlapping chunks (120/60 seconds by default). The default spectrum is
+ a 2-second Hann window, 0.5-second hop, 40–1600 Hz. Temporal whitening and
+ frequency-wise robust contrast retain the existing measured-peak method.
+ Peaks carry frequency, contrast, and measured spectral amplitude in dB.
+3. Retain provisional paths of at least four observations **before** duration or
+ motion rejection. Reconcile duplicate measurements across chunks and then
+ reciprocal-unique continuations. Distinct peaks in the same chunk are not
+ deleted as near-unison duplicates. A continuation requires compatible endpoint
+ slopes in either direction, forward and backward prediction, uncertainty,
+ observed support and a gap penalty. A turn hidden in a gap is unresolved.
+4. Assess resolved excursion, distributed motion, local prediction residual,
+ sustained turns, and total variation. Differences and slopes use timestamps
+ in seconds; missing intervals do not count as observed motion. Rising,
+ falling and one sustained turning trajectory are eligible. Stationary tones,
+ isolated tone switches, repeated modulation and wandering fail explicit gates.
+5. Fit constant frequency scales between components with substantial shared
+ support, resolved motion, high curve correlation and uncertainty-aware
+ residuals. A bounded integer-family model (orders 1–12, smallest total order)
+ supplies secondary harmonic evidence, including missing fundamentals.
+ Noninteger relationships require correlated measured amplitude envelopes.
+ A ratio match or temporal overlap alone is insufficient.
+6. Reconsider groups jointly. For three or more components each needs two strong
+ motion witnesses; every supported pair must agree with one joint scale fit,
+ including short overlaps that cannot establish motion independently. A weak
+ bridge cannot transitively collect unrelated groups. Sorting observations and
+ associations makes grouping invariant to input order. Unresolved strong links
+ are counted and traced separately; separate proposals do not certify separate
+ physical sources. The model has one scale per child, at most 32 children per
+ episode, and no free time warps or high-order rational search.
+7. Apply episode acceptance using the union of **observed hop-sized cells**, not
+ interval span or spectrogram-window footprints. Sustained acceptance needs
+ at least 12 observed seconds and 65% support. Brief acceptance needs at least
+ 3 observed seconds, 85% support, and a child with median contrast >=4.5,
+ excursion >=6 frequency-uncertainty units, distributed motion >=60%, and
+ prediction-error q95 <=0.75. These are uncalibrated engineering criteria,
+ not a blanket reduction of the duration/contrast thresholds.
+
+Gaps are never filled with invented ridge observations. Stitch gaps are at most
+20 seconds and at most 35% of the pair's observed support; aggregate acceptance
+also penalizes missing support. The output distinguishes the observed cells
+from the wider spectral-window timing uncertainty. Very brief accepted episodes
+with fewer than ten total samples are acoustic-review evidence only;
+`quality.matching_eligible=0` prevents the matching adapter from failing on them.
+The application returns `brief_acoustic_evidence_only` if none are matchable.
+Other proposals need not match any flight.
+
+## Coverage and resources
+
+The default coverage remains 40–1600 Hz for compatibility. Higher frequencies
+require decoding the original source at a sufficient rate; changing a scanner
+setting cannot restore frequencies discarded in an old cache. The automatic
+adapter and scanner CLI choose the rate before intake. Direct Python callers
+must use `required_sample_rate(settings)` when invoking `intake_recording`.
+Requested upper coverage must be <=45% of both source and decoded rates. A
+request outside this usable range fails with `unsupported_frequency_coverage`;
+it is not reported as an exhaustive negative. The lower limit requires at least
+two cycles per window. Longer windows improve low-frequency resolution; shorter
+windows improve temporal resolution while increasing frequency uncertainty.
+
+The `coverage` object records requested and analyzed bin ranges, eligible peak
+centers, source/decoded rates, resolution, hop, window, transition guard and edge
+policy. Outermost bins support interpolation and cannot be peak centers. Inputs
+shorter than a complete window have no spectral observations. The high-rate path
+uses a single spectrum per chunk, so there are no overlapping frequency bands
+that could duplicate proposals. Chunk grids must align to whole sample hops;
+chunk provenance is retained through deduplication and stitching.
+
+Default explicit limits: 60 peaks/frame, 4,096 provisional paths, 500,000 retained
+observations, 200,000 association work units, 8,000,000 full-spectrum cells per
+chunk and 100 diagnostic trace entries. Association work includes repeated joint
+fit checks, not only unique pairs. Joint fit matrices have at most 32 columns.
+Exceeding a processing budget raises a typed failure and records the limit;
+it does not silently drop candidates and claim a complete scan. There is no cap
+of two, three or four canonical proposals per recording. Increasing a work limit
+in the Python settings is an explicit resource choice.
+
+The decoded cache uses `4 * rate * seconds` bytes: about 55 MiB/hour at 4 kHz,
+220 MiB/hour at 16 kHz. Working spectra stay chunk-bounded; raising rate increases
+FFT memory and CPU, and reducing the hop increases frame count. Spectrum cells
+are checked before allocation. The [engineering report](ACOUSTIC_EPISODES_REPORT.md)
+records measured RSS, time and output volume, including regressions.
+
+## Outputs and compatibility
+
+`discovery.json` contains:
+
+| Field | Meaning |
+| --- | --- |
+| `status`, `failure` | `complete` with zero proposals is a valid in-band result; `failed` is an execution, coverage or budget failure |
+| `source_sha256`, `code`, `settings`, `runtime_s`, `coverage` | Input identity, git/implementation identity, library versions, effective configuration and runtime |
+| `events` | Compatible measured DetectedEvent children; ridge points are stored once here |
+| `episodes` | Canonical IDs, child IDs, raw track/chunk membership, fitted frequency scales, integer orders, observed intervals/gaps, confidence category and limitations |
+| `counts` | Separate raw per-chunk paths/observations, provisional/deduplicated/stitched candidates, accepted children and canonical proposals |
+| `ridge_count` | Accepted measured child count, retained for older count consumers |
+| `rejections` | Counts and measured/threshold ranges per rejection reason; optional bounded candidate traces |
+| `reconciliation_counts`, `reconciliation_decisions` | Counted and bounded duplicate, stitch, joint-group and unresolved-relation decisions; truncation is explicit |
+
+IDs hash source identity and ordered measured content, and do not depend on input
+ordering, proposal enumeration or unrelated accepted proposals. They are stable
+for the same measurements and effective configuration, not guaranteed invariant
+to codecs, alternate chunk whitening, or resampling that changes measurements.
+Each child retains original raw-track identifiers and amplitude measurements;
+observed frequency scales replace the old ratio dictionary in the matching
+adapter. Legacy event groups without these scales still use their old adapter.
+Equal aggregate weight per child preserves the existing consumer convention;
+additional components do not become additional independent time constraints.
+
+Confidence is categorical and uncalibrated: `uncalibrated_supported` or
+`uncalibrated_brief_limited`. Frequency uncertainty uses the spectral resolution
+and robust deviation, not a calibrated interval. `frequency_sigma_hz` and
+`frequency_uncertainty_hz` retain the same diagnostic; matching retains its
+>=1 Hz loss-scale floor. Every episode says `physical_motion_confirmed=false`
+and `physical_source_multiplicity=unresolved`.
+
+Schema 2 removes the duplicate `harmonic_ridges` array, legacy `fragment_links`
+and automatic per-child CSV files. Read measurements from `events` and decisions
+from the reconciliation fields. The plot benchmark reads both artifact versions.
+Old evidence is preserved; [the starting-version documentation](https://github.com/1337isnot1337/DOPPLERGEIST/blob/451b67f01dcd1dfe44d47c42407f404706a5a9b7/docs/AUDIO_DISCOVERY.md)
+contains historical counts and methodology, including cancelled synthetic work.
+
+## Commands
```bash
-PYTHONPATH=src .venv/bin/python scripts/discover_audio.py input.mp3 --output results/new-audio-scan
+# Default acoustic-only scan. A fresh output directory is required.
+PYTHONPATH=src .venv/bin/python scripts/discover_audio.py input.mp3 \
+ --output results/discovery --plot-summary
+
+# Global wider coverage; source must actually retain those frequencies.
+PYTHONPATH=src .venv/bin/python scripts/discover_audio.py input.wav \
+ --min-frequency-hz 20 --max-frequency-hz 6000 \
+ --output results/discovery-wide --plot-summary
+
+# Optional higher temporal resolution and bounded rejected-candidate traces.
+PYTHONPATH=src .venv/bin/python scripts/discover_audio.py input.wav \
+ --window-s .5 --hop-s .125 --max-frequency-hz 6000 \
+ --output results/discovery-fast --detailed-candidates --plot-event-details
+
+# Software properties and existing genuine recorded evidence; no audio synthesis.
+PYTHONPATH=src .venv/bin/python -m pytest -q tests/test_episodes.py \
+ tests/test_real_discovery_normalization.py tests/test_real_upload_verification.py \
+ tests/test_contracts.py tests/test_job_resume.py
```
-The script records source/code hashes, git state, settings, runtime and peak RSS.
-It does not consume evaluation labels. Run post-hoc evaluation in a separate
-process. Use a new output directory for every run.
-
-Development evidence before DG-02 controls: full synthetic WAV (1180 s), 2
-proposals spanning both planted passes, 7.24 s / 238044 KiB peak RSS; full public
-reference WAV (3600.1845 s after decode), 8 proposals, 31.05 s / 249528 KiB peak
-RSS. The two prominent known passes are covered, but are fragmented into multiple
-proposals; the documented faint pass is missed. Six proposals do not cover either documented pass center; some appear to be
-fragments of the same pass. Their identities remain unverified. They are not
-labelled aircraft or certified false alarms. No
-complete real event annotation exists to calculate reference precision/recall.
-Earlier v1/v2 diagnostics preserve a missed synthetic event and excess proposals.
-These are development checks, not independent location/nationwide validation.
-
-Known limits: fixed spectral scale and minimum duration can miss steep/short,
-faint, low-frequency or high-frequency events. A single wide ridge can fragment
-across missing spans, and acoustically similar harmonic sources can be grouped
-ambiguously. No algorithm can establish aircraft identity solely from an arbitrary
-matching synthetic chirp. The existing real recording has no independently
-verified UTC clock. This implementation does not establish 100 m localization.
-
-Focused tests: `PYTHONPATH=src .venv/bin/python -m pytest -q tests/test_discovery.py`.
-They cover complete-file seams, harmonic grouping including an absent fundamental,
-simultaneous unrelated events, noise/stationary/upward-sweep rejection, short
-unsupported clips, and crossfading stationary tones. Fixed-setting DG-02 corpus
-results are recorded separately in the audio evidence handoff.
-
-Fixed-code DG-02 control check at discovery commit `4a49646`: all 14 opaque-input
-cases processed, 8 planted events found, 0 misses and 0 false alarms at event
-IoU >= 0.3. Both the development and public deterministic held_out split find
-4/4 events. Noise, stationary, upward electronic sweep and short clips produce
-no event; harmonic, overlapping and MP3 positives are retained. Detection ran
-through the label-free subprocess runner; a later process read truth for scoring.
-No detector/settings changes followed the control run. These controls have no
-eligible location cases and do not establish real-aircraft identity or accuracy.
-
-Additional unchanged-code public-audio check: all 40 complete passage WAVs process
-without errors; 28 have at least one proposal, 12 have none, 102 proposals total.
-The run never supplied their frequency annotations. These counts do not establish
-true aircraft-event recall/precision: fragments and incidental sources remain
-unverified. Every prediction is saved under evidence/audio/w1-20261102/public40-4a49646.
-
-Fragment follow-up: compatible disjoint ridge fragments can join across at most
-20 s when the gap is at most 35% of the combined observed span, both endpoint
-slopes descend, the observed frequency drop/rate is compatible with those slopes,
-and each endpoint has only one compatible continuation. Competing continuations
-remain separate. No frequency samples are invented across missing intervals;
-harmonic comparison excludes unsupported spans. `fragment_count` and
-`max_observed_gap_s` retain missing-support evidence, and `fragment_links` records
-accepted/ambiguous continuations with reasons. This is a tonal continuity
-hypothesis; it does not prove a flight identity.
-
-`quality.frequency_sigma_hz` is the integration consumer's canonical robust-loss
-scale, with `frequency_uncertainty_hz` retained for compatibility. Both currently
-carry the same resolution/robust-deviation diagnostic, with no statistical or
-probability calibration implied by the field name. Matching applies its >=1 Hz
-loss-scale floor. Later tests against the already public DG-02 controls are
-regressions, not newly held-out accuracy evidence.
-
-Sensitivity follow-up lowers the global seed contrast threshold from3.0 to2.5.
-At this setting the full synthetic recording still yields2 groups, with first
-ridge support extended from143–180.5s to119–196s. The reference emits16 proposals,
-including a short1265.5–1282s ridge in the documented faint passage; post-run
-comparison with the archived manual ridge gives median absolute difference
-0.692Hz over23 points (90th percentile2.139Hz). This is acoustic overlap with an
-existing development measurement, not independently known aircraft or location
-truth. Fourteen known synthetic controls retain8 hits/0 misses/0 false alarms.
-The extra unverified reference proposals remain a matching/search cost. Earlier
-3.0-threshold evidence and the faint-pass miss stay preserved; no held-out or
-100m localization claim follows from this development tuning.
-
-Discovery returns all accepted independent groups; it has no global two- or
-three-event maximum. Four- and six-episode generated audio regressions check that
-later events survive and harmonics count once. These audio-only controls test
-retention, not geographic accuracy. The per-frame peak budget remains an explicit
-bounded-analysis setting, and unsupported or missed tones are not a completeness
-guarantee. The contrast 2.5 settings are frozen for the new public-camera batch.
+Summary plots use canonical episodes. Optional detail plots render one bounded
+spectrogram per episode, with all its child tracks, once. They do not render the
+same group repeatedly for each harmonic. Normal operation records aggregated
+noise rejection rather than writing a file per rejected fragment. Input/decoder
+failures exit unsuccessfully; discovery failures additionally write a failed
+artifact. No command above acquires flight data or estimates a location.
diff --git a/docs/INTERFACES.md b/docs/INTERFACES.md
index 17671d9..2245316 100644
--- a/docs/INTERFACES.md
+++ b/docs/INTERFACES.md
@@ -67,8 +67,10 @@ points. Each point has `time_s`, `frequency_hz` and an optional nonnegative `wei
lie within its band. Times are relative to the beginning of the decoded recording,
not to the beginning of an analysis chunk. Chunked discovery must add its offset.
-Optional `harmonic_group_id` identifies ridges believed to come from the same
-flyover. Harmonics must not become independent time constraints. Optional
+Optional `harmonic_group_id` is the canonical acoustic proposal key. Related
+measured components contribute once as an episode, with physical source
+multiplicity unresolved; a group does not establish a flyover or source motion.
+Harmonics must not become independent time constraints. Optional
`cpa_receive_s` is an estimated received CPA time, in audio-relative seconds, and
may lie outside a truncated observed event. Optional positive increasing ranges
`f0_range_hz`, `speed_range_mps` and `slant_range_m` are measured proposal bounds.
@@ -78,6 +80,16 @@ They are not guaranteed true geometry; the matcher must account for uncertainty.
(e.g. `ridge_rmse_hz`). A diagnostic score is not a calibrated probability.
There is no blanket `confidence` float whose semantics different workers can guess.
+Automatic discovery's separate artifact is now `schema_version: 2`; its `episodes`
+catalog references compatible `events` children. See [the artifact contract and
+migration](AUDIO_DISCOVERY.md#outputs-and-compatibility). Modern children carry
+`quality.episode_model_version=2`, positive `episode_frequency_scale`, and numeric
+`matching_eligible`. Matching normalizes these measured scales within each
+canonical group, with legacy rational-ratio behavior for old records. Accepted
+brief episodes with insufficient samples remain acoustic-review evidence. This
+additive numeric quality metadata does not change DetectedEvent's strict keys or
+the upload/result schema version.
+
## Location candidates
A `LocationCandidate` contains `lat`, `lon`, finite `score`, a descriptive
diff --git a/docs/PRODUCT.md b/docs/PRODUCT.md
index 737d40b..7957486 100644
--- a/docs/PRODUCT.md
+++ b/docs/PRODUCT.md
@@ -290,9 +290,12 @@ retained approaches prevent an affirmative confidence decision.
For chunk-boundary checks on real recordings, an operator can set
`"discovery": {"chunk_s": 120.0, "overlap_s": 60.0}` in this private configuration.
-These are the only discovery overrides supported by the product adapter. The
-existing detector validates the overlap and chunk constraints; spectral bands,
-acceptance thresholds and other scientific settings keep their defaults. The
+The operator may also set global `min_frequency_hz`, `max_frequency_hz`,
+`window_s` and `hop_s`. The adapter selects a sufficient decode rate before
+intake and rejects unsupported source coverage explicitly. These configure the
+whole-recording analysis, not individual event windows or bands; they remain
+outside the upload request. Acceptance thresholds keep their defaults. See
+[acoustic discovery](AUDIO_DISCOVERY.md) for coverage, time-grid and work limits. The
actual settings and analyzed chunks are saved in `discovery/discovery.json`, and
the operator configuration hash is retained in provenance. This configuration is
not an upload-request field. A chunk-size comparison must rerun detection and the