Skip to content

Python tool for aircraft Doppler audio geolocation using historical ADS-B flight data (Source-available)

10.1 KB
Raw Blame

Evidence checks and inconclusive outcomes

confidence.decide_location(events, candidates, coverage, matching_diagnostics=..., policy=None) consumes the version-one contracts and returns status, reason_code, message and diagnostics. The product stores the last field in an artifact and only adds accepted contract fields to JobResult. Ranked candidates may be retained for inspection when the outcome is inconclusive; the UI must not show them as an accepted recorder position.

The function validates the inputs without changing them or reading evaluation truth. Malformed producer evidence returns failed / invalid_evidence. Missing recording-clock input remains the orchestrator's needs_input; acquisition or execution exceptions remain execution failures, distinct from insufficient scientific evidence. Coverage means completion of the requested geographic/time scope and availability of requested archives, not capture of every aircraft.

When separately labeled matched groups have strongly overlapping measured ridge samples compatible with a rational harmonic ratio within a conservative frequency-uncertainty heuristic, the decision is inconclusive. The diagnostic records the pair and measured discrepancy; it does not relabel events or assert that they came from the same aircraft. The heuristic is uncalibrated and may cause false inconclusive outcomes. A missing flag does not establish event independence, particularly with missing uncertainties or poorly aligned samples. The screen examines at most 8,192 event pairs per decision. Finding a possible overlap is sufficient to abstain. If the matched-event screen reaches its pair budget without a finding, it reports an incomplete assessment and abstains rather than treating unchecked pairs as independent. The broader proposal screen reports its assessment state in diagnostics; this cap does not limit event discovery or change candidate scores and rankings.

Default policy and evidence

The current default DecisionPolicy is engineering-v3; the original v1 receipts remain preserved. Development negative controls motivated explicit abstention for noise, absent tracks, harmonic duplicates, incomplete coverage, bad residuals and multiple plausible hypotheses. The engineering thresholds are deliberately simple and predeclared: normally at least two independent flyovers, <=3 Hz event RMSE, <=25% outliers and >=95% trajectory support. The current precision target is 1,000 m, following the updated project acceptance requirement; 100–500 m or better remains preferred. Earlier 100 m runs retain their original policy/threshold. Sensitivity radius is an engineering envelope, not measured location error. For negative_robust_ridge_loss_hz, a competing loss within 0.5 Hz per independent event is a near tie. The declared subset-support score has a separate dimensionless near-tie threshold of 0.2 support units per independent group; it is not a Hz threshold or probability. A near-tied alternative must fit inside the actual returned radius, capped at the precision target; overlapping target circles do not establish containment. A 70 m envelope therefore cannot certify a near-tied mode 90 m or 150 m from its center. Other score kinds have no declared gap threshold and abstain. These are regression-tested engineering choices, not empirically calibrated thresholds for arbitrary aircraft audio. No held-out real recordings have established their false-localization rate.

One flyover can pass the event-count gate with a trusted recording-start window when the fitted clock lies within it and sensitivity tests cover that entire input window, not only a local clock perturbation. All residual, support, global-alternative and uncertainty gates still apply. A known clock is not inferred from the fitted candidate. W5 supplies the validated original request window as recording_start_window_s; W3 profiles expose the tested span in sensitivity.settings.clock_bounds_s.

When a candidate clock falls inside the supplied window, diagnostics also report recording_start_window_fit: the window width, fit offset from the midpoint, and distance to the nearest edge, all in seconds. These are geometric descriptions of the optimizer result, not a clock-error estimate or calibration. They do not change the score, ranking, or decision; the caller remains responsible for supplying a defensible clock window.

Matched flyovers use one locked recording clock; harmonics count once and must map to the same flight. The minimum independent-flyover gate counts distinct matched flight segments: multiple acoustic groups assigned to one segment may contribute to its fit, but count once toward independent support. A one-flyover known-clock exception applies only to one matched harmonic group and one segment. The explicit subset score may leave proposals unmatched when unmatched_event_ids exactly enumerates the original discovered-minus-matched IDs. Entire harmonic groups must be retained. These omitted proposals remain unconfirmed sounds, not labelled aircraft misses or silently deleted evidence. Different retained subsets remain comparable only using their declared support score and may still trigger flight/location ambiguity. Matching diagnostics may also supply supported_event_ids: independently supported flyovers must all appear in the winning association. Three, four and more matched groups are all checked; there is no decision-stage maximum event count. A four-event regression ensures the fourth event's inconsistent residual cannot be ignored.

Matching diagnostics use event-ID keys with rmse_hz, outlier_fraction, support_fraction and drift_at_bound. Other required evidence keys are clock_locked, refinement_boundary_limited, alternatives_evaluated and sensitivity. Alternatives must include the retained global competing hypotheses, not only a selected local winner. Failed/unrefined hypotheses or incomplete tile coverage must not silently become evidence that no competing hypothesis exists.

sensitivity must carry complete: true, boundary_limited: false, radius_m and sources including frequency, clock, altitude and sound_speed. Candidate uncertainty must cover those actual perturbations, with a radius at least as large as the tested envelope. Unverified calibrated uncertainty claims abstain; this policy accepts explicitly uncalibrated sensitivity only. The current W3 altitude-only profile does not meet this requirement, even if its radius is zero. Missing measurement/model sensitivity therefore yields uncertainty_not_established. An engineering pass can be succeeded, but is explicitly uncalibrated and carries no probability. The synthetic fixture that supplies complete hypothetical evidence tests that branch; it does not establish that the real producer has provided such evidence.

Machine-readable reason codes

Code Interpretation
invalid_evidence Malformed producer result; execution failed
no_doppler_events No supported discovered event
coverage_unknown No coverage report
missing_flight_data Archive data unavailable
search_incomplete Requested geographic/time search unfinished
partial_flight_coverage Some required data missing
no_matching_flights No event/track hypothesis
insufficient_independent_events Too few flyovers, or matched groups may be overlapping harmonics
inconsistent_harmonic_association Harmonics map to different flights
incomplete_harmonic_group Partial association omits a matched event's harmonic
unexplained_events Unmatched proposals lack explicit subset-association evidence
unmatched_supported_events The hypothesis omits independently supported flyovers
recording_clock_mismatch Fitted start is outside the trusted original window
timing_not_established Shared clock unverified
matching_diagnostics_missing Event-fit evidence unavailable
insufficient_track_support Incomplete trajectory support
inconsistent_event_match Residual/outlier threshold failed
source_drift_unresolved Drift estimate missing or on its bound
unsupported_score_kind No declared ambiguity rule for score
incomparable_candidates Competing hypotheses explain different event sets
ambiguous_locations Near-tied materially different locations
ambiguous_flight_match Near-tied different flight associations
alternatives_not_assessed Retained global alternatives not fully assessed
uncertainty_not_established Full measurement/model sensitivity unavailable
calibration_not_verified Candidate's calibration claim lacks independent verification
refinement_boundary_limited Refinement/sensitivity reaches search boundary
location_uncertainty_too_large Sensitivity radius exceeds precision target
supported_by_evidence_checks Passed declared checks; uncalibrated

Reproduce decision controls

PYTHONPATH=src .venv/bin/python scripts/evaluate_confidence.py --output results/confidence-new
PYTHONPATH=src .venv/bin/python -m pytest -q tests/test_confidence.py tests/test_evaluation.py

benchmarks/decision-inputs.json contains simulated evidence, separate from benchmarks/truth/decision-labels.json. Decisions are written before truth is opened. There are eight development and eight held-out fixture cases. Each split has one engineered supported case and seven expected inconclusive outcomes; observed abstention is 7/8, unsupported localization 0/8, and the engineered location error is zero by construction. All case-level outputs are retained. These public, author-constructed fixture splits verify decision logic only. The zero errors and false-localization counts do not estimate audio or real-world localization accuracy. The audio controls and the frozen audit remain separate; see benchmarks/README.md for their evidence limits and real-data gap.

New evaluation reports include successes and false localizations at each of 100, 500 and 1,000 m, with the active threshold stated separately. The frozen historical runner explicitly retains its original 100 m threshold. An abstained best candidate may have a diagnostic error at these tiers, but it never counts as an accepted localization success.