Skip to content

Python tool for aircraft Doppler audio geolocation using historical ADS-B flight data (Source-available)

Integrate cb996dc continuation with verified consumer restrictions

Cinna Davis32e2ad9cff42
diff --cc docs/verification/CB996DC_CONTINUATION.md
index 0000000,0000000..ccda075
new file mode 100644
--- /dev/null
+++ b/docs/verification/CB996DC_CONTINUATION.md
@@@ -1,0 -1,0 +1,112 @@@
++# Continuation of cb996dc checkpoints
++
++This profile continues stopped searches from `cb996dcb49b9d31a24c10b9b5418c35423f7bbcd` under a specific Python consumer.
++It ports the unique behavior from [PR #260](https://github.com/1337isnot1337/DOPPLERGEIST/pull/260) after the #263 compatibility changes.
++The profile preserves completed tile bytes, their producer identities, and the retained scores and bounds.
++
++The consumer must match the 30 source hashes in [the verification receipt](cb996dc-continuation-20261104.json).
++Those files are identical at `350c9bfa1a56ce2b46d0418b39779b12cdfa04de` and `be56c2fae81fe65159103ddb2bf3ccd59ca3bb67`.
++The consumer also needs the `checkpoint_resume.py` profile and tests from this change.
++The source list covers search orchestration, input interpretation, scoring, retention, and their Python dependencies.
++It excludes this adapter and files that only provide CLI, API, or UI behavior.
++
++The geometry change proposed in #293 changes which trajectory segments can produce scores.
++The shared-observation change for #275 changes which event groups can receive independent scores.
++Those consumers need a separate compatibility proof before they can reuse these scores.
++This profile rejects changes to `geometry.py`, `search.py`, `tiled_search.py`, or any other pinned file.
++It does not authorize continuation on arbitrary future versions of main.
++
++## Preserved behavior and restrictions
++
++The old scorer used two matched groups by default for a broad recording clock with multiple groups.
++The profile writes that effective minimum into both the search manifest and the operator configuration.
++An explicit saved minimum remains unchanged.
++The profile uses ordinary batched arithmetic and does not enable `legacy_first_batch`.
++
++The profile requires the Python ridge fitter, acoustic solver, CPA calculation, and flight decoder.
++The current native flight decoder has no environment switch, so the source checkout must have no compiled `_native_index` extension.
++The producer and consumer must have the same Python version, machine architecture, and recorded package versions.
++Audio, request, runtime, event preparation, and saved-file identity checks remain active.
++The job must have a terminal status of `cancelled`, `failed`, or `inconclusive`.
++The profile rejects fine checkpoints and the legacy first-batch mode because this producer did not create them.
++
++The adapter copies the retention database independently before it changes the run identity.
++It keeps the previous search directory and identity files for rollback.
++An interrupted metadata commit restores the original search, operator configuration, and audio identity.
++The compatibility reader binds the old producer to the new run hash and the pinned consumer fingerprint.
++
++## Real-data evidence
++
++Both replay pairs used the same recorded input and numerical runtime within each pair.
++All recordings, trajectories, terrain, and ridge measurements came from recorded or released data.
++The receiver location and regional bounds were source-informed.
++The historical regional index excluded known jets during its original ingestion and does not establish complete coverage.
++These comparisons establish software equivalence for these workloads, not independent location accuracy or nationwide completion time.
++
++| Replay | Candidate associations | Ranked payloads | Retained records | Retained bounds | Result |
++| --- | ---: | ---: | ---: | ---: | --- |
++| Regional tile, two points, 317 tracks, 16 groups | 1,699 | 20 | 112 | 1,617 | Exact scores, order, retention, and remaining payload fields |
++| Full-day query, one point, 1,354 tracks, 16 groups | 16,015 | 100 | 499 | 15,691 | Exact scores, order, retention, and remaining payload fields |
++
++The full-day replay used an unknown recording clock and one documented association-anchor partition.
++Its query covered `1788825600.17` through `1788911999.6` seconds UTC.
++Its anchor partition covered `1788864888.7151732` through `1788865236.7214916` seconds UTC.
++Physical fit intervals remained unchanged.
++This bounded partition satisfies the dense replay check without a complete nationwide search.
++
++The full-day retention checksum was `ca75b434b7776ca1507c9efed4e3e0f7df891a623c0b28b30fe19d0d1b5048e7` on both revisions.
++The producer exposed a `minimum_matched_events: 2` diagnostic that the consumer no longer emits.
++The consumer added an empty `clock_excluded_unsupported_tracks` diagnostic.
++All other compared payload fields were equal.
++
++The full-day runs finished within independent 180-second limits.
++An earlier two-point full-day attempt exceeded the same candidate budget on both revisions before scoring.
++Its failure receipts and partial artifacts remain available in the local evidence directory.
++The successful point partition used the existing documented benchmark configuration instead of an increased candidate budget.
++The shared machine ran concurrent work, so the elapsed times do not support a performance claim.
++
++A separate run used the unmodified cb996dc JobManager and the whole real MP3.
++It produced an actual inconclusive job with two nonempty coarse tiles.
++The continuation tests reused both tiles and reproduced the ordered ranked result exactly.
++The tests also rejected changed producer and geometry identities, invalid manifests, changed runtimes, native processing, and active jobs.
++The interrupted-commit control restored the original run identity and metadata.
++No original checkpoint or live job changed during these checks.
++
++## Continue a compatible stopped job
++
++1. Stop the job and wait for its worker to exit.
++2. Copy the complete stopped job directory, including its retention database and referenced inputs.
++3. Use the consumer files from `be56c2fae81fe65159103ddb2bf3ccd59ca3bb67` with this continuation change applied.
++4. Use the Python version and package versions recorded in the saved manifest.
++5. Verify that the checkout has no compiled `_native_index` extension.
++6. Run the continuation from that checkout with all three numerical backend variables set to `python`.
++
++```bash
++DOPPLERGEIST_RIDGE_BACKEND=python \
++DOPPLERGEIST_ACOUSTIC_BACKEND=python \
++DOPPLERGEIST_CPA_BACKEND=python \
++PYTHONPATH=src \
++python -m dopplergeist.cli resume JOB_ID --jobs /path/to/copied-jobs
++```
++
++The normal resume command performs the compatibility checks before it queues work.
++An unsupported consumer leaves the old checkpoint incompatible.
++An inconclusive result remains inconclusive unless the resumed search independently meets the normal decision requirements.
++
++## Repeat the continuation controls
++
++1. Use an actual stopped cb996dc job with at least one nonempty coarse tile.
++2. Run the tests in the pinned consumer checkout.
++
++```bash
++DOPPLERGEIST_RIDGE_BACKEND=python \
++DOPPLERGEIST_ACOUSTIC_BACKEND=python \
++DOPPLERGEIST_CPA_BACKEND=python \
++DOPPLERGEIST_REAL_CB996DC_JOB=/path/to/stopped-job \
++PYTHONPATH=src \
++python -m pytest -q tests/test_real_cb996dc_checkpoint.py
++```
++
++The tests copy the real checkpoint before fault injection or continuation.
++They skip when the environment does not provide a real job.
++They do not generate replacement recordings, trajectories, or scores.
diff --cc docs/verification/cb996dc-continuation-20261104.json
index 0000000,0000000..838ded7
new file mode 100644
--- /dev/null
+++ b/docs/verification/cb996dc-continuation-20261104.json
@@@ -1,0 -1,0 +1,586 @@@
++{
++ "scope": "Recorded-input software equivalence and checkpoint continuation. No independent location accuracy or complete-coverage claim.",
++ "original_pull_request": "https://github.com/1337isnot1337/DOPPLERGEIST/pull/260",
++ "producer_revision": "cb996dcb49b9d31a24c10b9b5418c35423f7bbcd",
++ "producer_code_fingerprint": "c66d402b750381303bd0c7d9c828d2294942227176ccb3cb157f0ef6299cbe40",
++ "validated_numerical_consumer_revision": "350c9bfa1a56ce2b46d0418b39779b12cdfa04de",
++ "integration_base_revision": "be56c2fae81fe65159103ddb2bf3ccd59ca3bb67",
++ "target_code_fingerprint": "2e6a9601eaf42be7ca2b440eef55eeafa894ab21f9f90adcbb216c9f08a16d2d",
++ "target_file_hashes": {
++ "__init__.py": "01428b7938817c1cfe7eb3cf7e186f35b3fa7bc2d85f14204d07e0aa251d2c88",
++ "acoustic_screen.py": "a0a479a46d6c48ab973ec539d1457868a7eb6f4b9c361a0426dc7993f84970ec",
++ "audio.py": "7ae1cbd3e682057096f103d34fa5664b11cf4e7a74e25af29736ed6618cc9c6d",
++ "automatic.py": "8de96b1450eb26ab33acac5d36680c743c3d2c537b4b0f7182aae9b6faf245bd",
++ "batch_ridge.py": "d8187c64c3fcfb2d084fbdaa302b78791c68fa6aed33e2d61f95537058ab7f3a",
++ "confidence.py": "6a35edb42bc5321999a179f09f33727122f6de2549924d47f1da99673dcae00d",
++ "contracts.py": "602735eef7de44e6410d79336e33c47d35e2c3d00d531729a4c137995ccd1a70",
++ "data_access.py": "77da0d39e1e95476fdf02e29e9a345914545af7e62227a93b4cb6786edb37a71",
++ "discovery.py": "0a973a1238b7d0c344efa23383ac42a7a8d5b37a2e975b4fad9ac878cee5b2a0",
++ "finite_search.py": "f499ba1f298b5869ac5ae375a2eddb46aabfabcfb9f3fd0d848e362c636c0367",
++ "flight_payload.py": "d74d218529d91db6397c5450a4b57f1f5efd2b3bc942120858dfabe6e69fd4e5",
++ "geometry.py": "85336bd0e3739f890386e9f93965cefb78c4477ae2e8a4fe489353501c5e26a2",
++ "index_compatibility.py": "a9aa315a3cdaf9b01bcac027bb4f7b5074864db8623402bc1afcbee24e6ecc83",
++ "intake.py": "c88f51f8c7edd6abb74fca830a839d0bbce758beb9a582a7791a8a174b603781",
++ "jobs.py": "40517440e0dc380898e0685b8c4bec3320e9cf121626983ee207e42729ad6ebc",
++ "matching.py": "777115ddc2752e27a4a4e69cc25eab727ff819d54398a688bccbf8b10ebdff5a",
++ "native_acoustic.py": "608475fc2254e9a80549ebd0fc337ba2893d938ee8d4834f1573d0022b1b6f7f",
++ "native_cpa.py": "32c02cd43f6a1a67bdaa17ee3cdaef385fd763a4781610aeada9e1d8d714566d",
++ "parallel_retention.py": "2861608faa3490601aec9870647ec2a5f9268ff051548565b6d657cb1e95155f",
++ "parallel_tiles.py": "3605621c3c5fd9b35008fe1680e1bea99f968ec841f69b4a5bc9e3ced3c52108",
++ "provenance.py": "b3dfda5b31f1aec195eef8ef763b08b2734265da06ce41c129daf70acce8c041",
++ "resource_usage.py": "2704ff96bc9ae5a39fa29b3f4cee590e38bc5e59f3ec369e6aa28aab307f5d04",
++ "scoring_inputs.py": "9cbc11678ba5a555acf234945e037662b494591df6a33d21b460e726c62c3588",
++ "search.py": "41b75f9f2d4a951cd2e1cac42b81ca5a7e15620fca4cd44e75767de5a025f9fb",
++ "spectral_grid.py": "67fe613a0ee9505f1d9cfd8db468ee03ccdf8664fe1608d6df3d484ebcaad27c",
++ "spectral_matching.py": "8de9273ea8090ce9e5a639c6d9fb4b88c2455afc02413643935983149fe9d2e6",
++ "terrain.py": "749d6e1d526324fc3015511e9f83fde3663d31a1fc863f069bd18f56601869f8",
++ "tiled_search.py": "592ebec1d077cc9b42c55ea07bccc4097c082e2c455a7a1580711deae2956eed",
++ "tracks.py": "543f483cee756fdb9efdd69c8e8f05b32e290f19187af3523160f19d2eba90e0",
++ "trial_prefix.py": "cd428883c036d52e9eb6f06cb453fa4cfa63614695ff229db21f3ca5ebfba285"
++ },
++ "backend_environment": {
++ "DOPPLERGEIST_RIDGE_BACKEND": "python",
++ "DOPPLERGEIST_ACOUSTIC_BACKEND": "python",
++ "DOPPLERGEIST_CPA_BACKEND": "python"
++ },
++ "runtime": {
++ "python": "3.12.14",
++ "machine": "x86_64",
++ "packages": {
++ "numpy": "2.3.5",
++ "scipy": "1.17.0",
++ "pandas": "2.2.3",
++ "pyproj": "3.8.0",
++ "rasterio": "1.5.2",
++ "pyarrow": "26.0.1"
++ }
++ },
++ "same_full_day_runtime": true,
++ "numerical_thread_environment": {
++ "OPENBLAS_NUM_THREADS": null,
++ "OMP_NUM_THREADS": null,
++ "MKL_NUM_THREADS": null,
++ "NUMEXPR_NUM_THREADS": null,
++ "VECLIB_MAXIMUM_THREADS": null
++ },
++ "regional_replay": {
++ "producer_revision": "cb996dcb49b9d31a24c10b9b5418c35423f7bbcd",
++ "consumer_revision": "06693a457f4f1dde0d194588232d19a459f946e1",
++ "scope": "Complete nonempty regional tile from real MP3 discovery and recorded ADS-B/DEM; 20-second source-informed clock; not the canonical full-day checkpoint/runtime.",
++ "points": 2,
++ "tracks": 317,
++ "groups": 16,
++ "candidates": 1699,
++ "ranked_count": 20,
++ "same_ordered_ranked_rows": true,
++ "same_points": true,
++ "same_tracks": true,
++ "same_events": true,
++ "same_runtime": true,
++ "same_retention": true,
++ "retention": {
++ "stages": {
++ "rows": 1,
++ "sha256": "93c084095e2e9ec2f6a36c6840d6879b0361fbe70b4705a382c289bb6f856b3d"
++ },
++ "records": {
++ "rows": 112,
++ "sha256": "dfe03cfaafdec30c9cec19a5e809bf59a232fa0acd278590008d57d0ca88910e"
++ },
++ "bounds": {
++ "rows": 1617,
++ "sha256": "660201d145a54d34970a129d9dff7cc0a983029452eb6779cc7dbc699075a3a4"
++ }
++ },
++ "retention_checksum": "62a465ba1ef79dab292143b8d7af4c41e0c9ed4e3672bbfd2eaa2d6b2b16e31f",
++ "counts_differences": {
++ "clock_excluded_unsupported_tracks": {
++ "cb996dc": null,
++ "current": []
++ },
++ "minimum_matched_events": {
++ "cb996dc": 2,
++ "current": null
++ }
++ },
++ "all_other_payload_fields_equal": true,
++ "source_checkpoint_preserved": {
++ "search/manifest.json": true,
++ "search/retention.sqlite": true,
++ "search/tile-58_68.json": true,
++ "discovery/discovery.json": true
++ },
++ "options": {
++ "bounds": [
++ -73.545,
++ 40.865,
++ -73.505,
++ 40.895
++ ],
++ "time_range_s": [
++ 1788898746.3333333,
++ 1788902290.1845
++ ],
++ "budget": {
++ "spacing_m": 2000,
++ "tile_cells": 16,
++ "max_tracks_per_tile": 2000,
++ "max_track_samples_per_tile": 500000,
++ "max_cpa_rows": 100000,
++ "max_candidates_per_tile": 20000,
++ "top_k": 20,
++ "refine_spacing_m": 500,
++ "refine_finalists": 3,
++ "cpa_step_s": 2.0,
++ "max_tiles": null
++ },
++ "scoring": {
++ "mode": "robust_ridge",
++ "c": 343.0,
++ "time_origin": null,
++ "time_sweep_s": 5,
++ "time_step_s": 0.5,
++ "recording_start_window_s": [
++ 1788898640.0,
++ 1788898660.0
++ ]
++ },
++ "filters": {
++ "association": "subsets",
++ "max_events": 16
++ },
++ "crs": "EPSG:5070",
++ "coarse_floor_policy": "global_and_per_tile",
++ "execution": "Direct production _tile_coordinates -> _tracks_for_points -> _sample_tile_terrain -> _score for all recorded cells in tile58_68; no candidate/event truncation and no initial score-floor override."
++ },
++ "inputs": [
++ {
++ "path": "/workspace/scratch/e20de6666de2/verification-main-ledger/jobs/6f9890441e4b4add8a7eb35890c185a5/search/manifest.json",
++ "bytes": 139863,
++ "sha256": "cb4510a238c5de5014ff0225a64380aa8ac26e32be6e252bb55364a79a530605"
++ },
++ {
++ "path": "/workspace/scratch/e20de6666de2/verification-main-ledger/jobs/6f9890441e4b4add8a7eb35890c185a5/search/tile-58_68.json",
++ "bytes": 2018801,
++ "sha256": "50294472363ac7e70d8a1005dfedd7b3110a9506c1889d15915fb6ecac10be4c"
++ },
++ {
++ "path": "/workspace/scratch/e20de6666de2/verification-main-ledger/jobs/6f9890441e4b4add8a7eb35890c185a5/discovery/discovery.json",
++ "bytes": 213364,
++ "sha256": "c640045ef8aedf29a0cf50f6cdef07d53aa3f024d403ee243978c7250314d80c"
++ },
++ {
++ "path": "/workspace/scratch/e20de6666de2/verification-main-ledger/jobs/6f9890441e4b4add8a7eb35890c185a5/input.mp3",
++ "bytes": 42085293,
++ "sha256": "e1c13dafd8d1634bbf84df843c31771318e15a31ebb86dd165af7ee312ab80a0"
++ },
++ {
++ "path": "/workspace/scratch/e20de6666de2/real-ledger-inputs/reference-region.sqlite",
++ "bytes": 85929984,
++ "sha256": "17ab5d9b860513c86fa850b6c7e7318cf5d507339dfa1921e86611439b214c6a"
++ },
++ {
++ "path": "/workspace/scratch/e20de6666de2/verification-pr260-parity/replay_tile.py",
++ "bytes": 4978,
++ "sha256": "cb431588a3502d0b75d8e716ea655b7a33c0b168b53cd3e8cfa4176ac834d2d0"
++ },
++ {
++ "path": "/workspace/scratch/e20de6666de2/DOPPLERGEIST/data/terrain/Copernicus_DSM_COG_10_N40_00_W074_00_DEM.tif",
++ "bytes": 15806993,
++ "sha256": "6a0f4273de0bbc0709cc9db4793a89ac40c811388f59f23f644c0304b66ab94d"
++ },
++ {
++ "path": "/workspace/scratch/e20de6666de2/DOPPLERGEIST/data/terrain/us_nga_egm96_15.tif",
++ "bytes": 2710815,
++ "sha256": "db493027562c9b004d7220fa881f5603adada4e1c5029b933fa7de4547b0e78d"
++ },
++ {
++ "path": "/workspace/scratch/e20de6666de2/verification-main-ledger/jobs/6f9890441e4b4add8a7eb35890c185a5/discovery/event-0001.csv",
++ "bytes": 1602,
++ "sha256": "db32c8e940b5512873ec18fe405eb06cf363d4567815db3ee2531f9190bf5ab4"
++ },
++ {
++ "path": "/workspace/scratch/e20de6666de2/verification-main-ledger/jobs/6f9890441e4b4add8a7eb35890c185a5/discovery/event-0002.csv",
++ "bytes": 1827,
++ "sha256": "afe993bfe19457324ee47ff8f2cf938214a90aa00fd4a518e1ec54050837e643"
++ },
++ {
++ "path": "/workspace/scratch/e20de6666de2/verification-main-ledger/jobs/6f9890441e4b4add8a7eb35890c185a5/discovery/event-0003.csv",
++ "bytes": 2727,
++ "sha256": "6ca82441bd4346d21827d99ca0910ea51ead05101d5261a44e5f1ad604d35a90"
++ },
++ {
++ "path": "/workspace/scratch/e20de6666de2/verification-main-ledger/jobs/6f9890441e4b4add8a7eb35890c185a5/discovery/event-0004.csv",
++ "bytes": 1677,
++ "sha256": "277d1c95b08dee58b17fd8ada90a38b9a2d7b0f0cc45ea1a6f4217de9dde81ce"
++ },
++ {
++ "path": "/workspace/scratch/e20de6666de2/verification-main-ledger/jobs/6f9890441e4b4add8a7eb35890c185a5/discovery/event-0005.csv",
++ "bytes": 2502,
++ "sha256": "9f74137474bf5daf8ea0a0386e4d8111ec20495082c798a42ba109f8c540049c"
++ },
++ {
++ "path": "/workspace/scratch/e20de6666de2/verification-main-ledger/jobs/6f9890441e4b4add8a7eb35890c185a5/discovery/event-0006.csv",
++ "bytes": 2277,
++ "sha256": "ae5abab11a3c5d546c2efb75a1d045b786a101e238ee4a9609284d6a5d2c7aa5"
++ },
++ {
++ "path": "/workspace/scratch/e20de6666de2/verification-main-ledger/jobs/6f9890441e4b4add8a7eb35890c185a5/discovery/event-0007.csv",
++ "bytes": 3627,
++ "sha256": "3fe3027c1985975537f70927e8f5490d0ab86465441a74888f2dc4ff418cb701"
++ },
++ {
++ "path": "/workspace/scratch/e20de6666de2/verification-main-ledger/jobs/6f9890441e4b4add8a7eb35890c185a5/discovery/event-0008.csv",
++ "bytes": 3327,
++ "sha256": "fb53735e349b179538fa65e86b0db40716159e84c1de43f51dc9ecd380186c51"
++ },
++ {
++ "path": "/workspace/scratch/e20de6666de2/verification-main-ledger/jobs/6f9890441e4b4add8a7eb35890c185a5/discovery/event-0009.csv",
++ "bytes": 1752,
++ "sha256": "a691392e4348390f0f681677699a536d4ff05e775abfcfee6827d5f961935301"
++ },
++ {
++ "path": "/workspace/scratch/e20de6666de2/verification-main-ledger/jobs/6f9890441e4b4add8a7eb35890c185a5/discovery/event-0010.csv",
++ "bytes": 2952,
++ "sha256": "cfb556b2203a4fe8949aad384db33d310d2158cba754ed422aaf054bded88be3"
++ },
++ {
++ "path": "/workspace/scratch/e20de6666de2/verification-main-ledger/jobs/6f9890441e4b4add8a7eb35890c185a5/discovery/event-0011-harmonic-2.csv",
++ "bytes": 4527,
++ "sha256": "9ef47f55d3b7972c1ec4e32cc123e01e91e4317c4154b9b72bbd392130efa89f"
++ },
++ {
++ "path": "/workspace/scratch/e20de6666de2/verification-main-ledger/jobs/6f9890441e4b4add8a7eb35890c185a5/discovery/event-0011-harmonic-3.csv",
++ "bytes": 1752,
++ "sha256": "bad92f16bbb3da88d85c6d23e4b9bfc7c6d917c6be78b034198a5c1140eb17d8"
++ },
++ {
++ "path": "/workspace/scratch/e20de6666de2/verification-main-ledger/jobs/6f9890441e4b4add8a7eb35890c185a5/discovery/event-0011.csv",
++ "bytes": 15027,
++ "sha256": "c2381e2d365fa15f02ed6389f19e3a3c8bda75147181ffed3f4e253f807cd1d4"
++ },
++ {
++ "path": "/workspace/scratch/e20de6666de2/verification-main-ledger/jobs/6f9890441e4b4add8a7eb35890c185a5/discovery/event-0012.csv",
++ "bytes": 4377,
++ "sha256": "cfec0566d225300f0d9a0bf6dc9b92551ebad62610c964afa00753ae57f9c474"
++ },
++ {
++ "path": "/workspace/scratch/e20de6666de2/verification-main-ledger/jobs/6f9890441e4b4add8a7eb35890c185a5/discovery/event-0013.csv",
++ "bytes": 6327,
++ "sha256": "b660a01a291000ad451e30ed44f0fb0cea327571c9a81a7437238d8c1df0c81d"
++ },
++ {
++ "path": "/workspace/scratch/e20de6666de2/verification-main-ledger/jobs/6f9890441e4b4add8a7eb35890c185a5/discovery/event-0014.csv",
++ "bytes": 1827,
++ "sha256": "ce724f76e6169ecc3b43526d300c653d92cc8222e786abaf7a10cfcd9d34d1be"
++ },
++ {
++ "path": "/workspace/scratch/e20de6666de2/verification-main-ledger/jobs/6f9890441e4b4add8a7eb35890c185a5/discovery/event-0015.csv",
++ "bytes": 4752,
++ "sha256": "0ecebac047804687a696c4153e2157baeffc8bfcb18afe04b2a1362788810e47"
++ },
++ {
++ "path": "/workspace/scratch/e20de6666de2/verification-main-ledger/jobs/6f9890441e4b4add8a7eb35890c185a5/discovery/event-0016.csv",
++ "bytes": 2127,
++ "sha256": "ecf83cc2872b207a9005a216a2ef58af453899c334aba4383104afbe64568d28"
++ }
++ ],
++ "outputs": {
++ "cb996dc": {
++ "report.json": {
++ "path": "/workspace/scratch/e20de6666de2/verification-pr260-parity/cb996dc/report.json",
++ "bytes": 7342,
++ "sha256": "d87a38b664773546b7bada3280a5ad93a3a2917bca56986a556ea58774480f1b"
++ },
++ "payload.json": {
++ "path": "/workspace/scratch/e20de6666de2/verification-pr260-parity/cb996dc/payload.json",
++ "bytes": 3108129,
++ "sha256": "14025e0d691cdba59b7d52b7f9d5b201f8106ad21ae39b5fe6f888bbdebdaf1a"
++ },
++ "retention.sqlite": {
++ "path": "/workspace/scratch/e20de6666de2/verification-pr260-parity/cb996dc/retention.sqlite",
++ "bytes": 2654208,
++ "sha256": "fc677b7a95621bef5797be1dc6b3220a7f7c07e8f222129a352c8898cbd553e7"
++ }
++ },
++ "current": {
++ "report.json": {
++ "path": "/workspace/scratch/e20de6666de2/verification-pr260-parity/current/report.json",
++ "bytes": 7328,
++ "sha256": "1df2c1c3f827a461eee32c6480376f0cbb4b4e02ccc601bd9483ae73a28ebc2f"
++ },
++ "payload.json": {
++ "path": "/workspace/scratch/e20de6666de2/verification-pr260-parity/current/payload.json",
++ "bytes": 3108141,
++ "sha256": "797900746ffd868ef6686189d7466d8031d655b3c235fa1e700c065503d5cb0a"
++ },
++ "retention.sqlite": {
++ "path": "/workspace/scratch/e20de6666de2/verification-pr260-parity/current/retention.sqlite",
++ "bytes": 2654208,
++ "sha256": "fc677b7a95621bef5797be1dc6b3220a7f7c07e8f222129a352c8898cbd553e7"
++ }
++ }
++ }
++ },
++ "full_day_replay": {
++ "producer_revision": "cb996dcb49b9d31a24c10b9b5418c35423f7bbcd",
++ "consumer_revision": "350c9bfa1a56ce2b46d0418b39779b12cdfa04de",
++ "scope": "Documented real full-day dense point/association-anchor partition; unknown recording clock, source-informed receiver, incomplete regional coverage; no accuracy claim.",
++ "fixture_path": "/workspace/scratch/e20de6666de2/verification-pr260-parity/documented-full-day-fixture.json",
++ "fixture_sha256": "d7ad8de1657d524b933454fb22c842c8d14bf4c35bb595f4e481269b6afefdcd",
++ "runtime": {
++ "python": "3.12.14",
++ "machine": "x86_64",
++ "packages": {
++ "numpy": "2.3.5",
++ "scipy": "1.17.0",
++ "pandas": "2.2.3",
++ "pyproj": "3.8.0",
++ "rasterio": "1.5.2",
++ "pyarrow": "26.0.1"
++ }
++ },
++ "points": 1,
++ "tracks": 1354,
++ "groups": 16,
++ "candidates": 16015,
++ "ranked_count": 100,
++ "retention": {
++ "stages": {
++ "rows": 1,
++ "sha256": "973d246388dd4523e770a52ba98edf77a9dcc684e6c463cfebb253a1a651ba02"
++ },
++ "records": {
++ "rows": 499,
++ "sha256": "316ed354465d0548133f84ab118b9d96f00dce8bf6f0df4d44c247df52a2e488"
++ },
++ "bounds": {
++ "rows": 15691,
++ "sha256": "3d72914f96e31f5b3c1954e292a1f295be40c1dedf6ac188bd75048e6469bfc0"
++ }
++ },
++ "retention_checksum": "ca75b434b7776ca1507c9efed4e3e0f7df891a623c0b28b30fe19d0d1b5048e7",
++ "same_ordered_ranked_rows": true,
++ "same_retention": true,
++ "same_points_tracks_events": true,
++ "all_other_payload_fields_equal": true,
++ "counts_differences": {
++ "minimum_matched_events": {
++ "producer": 2,
++ "consumer": null
++ },
++ "clock_excluded_unsupported_tracks": {
++ "producer": null,
++ "consumer": []
++ }
++ },
++ "recording_start_window_s": null,
++ "query_time_s": [
++ 1788825600.17,
++ 1788911999.6
++ ],
++ "anchor_window_s": [
++ 1788864888.7151732,
++ 1788865236.7214916
++ ],
++ "producer_outputs": {
++ "report.json": {
++ "path": "/workspace/scratch/e20de6666de2/verification-pr260-parity/documented-full-day-cb996dc/report.json",
++ "sha256": "30e1af59679cddd5bdf5515561d8aca779002c7b20479e871a9d0df1cac14eee",
++ "bytes": 24090
++ },
++ "payload.json": {
++ "path": "/workspace/scratch/e20de6666de2/verification-pr260-parity/documented-full-day-cb996dc/payload.json",
++ "sha256": "6f16fa437e1b9b67348b6f4979f907a19dcc73dfa414fff8913f52b10f4d4bae",
++ "bytes": 29506029
++ },
++ "retention.sqlite": {
++ "path": "/workspace/scratch/e20de6666de2/verification-pr260-parity/documented-full-day-cb996dc/retention.sqlite",
++ "sha256": "e893fc635ea60b15207f208d91c4a34f352b06472b93cbcd1039f40351f8942a",
++ "bytes": 25264128
++ }
++ },
++ "consumer_outputs": {
++ "report.json": {
++ "path": "/workspace/scratch/e20de6666de2/verification-pr260-parity/documented-full-day-current318/report.json",
++ "sha256": "2accec444f07f7ea1f951fedf16514927615be415f618ed12563198bfd27d913",
++ "bytes": 24075
++ },
++ "payload.json": {
++ "path": "/workspace/scratch/e20de6666de2/verification-pr260-parity/documented-full-day-current318/payload.json",
++ "sha256": "defc23048b7afd29670d512c70e8b1a23337dd9d99193e2a50a2f4476ae44b60",
++ "bytes": 29506041
++ },
++ "retention.sqlite": {
++ "path": "/workspace/scratch/e20de6666de2/verification-pr260-parity/documented-full-day-current318/retention.sqlite",
++ "sha256": "e893fc635ea60b15207f208d91c4a34f352b06472b93cbcd1039f40351f8942a",
++ "bytes": 25264128
++ }
++ }
++ },
++ "full_day_input_files": [
++ {
++ "path": "/workspace/scratch/e20de6666de2/DOPPLERGEIST-pr260-review/evidence/audio/w1-sensitivity-20261102/reference/discovery.json",
++ "sha256": "89281081ba16310e622a90001373bb1d21ad59eed5667842e98957753feab26c",
++ "bytes": 188675
++ },
++ {
++ "path": "/workspace/scratch/e20de6666de2/DOPPLERGEIST-pr260-review/evidence/results/reference-local/report.json",
++ "sha256": "29b01918f2bb20e9f9a3770bfcc229f1bd1b8e8862fe2f66483b988a73cbf090",
++ "bytes": 4311
++ },
++ {
++ "path": "/workspace/scratch/e20de6666de2/real-ledger-inputs/reference-region.sqlite",
++ "sha256": "17ab5d9b860513c86fa850b6c7e7318cf5d507339dfa1921e86611439b214c6a",
++ "bytes": 85929984
++ },
++ {
++ "path": "/workspace/scratch/e20de6666de2/verification-pr260-parity/documented-full-day-fixture.json",
++ "sha256": "d7ad8de1657d524b933454fb22c842c8d14bf4c35bb595f4e481269b6afefdcd",
++ "bytes": 358130
++ },
++ {
++ "path": "/workspace/scratch/e20de6666de2/verification-pr260-parity/replay_documented_full_day.py",
++ "sha256": "d3d5b45e71098e8c21eb3fe035ac21ef697595841f7238c1886887e8d8cb5d57",
++ "bytes": 4404
++ }
++ ],
++ "full_day_configuration": {
++ "index_input_hash": "229d57dc810fc3f2985bae5c7fc68eb1132912cf8b95fdea2376c739d66e22a2",
++ "query_bounds": [
++ -74.299988,
++ 40.100005,
++ -72.700011,
++ 41.599964
++ ],
++ "query_time_s": [
++ 1788825600.17,
++ 1788911999.6
++ ],
++ "budget": {
++ "spacing_m": 2000.0,
++ "tile_cells": 32,
++ "max_tracks_per_tile": 2000,
++ "max_track_samples_per_tile": 500000,
++ "max_cpa_rows": 100000,
++ "max_candidates_per_tile": 20000,
++ "top_k": 100,
++ "refine_spacing_m": 100.0,
++ "refine_finalists": 10,
++ "cpa_step_s": 2.0,
++ "max_tiles": null
++ },
++ "scoring": {
++ "mode": "robust_ridge"
++ },
++ "recording_start_window_s": null,
++ "anchor_window_s": [
++ 1788864888.7151732,
++ 1788865236.7214916
++ ],
++ "track_query_preparation": {
++ "queried_trajectories": 4694,
++ "queried_trajectory_samples": 713100,
++ "operations": {
++ "query_calls": 1,
++ "decoded_track_yields": 4694,
++ "decoded_sample_yields": 713100
++ }
++ },
++ "counts": {
++ "trajectories": 1354,
++ "trajectory_samples": 357324,
++ "ridge_records": 16,
++ "independent_groups": 16,
++ "ridge_samples": 724,
++ "cpa_rows": 944,
++ "unsupported_cpa_tracks": 62
++ }
++ },
++ "full_day_receipts": [
++ {
++ "label": "documented-full-day-cb996dc",
++ "source": "/workspace/scratch/e20de6666de2/DOPPLERGEIST-cb996dc-review",
++ "started_utc": "2026-11-04T03:10:52.091754+00:00",
++ "wall_cap_s": 180.0,
++ "passed": true,
++ "command": [
++ "/workspace/scratch/e20de6666de2/.venv/bin/python",
++ "/workspace/scratch/e20de6666de2/verification-pr260-parity/replay_documented_full_day.py",
++ "/workspace/scratch/e20de6666de2/DOPPLERGEIST-cb996dc-review",
++ "/workspace/scratch/e20de6666de2/verification-pr260-parity/documented-full-day-cb996dc"
++ ],
++ "backend_environment": {
++ "DOPPLERGEIST_RIDGE_BACKEND": "python",
++ "DOPPLERGEIST_ACOUSTIC_BACKEND": "python",
++ "DOPPLERGEIST_CPA_BACKEND": "python"
++ },
++ "source_revision": "cb996dcb49b9d31a24c10b9b5418c35423f7bbcd",
++ "returncode": 0,
++ "timed_out": false,
++ "wall_seconds": 69.19827941400035,
++ "partial_artifacts_preserved": true
++ },
++ {
++ "label": "documented-full-day-current318",
++ "source": "/workspace/scratch/e20de6666de2/DOPPLERGEIST-pr260-review",
++ "started_utc": "2026-11-04T03:10:52.086807+00:00",
++ "wall_cap_s": 180.0,
++ "passed": true,
++ "command": [
++ "/workspace/scratch/e20de6666de2/.venv/bin/python",
++ "/workspace/scratch/e20de6666de2/verification-pr260-parity/replay_documented_full_day.py",
++ "/workspace/scratch/e20de6666de2/DOPPLERGEIST-pr260-review",
++ "/workspace/scratch/e20de6666de2/verification-pr260-parity/documented-full-day-current318",
++ "--normalize-min-events"
++ ],
++ "backend_environment": {
++ "DOPPLERGEIST_RIDGE_BACKEND": "python",
++ "DOPPLERGEIST_ACOUSTIC_BACKEND": "python",
++ "DOPPLERGEIST_CPA_BACKEND": "python"
++ },
++ "source_revision": "350c9bfa1a56ce2b46d0418b39779b12cdfa04de",
++ "returncode": 0,
++ "timed_out": false,
++ "wall_seconds": 42.10764088199994,
++ "partial_artifacts_preserved": true
++ }
++ ],
++ "real_legacy_job": {
++ "job": "/workspace/scratch/e20de6666de2/verification-pr260-parity/genuine-cb996dc-job/jobs/c00d7c754bc845ccaca8f8c0a3d60c92",
++ "status": "inconclusive",
++ "reason_code": "search_incomplete",
++ "run_hash": "1305ac416229590a1133166f5ebab5af67432942636a4bf74c3c15da6bc2d91c",
++ "producer_code_fingerprint": "c66d402b750381303bd0c7d9c828d2294942227176ccb3cb157f0ef6299cbe40",
++ "coarse_floor_policy": "per_tile",
++ "nonempty_tiles": [
++ {
++ "name": "tile-58_68.json",
++ "candidates": 1699,
++ "ranked": 20,
++ "retention_checksum": "7a26d507ad4da1ec11e2e5361585eedd35c5ac69853bef3a100171fa158f0150"
++ },
++ {
++ "name": "tile-58_69.json",
++ "candidates": 1681,
++ "ranked": 20,
++ "retention_checksum": "27555384b433d759a2ff8525c043b5c23c1fa6511d5eb4423ebca8579f5491f7"
++ }
++ ]
++ },
++ "continuation_controls": {
++ "command": "DOPPLERGEIST_RIDGE_BACKEND=python DOPPLERGEIST_ACOUSTIC_BACKEND=python DOPPLERGEIST_CPA_BACKEND=python DOPPLERGEIST_REAL_CB996DC_JOB=/path/to/real-stopped-job PYTHONPATH=src python -m pytest -q tests/test_real_cb996dc_checkpoint.py",
++ "result": "10 passed",
++ "source_checkpoints_unchanged": true
++ },
++ "limitations": [
++ "Only this pinned Python consumer family is admitted.",
++ "The geometry changes proposed in #293 require a new compatibility proof and invalidate this target fingerprint.",
++ "The shared-observation changes for #275 invalidate the matching.py/search.py consumer pin and require a new compatibility proof.",
++ "The full-day comparison covers one documented association-anchor partition at a source-informed receiver, not a complete search.",
++ "The historical regional data omit known jets and do not establish complete coverage.",
++ "The replay used Python 3.12.14 and the recorded package versions. Continuation also requires exact producer/consumer runtime identity.",
++ "An earlier full-day two-point attempt exceeded the candidate budget on both revisions; its partial artifacts are preserved.",
++ "Elapsed times on a shared machine are not a performance result."
++ ]
++}
diff --cc src/dopplergeist/checkpoint_resume.py
index 69bb4ec,4219d4b..511d6ae
--- a/src/dopplergeist/checkpoint_resume.py
+++ b/src/dopplergeist/checkpoint_resume.py
@@@ -24,35 -25,43 +25,69 @@@ LEGACY_SOURCES =
"finite_search.py": "78ed9fca6cb13c9754ea0f4438f1bb54bbcdc488ead2a4d77a15f149b6d77b1c",
}
-# Complete code-map identity for the real combined scorer/scheduler used by
-# the 7ae saved continuation. Unlike the older legacy scorer above, this code
-# already used the batched scoring arithmetic now on main.
+
+# The already upgraded reference search has been replayed exactly against this
+# Python scoring family. Keep this transition bound to its complete input hash.
+BATCHED_REFERENCE_RUN = "9890647b03bc1f966f3b039727e7c0f00ccef0b4250b7312d5b423c191ca3d3f"
+ORIGINAL_REFERENCE_RUN = "d1ef045417a64a792b81c7355e508bb204046a70b614fb5815aba14b74a75456"
+THROUGHPUT_SOURCES = {
+ "matching.py": "8c2ba9b32ce035f07a8357bf0da05f63a5ac693baad5f6dc52930a82f94439c9",
+ "search.py": "be84feaef43fa356e3e77e0132e1e427b677216f4c3e84166e566a1d9787c99e",
+ "batch_ridge.py": "314d4837fadd80f94410d21c8d5115fa65aff0515e3de2cf4bb08fefd427680a",
+ "geometry.py": "f30f6aa755387f7c28f4bb49d0750e7afc42551f3e5af34f9d8be87c5f0cdc26",
+ "tiled_search.py": "8b54a363611863e896cb86b388daa0fd5722161eac7b19ea8dbe1937b10c45ea",
+ "acoustic_screen.py": "ce2b4e86fed8dfba92ae4742f8cd273f80711976df6ca0a399f1f233d6a6ff32",
+ "finite_search.py": "f1bae3b5a947a3d3234df3b3b1bc8edecca5cf7682ed561d45ad38fe34681fe6",
+ "parallel_tiles.py": "6b5ff3def69b2433c00f9c4afbd1c52ccc0a79e0f623bdf0dff3f61c0d7ea16a",
+ "data_access.py": "1b599aadb8cafa6be0c6aeec959e0c450c1b5c515c24fad6e33cb97719bce6c1",
+ "tracks.py": "543f483cee756fdb9efdd69c8e8f05b32e290f19187af3523160f19d2eba90e0",
+ "native_acoustic.py": "a048cb3bd2d308c91995c97e587f181fbf6d74d6dd9598ab3ab3fd1cc4a50796",
+}
+
++# cb996dc's combined scheduler/scorer uses ordinary batched arithmetic. Its
++# real broad-clock tile and a regional tile were replayed against this Python
++# consumer, including the complete ranked payload and retention records/bounds.
+ CB996DC_CODE_FINGERPRINT = "c66d402b750381303bd0c7d9c828d2294942227176ccb3cb157f0ef6299cbe40"
++CB996DC_TARGET_FINGERPRINT = "2e6a9601eaf42be7ca2b440eef55eeafa894ab21f9f90adcbb216c9f08a16d2d"
++CB996DC_TARGET_FILES = (
++ "__init__.py", "acoustic_screen.py", "audio.py", "automatic.py", "batch_ridge.py",
++ "confidence.py", "contracts.py", "data_access.py", "discovery.py", "finite_search.py",
++ "flight_payload.py", "geometry.py", "index_compatibility.py", "intake.py", "jobs.py",
++ "matching.py", "native_acoustic.py", "native_cpa.py", "parallel_retention.py",
++ "parallel_tiles.py", "provenance.py", "resource_usage.py", "scoring_inputs.py",
++ "search.py", "spectral_grid.py", "spectral_matching.py", "terrain.py", "tiled_search.py",
++ "tracks.py", "trial_prefix.py",
++)
+
+
-def _legacy_source_profile(configuration):
- code = configuration.get("code", {})
- if all(code.get(name) == digest for name, digest in LEGACY_SOURCES.items()):
- return "legacy-first-batch"
- if content_hash(code) == CB996DC_CODE_FINGERPRINT:
- return "combined-cb996dc"
- return None
++def _consumer_fingerprint(code):
++ # Pin orchestration, input interpretation, numerical scoring, retention and
++ # their Python dependencies. CLI/API/UI-only changes do not change this
++ # proof. The adapter cannot hash itself; native execution is gated below.
++ return content_hash({name: code.get(name) for name in CB996DC_TARGET_FILES})
+
+
+ def _combined_min_events(configuration):
- """Preserve cb996dc's broad-clock default when current main resumes it."""
++ """Preserve the producer's default instead of adopting main's singleton default."""
+ filters = configuration.get("filters", {})
+ if "min_events" in filters:
+ return filters["min_events"]
+ events = configuration["events"]
- max_events = filters.get("max_events") or len(events)
- start_window = configuration["scoring"].get("recording_start_window_s")
- clock_window_width = math.inf if start_window is None else start_window[1] - start_window[0]
- default = min(2, max_events) if len(events) > 1 and clock_window_width > 10.0 else 1
- return default
++ maximum = filters.get("max_events") or len(events)
++ window = configuration["scoring"].get("recording_start_window_s")
++ width = math.inf if window is None else window[1] - window[0]
++ return min(2, maximum) if len(events) > 1 and width > 10.0 else 1
+
def upgrade_legacy_job(directory):
- """Prepare a compatible replacement search before swapping a stopped job.
+ """Prepare a replacement search before swapping it into a stopped job.
Completed payloads and retention records stay unchanged. The old directory
- and audio/operator identities remain available for rollback. New work uses
- staged batching with the saved first-group arithmetic and fallback checks.
- and audio/operator identities remain available for rollback. The old
- first-batch arithmetic is retained only for the matching legacy source.
++ and audio/operator identities remain available for rollback. Each profile
++ preserves its scoring arithmetic and requires its validated consumer.
"""
from .automatic import _audio_identity, _digest
- from . import geometry, matching, native_cpa
- from . import geometry, matching
++ from . import data_access, geometry, matching, native_cpa
directory = Path(directory)
search = directory / "search"
@@@ -60,36 -69,18 +95,44 @@@
return None
manifest = json.loads((search / "manifest.json").read_text())
old = manifest["configuration"]
- source_profile = _legacy_source_profile(old)
- if source_profile is None:
+ if content_hash(old) != manifest.get("run_hash"):
return None
- legacy_first_batch = source_profile == "legacy-first-batch"
++ combined = content_hash(old.get("code", {})) == CB996DC_CODE_FINGERPRINT
+ previous_upgrade = None
- if any(old["code"].get(name) != digest for name, digest in LEGACY_SOURCES.items()):
++ if not combined and any(old["code"].get(name) != digest for name, digest in LEGACY_SOURCES.items()):
+ report_path = directory / "checkpoint-upgrade.json"
+ if manifest["run_hash"] != BATCHED_REFERENCE_RUN or not report_path.exists():
+ return None
+ previous_upgrade = json.loads(report_path.read_text())
+ if (previous_upgrade.get("version") != 1
+ or previous_upgrade.get("old_run_hash") != ORIGINAL_REFERENCE_RUN
+ or previous_upgrade.get("new_run_hash") != BATCHED_REFERENCE_RUN
+ or previous_upgrade.get("legacy_first_batch") is not True
+ or old["scoring"].get("legacy_first_batch") is not True):
+ return None
if getattr(matching, "_native_fit_irls", None) is not None:
- raise ValueError("Checkpoint compatibility requires the Python ridge fitter")
+ raise ValueError("Legacy checkpoint continuation requires the Python ridge fitter")
if geometry.ACOUSTIC_BACKEND != "python-acoustic-v1":
- raise ValueError("Checkpoint compatibility requires the Python acoustic solver")
+ raise ValueError("Legacy checkpoint continuation requires the Python acoustic solver")
+ if native_cpa.CPA_BACKEND != "python-cpa-v1":
+ raise ValueError("Legacy checkpoint continuation requires Python closest-approach processing")
++ if combined and getattr(data_access, "_native_payload_columns", None) is not None:
++ raise ValueError("The cb996dc continuation requires the verified Python flight decoder")
+ from .provenance import implementation_files
+ package_root = Path(__file__).parent
+ code = {p.relative_to(package_root).as_posix(): _digest(p)
+ for p in implementation_files(package_root, recursive=False)}
++ if combined and _consumer_fingerprint(code) != CB996DC_TARGET_FINGERPRINT:
++ return None
+ if previous_upgrade and any(code.get(name) != digest for name, digest in THROUGHPUT_SOURCES.items()):
+ return None
if (old["scoring"].get("mode") != "robust_ridge" or old.get("coarse_floor_policy") != "per_tile"
or old["scoring"].get("time_origin") is not None
or old.get("filters", {}).get("association") != "subsets"):
raise ValueError("This legacy checkpoint mode has no compatible upgrade")
++ if combined and (old["scoring"].get("legacy_first_batch")
++ or any(search.glob("fine-tile-*.json"))):
++ raise ValueError("The cb996dc producer has no compatible legacy-batch or fine-checkpoint mode")
runtime = {"python": platform.python_version(), "machine": platform.machine(),
"packages": {name: version(name) for name in old["runtime"]["packages"]}}
if runtime != old["runtime"]:
@@@ -111,8 -102,14 +154,11 @@@
if events != old["events"]:
raise ValueError("Prepared events no longer match the saved search")
current = deepcopy(old)
- if source_profile == "combined-cb996dc":
+ current["code"] = code
- current["scoring"]["legacy_first_batch"] = True
++ if combined:
+ current.setdefault("filters", {})["min_events"] = _combined_min_events(old)
- from .provenance import implementation_files
- package_root = Path(__file__).parent
- current["code"] = {p.relative_to(package_root).as_posix(): _digest(p)
- for p in implementation_files(package_root, recursive=False)}
- if legacy_first_batch:
++ else:
+ current["scoring"]["legacy_first_batch"] = True
new_hash = content_hash(current)
suffix = uuid.uuid4().hex[:12]
staged = directory / f".search-upgrade-{suffix}"
@@@ -141,21 -138,22 +187,29 @@@
# each payload and retention stage before reuse.
tiles = len(list(staged.glob("tile-*.json")))
write_json(staged / "manifest.json", {"run_hash": new_hash, "configuration": current})
- report = {"version": 2 if previous_upgrade else 1,
- report = {"version": 1, "created_utc": datetime.now(timezone.utc).isoformat(),
++ report = {"version": 3 if combined else (2 if previous_upgrade else 1),
+ "created_utc": datetime.now(timezone.utc).isoformat(),
"old_run_hash": manifest["run_hash"], "new_run_hash": new_hash,
- "tiles_preserved": tiles, "legacy_first_batch": True,
- "tiles_preserved": tiles, "legacy_first_batch": legacy_first_batch,
- "compatible_source": source_profile,
++ "tiles_preserved": tiles, "legacy_first_batch": not combined,
"previous_search": str(previous), "identity_backup": str(backup)}
+ if previous_upgrade:
+ report["compatible_run_hashes"] = sorted([ORIGINAL_REFERENCE_RUN, BATCHED_REFERENCE_RUN])
+ report["upgrade_kind"] = "exact_python_throughput_v1"
+ report["previous_upgrade_report"] = str(backup / "checkpoint-upgrade.json")
++ if combined:
++ report.update(upgrade_kind="exact_python_cb996dc_v1",
++ source_code_fingerprint=CB996DC_CODE_FINGERPRINT,
++ target_code_fingerprint=CB996DC_TARGET_FINGERPRINT)
write_json(backup / "upgrade.json", report)
search.rename(previous)
try:
staged.rename(search)
audio["identity"] = identity
- operator["resume_legacy_first_batch"] = True
- if legacy_first_batch:
- operator["resume_legacy_first_batch"] = True
- else:
++ if combined:
+ operator.pop("resume_legacy_first_batch", None)
- if source_profile == "combined-cb996dc":
- operator["filters"] = deepcopy(current["filters"])
++ operator["filters"] = deepcopy(current["filters"])
++ else:
++ operator["resume_legacy_first_batch"] = True
write_json(directory / "resume.json", audio)
write_json(directory / "operator.json", operator)
write_json(directory / "checkpoint-upgrade.json", report)
@@@ -181,11 -175,8 +235,21 @@@ def compatible_checkpoint_runs(output,
path = Path(output).parent / "checkpoint-upgrade.json"
if path.exists():
report = json.loads(path.read_text())
- if (report.get("version") == 1 and report.get("new_run_hash") == run_hash
- and (report.get("legacy_first_batch") is True
- or report.get("compatible_source") == "combined-cb996dc")):
- accepted.add(report["old_run_hash"])
++ if (report.get("version") == 3 and report.get("new_run_hash") == run_hash
++ and report.get("legacy_first_batch") is False
++ and report.get("upgrade_kind") == "exact_python_cb996dc_v1"
++ and report.get("source_code_fingerprint") == CB996DC_CODE_FINGERPRINT
++ and report.get("target_code_fingerprint") == CB996DC_TARGET_FINGERPRINT):
++ manifest = json.loads((Path(output) / "manifest.json").read_text())
++ configuration = manifest.get("configuration", {})
++ if (manifest.get("run_hash") == run_hash and content_hash(configuration) == run_hash
++ and _consumer_fingerprint(configuration.get("code", {})) == CB996DC_TARGET_FINGERPRINT):
++ accepted.add(report["old_run_hash"])
+ if report.get("new_run_hash") == run_hash and report.get("legacy_first_batch") is True:
+ if report.get("version") == 1:
+ accepted.add(report["old_run_hash"])
+ elif (report.get("version") == 2 and report.get("upgrade_kind") == "exact_python_throughput_v1"
+ and report.get("old_run_hash") == BATCHED_REFERENCE_RUN
+ and report.get("compatible_run_hashes") == sorted([ORIGINAL_REFERENCE_RUN, BATCHED_REFERENCE_RUN])):
+ accepted.update([ORIGINAL_REFERENCE_RUN, BATCHED_REFERENCE_RUN])
return accepted
diff --cc tests/test_real_cb996dc_checkpoint.py
index 0000000,0000000..3f37441
new file mode 100644
--- /dev/null
+++ b/tests/test_real_cb996dc_checkpoint.py
@@@ -1,0 -1,0 +1,157 @@@
++"""Continue a real cb996dc job; never generate audio, tracks or scored modes."""
++import hashlib
++import json
++import os
++from pathlib import Path
++import shutil
++import sqlite3
++
++import pytest
++
++from dopplergeist import checkpoint_resume, data_access, native_cpa
++from dopplergeist.automatic import _load_audio_checkpoint, _search
++from dopplergeist.data_access import content_hash
++from dopplergeist.finite_search import RetentionLedger
++
++
[email protected](scope="module")
++def real_job():
++ location = os.environ.get("DOPPLERGEIST_REAL_CB996DC_JOB")
++ if not location:
++ pytest.skip("Supply an actual stopped cb996dc job with nonempty coarse checkpoints")
++ source = Path(location)
++ manifest = json.loads((source / "search/manifest.json").read_text())
++ assert content_hash(manifest["configuration"]) == manifest["run_hash"]
++ assert content_hash(manifest["configuration"]["code"]) == checkpoint_resume.CB996DC_CODE_FINGERPRINT
++ assert json.loads((source / "result.json").read_text())["status"] in {"cancelled", "failed", "inconclusive"}
++ tiles = [path for path in sorted((source / "search").glob("tile-*.json"))
++ if json.loads(path.read_text())["payload"].get("ranked")]
++ assert tiles, "The recorded job must include actual scored candidates"
++ return source, manifest, tiles
++
++
++def _copy_job(source, destination):
++ # request.json continues to identify the unchanged original input.mp3.
++ # Copy metadata/ledgers independently so fault injection cannot alter source.
++ shutil.copytree(source, destination, ignore=shutil.ignore_patterns("input.mp3"))
++ return destination
++
++
++def _digest(path):
++ with path.open("rb") as stream:
++ return hashlib.file_digest(stream, "sha256").hexdigest()
++
++
++def test_real_nonempty_tiles_resume_after_pinned_upgrade(real_job, tmp_path):
++ source, original, tiles = real_job
++ original_files = {path: _digest(path) for path in [source / "search/manifest.json",
++ source / "search/retention.sqlite", source / "resume.json", source / "operator.json", *tiles]}
++ job = _copy_job(source, tmp_path / "job")
++ saved = {path.name: path.read_bytes() for path in tiles}
++ expected_ranked = json.loads((source / "search/search-report.json").read_text())["ranked"]
++ report = checkpoint_resume.upgrade_legacy_job(job)
++ assert report["version"] == 3
++ assert report["upgrade_kind"] == "exact_python_cb996dc_v1"
++ assert report["legacy_first_batch"] is False
++ assert report["old_run_hash"] == original["run_hash"]
++ assert report["target_code_fingerprint"] == checkpoint_resume.CB996DC_TARGET_FINGERPRINT
++ manifest = json.loads((job / "search/manifest.json").read_text())
++ operator = json.loads((job / "operator.json").read_text())
++ assert operator["filters"] == manifest["configuration"]["filters"]
++ assert operator["filters"]["min_events"] == checkpoint_resume._combined_min_events(original["configuration"])
++ assert not operator.get("resume_legacy_first_batch")
++ assert checkpoint_resume.compatible_checkpoint_runs(job / "search", report["new_run_hash"]) == {
++ report["new_run_hash"], report["old_run_hash"]}
++ assert checkpoint_resume.compatible_checkpoint_runs(job / "search", report["old_run_hash"]) == {report["old_run_hash"]}
++ for name, data in saved.items():
++ assert (job / "search" / name).read_bytes() == data
++ assert (Path(report["previous_search"]) / name).read_bytes() == data
++ assert checkpoint_resume.upgrade_legacy_job(job) is None
++ request = json.loads((job / "request.json").read_text())
++ recording, events = _load_audio_checkpoint(job, request, operator)
++ result, *_ = _search(request, recording, events, job, operator, lambda *args, **kwargs: None)
++ assert result["run_hash"] == report["new_run_hash"]
++ assert result["tiles_resumed"] == result["tiles_completed"] > 0
++ assert result["tile_failures"] == []
++ assert result["ranked"] == expected_ranked
++ ledger = RetentionLedger(job / "search/retention.sqlite", report["new_run_hash"],
++ "matched_group_support_minus_normalized_residual")
++ for name, data in saved.items():
++ assert (job / "search" / name).read_bytes() == data
++ payload = json.loads(data)["payload"]
++ stage = "coarse:" + Path(name).stem.removeprefix("tile-")
++ assert ledger.stage_valid(stage, payload["retention_checksum"])
++ assert {path: _digest(path) for path in original_files} == original_files
++
++
[email protected]("change", ["producer", "consumer", "manifest", "runtime", "native_cpa", "native_index", "active"])
++def test_unverified_continuations_leave_real_checkpoint_unchanged(real_job, tmp_path, monkeypatch, change):
++ source, original, _ = real_job
++ job = tmp_path / "job"
++ (job / "search").mkdir(parents=True)
++ shutil.copy2(source / "search/manifest.json", job / "search/manifest.json")
++ shutil.copy2(source / "result.json", job / "result.json")
++ manifest = json.loads((job / "search/manifest.json").read_text())
++ if change == "producer":
++ manifest["configuration"]["code"]["matching.py"] = "0" * 64
++ elif change == "consumer":
++ # A later geometry change must invalidate the measured consumer even
++ # when its checkpoint format and all other source files stay unchanged.
++ from dopplergeist import automatic
++ digest = automatic._digest
++ monkeypatch.setattr(automatic, "_digest",
++ lambda path: "0" * 64 if Path(path).name == "geometry.py" else digest(path))
++ elif change == "manifest":
++ manifest["run_hash"] = "0" * 64
++ elif change == "runtime":
++ manifest["configuration"]["runtime"]["python"] += ".changed"
++ elif change == "native_cpa":
++ monkeypatch.setattr(native_cpa, "CPA_BACKEND", "native-c-cpa-v1")
++ elif change == "native_index":
++ monkeypatch.setattr(data_access, "_native_payload_columns", object())
++ elif change == "active":
++ state = json.loads((job / "result.json").read_text())
++ state["status"] = "running"
++ (job / "result.json").write_text(json.dumps(state))
++ if change in {"producer", "runtime"}:
++ manifest["run_hash"] = content_hash(manifest["configuration"])
++ (job / "search/manifest.json").write_text(json.dumps(manifest))
++ before = (job / "search/manifest.json").read_bytes()
++ if change in {"producer", "consumer", "manifest"}:
++ assert checkpoint_resume.upgrade_legacy_job(job) is None
++ else:
++ with pytest.raises(ValueError, match={"runtime": "runtime changed", "native_cpa": "Python closest-approach", "native_index": "Python flight decoder", "active": "Stop the previous job"}[change]):
++ checkpoint_resume.upgrade_legacy_job(job)
++ assert (job / "search/manifest.json").read_bytes() == before
++ assert not list(job.glob("checkpoint-upgrade*"))
++
++
++def test_real_upgrade_rolls_back_operator_and_producer_binding(real_job, tmp_path, monkeypatch):
++ source, manifest, tiles = real_job
++ job = _copy_job(source, tmp_path / "job")
++ names = ("search/manifest.json", "resume.json", "operator.json")
++ before = {name: (job / name).read_bytes() for name in names}
++ write_json = checkpoint_resume.write_json
++
++ def fail_operator(path, value):
++ if Path(path) == job / "operator.json":
++ raise OSError("interrupted real checkpoint upgrade")
++ return write_json(path, value)
++
++ monkeypatch.setattr(checkpoint_resume, "write_json", fail_operator)
++ with pytest.raises(OSError, match="interrupted real checkpoint"):
++ checkpoint_resume.upgrade_legacy_job(job)
++ assert {name: (job / name).read_bytes() for name in names} == before
++ assert not (job / "checkpoint-upgrade.json").exists()
++ with sqlite3.connect((job / "search/retention.sqlite").as_uri() + "?mode=ro", uri=True) as db:
++ assert db.execute("SELECT identity FROM run").fetchone() == (manifest["run_hash"],)
++ assert (job / "search" / tiles[0].name).read_bytes() == tiles[0].read_bytes()
++
++
++def test_cb996dc_report_cannot_authorize_an_unverified_target(real_job, tmp_path):
++ source, _, _ = real_job
++ job = _copy_job(source, tmp_path / "job")
++ report = checkpoint_resume.upgrade_legacy_job(job)
++ report["target_code_fingerprint"] = "0" * 64
++ (job / "checkpoint-upgrade.json").write_text(json.dumps(report))
++ assert checkpoint_resume.compatible_checkpoint_runs(job / "search", report["new_run_hash"]) == {report["new_run_hash"]}