Python tool for aircraft Doppler audio geolocation using historical ADS-B flight data (Source-available)
Python acoustic-geolocationads-baircraft-trackingairplaneaudio-forensicsdoppler-effectdoppler-shiftgeolocationosintsignal-processing
Report saved coarse-search progress without replaying work
Cinna Daviseacf24bcfabe
diff --git a/docs/CHECKPOINT_PROGRESS.md b/docs/CHECKPOINT_PROGRESS.mdnew file mode 100644index 0000000..3c3986b--- /dev/null+++ b/docs/CHECKPOINT_PROGRESS.md@@ -0,0 +1,134 @@+# Read saved checkpoint progress++`scripts/report_checkpoint_progress.py` reports recorded coarse search work.+It does not resume the search or change its scoring, inputs, or checkpoints.+It is a bounded increment for [issue #265](https://github.com/1337isnot1337/DOPPLERGEIST/issues/265).+The original a904 report and measured stage CPU acceptance remain outstanding.++Run this from an installed checkout. Pass a saved job directory or its `search`+directory. Store the reports and optional cache outside that job:++```bash+python scripts/report_checkpoint_progress.py /path/to/job/search \+ --json /path/to/reports/progress.json \+ --markdown /path/to/reports/progress.md \+ --cache /path/to/reports/progress-cache.json+```++The same command refreshes the report. Unchanged receipt summaries and grid+counts come from the cache. The script refreshes canonical stage metadata on+every invocation. Without output arguments, it prints JSON to standard output.++## What the report counts++The report reconstructs the saved spacing, CRS, geographic bounds, root tile+size, `max_tiles`, and spatial split tree. Requested roots and the scheduled+prefix have separate denominators. Child tiles do not inflate the root count.++Receiver counts use the producer's integer projected grid and geographic bounds.+They do not use the checkpoint's `grid_points` counter. Empty-flight tiles can+record zero receiver-point visits after processing a nonempty receiver region.+Recorded points can also reach the CPA screen without receiving an acoustic fit.++Clock siblings share the same receiver cell. The cell counts once, after all+required clock descendants have receipts. An incomplete clock partition remains+outstanding. The script checks the recorded split geometry and child names; it+does not rerun clock interval or scoring calculations.++The output separates two accounting levels:++| Field | Required evidence | Remaining limit |+| --- | --- | --- |+| `receipt_recorded` | Matching manifest producer, payload hash, cell rectangle, and required split descendants | A receipt may precede its ledger commit. |+| `canonical_stage_agreed` | The same receipt plus a matching committed canonical stage checksum | Record and bound contents are not rehashed. |++Neither level certifies resumability, scientific completeness, data coverage, or+localization accuracy. Unsupported trajectories and unresolved trials remain+explicit counters. Terrain and source files are not rehashed. A failed receipt,+unknown producer, missing child, unreadable file, or read limit leaves work+unresolved. Canonical stages without reconciled receipts are counted separately.+Fine-grid files have a separate inventory and do not add to coarse coverage.++Each saved counter keeps its own unit. Track counts are recorded segment visits+after query filtering, not raw query counts or unique aircraft. CPA rows,+associations, clock trials, and retained mode+appearances are separate. Missing or invalid fields stay unknown; they are not+zero. There are no inferred pass rates across different units.++Historical checkpoints do not supply stage CPU/wall percentiles, stage+throughput, unique aircraft/sample counts, complete nonlinear fit counts, or+global retained-mode counts. Those output fields remain `null`.++## Read limits and active jobs++Each refresh reads at most 128 MiB of new JSON by default. A single JSON input+must be at most 64 MiB. Split metadata has a 64 KiB limit; manifest/workload input+has an 8 MiB limit. Change the receipt limits explicitly when necessary:++```bash+python scripts/report_checkpoint_progress.py /path/to/job/search \+ --max-read-mib 256 --max-file-mib 96 \+ --cache /path/to/reports/progress-cache.json \+ --json /path/to/reports/progress.json+```++Small receipts are read first. A limited refresh is a biased partial inventory.+Later refreshes can read additional changed files because cached summaries do+not consume the JSON read budget. Oversized files remain unresolved until the+individual limit permits them. JSON decoding can use more memory than file size.++The external cache keys summaries by device, inode, size, modification time,+change time, and reader source hash. It is a local performance cache, not an+independent content-integrity proof. The script checks input stats before and+after reading and reports whether the receipt inventory changed during refresh.++SQLite opens with `mode=ro` and `query_only`, in one read transaction. The script+reads `run` and the indexed coarse range of `stages`, capped at 100,000 rows.+It never scans `records` or `bounds`, hashes the database, checkpoints the WAL,+or opens worker ledgers. Standard SQLite WAL shared-memory bookkeeping can still+occur; no application rows are written. The database header identifies WAL mode+even if its sidecars are absent. A WAL-mode database needs existing readable+WAL and SHM files of at least 32 bytes and 32 KiB, respectively. Otherwise,+canonical agreement stays unknown without opening a SQLite connection.+Pre-open checks cannot atomically prevent sidecar races during an active+producer's shutdown. Normal SQLite shared-memory reader bookkeeping is allowed.+Do not copy an active database file alone and treat it as a consistent snapshot.+See SQLite's [read-only WAL rules](https://www.sqlite.org/wal.html#read_only_databases)+and [WAL file format](https://www.sqlite.org/walformat.html).++The SQLite transaction cannot atomically include separate JSON files. A report+from an active producer is an observation across those files and that database+snapshot. Changed inputs are marked provisional. Empty-tile receipts can exist+before their transaction commits; they do not receive canonical agreement until+the committed stage metadata is visible.++The script accepts only the exact manifest run identity. It lists other payload+producer identities but does not approve them through an upgrade receipt alone.+That compatibility review remains separate. Geometry reconstruction reports the+current NumPy, pyproj, and PROJ versions; it does not reproduce a historic runtime.+Known saved/current NumPy or pyproj version differences are flagged explicitly.++Receiver reconstruction is capped at 20 million projected grid points and+100,000 roots. `--max-grid-points` changes the first limit. A missing pyproj+installation, `--skip-grid`, or a saved receiver mask leaves receiver counts+unknown. The script does not approximate a country mask with its bounding box.++## Existing recorded workload check++The existing fixture contains 256 completed empty coarse checkpoints from the+stopped original automatic baseline. Its source and recording hashes are saved+inside the fixture. It is not a904, a completed national run, or an accuracy test.++```bash+python scripts/report_checkpoint_progress.py \+ tests/data/real-empty-tile-workload.json \+ --json /path/to/reports/recorded-empty-progress.json \+ --markdown /path/to/reports/recorded-empty-progress.md \+ --cache /path/to/reports/recorded-empty-cache.json+```++The script verifies each recorded payload hash. The recorded point and track+counters are zero, while reconstructed receiver coverage is nonzero. CPA and+trial counters are absent and stay unknown. The portable fixture has no canonical+ledger, so canonical agreement remains unknown. Repeating the command exercises+the incremental cache without generating recordings, tracks, or fixture data.diff --git a/scripts/report_checkpoint_progress.py b/scripts/report_checkpoint_progress.pynew file mode 100644index 0000000..231c52b--- /dev/null+++ b/scripts/report_checkpoint_progress.py@@ -0,0 +1,569 @@+#!/usr/bin/env python3+"""Read-only, bounded accounting of saved coarse search checkpoints."""+from __future__ import annotations++import argparse+from collections import Counter+from datetime import datetime, timezone+import hashlib+import json+import math+import os+from pathlib import Path+import re+import sqlite3+import sys+import tempfile+import time++MIB = 1024 * 1024+TILE_ID = r"-?\d+_-?\d+(?:~(?:[0-3]|t[01]))*"+RECEIPT_NAME = re.compile(rf"tile-({TILE_ID})\.json\Z")+SPLIT_NAME = re.compile(rf"split-({TILE_ID})\.json\Z")+METRICS = {+ "grid_points": "recorded receiver-point visits, not geographic coverage",+ "tracks": "recorded track-segment visits, not unique aircraft",+ "cpa_rows": "CPA rows",+ "candidates": "association candidates",+ "hypotheses_examined": "association hypotheses examined",+ "work_units": "association work units",+ "unclassified_score_candidates": "unclassified candidate assessments",+ "bounded_score_candidates": "candidate assessments with a score bound",+ "unbounded_score_candidates": "candidate assessments without a score bound",+ "evaluated_rejections": "evaluated candidate rejections",+ "evaluated_trials": "clock trial visits evaluated",+ "rejected_trials": "rejected clock-trial visits",+ "unknown_trials": "unresolved clock trial visits",+ "unreported_trial_envelopes": "candidate assessments without trial accounting",+ "unsupported_cpa_tracks": "unsupported track evaluations",+ "ranked": "locally retained mode appearances, not globally unique modes",+}+++def content_hash(value):+ # Exact data_access.content_hash convention, including JSON whitespace.+ return hashlib.sha256(json.dumps(value, sort_keys=True, allow_nan=False).encode()).hexdigest()+++def nonnegative_int(value):+ return type(value) is int and value >= 0+++def cells_tuple(value):+ if (not isinstance(value, (list, tuple)) or len(value) != 4+ or not all(type(n) is int for n in value)+ or value[0] >= value[2] or value[1] >= value[3]):+ raise ValueError("invalid half-open cell rectangle")+ return tuple(value)+++def fingerprint(path):+ if path.is_symlink():+ raise ValueError("symlinked input files are not read")+ st = path.stat()+ if not path.is_file():+ raise ValueError("input is not a regular file")+ return [st.st_dev, st.st_ino, st.st_size, st.st_mtime_ns, st.st_ctime_ns]+++def strict_json(raw):+ def invalid_constant(value):+ raise ValueError(f"non-finite JSON number: {value}")+ return json.loads(raw, parse_constant=invalid_constant)+++def receipt_summary(record):+ payload = record["payload"]+ if not isinstance(record.get("run_hash"), str) or re.fullmatch(r"[0-9a-f]{64}", record["run_hash"]) is None:+ raise ValueError("invalid producer identity")+ if record.get("payload_hash") != content_hash(payload):+ raise ValueError("payload hash mismatch")+ cells = cells_tuple(payload["cells"])+ if (not isinstance(payload.get("counts"), dict) or not isinstance(payload.get("terrain_failures"), list)+ or not isinstance(payload.get("ranked"), list)):+ raise ValueError("unrecognized checkpoint schema")+ metrics = {}+ for name in METRICS:+ value = (payload if name in {"grid_points", "tracks", "ranked"} else payload["counts"]).get(name)+ if name in {"ranked", "unsupported_cpa_tracks"}:+ value = len(value) if isinstance(value, list) else None+ metrics[name] = value if nonnegative_int(value) else None+ return {"run_hash": record.get("run_hash"), "cells": list(cells),+ "retention_checksum": payload.get("retention_checksum"),+ "terrain_failures": len(payload["terrain_failures"]), "metrics": metrics}+++class Reader:+ """Reuse reduced summaries while file identity and timestamps match."""++ def __init__(self, remaining, file_limit, entries):+ self.remaining, self.file_limit, self.entries = remaining, file_limit, entries+ self.stats = Counter(bytes_read=0, files_read=0, files_reused=0)++ def read(self, path, reducer=lambda value: value, limit=None):+ key = str(path)+ try:+ before = fingerprint(path)+ prior = self.entries.get(key, {})+ if prior.get("fingerprint") == before and "result" in prior:+ self.stats["files_reused"] += 1+ return prior["result"]+ maximum = min(self.file_limit, limit if limit is not None else self.file_limit)+ if before[2] > maximum:+ return {"status": "file_size_limit"}+ if before[2] > self.remaining:+ return {"status": "refresh_byte_budget"}+ with path.open("rb") as handle:+ raw = handle.read(before[2])+ self.remaining -= len(raw)+ self.stats["bytes_read"] += len(raw)+ self.stats["files_read"] += 1+ if len(raw) != before[2] or fingerprint(path) != before:+ return {"status": "changed_during_read"}+ try:+ result = {"status": "ok", "value": reducer(strict_json(raw))}+ except (KeyError, TypeError, ValueError, OverflowError, RecursionError) as error:+ result = {"status": "invalid_content", "reason": str(error)}+ self.entries[key] = {"fingerprint": before, "result": result}+ return result+ except (OSError, ValueError) as error:+ return {"status": "unreadable", "reason": str(error)}+++def inventory(directory):+ result = {}+ for path in directory.iterdir():+ if path.name.startswith(("tile-", "split-", "fine-tile-")) and path.name.endswith(".json"):+ try:+ result[path.name] = fingerprint(path)+ except (OSError, ValueError) as error:+ result[path.name] = str(error)+ return result+++def ledger_snapshot(path, run_hash):+ """Read only the indexed stages metadata; never records or bounds."""+ result = {"status": "missing", "stages": {}, "record_and_bound_content_rehashed": False}+ if not path.exists():+ return result+ try:+ fingerprint(path)+ with path.open("rb") as handle:+ header = handle.read(100)+ if len(header) != 100 or header[:16] != b"SQLite format 3\x00":+ raise ValueError("invalid SQLite database header")+ if header[18:20] not in (b"\x01\x01", b"\x02\x02"):+ raise ValueError("unsupported SQLite journal format")+ wal, shm = Path(str(path) + "-wal"), Path(str(path) + "-shm")+ result["journal_mode_observed"] = "wal" if header[18] == 2 else "rollback"+ if header[18] == 2:+ # mode=ro can still create sidecars for a writable directory. Inspect+ # the database header even when its WAL is missing or empty.+ for sidecar, minimum in ((wal, 32), (shm, 32768)):+ if not sidecar.exists() or fingerprint(sidecar)[2] < minimum:+ raise ValueError("WAL database needs existing readable nonempty WAL/SHM files; canonical agreement is unavailable")+ with sidecar.open("rb") as handle:+ if len(handle.read(32)) != 32:+ raise ValueError("unreadable WAL/SHM header")+ db = sqlite3.connect(path.resolve().as_uri() + "?mode=ro", uri=True, timeout=1)+ try:+ db.execute("PRAGMA query_only=ON")+ db.execute("PRAGMA cache_size=-2048")+ db.execute("BEGIN")+ identities = db.execute("SELECT identity,score_kind FROM run LIMIT 2").fetchall()+ if len(identities) != 1 or identities[0][0] != run_hash:+ raise ValueError("ledger identity differs from manifest")+ result["score_kind"] = identities[0][1]+ rows = db.execute("SELECT name,checksum FROM stages "+ "WHERE name >= 'coarse:' AND name < 'coarse;' LIMIT 100001").fetchall()+ if len(rows) > 100000:+ raise ValueError("stage metadata exceeds the 100000-row report limit")+ result["stages"] = {name.removeprefix("coarse:"): checksum for name, checksum in rows}+ finally:+ db.close()+ result["status"] = "metadata_snapshot"+ except (OSError, ValueError, sqlite3.Error) as error:+ result.update(status="unavailable", reason=str(error), stages={})+ return result+++class Grid:+ """Reproduce the integer grid with at most one root's coordinate arrays."""++ def __init__(self, config, cache, max_points, skip=False):+ self.roots, self.reason, self.point_reason = None, None, None+ self.cache, self.runtime = cache, {}+ try:+ if skip:+ raise ValueError("grid reconstruction disabled by --skip-grid")+ import numpy as np+ import pyproj+ self.np = np+ self.runtime = {"numpy": np.__version__, "pyproj": pyproj.__version__, "proj": pyproj.proj_version_str}+ self.bounds, self.spacing = config["bounds"], config["budget"]["spacing_m"]+ size = config["budget"]["tile_cells"]+ if (not isinstance(self.bounds, list) or len(self.bounds) != 4+ or not all(type(x) in (int, float) and math.isfinite(x) for x in self.bounds)+ or not -180 <= self.bounds[0] < self.bounds[2] <= 180+ or not -90 <= self.bounds[1] < self.bounds[3] <= 90+ or type(self.spacing) not in (int, float) or not math.isfinite(self.spacing)+ or self.spacing <= 0 or type(size) is not int or not 1 <= size <= 256):+ raise ValueError("invalid saved grid configuration")+ forward = pyproj.Transformer.from_crs("EPSG:4326", config["crs"], always_xy=True)+ bounds = forward.transform_bounds(*self.bounds, densify_pts=101)+ x0, y0 = (math.floor(value / self.spacing) for value in bounds[:2])+ x1, y1 = (math.ceil(value / self.spacing) for value in bounds[2:])+ if (x1 // size - x0 // size + 1) * (y1 // size - y0 // size + 1) > 100000:+ raise ValueError("grid exceeds the 100000-root report limit")+ self.roots = {f"{ix}_{iy}": (max(x0, ix * size), max(y0, iy * size),+ min(x1 + 1, (ix + 1) * size), min(y1 + 1, (iy + 1) * size))+ for iy in range(y0 // size, y1 // size + 1)+ for ix in range(x0 // size, x1 // size + 1)}+ self.inverse = pyproj.Transformer.from_crs(config["crs"], "EPSG:4326", always_xy=True)+ self.key = content_hash([self.bounds, self.spacing, config["crs"], self.runtime, config.get("receiver_mask")])+ if config.get("receiver_mask") is not None:+ self.point_reason = "receiver mask present; its geometry is not loaded or approximated"+ elif (x1 - x0 + 1) * (y1 - y0 + 1) > max_points:+ self.point_reason = "projected grid exceeds --max-grid-points"+ except ImportError as error:+ self.reason = f"grid dependency unavailable: {error}; install the project or use --skip-grid"+ except (KeyError, TypeError, ValueError, OverflowError, RuntimeError) as error:+ self.roots = None+ self.reason = str(error)++ def count(self, cells):+ if self.roots is None or self.point_reason is not None:+ return None+ key = self.key + ":" + ",".join(map(str, cells))+ area = (cells[2] - cells[0]) * (cells[3] - cells[1])+ saved = self.cache.get(key)+ if nonnegative_int(saved) and saved <= area:+ return saved+ np = self.np+ x, y = np.meshgrid(np.arange(cells[0], cells[2], dtype=np.int64), np.arange(cells[1], cells[3], dtype=np.int64))+ lon, lat = self.inverse.transform(x.ravel() * self.spacing, y.ravel() * self.spacing)+ value = int(np.count_nonzero((lon >= self.bounds[0]) & (lon <= self.bounds[2])+ & (lat >= self.bounds[1]) & (lat <= self.bounds[3])))+ self.cache[key] = value+ return value+++def children_for(tile_id, cells):+ x0, y0, x1, y1 = cells+ xs = [x0, (x0 + x1) // 2, x1] if x1 - x0 > 1 else [x0, x1]+ ys = [y0, (y0 + y1) // 2, y1] if y1 - y0 > 1 else [y0, y1]+ if len(xs) == len(ys) == 2:+ if sum(part.startswith("t") for part in tile_id.split("~")[1:]) >= 24:+ return [], True+ return [[f"{tile_id}~t{i}", list(cells)] for i in range(2)], True+ parts = [(xs[i], ys[j], xs[i + 1], ys[j + 1])+ for j in range(len(ys) - 1) for i in range(len(xs) - 1)]+ return [[f"{tile_id}~{i}", list(part)] for i, part in enumerate(parts)], False+++def sum_known(values):+ values = list(values)+ return sum(values) if all(value is not None for value in values) else None+++class Coverage:+ def __init__(self, grid, receipts, splits, run_hash, ledger):+ self.grid, self.receipts, self.splits, self.run_hash = grid, receipts, splits, run_hash+ self.stages = ledger["stages"] if ledger["status"] == "metadata_snapshot" else None+ self.active, self.agreed, self.visited, self.superseded = set(), set(), set(), set()+ self.reasons, self.clock_splits = Counter(), 0++ def empty(self, cells, reason):+ self.reasons[reason] += 1+ zero = 0 if self.grid.count(cells) is not None else None+ return {"receipt_full": False, "ledger_full": False, "receipt_any": False, "ledger_any": False,+ "receipt_points": zero, "ledger_points": zero if self.stages is not None else None}++ def walk(self, tile_id, cells):+ self.visited.add(tile_id)+ if tile_id in self.splits:+ if tile_id in self.receipts:+ self.superseded.add(tile_id)+ saved = self.splits[tile_id]+ if saved["status"] != "ok":+ return self.empty(cells, "split_" + saved["status"])+ children, temporal = children_for(tile_id, cells)+ if not children or saved["value"] != {"run_hash": self.run_hash, "cells": list(cells), "children": children}:+ return self.empty(cells, "invalid_or_foreign_split")+ self.clock_splits += int(temporal)+ pieces = [self.walk(child, tuple(child_cells)) for child, child_cells in children]+ result = {}+ for scope in ("receipt", "ledger"):+ full = all(piece[scope + "_full"] for piece in pieces)+ result[scope + "_full"] = full+ result[scope + "_any"] = any(piece[scope + "_any"] for piece in pieces)+ if temporal:+ points = self.grid.count(cells)+ result[scope + "_points"] = points if full else 0 if points is not None else None+ else:+ result[scope + "_points"] = sum_known(piece[scope + "_points"] for piece in pieces)+ if self.stages is None:+ result["ledger_points"] = None+ return result+ saved = self.receipts.get(tile_id, {"status": "missing_receipt"})+ if saved["status"] != "ok":+ return self.empty(cells, saved["status"])+ value = saved["value"]+ if value["run_hash"] != self.run_hash:+ return self.empty(cells, "foreign_producer_identity")+ if tuple(value["cells"]) != cells:+ return self.empty(cells, "cell_rectangle_mismatch")+ if value["terrain_failures"]:+ return self.empty(cells, "recorded_terrain_failure")+ self.active.add(tile_id)+ matched = (self.stages is not None and tile_id in self.stages+ and isinstance(value["retention_checksum"], str)+ and value["retention_checksum"] == self.stages[tile_id])+ if matched:+ self.agreed.add(tile_id)+ elif self.stages is not None:+ self.reasons["missing_canonical_stage" if tile_id not in self.stages else "canonical_checksum_mismatch"] += 1+ points = self.grid.count(cells)+ return {"receipt_full": True, "ledger_full": matched, "receipt_any": True, "ledger_any": matched,+ "receipt_points": points,+ "ledger_points": (points if matched else 0 if points is not None else None)+ if self.stages is not None else None}++ def report(self, max_tiles):+ if self.grid.roots is None:+ self.active = {name for name, saved in self.receipts.items()+ if name not in self.splits and saved["status"] == "ok"+ and saved["value"]["run_hash"] == self.run_hash and not saved["value"]["terrain_failures"]}+ return {"status": "unavailable", "reason": self.grid.reason}+ roots = list(self.grid.roots.items())+ scheduled = roots[:max_tiles] if max_tiles is not None else roots+ requested_points = sum_known(self.grid.count(cells) for _, cells in roots)+ scheduled_points = sum_known(self.grid.count(cells) for _, cells in scheduled)+ states = [self.walk(name, cells) for name, cells in scheduled]+ result = {"status": "recorded_geometry_only", "requested_root_tiles": len(roots),+ "scheduled_root_tiles": len(scheduled), "max_tiles": max_tiles,+ "requested_receiver_points": requested_points, "scheduled_receiver_points": scheduled_points,+ "receiver_count_limitation": self.grid.point_reason, "clock_split_nodes": self.clock_splits,+ "unresolved_piece_reasons": dict(sorted(self.reasons.items()))}+ for scope, label in (("receipt", "receipt_recorded"), ("ledger", "canonical_stage_agreed")):+ points = sum_known(state[scope + "_points"] for state in states)+ result[label] = {"fully_receipted_roots": sum(state[scope + "_full"] for state in states),+ "partial_roots": sum(state[scope + "_any"] and not state[scope + "_full"] for state in states),+ "roots_without_accepted_receipts": sum(not state[scope + "_any"] for state in states),+ "unique_receiver_points": points,+ "outstanding_scheduled_receiver_points": scheduled_points - points+ if scheduled_points is not None and points is not None else None,+ "percent_of_scheduled_receiver_points": round(100 * points / scheduled_points, 6)+ if points is not None and scheduled_points else None,+ "percent_of_requested_receiver_points": round(100 * points / requested_points, 6)+ if points is not None and requested_points else None}+ if self.stages is None:+ result["canonical_stage_agreed"] = None+ return result+++def funnel(receipts, selected):+ result = []+ for name, unit in METRICS.items():+ values = [receipts[tile]["value"]["metrics"][name] for tile in selected]+ known = [value for value in values if value is not None]+ result.append({"metric": name, "unit": unit, "known_sum": sum(known) if known else None,+ "receipts_with_counter": len(known), "receipts_missing_or_invalid": len(values) - len(known)})+ return {"receipt_count": len(selected), "metrics": result}+++def atomic_write(path, text):+ path.parent.mkdir(parents=True, exist_ok=True)+ name = None+ try:+ with tempfile.NamedTemporaryFile("w", encoding="utf-8", dir=path.parent, delete=False) as handle:+ name = handle.name+ handle.write(text)+ os.replace(name, path)+ finally:+ if name is not None and os.path.exists(name):+ os.unlink(name)+++def markdown(report):+ def display(value):+ return "unknown" if value is None else str(value)+ coverage = report["coverage"]+ lines = ["# Saved checkpoint progress", "", f"Run: `{report['source']['run_hash']}`", "",+ "Recorded coarse work does not certify scientific completeness or a location.", "",+ f"Scheduled roots: {display(coverage.get('scheduled_root_tiles'))}; "+ f"requested roots: {display(coverage.get('requested_root_tiles'))}.", "",+ "| Basis | Fully receipted roots | Partial roots | Unique receiver points | Scheduled percent |",+ "| --- | ---: | ---: | ---: | ---: |"]+ for key in ("receipt_recorded", "canonical_stage_agreed"):+ row = coverage.get(key) or {}+ lines.append("| " + key + " | " + " | ".join(display(row.get(field)) for field in (+ "fully_receipted_roots", "partial_roots", "unique_receiver_points", "percent_of_scheduled_receiver_points")) + " |")+ lines.extend(["", "## Recorded counters", "", f"Scope: {report['funnel_scope']}.", "",+ "Sums are visits within included receipts. Missing or invalid fields remain unknown.", "",+ "| Counter | Unit | Known sum | Present receipts | Missing or invalid receipts |",+ "| --- | --- | ---: | ---: | ---: |"])+ for row in report["receipt_funnel"]["metrics"]:+ lines.append("| " + " | ".join(display(row[key]) for key in (+ "metric", "unit", "known_sum", "receipts_with_counter", "receipts_missing_or_invalid")) + " |")+ lines.extend(["", "## Limits", ""] + ["- " + item for item in report["limitations"]])+ return "\n".join(lines) + "\n"+++def main(argv=None):+ parser = argparse.ArgumentParser(description=__doc__)+ parser.add_argument("source", type=Path, help="saved job/search directory or existing real-empty-tile workload JSON")+ parser.add_argument("--json", dest="json_path", help="JSON output outside the input job (default: stdout)")+ parser.add_argument("--markdown", dest="markdown_path", help="Markdown output outside the input job")+ parser.add_argument("--cache", help="incremental summary cache outside the input job")+ parser.add_argument("--max-read-mib", type=float, default=128, help="new JSON bytes per refresh (default: 128 MiB)")+ parser.add_argument("--max-file-mib", type=float, default=64, help="individual JSON size limit (default: 64 MiB)")+ parser.add_argument("--max-grid-points", type=int, default=20000000, help="projected receiver reconstruction limit")+ parser.add_argument("--skip-grid", action="store_true", help="only report recorded counters and inventory")+ args = parser.parse_args(argv)+ started = time.perf_counter()+ observed_at = datetime.now(timezone.utc).isoformat()+ if any(not math.isfinite(v) or v <= 0 for v in (args.max_read_mib, args.max_file_mib)) or args.max_grid_points < 1:+ parser.error("read, file, and grid budgets must be positive and finite")+ source = args.source.expanduser().resolve(strict=True)+ directory = source / "search" if source.is_dir() and (source / "search" / "manifest.json").exists() else source+ protected = (directory.parent if directory.name == "search" else directory) if directory.is_dir() else None+ targets = [Path(value).expanduser().resolve() if value else None+ for value in (args.json_path, args.markdown_path, args.cache)]+ if any(path == source or protected is not None and path.is_relative_to(protected) for path in targets if path):+ raise ValueError("report and cache paths must be outside the input job")+ if len({p for p in targets if p}) != sum(p is not None for p in targets):+ raise ValueError("JSON, Markdown, and cache paths must be distinct")+ json_path, markdown_path, cache_path = targets+ reader_hash = hashlib.sha256(Path(__file__).read_bytes()).hexdigest()+ cache = {"schema_version": 1, "reader_sha256": reader_hash, "source": str(source), "entries": {}, "grid_counts": {}}+ if cache_path is not None and cache_path.exists():+ if cache_path.stat().st_size > 64 * MIB:+ raise ValueError("summary cache exceeds 64 MiB; choose a fresh cache path")+ old = strict_json(cache_path.read_bytes())+ if not isinstance(old, dict) or old.get("schema_version") != 1 or old.get("source") != str(source):+ raise ValueError("cache belongs to another source or schema; choose a fresh cache path")+ if old.get("reader_sha256") == reader_hash:+ if not isinstance(old.get("entries"), dict) or not isinstance(old.get("grid_counts"), dict):+ raise ValueError("invalid summary cache; choose a fresh cache path")+ cache = old+ reader = Reader(int(args.max_read_mib * MIB), int(args.max_file_mib * MIB), cache["entries"])+ receipts, splits = {}, {}+ if directory.is_dir():+ manifest_path = directory / "manifest.json"+ saved = reader.read(manifest_path, limit=8 * MIB)+ if saved["status"] != "ok":+ raise ValueError(f"cannot read manifest: {saved}")+ config, run_hash = saved["value"]["configuration"], saved["value"]["run_hash"]+ if content_hash(config) != run_hash:+ raise ValueError("manifest configuration hash mismatch")+ before = inventory(directory)+ for name in sorted(before):+ match = SPLIT_NAME.fullmatch(name)+ if match:+ splits[match[1]] = reader.read(directory / name, limit=64 * 1024)+ for name in sorted(before, key=lambda n: (before[n][2] if isinstance(before[n], list) else math.inf, n)):+ match = RECEIPT_NAME.fullmatch(name)+ if match:+ receipts[match[1]] = reader.read(directory / name, receipt_summary)+ ledger = ledger_snapshot(directory / "retention.sqlite", run_hash)+ source_info = {"kind": "search_directory", "path": str(directory), "run_hash": run_hash,+ "manifest_hash_verified": True,+ "legacy_upgrade_present": (directory.parent / "checkpoint-upgrade.json").exists()}+ else:+ saved = reader.read(source, limit=8 * MIB)+ if saved["status"] != "ok":+ raise ValueError(f"cannot read recorded workload: {saved}")+ fixture = saved["value"]+ if (fixture.get("schema_version") != 1 or not isinstance(fixture.get("tiles"), list)+ or not isinstance(fixture.get("empty_payload"), dict) or "provenance" not in fixture):+ raise ValueError("expected the existing recorded real-empty-tile workload format")+ config, run_hash = fixture["query"], fixture["run_hash"]+ for tile in fixture["tiles"]:+ tile_id = tile["tile_id"]+ if re.fullmatch(TILE_ID, tile_id) is None or tile_id in receipts:+ raise ValueError("invalid or duplicated recorded tile ID")+ payload = dict(fixture["empty_payload"], cells=tile["cells"])+ receipts[tile_id] = {"status": "ok", "value": receipt_summary({+ "run_hash": run_hash, "payload": payload, "payload_hash": tile["payload_hash"]})}+ before = {}+ ledger = {"status": "unavailable_in_portable_workload", "stages": {}, "record_and_bound_content_rehashed": False}+ source_info = {"kind": "recorded_workload", "path": str(source), "run_hash": run_hash,+ "manifest_hash_verified": False, "provenance": fixture["provenance"]}+ max_tiles = config["budget"].get("max_tiles")+ if max_tiles is not None and (type(max_tiles) is not int or max_tiles <= 0):+ raise ValueError("invalid saved max_tiles")+ grid = Grid(config, cache["grid_counts"], args.max_grid_points, args.skip_grid)+ saved_packages = config.get("runtime", {}).get("packages", {})+ runtime_mismatches = {name: {"saved": saved_packages[name], "current": grid.runtime[name]}+ for name in ("numpy", "pyproj") if name in saved_packages and name in grid.runtime+ and saved_packages[name] != grid.runtime[name]}+ coverage = Coverage(grid, receipts, splits, run_hash, ledger)+ progress = coverage.report(max_tiles)+ producers = Counter(value["value"]["run_hash"] for value in receipts.values() if value["status"] == "ok")+ if directory.is_dir():+ changed = before != inventory(directory) or fingerprint(manifest_path) != cache["entries"][str(manifest_path)]["fingerprint"]+ else:+ changed = fingerprint(source) != cache["entries"][str(source)]["fingerprint"]+ stages = ledger.pop("stages")+ ledger.update(coarse_stage_rows=len(stages), stage_rows_without_reconciled_receipts=len(set(stages) - coverage.agreed))+ limitations = [+ "Payload hashes are checked or reused from an unchanged stat identity. Terrain/source files and retention records/bounds are not rehashed.",+ "Canonical agreement means matching committed stage metadata, not independently verified scientific contents or a resumability guarantee.",+ "SQLite uses one mode=ro WAL-aware transaction. Receipts and split markers are separate files; the combined observation is not atomic.",+ "Pre-open checks cannot atomically prevent sidecar races during producer shutdown. SQLite shared-memory reader bookkeeping is allowed; application rows and checkpoints are not written.",+ "Fully receipted regions record work. Unsupported trajectories, unresolved trials, or missing data can still prevent a scientific conclusion.",+ "Only the exact manifest producer identity is counted. Other producer identities need separate compatibility validation.",+ "Small receipts are read first. Read limits and absent counters make known sums a biased partial inventory, not a national extrapolation.",+ "Clock siblings share cells. All required clock descendants must be present before the receiver counts once; clock/scoring semantics are not replayed.",+ "Fine receipts are inventoried separately. grid_points counts recorded receiver visits and can be zero after an empty-flight shortcut; it never supplies geographic coverage.",+ "Receiver geometry is reconstructed with the current NumPy/pyproj/PROJ runtime, not a reproduced historical runtime.",+ "Stage CPU/wall percentiles, stage throughput, unique aircraft/samples, nonlinear fits, and globally retained modes remain unknown.",+ ]+ if grid.reason or grid.point_reason:+ limitations.append(grid.reason or grid.point_reason)+ if runtime_mismatches:+ limitations.append("Saved and current grid package versions differ: " + ", ".join(sorted(runtime_mismatches)) + ".")+ if changed:+ limitations.append("Input inventory changed during this refresh. Observations are provisional; refresh after it stabilizes.")+ if source_info["kind"] == "recorded_workload":+ limitations.append("This portable workload has no canonical ledger. It is not the original a904 national search or an accuracy result.")+ report = {"schema_version": 1, "observed_at": observed_at, "source": source_info,+ "grid_runtime": grid.runtime, "saved_runtime": config.get("runtime"), "grid_runtime_mismatches": runtime_mismatches,+ "configuration": {key: config.get(key) for key in ("bounds", "crs", "budget", "receiver_mask")},+ "snapshot": {"receipt_inventory_unchanged": not changed, "atomic_across_sqlite_and_files": False, "canonical_ledger": ledger},+ "read_budget": dict(reader.stats, max_read_bytes=int(args.max_read_mib * MIB), max_file_bytes=int(args.max_file_mib * MIB)),+ "inventory": {"coarse_receipt_files": len(receipts), "split_markers": len(splits),+ "fine_receipt_files": sum(n.startswith("fine-tile-") and not n.endswith(".error.json") for n in before),+ "error_files": sum(n.endswith(".error.json") for n in before),+ "receipt_read_statuses": dict(Counter(v["status"] for v in receipts.values())),+ "payload_verified_producer_counts": dict(sorted(producers.items())),+ "superseded_parent_receipts": len(coverage.superseded),+ "receipt_files_outside_reconciled_tree": len(set(receipts) - coverage.visited) if grid.roots is not None else None},+ "coverage": progress,+ "funnel_scope": "active coarse leaves with matching geometry and producer" if grid.roots is not None+ else "matching-producer receipts without grid/hierarchy reconciliation",+ "receipt_funnel": funnel(receipts, coverage.active),+ "canonical_stage_agreed_funnel": funnel(receipts, coverage.agreed)+ if grid.roots is not None and ledger["status"] == "metadata_snapshot" else None,+ "unknown_telemetry": {key: None for key in ("unique_aircraft", "unique_track_samples", "nonlinear_fits",+ "global_retained_modes", "stage_cpu_p50_p95_max_s", "stage_wall_p50_p95_max_s", "stage_throughput")},+ "limitations": limitations, "analysis_wall_s": time.perf_counter() - started}+ if cache_path is not None:+ atomic_write(cache_path, json.dumps(cache, sort_keys=True, separators=(",", ":"), allow_nan=False) + "\n")+ formatted = json.dumps(report, indent=2, sort_keys=True, allow_nan=False) + "\n"+ if json_path is not None:+ atomic_write(json_path, formatted)+ elif markdown_path is None:+ sys.stdout.write(formatted)+ if markdown_path is not None:+ atomic_write(markdown_path, markdown(report))+ return 0+++if __name__ == "__main__":+ try:+ raise SystemExit(main())+ except (OSError, KeyError, TypeError, ValueError, sqlite3.Error) as error:+ print(f"checkpoint report: {error}", file=sys.stderr)+ raise SystemExit(2)