Give satellite imagery, terrain data, buildings, streetview photos to your LLM
Complete map multipolygons
Belluxx5f03ebbc606c
diff --git a/README.md b/README.mdindex 275cdef..1b83a81 100644--- a/README.md+++ b/README.md@@ -56,6 +56,8 @@ Here is an overview of the structure of the output dir after a capture:OSM data comes from [Geofabrik](https://download.geofabrik.de/). Aleph downloads the smallest regional file covering your selection, then extracts the selected rectangle locally with its bundled [PBF reader](src/pbf.py). The first download can be hundreds of megabytes even for a small selection. Regional files are reused from `$XDG_CACHE_HOME/aleph/geofabrik` (usually `~/.cache/aleph/geofabrik`); use `--refresh` on CLI requests to download the latest snapshot. Map manifests record the source and snapshot date. Geofabrik omits contributor names, user IDs, and changeset IDs.+Map exports include complete ways and complete multipolygons selected by the rectangle, preserving building courtyards, lake islands, and boundaries assembled from several ways. Their geometry can extend beyond the selection. Other relations retain their original references and may remain incomplete.+The extractor also runs independently, with no third-party packages:```shdiff --git a/docs/cli.md b/docs/cli.mdindex 52854ce..fe7981d 100644--- a/docs/cli.md+++ b/docs/cli.md@@ -64,6 +64,6 @@ alephgeo capture create --bbox 41.8895 12.4910 41.8910 12.4940 --no-plan -o capt`--bbox` takes two opposite corners: `lat1 lon1 lat2 lon2`. All sources are included; use `--sources satellite osm` to download only satellite imagery, map data, and terrain.-`capture create` and `capture resume` also accept `--cache-dir PATH` and `--refresh`. Geofabrik publishes daily snapshots; repeated captures reuse the regional file until you refresh it. Estimates do not include the first regional download. Map exports retain complete local ways and original relation references; distant relation members may be absent. Named-street routes are joined using OSM node IDs, and nearby ways/relations are represented by their bounding-box centers.+`capture create` and `capture resume` also accept `--cache-dir PATH` and `--refresh`. Geofabrik publishes daily snapshots; repeated captures reuse the regional file until you refresh it. Estimates do not include the first regional download. Map exports complete selected ways and multipolygons, including outer boundaries and holes that extend beyond the selection. Other relations retain their original references and may remain incomplete. Named-street routes are joined using OSM node IDs, and nearby ways/relations are represented by their bounding-box centers.`--no-plan` starts downloading without confirmation (not recommended, as it may take a lot of time). Omit it to review the estimate first, or use `--plan` to save a plan without downloading. See `alephgeo capture create --help` for resolution and spacing options.diff --git a/docs/osm-benchmark.json b/docs/osm-benchmark.jsonindex 7072b3b..090c8ba 100644--- a/docs/osm-benchmark.json+++ b/docs/osm-benchmark.json@@ -389,5 +389,70 @@"cancellation_seconds": 0.86,"validation_method": "One final SHA-256 comparison of canonical OPL output against the native baseline, after performance iteration.","validation_seconds": 23.16+ },+ "stdlib_pbf_multipolygons": {+ "python_site_packages_enabled": false,+ "workers": 8,+ "regional_file_cached": true,+ "network_requests": 0,+ "tests_passed": 53,+ "results": [+ {+ "area_km2": 10,+ "bounds": [+ 41.875980496522295,+ 12.473098695094054,+ 41.904419503477705,+ 12.511301304905947+ ],+ "previous_complete_ways": {+ "map_seconds": 16.69,+ "map_bytes": 39695725,+ "osm_data_at": "2026-09-20T20:22:06Z"+ },+ "map_seconds": 16.86,+ "map_bytes": 39855706,+ "osm_data_at": "2026-09-20T20:22:06Z",+ "normalized_opl_sha256": "877cbe1724432fc538fc309a0982be75577dddfe65c08b66c96408d9a29493e9",+ "counts": {+ "node": 178481,+ "way": 32912,+ "relation": 3219+ },+ "matches_native_smart": true+ },+ {+ "area_km2": 1000,+ "bounds": [+ 41.74800496522297,+ 12.301186950940533,+ 42.03239503477703,+ 12.683213049059468+ ],+ "previous_complete_ways": {+ "map_seconds": 29.51,+ "map_bytes": 560701743,+ "osm_data_at": "2026-09-20T20:22:06Z"+ },+ "map_seconds": 28.29,+ "map_bytes": 561502181,+ "osm_data_at": "2026-09-20T20:22:06Z",+ "normalized_opl_sha256": "d68c36c8f80c1b4802ff017380f01769adea211ffd583eb380cef2410c15c532",+ "counts": {+ "node": 3178420,+ "way": 469640,+ "relation": 21004+ },+ "matches_native_smart": true+ }+ ],+ "validation_method": "One final SHA-256 comparison of canonical OPL against native Osmium smart extracts.",+ "validation_seconds": 30.46,+ "completion_profile": {+ "area_km2": 1000,+ "total_way_blocks": 676,+ "revisited_way_blocks": 61,+ "way_completion_seconds": 0.254+ }}}diff --git a/docs/osm-benchmark.md b/docs/osm-benchmark.mdindex a375395..457956b 100644--- a/docs/osm-benchmark.md+++ b/docs/osm-benchmark.md@@ -10,7 +10,29 @@ The baseline is commit `38aa50015fe347a71bcd1c0906376a8a46e3779d`. Its road andThese are single live trials, so network timings depend on server load. Local files were available in the filesystem cache. Software installation time is excluded.-## Current standard-library extractor+## Current extractor with multipolygon completion++Map exports now complete multipolygons selected by the capture rectangle, including outer boundaries and holes. This runs in the bundled standard-library script, with no Osmium dependency or persistent derived cache. The first way scan records each block's ID range; completion reads only blocks containing missing member ways and merges their XML into the existing sorted output. Node coordinates and relation records continue to be reused within the operation. Other relation types retain their existing partial-reference behavior.++A fresh paired run measured the previous complete-way exporter and the new exporter consecutively, using the same cached central Italy PBF and `python -S`:++| Area | Before: complete ways | Now: complete multipolygons | Additional XML |+| --- | ---: | ---: | ---: |+| 10 km² | 16.69 s | 16.86 s | 0.16 MB |+| 1,000 km² | 29.51 s | 28.29 s | 0.80 MB |++These single-trial total-time differences are within observed run-to-run variation. A separate 1,000 km² profile measured 0.254 seconds for the missing-way lookup and merge, revisiting 61 of 676 way blocks. Total export times include node completion and XML output; downloads and validation are excluded. The existing 30-second map estimate remains appropriate for this region.++After performance iteration, one final integrity pass compared SHA-256 hashes of canonical OPL from both exports against fresh native Osmium `smart` extracts. Both match in all objects, metadata, tags, coordinates, way references and relation members. The native tool is used only as a validation oracle. The exports contain:++| Area | Nodes | Ways | Relations |+| --- | ---: | ---: | ---: |+| 10 km² | 178,481 | 32,912 | 3,219 |+| 1,000 km² | 3,178,420 | 469,640 | 21,004 |++All 53 tests pass. The added fixtures verify a split outer boundary and courtyard through the map viewer, sorted output without duplicate ways, preserving the existing scope of road selection and unrelated relations, and preserving the previous output when a required member way is missing from the source.++## Earlier standard-library extractor (complete ways)`src/pbf.py` is a standalone script using only Python's standard library. It decodes packed PBF arrays in bulk with standard-library byte operations and integer arithmetic, processes blocks across CPU cores, and decodes strings only when selected objects reference them. XML ways are formatted in workers; selected coordinates and relation records are reused within the operation. No derived files or indexes are cached on disk. It supports the sorted snapshot PBFs used by Geofabrik, including raw/zlib blobs, ordinary/dense nodes, complete ways, parent relations, and recursive POI references. It does not implement Osmium's unrelated commands, history processing, or other optional compression formats.diff --git a/src/capture.py b/src/capture.pyindex ccbe767..74d8344 100644--- a/src/capture.py+++ b/src/capture.py@@ -140,8 +140,8 @@ def estimate(run):2026-09-19 17:26:41 UTC: 66 Street View photos in 36 s,including panorama metadata. 2026-09-19 17:09:18 UTC:1,020 satellite tiles in 374 s and 9 terrain tiles in 10 s.- 2026-09-21: the bundled PBF extractor took 15–24 s for 10–1,000 km² from- the 384 MB central Italy region. Region size affects this cost.+ 2026-09-21: map exports with completed multipolygons took 17–29 s for+ 10–1,000 km² from the 384 MB central Italy region. Region size affects this cost.Planning, initial regional downloads and final exports are excluded."""counts = dict(streetview_photos=0, streetview_stops=0, satellite_tiles=0,diff --git a/src/pbf.py b/src/pbf.pyindex a26085e..f23882a 100644--- a/src/pbf.py+++ b/src/pbf.py@@ -333,15 +333,24 @@ def references(data):def scan_ways(task):- path, entry, xml = task+ """Select spatial ways or requested member IDs and record the block's ID range."""+ path, entry, xml, wanted = taskvalues, groups = block(path, entry)selected, identities, nodes = [], array("q"), set()+ first = last = Nonestrings = Nonefor group in groups:for kind, message in fields(group):if kind == 3:+ if first is None:+ first = message+ last = message+ if wanted is not None:+ identity = varint(message, 1)[0] if message[:1] == b"\x08" else columns(message)[1]+ if identity not in wanted:+ continuepacked = way_references(message)- if not _SPATIAL.isdisjoint(references(packed)):+ if wanted is not None or not _SPATIAL.isdisjoint(references(packed)):row = columns(message)refs = list(references(packed))if strings is None:@@ -354,7 +363,9 @@ def scan_ways(task):selected.append(xml_object(item))else:selected.append(item)- return (identities, array("q", nodes), "".join(selected).encode()) if xml else selected+ bounds = tuple(varint(m, 1)[0] if m[:1] == b"\x08" else columns(m)[1] for m in (first, last))+ result = (identities, array("q", nodes), "".join(selected).encode()) if xml else selected+ return bounds, resultclass StringTable:@@ -644,7 +655,7 @@ class PBF:attrs=info(dict(fields(row.get(4, b""))), strings, date_gran))def select(self, area, xml=False):- self.way_xml = []+ self.way_xml = {}self.selected_coordinates.clear()spatial, nodes, ways, relations = set(), set(), set(), set()tasks = ((self.path, entry, area, xml) for entry in self.entries)@@ -659,35 +670,69 @@ class PBF:if coordinates is not None and selected:self.selected_coordinates[entry[0]] = selected, coordinatesnodes.update(spatial)- tasks = ((self.path, entry, xml) for entry in self.entries if entry[2] & 8)- for selected in self.parallel(scan_ways, tasks, spatial_nodes, (array("q", spatial),)):+ entries = [entry for entry in self.entries if entry[2] & 8]+ tasks = ((self.path, entry, xml, None) for entry in entries)+ way_blocks = []+ results = self.parallel(scan_ways, tasks, spatial_nodes, (array("q", spatial),))+ for entry, (bounds, selected) in zip(entries, results):if xml:identities, refs, chunk = selectedways.update(identities)nodes.update(refs)- self.way_xml.append(chunk)+ self.way_xml[entry[0]] = identities, chunk+ way_blocks.append((entry, bounds))continuefor way in selected:ways.add(way["id"])nodes.update(way["nodes"])self.selected_ways[way["id"]] = wayparents, pending = defaultdict(list), []- rows = {}+ rows, extra_ways = {}, set()for _, row, strings, date_gran in self.raw((4,)):identity = row[1]rows[identity] = row, strings, date_gran- refs = accumulate(unpack(row.get(9, b""), True))- for kind, ref in zip(unpack(row.get(10, b"")), refs):+ refs = list(accumulate(unpack(row.get(9, b""), True)))+ kinds = unpack(row.get(10, b""))+ matched = False+ for kind, ref in zip(kinds, refs):if kind == 2:parents[ref].append(identity)elif ref in (spatial if kind == 0 else ways):- pending.append(identity)+ matched = True+ if matched:+ pending.append(identity)+ if xml and tags(row, strings).get("type") == "multipolygon":+ for kind, ref in zip(kinds, refs):+ if kind == 0:+ nodes.add(ref)+ elif kind == 1:+ extra_ways.add(ref)while pending:identity = pending.pop()if identity not in relations:relations.add(identity)pending.extend(parents[identity])self.selected_relations = {i: rows[i] for i in relations}+ if xml:+ # Revisit only blocks containing missing polygon members. Keep the+ # existing XML and merge added ways in ID order without duplicates.+ missing = sorted(extra_ways - ways)+ tasks, entries = [], []+ for entry, (first, last) in way_blocks:+ wanted = missing[bisect.bisect_left(missing, first):bisect.bisect_right(missing, last)]+ if wanted:+ entries.append(entry)+ tasks.append((self.path, entry, True, set(wanted)))+ if tasks:+ for entry, (_, (identities, refs, chunk)) in zip(entries, self.parallel(scan_ways, tasks)):+ nodes.update(refs)+ ways.update(identities)+ previous, original = self.way_xml[entry[0]]+ merged = sorted(chain(zip(previous, original.splitlines(keepends=True)),+ zip(identities, chunk.splitlines(keepends=True))))+ self.way_xml[entry[0]] = (), b"".join(part for _, part in merged)+ if not extra_ways <= ways:+ raise ValueError("PBF source is missing multipolygon member ways.")return nodes, ways, relationsdef nodes(self, wanted, mode="points"):@@ -785,7 +830,7 @@ class PBF:written += chunk.count(b"<node ")if written != len(selected[0]):raise ValueError("PBF extract contains incomplete way geometry.")- for chunk in self.way_xml:+ for _, chunk in self.way_xml.values():stream.write(chunk)for item in self.relations(selected[2]):stream.write(xml_object(item).encode("utf-8"))diff --git a/tests/test_pbf.py b/tests/test_pbf.pyindex 32fd2a7..0bea085 100644--- a/tests/test_pbf.py+++ b/tests/test_pbf.py@@ -10,6 +10,7 @@ import zlibfrom pathlib import Pathfrom xml.etree import ElementTree as ET+from src.dashboard_osm import display_osmfrom src.pbf import PBF, unpack@@ -37,12 +38,43 @@ def blob(kind, payload, compressed=False):return struct.pack(">I", len(header)) + header + data-def snapshot(group, *, options=b"", features=(b"OsmSchema-V0.6", b"DenseNodes")):+def snapshot(group, *, options=b"", features=(b"OsmSchema-V0.6", b"DenseNodes"), strings=(b"",)):header = b"".join(field(4, feature) for feature in features)- data = field(1, field(1, b"")) + field(2, group) + options+ data = field(1, b"".join(field(1, value) for value in strings)) + field(2, group) + optionsreturn blob(b"OSMHeader", header) + blob(b"OSMData", data, True)+def multipolygon_snapshot(missing_way=False):+ strings = (b"", b"type", b"multipolygon", b"building", b"yes", b"outer", b"inner", b"label", b"route")+ coordinates = [(-10000, -10000), (-10000, 10000), (10000, 10000), (10000, -10000),+ (-5000, -5000), (-5000, 5000), (5000, 5000), (5000, -5000),+ (100000, 100000), (100000, 110000), (20000, 20000)]+ nodes = b"".join(field(1, field(1, identity * 2) + field(8, (lat << 1) ^ (lat >> 63))+ + field(9, (lon << 1) ^ (lon >> 63)))+ for identity, (lat, lon) in enumerate(coordinates, 1))+ result = snapshot(nodes, strings=strings)+ table = field(1, b"".join(field(1, value) for value in strings))++ def deltas(refs):+ return packed([refs[0], *(b - a for a, b in zip(refs, refs[1:]))], True)++ for ways in ([(10, [1, 2]), (15, [5, 6, 7, 8, 5]), (20, [2, 3, 4])],+ [(30, [4, 1])], [(40, [9, 10]), (50, [9, 10])]):+ group = b"".join(field(3, field(1, identity) + field(8, deltas(refs)))+ for identity, refs in ways if not (missing_way and identity == 20))+ result += blob(b"OSMData", table + field(2, group), True)+ relations = b""+ # Only 100 directly intersects the capture and needs polygon completion.+ for identity, keys, vals, refs, roles, kinds in (+ (100, [1, 3], [2, 4], [10, 20, 30, 15, 11], [5, 5, 5, 6, 7], [1, 1, 1, 1, 0]),+ (101, [1], [8], [10, 40], [0, 0], [1, 1]),+ (102, [1], [2], [100, 50], [0, 5], [2, 1]),+ ):+ relations += field(4, field(1, identity) + field(2, packed(keys)) + field(3, packed(vals))+ + field(8, packed(roles)) + field(9, deltas(refs)) + field(10, packed(kinds)))+ return result + blob(b"OSMData", table + field(2, relations), True)++class PBFTests(unittest.TestCase):def setUp(self):self.folder = Path(self.enterContext(tempfile.TemporaryDirectory()))@@ -115,6 +147,29 @@ class PBFTests(unittest.TestCase):self.assertEqual(self.output.read_bytes(), b"previous")self.assertEqual(list(self.folder.glob(".map.osm-*")), [])+ def test_map_completes_multipolygon_outer_and_hole_without_expanding_other_relations(self):+ self.path.write_bytes(multipolygon_snapshot())+ area = (-.0011, -.0011, -.0009, -.0009)+ source = PBF(self.path)+ self.assertEqual(source.select(area)[1], {10, 30})+ source.export(area, self.output)+ root = ET.parse(self.output).getroot()+ self.assertEqual([int(w.attrib["id"]) for w in root.findall("way")], [10, 15, 20, 30])+ self.assertEqual({int(n.attrib["id"]) for n in root.findall("node")}, {*range(1, 9), 11})+ self.assertEqual([int(r.attrib["id"]) for r in root.findall("relation")], [100, 101, 102])+ features = display_osm(self.output)["features"]+ self.assertEqual(len(features), 1)+ geometry = features[0]["geometry"]+ self.assertEqual(geometry["type"], "MultiPolygon")+ self.assertEqual(len(geometry["coordinates"][0]), 2) # Closed outer and courtyard.++ def test_missing_multipolygon_way_preserves_previous_output(self):+ self.path.write_bytes(multipolygon_snapshot(missing_way=True))+ self.output.write_bytes(b"previous")+ with self.assertRaisesRegex(ValueError, "missing multipolygon member ways"):+ PBF(self.path).export((-.0011, -.0011, -.0009, -.0009), self.output)+ self.assertEqual(self.output.read_bytes(), b"previous")+def test_standalone_script_runs_without_site_packages(self):source = Path(__file__).parent / "data" / "dense.osm.pbf"script = Path(__file__).resolve().parents[1] / "src" / "pbf.py"