Skip to content

armgit / Aleph

Public Imported from GitHub
0

Give satellite imagery, terrain data, buildings, streetview photos to your LLM

Cleanup

Belluxx6034ab779fc3
diff --git a/README.md b/README.md
index 1b83a81..f1dbf13 100644
--- a/README.md
+++ b/README.md
@@ -19,7 +19,7 @@ This was born as a way to let LLMs quickly gather information from the physical
## Install
-Requires Python 3.11+. Pillow is the only dependency and is installed automatically with Aleph. OSM processing uses the bundled Python script and standard library. No API keys required.
+Requires Python 3.11+. Pillow is the only dependency. No API keys are required.
```sh
python3 -m venv .venv
@@ -54,15 +54,7 @@ Here is an overview of the structure of the output dir after a capture:
- `map.osm`: OpenStreetMap roads, buildings, water, and other features
- `terrain.tif`: Elevation as a GeoTIFF (original tiles in `terrain/`)
-OSM data comes from [Geofabrik](https://download.geofabrik.de/). Aleph downloads the smallest regional file covering your selection, then extracts the selected rectangle locally with its bundled [PBF reader](src/pbf.py). The first download can be hundreds of megabytes even for a small selection. Regional files are reused from `$XDG_CACHE_HOME/aleph/geofabrik` (usually `~/.cache/aleph/geofabrik`); use `--refresh` on CLI requests to download the latest snapshot. Map manifests record the source and snapshot date. Geofabrik omits contributor names, user IDs, and changeset IDs.
-
-Map exports include complete ways and complete multipolygons selected by the rectangle, preserving building courtyards, lake islands, and boundaries assembled from several ways. Their geometry can extend beyond the selection. Other relations retain their original references and may remain incomplete.
-
-The extractor also runs independently, with no third-party packages:
-
-```sh
-python3 src/pbf.py region.osm.pbf 41.88 12.48 41.90 12.51 -o map.osm
-```
+Map data comes from [Geofabrik](https://download.geofabrik.de/). The first use will download a large regional file. It will be reused for later requests.
## From the terminal (for scripts and agents)
diff --git a/docs/cli.md b/docs/cli.md
index fe7981d..20537ef 100644
--- a/docs/cli.md
+++ b/docs/cli.md
@@ -20,9 +20,9 @@ alephgeo resolve --at 41.8902 12.4922 --json
alephgeo resolve --at 41.8902 12.4922 --nearby --radius 100 --limit 10 --json
```
-Reverse lookup returns the nearest address or named feature. Nearby search returns POIs ordered by distance.
+Reverse lookup returns the nearest address or named place. Nearby search lists places by distance, using the center of buildings and other areas.
-Name lookup uses [Photon](https://github.com/komoot/photon). Set `ALEPH_GEOCODER_URL` or `--geocoder URL` to use another Photon server. Nearby POIs and street geometry use local Geofabrik extracts.
+Name lookup uses [Photon](https://github.com/komoot/photon). Set `ALEPH_GEOCODER_URL` or `--geocoder URL` to use another Photon server. Nearby places and streets use Geofabrik map data.
## Quick imagery
@@ -49,10 +49,10 @@ alephgeo streetview --street "Via del Corso, Rome" --stops 10 --view forward -o
For example, `--stops 10` requests ten positions along the street. Choose a viewing direction with `--view forward`, `backward`, `left`, or `right`; `both` takes left and right photos at each stop. Use `--reverse` to go in the opposite direction.
-If Aleph finds several places or street branches, choose from the returned options with `--match ID` or `--route N`, respectively.
+If several results match, use `--match ID` to pick a place or `--route N` to pick a street branch.
> [!TIP]
-> Quick requests start immediately and cache imagery/geocoder responses for 24 hours. OSM roads, nearby POIs, and maps use Geofabrik regional downloads, processed locally by Aleph's standard-library PBF reader. These regional files are retained until explicitly refreshed; the first request may download hundreds of megabytes. Use `--refresh` to fetch fresh responses and regional files, or `--cache-dir PATH` to change the cache location (default: `$XDG_CACHE_HOME/aleph` or `~/.cache/aleph`).
+> Images and Photon results are cached for 24 hours. Geofabrik files are kept until refreshed. Use `--refresh` to fetch new data or `--cache-dir PATH` to choose the cache folder (default: `$XDG_CACHE_HOME/aleph` or `~/.cache/aleph`).
## Capture an area
@@ -64,6 +64,6 @@ alephgeo capture create --bbox 41.8895 12.4910 41.8910 12.4940 --no-plan -o capt
`--bbox` takes two opposite corners: `lat1 lon1 lat2 lon2`. All sources are included; use `--sources satellite osm` to download only satellite imagery, map data, and terrain.
-`capture create` and `capture resume` also accept `--cache-dir PATH` and `--refresh`. Geofabrik publishes daily snapshots; repeated captures reuse the regional file until you refresh it. Estimates do not include the first regional download. Map exports complete selected ways and multipolygons, including outer boundaries and holes that extend beyond the selection. Other relations retain their original references and may remain incomplete. Named-street routes are joined using OSM node IDs, and nearby ways/relations are represented by their bounding-box centers.
+`capture create` and `capture resume` also accept `--cache-dir PATH` and `--refresh`. Time estimates do not include the first Geofabrik download.
`--no-plan` starts downloading without confirmation (not recommended, as it may take a lot of time). Omit it to review the estimate first, or use `--plan` to save a plan without downloading. See `alephgeo capture create --help` for resolution and spacing options.
diff --git a/docs/osm-benchmark.json b/docs/osm-benchmark.json
deleted file mode 100644
index 090c8ba..0000000
--- a/docs/osm-benchmark.json
+++ /dev/null
@@ -1,458 +0,0 @@
-{
- "date": "2026-09-21",
- "baseline_commit": "38aa50015fe347a71bcd1c0906376a8a46e3779d",
- "environment": {
- "python": "3.12.12 (main, Oct 16 2025, 14:57:37) [Clang 17.0.0 (clang-1700.3.19.1)]",
- "platform": "macOS-15.7.1-arm64-arm-64bit",
- "center": [
- 41.8902,
- 12.4922
- ],
- "regions": [
- {
- "requested_km2": 10,
- "spherical_km2": 9.99999989734788,
- "bbox_south_west_north_east": [
- 41.875980496522295,
- 12.473098695094054,
- 41.904419503477705,
- 12.511301304905947
- ]
- },
- {
- "requested_km2": 1000,
- "spherical_km2": 999.9989734686736,
- "bbox_south_west_north_east": [
- 41.74800496522297,
- 12.301186950940533,
- 42.03239503477703,
- 12.683213049059468
- ]
- }
- ]
- },
- "network_deadline_seconds": 600,
- "standalone_geofabrik_download": {
- "http_code": 200,
- "size_download": 383641858,
- "time_total": 13.592244,
- "url_effective": "https://download.geofabrik.de/europe/italy/centro-260920.osm.pbf"
- },
- "overpass": [
- {
- "provider": "current_overpass",
- "kind": "roads",
- "requested_km2": 10,
- "spherical_km2": 9.99999989734788,
- "bbox_south_west_north_east": [
- 41.875980496522295,
- 12.473098695094054,
- 41.904419503477705,
- 12.511301304905947
- ],
- "query": "[out:json][timeout:25][maxsize:1073741824];way[\"highway\"](41.875980496522295,12.473098695094054,41.904419503477705,12.511301304905947);out body geom;",
- "started_at": "2026-09-21T12:02:22.944196+00:00",
- "attempts": [
- {
- "url": "https://overpass-api.de/api/interpreter",
- "status": 504,
- "headers_or_error_seconds": 8.897348375001457
- },
- {
- "url": "https://overpass-api.de/api/interpreter",
- "status": 504,
- "headers_or_error_seconds": 7.583786708000844
- },
- {
- "url": "https://overpass-api.de/api/interpreter",
- "status": 504,
- "headers_or_error_seconds": 8.210137291000137
- },
- {
- "url": "https://overpass-api.de/api/interpreter",
- "status": 504,
- "headers_or_error_seconds": 16.630713708000258
- },
- {
- "url": "https://overpass.private.coffee/api/interpreter",
- "status": 504,
- "headers_or_error_seconds": 107.65975120900112
- },
- {
- "url": "https://overpass.private.coffee/api/interpreter",
- "error": "TimeoutError('The read operation timed out')",
- "headers_or_error_seconds": 240.06496191599945
- },
- {
- "url": "https://overpass.private.coffee/api/interpreter",
- "status": 504,
- "headers_or_error_seconds": 86.85915491600099
- },
- {
- "url": "https://overpass.private.coffee/api/interpreter",
- "status": 504,
- "headers_or_error_seconds": 102.47538841699861
- },
- {
- "url": "https://maps.mail.ru/osm/tools/overpass/api/interpreter",
- "headers_or_error_seconds": 7.599620124998182
- }
- ],
- "status": "failed",
- "error": "Deadline('Benchmark stopped this query after 600 seconds')",
- "total_seconds": 600.0076005420015
- },
- {
- "provider": "current_overpass",
- "kind": "map",
- "requested_km2": 10,
- "spherical_km2": 9.99999989734788,
- "bbox_south_west_north_east": [
- 41.875980496522295,
- 12.473098695094054,
- 41.904419503477705,
- 12.511301304905947
- ],
- "query": "[out:xml][timeout:180][maxsize:536870912];nwr(41.875980496522295,12.473098695094054,41.904419503477705,12.511301304905947);(._;node(w););out meta qt;",
- "started_at": "2026-09-21T12:12:22.925162+00:00",
- "attempts": [
- {
- "url": "https://overpass-api.de/api/interpreter",
- "status": 200,
- "headers_or_error_seconds": 2.9656944579983247
- }
- ],
- "download_seconds": 7.840124750000541,
- "bytes": 52404541,
- "valid": true,
- "counts": {
- "node": 177570,
- "way": 32788,
- "relation": 3082
- },
- "highway_ways": 12412,
- "validation_seconds": 1.7904330419987673,
- "osm_timestamp": {
- "osm_base": "2026-09-21T12:10:20Z"
- },
- "sha256": "74b078fa9f3d1ddc958969a814d83d62f9bea2adbd7182b1e9c38d83cd62fec8",
- "status": "ok",
- "total_seconds": 10.694375208000565
- },
- {
- "provider": "current_overpass",
- "kind": "roads",
- "requested_km2": 1000,
- "spherical_km2": 999.9989734686736,
- "bbox_south_west_north_east": [
- 41.74800496522297,
- 12.301186950940533,
- 42.03239503477703,
- 12.683213049059468
- ],
- "query": "[out:json][timeout:25][maxsize:1073741824];way[\"highway\"](41.74800496522297,12.301186950940533,42.03239503477703,12.683213049059468);out body geom;",
- "started_at": "2026-09-21T12:12:33.620171+00:00",
- "attempts": [
- {
- "url": "https://overpass-api.de/api/interpreter",
- "status": 504,
- "headers_or_error_seconds": 8.826568917000259
- },
- {
- "url": "https://overpass-api.de/api/interpreter",
- "status": 504,
- "headers_or_error_seconds": 9.151510834002693
- },
- {
- "url": "https://overpass-api.de/api/interpreter",
- "status": 504,
- "headers_or_error_seconds": 8.84591270799865
- },
- {
- "url": "https://overpass-api.de/api/interpreter",
- "status": 504,
- "headers_or_error_seconds": 16.052941875001125
- },
- {
- "url": "https://overpass.private.coffee/api/interpreter",
- "status": 504,
- "headers_or_error_seconds": 92.31481312500182
- },
- {
- "url": "https://overpass.private.coffee/api/interpreter",
- "status": 504,
- "headers_or_error_seconds": 98.98106058299891
- },
- {
- "url": "https://overpass.private.coffee/api/interpreter",
- "status": 504,
- "headers_or_error_seconds": 224.4021728330008
- },
- {
- "url": "https://overpass.private.coffee/api/interpreter",
- "status": 504,
- "headers_or_error_seconds": 99.43211099999826
- },
- {
- "url": "https://maps.mail.ru/osm/tools/overpass/api/interpreter",
- "headers_or_error_seconds": 27.963263708999875
- }
- ],
- "status": "failed",
- "error": "Deadline('Benchmark stopped this query after 600 seconds')",
- "total_seconds": 600.0040845830008
- },
- {
- "provider": "current_overpass",
- "kind": "map",
- "requested_km2": 1000,
- "spherical_km2": 999.9989734686736,
- "bbox_south_west_north_east": [
- 41.74800496522297,
- 12.301186950940533,
- 42.03239503477703,
- 12.683213049059468
- ],
- "query": "[out:xml][timeout:180][maxsize:536870912];nwr(41.74800496522297,12.301186950940533,42.03239503477703,12.683213049059468);(._;node(w););out meta qt;",
- "started_at": "2026-09-21T12:22:33.619508+00:00",
- "attempts": [
- {
- "url": "https://overpass-api.de/api/interpreter",
- "status": 504,
- "headers_or_error_seconds": 7.800043290997564
- },
- {
- "url": "https://overpass-api.de/api/interpreter",
- "status": 200,
- "headers_or_error_seconds": 16.731868375001795
- }
- ],
- "download_seconds": 62.22662429100092,
- "bytes": 769374728,
- "valid": true,
- "counts": {
- "node": 3172882,
- "way": 469157,
- "relation": 20521
- },
- "highway_ways": 163195,
- "validation_seconds": 113.68387441699815,
- "osm_timestamp": {
- "osm_base": "2026-09-21T12:21:34Z"
- },
- "sha256": "62fd2a8206a9a2e090cce667d442cd2780539f9877610e6619545d27d0bdc9a5",
- "status": "ok",
- "total_seconds": 226.08120658299958
- }
- ],
- "geofabrik": {
- "results": [
- {
- "area_km2": 10,
- "map_seconds": 15.256952792002267,
- "map_bytes": 41319499,
- "source_url": "https://download.geofabrik.de/europe/italy/centro-latest.osm.pbf",
- "region": "Centro",
- "osm_data_at": "2026-09-20T20:22:06Z",
- "cached_map_seconds": 3.3866028749980615,
- "road_seconds": 3.637508791998698,
- "highway_ways": 12412
- },
- {
- "area_km2": 1000,
- "map_seconds": 3.6622887500016077,
- "map_bytes": 571428870,
- "source_url": "https://download.geofabrik.de/europe/italy/centro-latest.osm.pbf",
- "region": "Centro",
- "osm_data_at": "2026-09-20T20:22:06Z",
- "road_seconds": 11.716084957999556,
- "highway_ways": 163195
- }
- ],
- "requests": [
- [
- "GET",
- "https://download.geofabrik.de/index-v1.json",
- 0
- ],
- [
- "GET",
- "https://download.geofabrik.de/europe/italy/centro-latest.osm.pbf",
- 0
- ]
- ]
- },
- "checks": {
- "missing_way_nodes": {
- "10": 0,
- "1000": 0
- },
- "highway_way_id_sets_match": {
- "10": true,
- "1000": true
- },
- "tests_passed": 44
- },
- "python_osmium": {
- "version": "4.3.1",
- "regional_file_cached": true,
- "network_requests": 0,
- "results": [
- {
- "area_km2": 10,
- "map_seconds": 161.65,
- "map_bytes": 41319502,
- "source_url": "https://download.geofabrik.de/europe/italy/centro-latest.osm.pbf",
- "region": "Centro",
- "osm_data_at": "2026-09-20T20:22:06Z",
- "counts": {
- "nodes": 177525,
- "ways": 32787,
- "relations": 3219
- },
- "object_ids_match_native_cli": true,
- "missing_way_nodes": 0
- },
- {
- "area_km2": 1000,
- "map_seconds": 173.88,
- "map_bytes": 571428873,
- "source_url": "https://download.geofabrik.de/europe/italy/centro-latest.osm.pbf",
- "region": "Centro",
- "osm_data_at": "2026-09-20T20:22:06Z",
- "counts": {
- "nodes": 3172834,
- "ways": 469155,
- "relations": 21004
- },
- "object_ids_match_native_cli": true,
- "missing_way_nodes": 0
- }
- ]
- },
- "stdlib_pbf": {
- "python_site_packages_enabled": false,
- "workers": 8,
- "regional_file_cached": true,
- "network_requests": 0,
- "tests_passed": 51,
- "results": [
- {
- "area_km2": 10,
- "bounds": [
- 41.875980496522295,
- 12.473098695094054,
- 41.904419503477705,
- 12.511301304905947
- ],
- "map_seconds": 15.08,
- "map_bytes": 39695725,
- "osm_data_at": "2026-09-20T20:22:06Z",
- "counts": {
- "node": 177525,
- "way": 32787,
- "relation": 3219
- },
- "matches_native": true,
- "missing_way_nodes": 0,
- "previous_map_seconds": 22.49,
- "normalized_opl_sha256": "6d15bc46365c66559a3f61001616771714517778534fb6afb4d47664ec5fcbb6"
- },
- {
- "area_km2": 1000,
- "bounds": [
- 41.74800496522297,
- 12.301186950940533,
- 42.03239503477703,
- 12.683213049059468
- ],
- "map_seconds": 23.95,
- "map_bytes": 560701743,
- "osm_data_at": "2026-09-20T20:22:06Z",
- "counts": {
- "node": 3172834,
- "way": 469155,
- "relation": 21004
- },
- "matches_native": true,
- "missing_way_nodes": 0,
- "previous_map_seconds": 40.54,
- "normalized_opl_sha256": "a868eaa29e95cc6cb375925e3eeb0becfd6ebabf1fe96b42fd78dd6175d2c2c6"
- }
- ],
- "query_checks": {
- "highway_ways": 12412,
- "road_geometry_matches": true,
- "nearby_ids_match": true,
- "connected_street_ways": 11
- },
- "cancellation_seconds": 0.86,
- "validation_method": "One final SHA-256 comparison of canonical OPL output against the native baseline, after performance iteration.",
- "validation_seconds": 23.16
- },
- "stdlib_pbf_multipolygons": {
- "python_site_packages_enabled": false,
- "workers": 8,
- "regional_file_cached": true,
- "network_requests": 0,
- "tests_passed": 53,
- "results": [
- {
- "area_km2": 10,
- "bounds": [
- 41.875980496522295,
- 12.473098695094054,
- 41.904419503477705,
- 12.511301304905947
- ],
- "previous_complete_ways": {
- "map_seconds": 16.69,
- "map_bytes": 39695725,
- "osm_data_at": "2026-09-20T20:22:06Z"
- },
- "map_seconds": 16.86,
- "map_bytes": 39855706,
- "osm_data_at": "2026-09-20T20:22:06Z",
- "normalized_opl_sha256": "877cbe1724432fc538fc309a0982be75577dddfe65c08b66c96408d9a29493e9",
- "counts": {
- "node": 178481,
- "way": 32912,
- "relation": 3219
- },
- "matches_native_smart": true
- },
- {
- "area_km2": 1000,
- "bounds": [
- 41.74800496522297,
- 12.301186950940533,
- 42.03239503477703,
- 12.683213049059468
- ],
- "previous_complete_ways": {
- "map_seconds": 29.51,
- "map_bytes": 560701743,
- "osm_data_at": "2026-09-20T20:22:06Z"
- },
- "map_seconds": 28.29,
- "map_bytes": 561502181,
- "osm_data_at": "2026-09-20T20:22:06Z",
- "normalized_opl_sha256": "d68c36c8f80c1b4802ff017380f01769adea211ffd583eb380cef2410c15c532",
- "counts": {
- "node": 3178420,
- "way": 469640,
- "relation": 21004
- },
- "matches_native_smart": true
- }
- ],
- "validation_method": "One final SHA-256 comparison of canonical OPL against native Osmium smart extracts.",
- "validation_seconds": 30.46,
- "completion_profile": {
- "area_km2": 1000,
- "total_way_blocks": 676,
- "revisited_way_blocks": 61,
- "way_completion_seconds": 0.254
- }
- }
-}
diff --git a/docs/osm-benchmark.md b/docs/osm-benchmark.md
deleted file mode 100644
index 457956b..0000000
--- a/docs/osm-benchmark.md
+++ /dev/null
@@ -1,129 +0,0 @@
-# OSM provider benchmark — 21 September 2026
-
-Aleph now uses its bundled standard-library [PBF reader](../src/pbf.py) for Geofabrik extraction and local road geometry. The Python `osmium` and native Osmium CLI measurements below describe earlier implementations. None of the Geofabrik implementations uses Overpass.
-
-## Method
-
-Both selections are centered on the Colosseum in Rome, at 41.8902° N, 12.4922° E. They are squares approximately 3.162 km and 31.623 km wide, with spherical areas of 10.000 and 999.999 km². Tests ran on macOS 15.7.1 ARM64 with Python 3.12.12 and eight available CPUs. The earlier implementations used Osmium CLI 1.19.1 and Python `osmium` 4.3.1.
-
-The baseline is commit `38aa50015fe347a71bcd1c0906376a8a46e3779d`. Its road and map queries, memory budgets, HTTP retries, and three-server fallback were unchanged. Each network operation had an outer 600-second benchmark deadline. A deadline result means no usable response arrived within the test budget, not that the client exhausted every possible retry. Imagery and terrain downloads were excluded.
-
-These are single live trials, so network timings depend on server load. Local files were available in the filesystem cache. Software installation time is excluded.
-
-## Current extractor with multipolygon completion
-
-Map exports now complete multipolygons selected by the capture rectangle, including outer boundaries and holes. This runs in the bundled standard-library script, with no Osmium dependency or persistent derived cache. The first way scan records each block's ID range; completion reads only blocks containing missing member ways and merges their XML into the existing sorted output. Node coordinates and relation records continue to be reused within the operation. Other relation types retain their existing partial-reference behavior.
-
-A fresh paired run measured the previous complete-way exporter and the new exporter consecutively, using the same cached central Italy PBF and `python -S`:
-
-| Area | Before: complete ways | Now: complete multipolygons | Additional XML |
-| --- | ---: | ---: | ---: |
-| 10 km² | 16.69 s | 16.86 s | 0.16 MB |
-| 1,000 km² | 29.51 s | 28.29 s | 0.80 MB |
-
-These single-trial total-time differences are within observed run-to-run variation. A separate 1,000 km² profile measured 0.254 seconds for the missing-way lookup and merge, revisiting 61 of 676 way blocks. Total export times include node completion and XML output; downloads and validation are excluded. The existing 30-second map estimate remains appropriate for this region.
-
-After performance iteration, one final integrity pass compared SHA-256 hashes of canonical OPL from both exports against fresh native Osmium `smart` extracts. Both match in all objects, metadata, tags, coordinates, way references and relation members. The native tool is used only as a validation oracle. The exports contain:
-
-| Area | Nodes | Ways | Relations |
-| --- | ---: | ---: | ---: |
-| 10 km² | 178,481 | 32,912 | 3,219 |
-| 1,000 km² | 3,178,420 | 469,640 | 21,004 |
-
-All 53 tests pass. The added fixtures verify a split outer boundary and courtyard through the map viewer, sorted output without duplicate ways, preserving the existing scope of road selection and unrelated relations, and preserving the previous output when a required member way is missing from the source.
-
-## Earlier standard-library extractor (complete ways)
-
-`src/pbf.py` is a standalone script using only Python's standard library. It decodes packed PBF arrays in bulk with standard-library byte operations and integer arithmetic, processes blocks across CPU cores, and decodes strings only when selected objects reference them. XML ways are formatted in workers; selected coordinates and relation records are reused within the operation. No derived files or indexes are cached on disk. It supports the sorted snapshot PBFs used by Geofabrik, including raw/zlib blobs, ordinary/dense nodes, complete ways, parent relations, and recursive POI references. It does not implement Osmium's unrelated commands, history processing, or other optional compression formats.
-
-These exports ran with `python -S`, so no site packages were available, using the same cached 383.6 MB central Italy file:
-
-| Area | Optimized script | Initial script | Python `osmium` | Native CLI | New XML size |
-| --- | ---: | ---: | ---: | ---: | ---: |
-| 10 km² | 15.08 s | 22.49 s | 161.65 s | 3.39 s | 39.7 MB |
-| 1,000 km² | 23.95 s | 40.54 s | 173.88 s | 3.66 s | 560.7 MB |
-
-The optimized script takes 33% and 41% less time than the initial bundled script, respectively, and is about 10.7× and 7.3× faster than the earlier Python `osmium` implementation. The native CLI remains faster. Timings include parsing the PBF index, selecting objects, completing ways, and writing XML, and exclude downloads and subsequent validation. Capture estimates now allow 30 seconds for map extraction based on this region.
-
-After performance iteration, one final comparison normalized both exports and the native baseline to OPL with the native CLI and compared SHA-256 hashes. Both match, including object IDs, coordinates, tags, metadata, way references and relation members. The matching native baseline has zero missing way nodes. The native CLI was used only as a validation oracle; extraction still runs with no third-party packages. The real 10 km² road query matches all 12,412 ways, and the Colosseum nearby results and Via del Corso's 11 connected ways match as well. A final live cancellation check joined all workers in 0.86 seconds and preserved the previous output. XML sizes differ from the CLI output because formatting differs.
-
-The test suite has 51 passing tests with `osmium` uninstalled. Tests cover bulk integer decoding, both node encodings, UTF-8/XML escaping, metadata, non-default coordinate offsets/granularity, split packed fields, malformed input, nested relations, cancellation, atomic output, and standalone execution without site packages.
-
-To run the script independently:
-
-```sh
-python3 src/pbf.py centro.osm.pbf \
- 41.875980496522295 12.473098695094054 41.904419503477705 12.511301304905947 \
- -o rome-10.osm
-```
-
-## Earlier Python Osmium results
-
-The Python implementation installs through `pip install .` and uses no external Osmium executable. It reads PBF files directly, preserves complete ways and parent relations, and completes nested POI relations. Roads and nearby places are read directly into Python without intermediate XML files.
-
-Both exports reused the same cached central Italy regional file, with network access disabled in the client:
-
-| Area | Map export | Map XML size |
-| --- | ---: | ---: |
-| 10 km² | 161.65 s | 41.3 MB |
-| 1,000 km² | 173.88 s | 571.4 MB |
-
-These single-trial timings exclude downloading the regional file and subsequent validation. This implementation scanned the entire regional file through Python OSM objects to select the rectangle, so the regional file's size mattered even for small selections.
-
-Both exports have identical node, way, and relation IDs to the earlier CLI exports, and both pass way-reference validation through Python Osmium. The 10 km² export's 12,412 highway ways match the earlier road geometry exactly. Nearby results around the Colosseum and Via del Corso's 11 connected ways also matched when read from the new export. All 44 tests pass with the native tool absent from `PATH`, including cancellation, nested relations, offline reuse, and preserving previous files on failure.
-
-## Earlier native Osmium CLI results
-
-Both selections use [central Italy's extract](https://download.geofabrik.de/europe/italy/centro.html), containing data through `2026-09-20T20:22:06Z`. The regional PBF is 383,641,858 bytes (383.6 MB). Its initial standalone download took 13.59 seconds.
-
-The earlier CLI-backed application path also completed a fresh index download, regional download, and 10 km² map export in **15.26 seconds total**. Its request log contained exactly two GETs, both to Geofabrik. Subsequent operations reused the same regional file:
-
-| Area | Map export | Road loading | Map XML size | Highway ways |
-| --- | ---: | ---: | ---: | ---: |
-| 10 km² | 3.39 s | 3.64 s | 41.3 MB | 12,412 |
-| 1,000 km² | 3.66 s | 11.72 s | 571.4 MB | 163,195 |
-
-Map timings include selecting the region and writing the selected rectangle as XML. Road timings include extracting the rectangle, filtering highways, and decoding geometry and OSM node IDs into Python. The larger area reused the first area's regional download; it did not require downloading a larger file. Each operation creates temporary local files, which are removed afterward. Only the regional PBF and region index are retained.
-
-## Baseline results
-
-The 10 km² road query reached the 600-second deadline after repeated HTTP 504 overload responses and a socket timeout. The 1,000 km² road query also reached the 600-second deadline after repeated HTTP 504 responses. Both reached the third endpoint before their deadlines.
-
-Both map queries succeeded:
-
-| Area | Download, including retries | Original validation | XML size |
-| --- | ---: | ---: | ---: |
-| 10 km² | 7.84 s | 1.79 s | 52.4 MB |
-| 1,000 km² | 62.23 s | 113.68 s | 769.4 MB |
-
-The smaller map succeeded on its first attempt. The larger map succeeded on its second attempt, after an HTTP 504 response. The recorded 10.69 and 226.08 second benchmark totals also include an additional XML parse for statistics, so they should not be presented as application timings. The new exporter streams Osmium's output to disk and uses its complete-way extraction, avoiding the original full-tree Python validation.
-
-## Data checks
-
-Both Geofabrik map exports passed `osmium check-refs`, with zero missing nodes in ways. Their counts were:
-
-| Area | Nodes | Ways | Relations |
-| --- | ---: | ---: | ---: |
-| 10 km² | 177,525 | 32,787 | 3,219 |
-| 1,000 km² | 3,172,834 | 469,155 | 21,004 |
-
-All highway way IDs matched between the baseline map and Geofabrik at both sizes: 12,412 and 163,195 respectively. The snapshots are approximately 16 hours apart, and Geofabrik omits contributor metadata. Osmium also retains parent relations, so the files are not expected to have identical object counts or bytes. Distant relation members may remain unresolved; local ways are complete.
-
-A real nearby search returned the Colosseum relation and surrounding POIs from the local file. Resolving Via del Corso's connected street geometry took 2.69 seconds. Offline fixtures cover boundary-crossing ways, disconnected streets with the same name, relation centers, empty nearby results, cancellation, and preserving existing files when refresh or extraction fails.
-
-All 44 tests passed with `python -m unittest discover -s tests -v`. [Machine-readable measurements](osm-benchmark.json) retain the recorded timings, bounds, source dates, and attempt outcomes.
-
-## Reproduce the historical CLI extraction
-
-With the central Italy PBF downloaded as `centro.osm.pbf`:
-
-```sh
-osmium extract centro.osm.pbf -s complete_ways --set-bounds \
- -b 12.473098695094054,41.875980496522295,12.511301304905947,41.904419503477705 \
- -o rome-10.osm
-osmium extract centro.osm.pbf -s complete_ways --set-bounds \
- -b 12.301186950940533,41.74800496522297,12.683213049059468,42.03239503477703 \
- -o rome-1000.osm
-osmium check-refs rome-10.osm
-osmium check-refs rome-1000.osm
-```
diff --git a/src/capture.py b/src/capture.py
index 74d8344..ab66155 100644
--- a/src/capture.py
+++ b/src/capture.py
@@ -43,7 +43,7 @@ DEFAULT_OPTIONS = dict(
OSM_NOTES = {
"map": "OSM XML from Geofabrik with original tags and topology; coordinates are WGS84. Contributor names, IDs and changeset IDs are omitted by Geofabrik.",
- "selection": "Bounding-box selection with complete ways. Relations retain their original member references, but members outside the extract are not downloaded recursively. Objects may extend outside the rectangle; crossings without inside nodes and enclosing polygons may be absent.",
+ "selection": "Bounding-box selection with complete ways and directly selected multipolygons, including outer boundaries and holes. Other relations retain their original references and may remain incomplete. Objects may extend outside the rectangle; crossings without inside nodes and enclosing polygons may be absent.",
"terrain": "Float32 heights in meters, EPSG:3857. Original compressed blocks retained without resampling. Full edge tiles extend beyond the rectangle. Tiles are checkpointed individually; terrain.tif is built after all terrain tiles are saved.",
"quality": "Terrain resolution, dates, accuracy and vertical reference vary by source. Higher zoom does not guarantee more detail.",
}
diff --git a/src/dashboard_osm.py b/src/dashboard_osm.py
index 48d67b0..3ff7dce 100644
--- a/src/dashboard_osm.py
+++ b/src/dashboard_osm.py
@@ -3,9 +3,9 @@
import math
import re
from collections import defaultdict
+from xml.etree import ElementTree as ET
from .geo import feature, ring_contains, signed_area
-from .osm import objects
DISPLAY_TAGS = {"building", "landuse", "natural", "water", "highway", "waterway"}
TAGS = DISPLAY_TAGS | {"name", "height", "building:levels", "min_height", "type"}
@@ -65,6 +65,22 @@ def properties(tags, polygon):
return result
+def objects(path):
+ """Stream complete OSM elements without retaining the XML tree."""
+ try:
+ with path.open("rb") as stream:
+ parser = ET.iterparse(stream, events=("start", "end"))
+ _, root = next(parser)
+ if root.tag != "osm":
+ raise ValueError("OSM file contains no osm root.")
+ for event, obj in parser:
+ if event == "end" and obj.tag in ("node", "way", "relation"):
+ yield obj
+ root.clear()
+ except (ET.ParseError, StopIteration) as error:
+ raise ValueError("OSM file contains invalid XML.") from error
+
+
def read_osm(path):
nodes, ways, relations = {}, {}, []
for element in objects(path):
diff --git a/src/osm.py b/src/osm.py
index d950596..9c20eb4 100644
--- a/src/osm.py
+++ b/src/osm.py
@@ -5,13 +5,13 @@ import json
from collections import defaultdict
from contextlib import nullcontext
from itertools import pairwise
-from xml.etree import ElementTree as ET
from .common import APP_AGENT, RequestError, atomic_path, write_bytes
from .geo import clip, extent, ring_contains, signed_area
-from .pbf import PBF, POI_KEYS
+from .pbf import PBF
INDEX = "https://download.geofabrik.de/index-v1.json"
+POI_KEYS = ("amenity", "tourism", "shop", "leisure", "historic", "office")
def polygons(geometry):
@@ -42,24 +42,6 @@ def region_size(feature):
return sum(abs(signed_area(polygon[0])) for polygon in polygons(feature["geometry"]))
-def objects(path, cancel=lambda: None):
- """Stream complete OSM elements without retaining the XML tree."""
- try:
- with path.open("rb") as stream:
- parser = ET.iterparse(stream, events=("start", "end"))
- _, root = next(parser)
- if root.tag != "osm":
- raise ValueError("OSM file contains no osm root.")
- for index, (event, obj) in enumerate(parser):
- if index % 65536 == 0:
- cancel()
- if event == "end" and obj.tag in ("node", "way", "relation"):
- yield obj
- root.clear()
- except (ET.ParseError, StopIteration) as error:
- raise ValueError("OSM file contains invalid XML.") from error
-
-
class Source:
def __init__(self, client):
self.client = client
@@ -101,13 +83,10 @@ class Source:
self.regions[url] = path, dict(source_url=url, region=props["name"], osm_data_at=stamp or None)
return self.regions[url]
- def extract(self, source, area, output):
- PBF(source, self.client.check_cancel).export(area, output)
-
def export(self, area, output, progress):
source, metadata = self.region(area, progress)
progress("Extracting OSM map")
- self.extract(source, area, output)
+ PBF(source, self.client.check_cancel).export(area, output)
return metadata
def data(self, area, *, poi=False, progress=lambda *args: None):
@@ -145,11 +124,7 @@ class Source:
if not tags.get("highway") or not tags.get("name"):
raise RequestError("place_not_found", "The selected street is absent or unnamed in the Geofabrik snapshot.")
name = tags["name"]
- nodes, identities = set(), set()
- for way in pbf.ways(name=name):
- identities.add(way["id"])
- nodes.update(way["nodes"])
- ways = {w["id"]: w for w in pbf.read((nodes, identities, set()))}
+ ways = {way["id"]: way for way in pbf.ways(name=name)}
by_node = defaultdict(list)
for way in ways.values():
for node in way["nodes"]:
@@ -162,7 +137,8 @@ class Source:
connected.add(current)
for node in ways[current]["nodes"]:
pending.extend(by_node.pop(node, ()))
- selected = [ways[i] for i in sorted(connected)]
+ nodes = {node for identity in connected for node in ways[identity]["nodes"]}
+ selected = pbf.read((nodes, connected, set()))
expanded = extent([area[:2], area[2:]] + [p for w in selected for p in w["points"]])
next_source, _ = self.region(expanded, progress)
if next_source == source:
diff --git a/src/pbf.py b/src/pbf.py
index f23882a..5f4915d 100644
--- a/src/pbf.py
+++ b/src/pbf.py
@@ -25,9 +25,7 @@ from pathlib import Path
from xml.sax.saxutils import quoteattr
MAX_BLOB = 32 * 1024 * 1024
-KINDS = {1: "node", 2: "node", 3: "way", 4: "relation"}
MEMBERS = ("node", "way", "relation")
-POI_KEYS = ("amenity", "tourism", "shop", "leisure", "historic", "office")
SIGNED_SMALL = tuple((v >> 1) ^ -(v & 1) for v in range(128))
SIGNED_BYTES = bytes(((v >> 1) ^ -(v & 1)) & 255 for v in range(256))
MULTIBYTE = re.compile(rb"[\x80-\xff]")
@@ -188,6 +186,14 @@ def signed64(value):
return value - (1 << 64) if value >= 1 << 63 else value
+def deltas(data):
+ """Separate the first absolute value so small deltas use the fast path."""
+ if not data:
+ return iter(())
+ first, offset = varint(data, 0)
+ return accumulate(chain((zigzag(first),), unpack(data[offset:], True)))
+
+
def inflate(data):
blob = dict(fields(data))
if 1 in blob:
@@ -239,9 +245,9 @@ def columns(message):
def dense(message):
values = columns(message)
- ids = list(accumulate(unpack(values.get(1, b""), True)))
- lats = list(accumulate(unpack(values.get(8, b""), True)))
- lons = list(accumulate(unpack(values.get(9, b""), True)))
+ ids = list(deltas(values.get(1, b"")))
+ lats = list(deltas(values.get(8, b"")))
+ lons = list(deltas(values.get(9, b"")))
if not len(ids) == len(lats) == len(lons):
raise ValueError("Mismatched dense PBF coordinate columns.")
return values, ids, lats, lons
@@ -252,8 +258,6 @@ def scan_nodes(task):
values, groups = block(path, entry)
gran = values.get(17, 100)
lat_offset, lon_offset = signed64(values.get(19, 0)), signed64(values.get(20, 0))
- if gran <= 0:
- raise ValueError("Invalid PBF coordinate granularity.")
if area is not None:
south, west, north, east = area
south, north = math.ceil((south * 1e9 - lat_offset) / gran), math.floor((north * 1e9 - lat_offset) / gran)
@@ -262,12 +266,12 @@ def scan_nodes(task):
coordinates = (array("q"), array("q")) if retain else None
for group in groups:
for kind, message in fields(group):
- if kind not in KINDS:
+ if kind not in (1, 2, 3, 4):
raise ValueError("Unsupported OSM primitive in PBF snapshot.")
kinds |= 1 << kind
if kind == 2:
if area is None:
- ids = list(accumulate(unpack(columns(message).get(1, b""), True)))
+ ids = list(deltas(columns(message).get(1, b"")))
else:
_, ids, lats, lons = dense(message)
elif kind == 1:
@@ -325,13 +329,6 @@ def way_references(message):
return refs
-def references(data):
- if not data:
- return iter(())
- first, offset = varint(data, 0)
- return accumulate(chain((zigzag(first),), unpack(data[offset:], True)))
-
-
def scan_ways(task):
"""Select spatial ways or requested member IDs and record the block's ID range."""
path, entry, xml, wanted = task
@@ -350,9 +347,9 @@ def scan_ways(task):
if identity not in wanted:
continue
packed = way_references(message)
- if wanted is not None or not _SPATIAL.isdisjoint(references(packed)):
+ if wanted is not None or not _SPATIAL.isdisjoint(deltas(packed)):
row = columns(message)
- refs = list(references(packed))
+ refs = list(deltas(packed))
if strings is None:
strings = StringTable(values[1])
item = dict(type="way", id=row[1], nodes=refs, tags=tags(row, strings),
@@ -416,13 +413,16 @@ def info(values, strings, date_gran):
return attrs
+def xml_attrs(attrs):
+ # Metadata values other than user names are generated numbers/ISO dates.
+ return "".join(
+ f' {key}={quoted(str(value))}' if key == "user" else f' {key}="{value}"'
+ for key, value in attrs.items())
+
+
def xml_object(item):
kind = item["type"]
- # IDs, coordinates and metadata values are generated numbers/ISO dates.
- # Only user names, tags and member roles need XML escaping.
- attributes = f'id="{item["id"]}"' + "".join(
- f' {key}={quoted(str(value))}' if key == "user" else f' {key}="{value}"'
- for key, value in item.get("attrs", {}).items())
+ attributes = f'id="{item["id"]}"' + xml_attrs(item.get("attrs", {}))
children = []
if kind == "node":
lat, lon = (f"{item[key]:.9f}".rstrip("0").rstrip(".") for key in ("lat", "lon"))
@@ -463,7 +463,7 @@ def read_nodes(task):
node, ids, lats, lons = dense(message)
else:
node = columns(message)
- ids = list(accumulate(unpack(node.get(1, b""), True)))
+ ids = list(deltas(node.get(1, b"")))
lats, lons = [0] * len(ids), [0] * len(ids)
if len(wanted) * 8 < len(ids):
selected = sorted(i for identity in wanted
@@ -495,7 +495,7 @@ def read_nodes(task):
if not data[offset:].strip(b"\0"):
metadata[key] = [zigzag(first)] * (len(data) - offset + 1)
else:
- metadata[key] = list(accumulate(unpack(data, True)))
+ metadata[key] = list(deltas(data))
else:
metadata[key] = unpack(data)
if len(metadata[key]) != len(ids):
@@ -503,8 +503,7 @@ def read_nodes(task):
if mode == "xml":
extra = {k: v for k, v in metadata.items() if k not in (1, 2)}
simple = 1 in metadata and 2 in metadata and all(v.count(v[0]) == len(v) for v in extra.values())
- suffix = "".join(f' {k}={quoted(str(v))}' for k, v in
- info({k: v[0] for k, v in extra.items()}, strings, date_gran).items()) if simple else ""
+ suffix = xml_attrs(info({k: v[0] for k, v in extra.items()}, strings, date_gran)) if simple else ""
precision = 7 if gran % 100 == lat_offset % 100 == lon_offset % 100 == 0 else 9
for i in selected:
lat, lon = (lat_offset + gran * lats[i]) / 1e9, (lon_offset + gran * lons[i]) / 1e9
@@ -513,8 +512,7 @@ def read_nodes(task):
if simple:
attrs = f' version="{metadata[1][i]}" timestamp="{timestamp(metadata[2][i], date_gran)}"{suffix}'
else:
- attrs = "".join(f' {k}={quoted(str(v))}' for k, v in
- info({k: v[i] for k, v in metadata.items()}, strings, date_gran).items())
+ attrs = xml_attrs(info({k: v[i] for k, v in metadata.items()}, strings, date_gran))
head = f'<node id="{ids[i]}"{attrs} lat="{lat:.{precision}f}" lon="{lon:.{precision}f}"'
node_tags = tag_rows.get(i)
output.append(head + ">" + "".join(f'<tag k={quoted(k)} v={quoted(v)}/>'
@@ -634,7 +632,7 @@ class PBF:
if name is not None and (properties.get("name") != name or not properties.get("highway")):
continue
way = dict(type="way", id=row[1], tags=properties,
- nodes=list(accumulate(unpack(row.get(8, b""), True))),
+ nodes=list(deltas(row.get(8, b""))),
attrs=info(dict(fields(row.get(4, b""))), strings, date_gran))
if wanted is not None or name is not None:
self.selected_ways[way["id"]] = way
@@ -644,7 +642,7 @@ class PBF:
rows = ((4, *self.selected_relations[i]) for i in sorted(wanted)) if (
wanted is not None and wanted <= self.selected_relations.keys()) else self.raw((4,), wanted)
for _, row, strings, date_gran in rows:
- refs = list(accumulate(unpack(row.get(9, b""), True)))
+ refs = list(deltas(row.get(9, b"")))
roles, kinds = unpack(row.get(8, b"")), unpack(row.get(10, b""))
if not len(refs) == len(roles) == len(kinds):
raise ValueError("Mismatched PBF relation member columns.")
@@ -691,7 +689,7 @@ class PBF:
for _, row, strings, date_gran in self.raw((4,)):
identity = row[1]
rows[identity] = row, strings, date_gran
- refs = list(accumulate(unpack(row.get(9, b""), True)))
+ refs = list(deltas(row.get(9, b"")))
kinds = unpack(row.get(10, b""))
matched = False
for kind, ref in zip(kinds, refs):
diff --git a/tests/test_osm.py b/tests/test_osm.py
index e77bf90..bf4719a 100644
--- a/tests/test_osm.py
+++ b/tests/test_osm.py
@@ -3,11 +3,11 @@ import json
from pathlib import Path
import tempfile
import unittest
-from unittest.mock import Mock, patch
+from unittest.mock import Mock
from xml.etree import ElementTree as ET
from src.common import Client
-from src.osm import Source, covers
+from src.osm import covers
def polygon(ring, *holes):
@@ -65,15 +65,6 @@ class ExtractTests(unittest.TestCase):
self.assertEqual(self.source.data((1, 1, 1.1, 1.1), poi=True)[0], [])
self.client.get.assert_not_called()
- def test_failed_extraction_keeps_previous_map(self):
- output = self.folder / "map.osm"
- output.write_bytes(b"previous")
- with patch.object(Source, "extract", side_effect=OSError("failed")):
- with self.assertRaises(OSError):
- self.source.export(self.area, output, lambda *args: None)
- self.assertEqual(output.read_bytes(), b"previous")
- self.assertEqual(list(self.folder.glob(".map.osm-*")), [])
-
def test_refresh_rejects_invalid_download_without_replacing_cached_region(self):
original = self.path.read_bytes()
self.client.refresh = True