The OSINT investigator's workbench. One case, one folder, every tool. Local-first: your media and your investigations never leave your machine.
JavaScript
Retry imagery tiles and stabilize browser navigation checks (#111)
OsintMeThat518a4553d4ca
diff --git a/docs/IMAGERY_PROVIDERS.md b/docs/IMAGERY_PROVIDERS.mdindex 69016cd..60a3a3f 100644--- a/docs/IMAGERY_PROVIDERS.md+++ b/docs/IMAGERY_PROVIDERS.md@@ -160,6 +160,32 @@ withheld from `all_providers()`, so the basemap vanishes from the selector (thelive map falls back to Esri) until the key changes (verdict cleared) or a testpasses. Settings shows the stored reason inline.+## Tile retries and failure messages++Three rules, shared by the live proxy and `fetch_crop` so a provider that is+slow on the map is not quietly fast on the evidence.++**A ceiling per provider.** `TILE_TIMEOUT` is 20s; `Provider.timeout` overrides+it, and only Sentinel-1 does (45s). Radar is drawn from the GRD through our+evalscript at request time where the optical layer is served already baked, so+a cold 512px tile regularly outran the shared ceiling.++**One more try, twice.** `MAX_TILE_TRIES` is 3, for a transport error or a 502+/503/504, never for a 4xx (a verdict about the request) and never for a 500+(the service's verdict on this render — on 2026-10-07 every Sentinel-1 tile+answered 500 for hours, identically). Retries cost nothing on the meter, which+only ever counts a 2xx. Before this, one slow tile left a hole in the live map+and took a whole *capture* down with it.++**The provider's own sentence reaches the panel.** Sentinel Hub answers a+render it could not do with an OGC `ExceptionReport`; `upstream_reason()`+unwraps it from its Java trace, strips addresses and credentials, caps it at+240 characters and the proxy returns *that* as the failed tile's body, as+`text/plain`. The engine hands it to the map as a Blob, so the reason arrives a+tick after the trouble. Without it, a Copernicus outage is indistinguishable+from a bug in Azimut — the DEM outage above read as "try again" for an hour.+Our own sentences still win for 401/403 and 429, which name what to go and do.+## Per-provider eco thresholds`eco_max_zooms` in settings.json lets each tile basemap carry its own eco@@ -475,6 +501,14 @@ why. One request per probe, on the meter. The first layer that answers is kept a`sentinel1_layer` in settings.json, carried by the backup, and forgotten when theinstance id changes. It can also be named and checked by hand.+**Once it is known, it leaves the display list.** The same silence in+GetCapabilities that makes Find necessary means the instance hands the radar+layer back among the optical ones, often first, so+`/api/satellite/sentinel/layers` drops `sentinel1_layer` from what it offers —+picked as a Sentinel-2 display it would render radar dated and cloud-filtered by+a calendar that is not its own. Radar is reached through `sentinel1_layer`+alone; nothing of its road reads that list.+- **A pass is a day and a time.** Sentinel-1 passes a place at dawn flying southand at dusk flying north, so one day can hold two looks from opposite sides.Radar lookups (`collection=sentinel1`) group granules by pass rather than bydiff --git a/frontend/e2e/command-palette.spec.js b/frontend/e2e/command-palette.spec.jsindex 69884c1..ee3a4d2 100644--- a/frontend/e2e/command-palette.spec.js+++ b/frontend/e2e/command-palette.spec.js@@ -39,6 +39,16 @@ async function find(page, term) {await page.keyboard.press('Control+k');await expect(search(page)).toBeFocused();await search(page).fill(term);+ // Recently added rows may already match the words, before the search answers.+ await expect(dialog(page).locator('.search')).not.toHaveClass(/busy/);+}++async function chooseProof(page, name) {+ await find(page, name);+ await expect(dialog(page).getByRole('option', { name: new RegExp(name) }))+ .toHaveAttribute('aria-selected', 'true');+ await page.keyboard.press('Enter');+ await expect(dialog(page)).toBeHidden();}test('Ctrl+K reads only on opening and reaches a tool with the keyboard', async ({ page }) => {@@ -102,6 +112,8 @@ test('a case result changes the case and opens its overview', async ({ page }) =});async function proofCase(page) {+ // This path includes the composer's first import and two navigation guards.+ test.setTimeout(60_000);const proofs = { 'proof-a': 'Proof A', 'proof-b': 'Proof B' };const catalog = Object.entries(proofs).map(([name, label]) => ({id: `${name}-entity`, type: 'proof', label, attrs: { spec: `proofs/.meta/${name}.json` },@@ -115,10 +127,9 @@ async function proofCase(page) {if (route.request().method() !== 'GET' || !proofs[name]) return route.fallback();return route.fulfill({ json: { title: proofs[name], panels: [], pastes: [], shapes: [] } });});- await find(page, 'Proof A');- await dialog(page).getByRole('option', { name: /Proof A/ }).click();+ await chooseProof(page, 'Proof A');const title = page.getByRole('textbox', { name: 'Proof name' });- await expect(title).toHaveValue('Proof A', { timeout: 15_000 });+ await expect(title).toHaveValue('Proof A');await title.fill('Proof A revised');await expect(page.locator('.tool-header .badge', { hasText: 'unsaved' })).toBeVisible();return { fixture, title };@@ -126,8 +137,7 @@ async function proofCase(page) {test('opening another proof retains its existing unsaved-work confirmation', async ({ page }) => {const { fixture, title } = await proofCase(page);- await find(page, 'Proof B');- await dialog(page).getByRole('option', { name: /Proof B/ }).click();+ await chooseProof(page, 'Proof B');const ask = page.getByRole('alertdialog', { name: 'Leave this proof?' });await expect(ask).toBeVisible();await ask.getByRole('button', { name: 'Keep editing' }).click();diff --git a/frontend/e2e/detect.spec.js b/frontend/e2e/detect.spec.jsindex a0a63f6..ec193ea 100644--- a/frontend/e2e/detect.spec.js+++ b/frontend/e2e/detect.spec.js@@ -908,7 +908,7 @@ test('failed check imagery says why on both maps and can be retried or replacedawait answerBuilder(page);let failing = true;await page.route('**/api/tiles/sentinel2*/**', (route) => failing- ? route.fulfill({ status: 400, contentType: 'application/xml', body: '<ServiceException>Layer not found</ServiceException>' })+ ? route.fulfill({ status: 400, contentType: 'text/plain', body: 'Layer not found' }): route.fallback());await page.getByRole('button', { name: 'Analyzers', exact: true }).click();await page.getByRole('button', { name: 'New analyzer', exact: true }).click();@@ -921,7 +921,7 @@ test('failed check imagery says why on both maps and can be retried or replacedawait deck.getByRole('button', { name: 'Place the pins' }).click();const notices = page.getByRole('status', { name: 'Imagery loading error' });await expect(notices).toHaveCount(2);- await expect(notices.first()).toContainText('The imagery service refused this display layer.');+ await expect(notices.first()).toContainText('Layer not found');await expect(notices.first()).toContainText('TRUE_COLOR');await expect(notices.first()).toContainText('2026-09-01');await expect(notices.last()).toContainText('2026-09-10');diff --git a/frontend/e2e/graph.spec.js b/frontend/e2e/graph.spec.jsindex 632a332..7b087aa 100644--- a/frontend/e2e/graph.spec.js+++ b/frontend/e2e/graph.spec.js@@ -116,6 +116,11 @@ async function inkLost(page) {*/async function drawn(page) {await expect.poll(() => ink(page), { timeout: 5000 }).toBeGreaterThan(0.001);+ // Existing pixels can belong to the previous camera or selection. Let Konva+ // paint its queued frame before reading pixels or using its canvas hit regions.+ await page.evaluate(() => new Promise((resolve) => {+ requestAnimationFrame(() => requestAnimationFrame(resolve));+ }));}/**@@ -962,6 +967,7 @@ test('the walk narrows the drawing to itself and what it can reach', async ({ paawait page.getByRole('button', { name: 'Path to…' }).click();await page.locator('.panel .link-row', { hasText: '3rd Battalion' }).click();+ await expect(steps(page)).toHaveCount(2);await expect.poll(() => ink(page)).not.toBeCloseTo(before, 4);// And the case comes back whole when the walk is given up.@@ -1632,7 +1638,7 @@ test('opening a node leaves everything already drawn exactly where it is', async// Not "roughly the same picture": every node is where it was, so every pixel that// was drawn is still drawn. What the arrival adds is deliberately not counted.// Without the drawing being held, this reads 0.96 — the whole case moves.- expect(await inkLost(page)).toBeLessThan(0.05);+ await expect.poll(() => inkLost(page)).toBeLessThan(0.05);});test('the whole picture is put back by one control, not one per list', async ({ page }) => {diff --git a/frontend/src/lib/map/basemap.js b/frontend/src/lib/map/basemap.jsindex d3797ae..eabb324 100644--- a/frontend/src/lib/map/basemap.js+++ b/frontend/src/lib/map/basemap.js@@ -29,6 +29,8 @@ const DEFAULT_SUBDOMAINS = ['a', 'b', 'c'];const IMAGERY = 'basemap-imagery';/** A second picture of the same ground, laid just over the first (`setAlternate`). */const ALTERNATE = 'basemap-alternate';+/** Failed tiles after which trouble is reported without waiting for the settle. */+const TROUBLE_SPEAKS_UP = 4;/*** What can be laid over the imagery, in the order they stack — first is@@ -274,13 +276,33 @@ export function rasterSource(provider, providerId, cell) {};}-export function imageryError(error) {+/**+ * A provider's sentence, shown only if it still looks like one.+ *+ * The proxy strips addresses and credentials before it sends this on, so the+ * checks here are the second pair of eyes rather than the first: a page is+ * where a leaked instance id would end up in a screenshot.+ */+export function readableReason(text) {+ const reason = (text ?? '').trim();+ if (!reason || reason.length > 240) return '';+ return /https?:\/\//i.test(reason) ? '' : reason;+}++export function imageryError(error, failed = 1, reason = '') {const status = error?.status;if (status === 404) return 'No imagery is available here for this pass.';- if (status === 400 || status === 422) return 'The imagery service refused this display layer.';+ // Ours name what to go and do about it, which the service cannot know. Its+ // own sentence wins everywhere else, including a refused layer, where what+ // it objected to is the whole answer.if (status === 401 || status === 403) return 'The imagery service refused the key; check Settings → Imagery.';if (status === 429) return 'The imagery allowance is exhausted; check Settings → Imagery.';- return 'The imagery could not be loaded; try again.';+ if (reason) return reason;+ if (status === 400 || status === 422) return 'The imagery service refused this display layer.';+ // How much of the view is missing is the difference between a hole to pan+ // past and a picture that never came, so the count is the message.+ if (failed > 1) return `${failed} tiles could not be loaded; try again.`;+ return 'A tile could not be loaded; try again.';}/**@@ -353,8 +375,40 @@ export function createBasemaps(engine, hooks = {}) {return above ? overlayLayers(above)[0] : ceiling();}+ // Tiles that failed since the imagery source last settled, by source id.+ // The report is a verdict on a whole settle rather than on one tile: a lone+ // hole in a full screen is overzoomed away and worth no panel, and a panel+ // that outlives the trouble it named is worse than no panel at all.+ const troubles = new Map();++ function report(source, key) {+ const trouble = troubles.get(source);+ troubles.delete(source);+ const tell = (reason) =>+ onImageryTrouble({+ id: key.slice(0, key.lastIndexOf('@')),+ error: trouble?.error ?? null,+ failed: trouble?.failed ?? 0,+ reason,+ });+ // The proxy puts the provider's own account of a refusal in the body of+ // the failed tile, and the engine hands that over as a Blob — so when+ // there is one to read, the reason arrives a tick after the trouble.+ const body = trouble?.error?.body;+ if (body && typeof body.text === 'function') {+ body.text().then(+ (text) => tell(readableReason(text)),+ () => tell(''),+ );+ } else {+ tell('');+ }+ }+function onSourceData(event) {if (!event.isSourceLoaded) return;+ const key = event.sourceId === IMAGERY ? live : event.sourceId === ALTERNATE ? alternate : null;+ if (key) report(event.sourceId, key);// every visible tile is in, so what the proxy counted is now finalconst billed = event.sourceId === IMAGERY ? metered : event.sourceId === ALTERNATE ? alternateMetered : null;if (billed) onMeteredTiles(billed);@@ -364,7 +418,14 @@ export function createBasemaps(engine, hooks = {}) {function onError(event) {const source = event?.sourceId;const key = source === IMAGERY ? live : source === ALTERNATE ? alternate : null;- if (key) onImageryTrouble({ id: key.slice(0, key.lastIndexOf('@')), error: event.error });+ if (key) {+ const trouble = troubles.get(source) ?? { error: null, failed: 0 };+ troubles.set(source, { error: event.error, failed: trouble.failed + 1 });+ // A source with nothing left to show may never report itself loaded, so+ // the settle that would speak for it never comes. Past this many the+ // trouble is the picture, not a gap in it: say so without waiting.+ if (trouble.failed + 1 >= TROUBLE_SPEAKS_UP) report(source, key);+ }const overlay = source && OVERLAYS.find((entry) => overlaySourceIds(entry).includes(source));if (overlay) onOverlayTrouble(overlay.id);}diff --git a/frontend/src/lib/map/basemap.test.js b/frontend/src/lib/map/basemap.test.jsindex 6196e04..68406a0 100644--- a/frontend/src/lib/map/basemap.test.js+++ b/frontend/src/lib/map/basemap.test.js@@ -1,6 +1,14 @@// @vitest-environment happy-domimport { describe, expect, it, vi } from 'vitest';-import { OVERLAY_IDS, imageryError, rasterSource, sourceMaxZoom, tileTemplate, tileUrls } from './basemap.js';+import {+ OVERLAY_IDS,+ imageryError,+ rasterSource,+ readableReason,+ sourceMaxZoom,+ tileTemplate,+ tileUrls,+} from './basemap.js';/*** The provider shapes are the ones `/api/satellite/providers` really answers@@ -55,7 +63,32 @@ it('explains imagery failures without exposing the upstream URL or response', ()expect(imageryError({ status: 404 })).toContain('No imagery');expect(imageryError({ status: 403 })).toContain('key');expect(imageryError({ status: 429 })).toContain('allowance');- expect(imageryError(new Error('private upstream URL'))).toBe('The imagery could not be loaded; try again.');+ expect(imageryError(new Error('private upstream URL'))).toBe('A tile could not be loaded; try again.');+ // how much of the view is missing is the difference between a hole and a blank+ expect(imageryError(new Error('private upstream URL'), 7)).toBe(+ '7 tiles could not be loaded; try again.'+ );+ // a named failure says why, whatever it took down with it+ expect(imageryError({ status: 429 }, 12)).toContain('allowance');+});++it('prefers what the service said to anything we could write for it', () => {+ const said = 'Could not load necessary digital elevation model.';+ // without it a Copernicus outage is indistinguishable from a bug in Azimut+ expect(imageryError({ status: 500 }, 9, said)).toBe(said);+ expect(imageryError({ status: 400 }, 1, said)).toBe(said);+ // except where ours names what to go and do, which the service cannot know+ expect(imageryError({ status: 403 }, 1, said)).toContain('Settings');+ expect(imageryError({ status: 429 }, 1, said)).toContain('Settings');+});++it('shows a provider sentence only while it still looks like one', () => {+ expect(readableReason(' Orthorectification failed. ')).toBe('Orthorectification failed.');+ expect(readableReason('')).toBe('');+ expect(readableReason(null)).toBe('');+ // a second pair of eyes on what the proxy already stripped+ expect(readableReason('failed for https://services.sentinel-hub.com/ogc/wmts/abc')).toBe('');+ expect(readableReason('x'.repeat(241))).toBe('');});describe('where a provider’s tiles are fetched from', () => {@@ -451,11 +484,19 @@ describe('the layers on a map', () => {basemaps.show(SENTINEL, id, 512);basemaps.setAlternate(SENTINEL, 'sentinel2~SWIR', 512);const [, onError] = map.calls.on.find(([name]) => name === 'error');+ const settle = map.calls.on.find(([name]) => name === 'sourcedata')[1];const error = { status: 400 };onError({ sourceId: 'basemap-imagery', error });onError({ sourceId: 'basemap-alternate', error });onError({ sourceId: 'basemap-roads', error });- expect(trouble.mock.calls).toEqual([[{ id, error }], [{ id: 'sentinel2~SWIR', error }]]);+ // one tile is a gap, not a verdict: the view it belongs to speaks first+ expect(trouble).not.toHaveBeenCalled();+ settle({ sourceId: 'basemap-imagery', isSourceLoaded: true });+ settle({ sourceId: 'basemap-alternate', isSourceLoaded: true });+ expect(trouble.mock.calls).toEqual([+ [{ id, error, failed: 1, reason: '' }],+ [{ id: 'sentinel2~SWIR', error, failed: 1, reason: '' }],+ ]);const source = map.sources.get('basemap-imagery');basemaps.show(SENTINEL, id, 512);expect(map.sources.get('basemap-imagery')).toBe(source);@@ -465,6 +506,62 @@ describe('the layers on a map', () => {});});+ it('takes back the trouble it reported once a settle comes through clean', async () => {+ await withStubbedGoogle(async ({ createBasemaps }) => {+ const map = stubMap();+ const trouble = vi.fn();+ const basemaps = createBasemaps(stubEngine(map), { onImageryTrouble: trouble });+ const id = 'sentinel2~TRUE_COLOR~2026-09-25~2026-09-25';+ basemaps.show(SENTINEL, id, 512);+ const [, onError] = map.calls.on.find(([name]) => name === 'error');+ const settle = map.calls.on.find(([name]) => name === 'sourcedata')[1];+ onError({ sourceId: 'basemap-imagery', error: { status: 500 } });+ settle({ sourceId: 'basemap-imagery', isSourceLoaded: true });+ expect(trouble).toHaveBeenLastCalledWith({ id, error: { status: 500 }, failed: 1, reason: '' });+ // the analyst pans, the tiles come through: nothing is wrong any more+ settle({ sourceId: 'basemap-imagery', isSourceLoaded: true });+ expect(trouble).toHaveBeenLastCalledWith({ id, error: null, failed: 0, reason: '' });+ });+ });++ it('passes on what the provider said about the tile it refused', async () => {+ await withStubbedGoogle(async ({ createBasemaps }) => {+ const map = stubMap();+ const trouble = vi.fn();+ const basemaps = createBasemaps(stubEngine(map), { onImageryTrouble: trouble });+ const id = 'sentinel1~2026-10-06~151800';+ basemaps.show(SENTINEL, id, 512);+ const [, onError] = map.calls.on.find(([name]) => name === 'error');+ const settle = map.calls.on.find(([name]) => name === 'sourcedata')[1];+ // what the proxy hands over: the service's own sentence, already stripped+ const said = 'Could not load necessary digital elevation model.';+ const error = { status: 500, body: { text: () => Promise.resolve(said) } };++ onError({ sourceId: 'basemap-imagery', error });+ settle({ sourceId: 'basemap-imagery', isSourceLoaded: true });+ await Promise.resolve();+ await Promise.resolve();++ expect(trouble).toHaveBeenLastCalledWith({ id, error, failed: 1, reason: said });+ });+ });++ it('says so without waiting when enough tiles fail that nothing will settle', async () => {+ await withStubbedGoogle(async ({ createBasemaps }) => {+ const map = stubMap();+ const trouble = vi.fn();+ const basemaps = createBasemaps(stubEngine(map), { onImageryTrouble: trouble });+ const id = 'sentinel2~TRUE_COLOR~2026-09-25~2026-09-25';+ basemaps.show(SENTINEL, id, 512);+ const [, onError] = map.calls.on.find(([name]) => name === 'error');+ const error = { status: 502 };+ for (const _ of [1, 2, 3]) onError({ sourceId: 'basemap-imagery', error });+ expect(trouble).not.toHaveBeenCalled();+ onError({ sourceId: 'basemap-imagery', error });+ expect(trouble.mock.calls).toEqual([[{ id, error, failed: 4, reason: '' }]]);+ });+ });+it('adds an overlay once and takes it off when asked', async () => {await withStubbedGoogle(async ({ createBasemaps }) => {const map = stubMap();diff --git a/frontend/src/tools/satellite/MapSurface.svelte b/frontend/src/tools/satellite/MapSurface.svelteindex b8132c5..16b570c 100644--- a/frontend/src/tools/satellite/MapSurface.svelte+++ b/frontend/src/tools/satellite/MapSurface.svelte@@ -207,7 +207,16 @@onWidgetAuthFailure: onwidgetauthfailure,onWidgetFailed: onwidgetfailed,onOverlayTrouble: (id) => onoverlaytrouble(id),- onImageryTrouble: (failure) => { tileFailure = { id: failure.id, message: imageryError(failure.error) }; },+ // One report per settle: tiles that came through clear the panel the+ // ones before them raised, so it never outlives the trouble it named.+ onImageryTrouble: (failure) => {+ tileFailure = failure.failed+ ? {+ id: failure.id,+ message: imageryError(failure.error, failure.failed, failure.reason),+ }+ : null;+ },});basemaps.setZoomCeiling(zoomCeiling);showBasemap();@@ -450,13 +459,20 @@</div><style>+ /* Clear of the left column, not over it and not under it: the zoom buttons+ (12px in, 29px wide) and a tool's own rail live there, and whichever wins+ the stack the analyst loses — either the panel is unreadable or the+ buttons are unpressable. Reading why the imagery failed and zooming out of+ it are the same gesture, so both stay available. */.tile-trouble {- position: absolute; top: 154px; left: 12px; z-index: 600;- display: grid; gap: 6px; max-width: min(330px, calc(100% - 24px));+ position: absolute; top: 154px; left: 53px; z-index: 600;+ display: grid; gap: 6px; max-width: min(330px, calc(100% - 65px));padding: 12px; border: 1px solid var(--warn); border-radius: var(--r-sm);background: var(--bg-1); color: var(--text-1); font-size: var(--fs-xs);}- .tile-trouble.right { left: auto; right: 12px; }+ .tile-trouble.right {+ left: auto; right: 12px; max-width: min(330px, calc(100% - 24px));+ }.tile-trouble span, .tile-trouble p { margin: 0; color: var(--text-2); }.error-actions { display: flex; flex-wrap: wrap; gap: 7px; }.map-wrap {diff --git a/src/azimut/api/satellite.py b/src/azimut/api/satellite.pyindex e760006..f71a236 100644--- a/src/azimut/api/satellite.py+++ b/src/azimut/api/satellite.py@@ -67,7 +67,7 @@ def _client() -> httpx.Client:if _tile_client is None:_tile_client = httpx.Client(follow_redirects=True,- timeout=20,+ timeout=tiles.TILE_TIMEOUT,limits=httpx.Limits(max_keepalive_connections=16, max_connections=32),headers={"User-Agent": tiles.USER_AGENT},)@@ -245,13 +245,24 @@ def _offered_layers(found: list[dict[str, Any]]) -> list[dict[str, Any]]:back to. They carry ``custom`` so a picker can offer them even where theinstance has not been asked — a script on this machine needs no permissionfrom GetCapabilities to exist — and so the editor knows which rows it owns.++ The Sentinel-1 layer is dropped. Radar lives in the same configuration as+ the optical layers and GetCapabilities never says which collection a layer+ reads, so the instance hands it back among the rest; offered here it would+ be picked as a Sentinel-2 display, dated and cloud-filtered by a calendar+ that is not its own. Radar is reached through ``sentinel1_layer``, never+ through this list."""- customs = sentinel.custom_layers(config.load_settings())+ settings = config.load_settings()+ customs = sentinel.custom_layers(settings)+ radar = str(settings.get("sentinel1_layer") or "")names = {entry["id"] for entry in customs}kept = [entryfor entry in found- if entry.get("id") not in names and not sentinel.is_draft_layer(entry.get("id", ""))+ if entry.get("id") not in names+ and entry.get("id") != radar+ and not sentinel.is_draft_layer(entry.get("id", ""))]return kept + [{"id": entry["id"], "label": entry["label"], "hint": entry["hint"], "custom": True}@@ -673,7 +684,8 @@ def _serve_tile(return cached[0], cached[1], {"Cache-Control": "private, max-age=86400"}upstream: httpx.Response | None = None- for attempt in (1, 2):+ for attempt in range(1, tiles.MAX_TILE_TRIES + 1):+ last = attempt == tiles.MAX_TILE_TRIEStry:url = tiles.tile_url(tiles.resolve_url(provider), z, x, y, provider.zoom_offset)except tiles.TileFetchError as exc:@@ -686,13 +698,21 @@ def _serve_tile(config.record_provider_status(provider.meter, False, str(exc.__cause__))raise HTTPException(status_code=502, detail=str(exc)) from exctry:- upstream = _client().get(url)+ upstream = _client().get(url, timeout=provider.timeout or tiles.TILE_TIMEOUT)except httpx.HTTPError as exc:- raise HTTPException(status_code=502, detail=f"tile fetch failed: {tiles.upstream_failure(exc)}") from exc+ # A renderer that misses its window on a cold tile usually answers+ # the moment after, with its own cache now warm. Asking again beats+ # a hole the viewer has to notice and press Retry on.+ if last:+ raise HTTPException(status_code=502, detail=f"tile fetch failed: {tiles.upstream_failure(exc)}") from exc+ continue# a stale Google session token answers 401/403 — re-mint once, transparentlyif attempt == 1 and provider.session and upstream.status_code in (401, 403):google_tiles.invalidate(google_tiles.key_from_url(provider.url))continue+ if upstream.status_code in tiles.TRANSIENT_STATUSES and not last:+ time.sleep(tiles.RETRY_PAUSE)+ continuebreakassert upstream is not None # the loop always assigns it before breaking@@ -706,10 +726,13 @@ def _serve_tile(config.record_provider_status(provider.meter, False, google_tiles.error_message(upstream))+ # The provider's own sentence, not the bytes it came in: a failed tile+ # is never drawn, so the body is only ever read as an explanation, and+ # the raw one carries its address and our credentials with it.return Response(- content=upstream.content,+ content=tiles.upstream_reason(upstream.content).encode("utf-8"),status_code=upstream.status_code,- media_type=upstream.headers.get("content-type", "image/png"),+ media_type="text/plain; charset=utf-8",)if provider.meter:config.record_usage(provider.meter, 1)diff --git a/src/azimut/engine/tiles.py b/src/azimut/engine/tiles.pyindex 6828dcc..36d9398 100644--- a/src/azimut/engine/tiles.py+++ b/src/azimut/engine/tiles.py@@ -12,6 +12,7 @@ from __future__ import annotationsimport hashlibimport mathimport re+import timefrom concurrent.futures import ThreadPoolExecutorfrom dataclasses import dataclass, field, replacefrom datetime import datetime, timezone@@ -32,6 +33,18 @@ SIZE_MAX = 4096 # hard cap on a capture's width/height, in pxMAX_TILES_PER_CROP = (SIZE_MAX // TILE_SIZE) ** 2USER_AGENT = "Azimut/0.1 (+local OSINT workbench; single-user)"+# How long one tile may take, and how many times it may be asked for. Both the+# live proxy and the capture stitcher work to these, so a provider that is slow+# on the map is not quietly fast on the evidence.+TILE_TIMEOUT = 20.0+MAX_TILE_TRIES = 3+# Upstream answers worth asking again: a gateway hiccup, or a renderer that ran+# out of time. A 4xx is a verdict about the request and is never retried. A+# retry costs nothing on the meter — only a 2xx is ever counted.+TRANSIENT_STATUSES = frozenset({502, 503, 504})+RETRY_PAUSE = 0.5 # before asking an overloaded service again, in seconds+RADAR_TILE_TIMEOUT = 45.0 # Sentinel-1, rendered from the GRD per request+@dataclass(frozen=True)class Provider:@@ -73,6 +86,11 @@ class Provider:widget: str | None = None# Provider-specific eco threshold; None uses the global setting.eco_max_zoom: int | None = None+ # Seconds one tile may take, when this provider needs longer than+ # TILE_TIMEOUT. A service that renders on demand rather than serving a tile+ # it already baked answers in its own time, and the viewer is waiting+ # either way: a tile that arrives late beats a hole that arrives on time.+ timeout: float | None = None# Built-in keyed providers (docs/IMAGERY_PROVIDERS.md): only surfaced from@@ -279,6 +297,11 @@ def all_providers() -> list[Provider]:tile_size=512,zoom_offset=1,eco_max_zoom=SENTINEL_ECO_MAX_ZOOM,+ # Radar is drawn from the GRD through our evalscript at request+ # time, where the optical layer is served from one Sentinel Hub+ # already baked. A cold 512px tile regularly outruns the shared+ # ceiling, which is what left holes in the map it was asked for.+ timeout=RADAR_TILE_TIMEOUT,))@@ -475,6 +498,42 @@ def is_placeholder_tile(content: bytes) -> bool:_ADDRESS = re.compile(r"https?://\S+")+# An instance id, a Mapbox token, a Google key: the three shapes a credential+# takes in this app, in case one is ever quoted back outside a URL.+_SECRET = re.compile(+ r"\b[0-9a-f]{8}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{12}\b|\bpk\.[\w.-]+|\bAIza[\w-]+",+ re.I,+)+_EXCEPTION_TEXT = re.compile(r"<(?:\w+:)?ExceptionText>(.*?)</(?:\w+:)?ExceptionText>", re.S)+# "java.lang.RuntimeException: " and the chain of wrappers in front of it: the+# service's plumbing, never the analyst's business.+_THROWN = re.compile(r"^[\w.$]+(?:Exception|Error):\s*")+REASON_MAX = 240 # one sentence in a panel, not a stack trace+++def upstream_reason(content: bytes) -> str:+ """The provider's own account of a refusal, fit to put in front of someone.++ A service that says why is worth more than any sentence we could write for+ it: Sentinel Hub answers a render it could not do with an OGC+ ExceptionReport naming what was missing, and without it a Copernicus outage+ is indistinguishable from a bug in Azimut. Quoted raw it would be a Java+ trace, and anything a provider echoes may carry what we sent it, so it is+ unwrapped, stripped of addresses and credentials, and capped.++ Empty when the body holds no such report — the caller then says what it+ can from the status alone.+ """+ found = _EXCEPTION_TEXT.search(content[:4096].decode("utf-8", "replace"))+ if not found:+ return ""+ reason = " ".join(found.group(1).split())+ while True:+ unwrapped = _THROWN.sub("", reason)+ if unwrapped == reason:+ break+ reason = unwrapped+ return _SECRET.sub("<id>", _ADDRESS.sub("<address>", reason))[:REASON_MAX]def upstream_failure(exc: BaseException) -> str:@@ -518,8 +577,24 @@ def tile_url(url_template: str, z: int, x: int, y: int, zoom_offset: int = 0) ->def _default_fetch(client: httpx.Client, url: str) -> Image.Image | None:"""Fetch one tile; None for 'no imagery here' (404 or a known placeholder- tile), raise on other errors."""- response = client.get(url)+ tile), raise on other errors.++ A timeout or a gateway answer is asked again rather than raised: one slow+ tile used to take the whole capture down with it, which is the worst way+ to lose a scene that was only ever going to be available for this pass.+ """+ for attempt in range(1, MAX_TILE_TRIES + 1):+ last = attempt == MAX_TILE_TRIES+ try:+ response = client.get(url)+ except httpx.HTTPError:+ if last:+ raise+ continue+ if response.status_code in TRANSIENT_STATUSES and not last:+ time.sleep(RETRY_PAUSE)+ continue+ breakif response.status_code == 404:return Noneresponse.raise_for_status()@@ -642,7 +717,9 @@ def fetch_crop(return tx, ty, fetch(client, url)with httpx.Client(- headers={"User-Agent": USER_AGENT}, timeout=20, follow_redirects=True+ headers={"User-Agent": USER_AGENT},+ timeout=provider.timeout or TILE_TIMEOUT,+ follow_redirects=True,) as client:with ThreadPoolExecutor(max_workers=6) as pool:return list(pool.map(lambda xy: grab(*xy), coords))@@ -683,7 +760,7 @@ def fetch_crop(try:filled, parent_served = _overzoom_fill(resolve_url(provider), gaps, tile_z, native_ts, fetch,- provider.zoom_offset, out_size=ts,+ provider.zoom_offset, out_size=ts, timeout=provider.timeout,)except TileFetchError:filled, parent_served = {}, 0@@ -789,6 +866,7 @@ def _overzoom_fill(fetch: Callable[[httpx.Client, str], Image.Image | None],zoom_offset: int = 0,out_size: int | None = None,+ timeout: float | None = None,) -> tuple[dict[tuple[int, int], Image.Image], int]:"""Best-effort fill for missing tiles from parent-level imagery.@@ -807,7 +885,9 @@ def _overzoom_fill(unresolved = list(gaps)served = 0with httpx.Client(- headers={"User-Agent": USER_AGENT}, timeout=20, follow_redirects=True+ headers={"User-Agent": USER_AGENT},+ timeout=timeout or TILE_TIMEOUT,+ follow_redirects=True,) as client:for up in range(1, OVERZOOM_LEVELS + 1):sub = ts >> up # the child's footprint inside its parent, in pxdiff --git a/tests/test_sentinel_custom_layers.py b/tests/test_sentinel_custom_layers.pyindex 50b7e54..f754a23 100644--- a/tests/test_sentinel_custom_layers.py+++ b/tests/test_sentinel_custom_layers.py@@ -348,6 +348,36 @@ def test_the_editor_is_told_which_layer_is_the_radar_one(client):assert client.get("/api/satellite/sentinel/custom-layers").json()["radar_layer"] == "RADAR"+def test_the_radar_layer_is_never_offered_as_a_display(client, monkeypatch):+ """Radar lives in the same configuration as the optical layers and+ GetCapabilities does not say which collection a layer reads, so the+ instance hands it back among the rest — often first. Offered as a display+ it would be picked, dated and cloud-filtered by a calendar that is not its+ own; the radar road reads ``sentinel1_layer``, never this list.+ """+ config.update_settings(+ lambda settings: settings.update(+ api_keys={"sentinelhub": "inst-uuid"}, sentinel1_layer="RADAR"+ )+ )+ monkeypatch.setattr(+ sentinel,+ "capabilities_layers",+ lambda *args, **kw: [+ {"id": "RADAR", "label": "RADAR", "hint": ""},+ {"id": "TRUE_COLOR", "label": "True colour", "hint": ""},+ ],+ )++ body = client.get("/api/satellite/sentinel/layers?check=true").json()+ assert body["source"] == "instance"+ assert [entry["id"] for entry in body["layers"]] == ["TRUE_COLOR"]+ # so the first entry every picker falls back to is an optical layer again+ assert body["layers"][0]["id"] == sentinel.DEFAULT_LAYER+ # and nothing was taken away from radar: it is reached from settings+ assert config.load_settings()["sentinel1_layer"] == "RADAR"++def test_the_route_takes_exactly_the_body_the_editor_sends(client):"""The editor's draft holds more than a layer, and this route forbids whatit does not know. The two lists are written out separately, so they arediff --git a/tests/test_settings_api.py b/tests/test_settings_api.pyindex 7dce754..6449c24 100644--- a/tests/test_settings_api.py+++ b/tests/test_settings_api.py@@ -170,6 +170,117 @@ def test_tile_proxy_unknown_provider_404(client):assert client.get("/api/tiles/nope/14/0/0").status_code == 404+def _flaky_upstream(monkeypatch, failures, content=b"JPEGDATA"):+ """An upstream that fails *failures* times (a status, or an exception to+ raise) before serving a tile. Returns the list of urls it was asked for."""+ from azimut.engine import tiles++ monkeypatch.setattr(tiles, "RETRY_PAUSE", 0) # the wait is real, not the test's+ asked: list[str] = []++ def get(url, **kwargs):+ asked.append(url)+ if len(asked) <= len(failures):+ trouble = failures[len(asked) - 1]+ if isinstance(trouble, Exception):+ raise trouble+ return httpx.Response(+ trouble, content=b"", headers={"content-type": "text/plain"},+ request=httpx.Request("GET", url),+ )+ return httpx.Response(+ 200, content=content, headers={"content-type": "image/jpeg"},+ request=httpx.Request("GET", url),+ )++ _use_tile_upstream(monkeypatch, get)+ return asked+++def test_tile_proxy_asks_again_for_a_tile_that_timed_out(client, monkeypatch):+ """A renderer that misses its window usually answers the moment after. The+ tile the analyst asked for beats the hole they would have to notice."""+ client.put("/api/settings/keys", json={"mapbox": "pk.abc"})+ asked = _flaky_upstream(monkeypatch, [httpx.ReadTimeout("too slow")])++ r = client.get("/api/tiles/mapbox-satellite/14/8298/5639")+ assert r.status_code == 200+ assert r.content == b"JPEGDATA"+ assert len(asked) == 2+ # the lost attempt was never served, so it is not on the meter+ body = client.get("/api/settings").json()+ assert body["usage"]["mapbox"][body["month"]] == 1+++def test_tile_proxy_asks_again_after_a_gateway_answer(client, monkeypatch):+ client.put("/api/settings/keys", json={"mapbox": "pk.abc"})+ asked = _flaky_upstream(monkeypatch, [503, 502])++ r = client.get("/api/tiles/mapbox-satellite/14/8298/5639")+ assert r.status_code == 200+ assert len(asked) == 3 # tiles.MAX_TILE_TRIES, all of them used+ body = client.get("/api/settings").json()+ assert body["usage"]["mapbox"][body["month"]] == 1+++def test_tile_proxy_gives_up_rather_than_asking_forever(client, monkeypatch):+ from azimut.engine import tiles++ client.put("/api/settings/keys", json={"mapbox": "pk.abc"})+ asked = _flaky_upstream(monkeypatch, [503] * 5)++ r = client.get("/api/tiles/mapbox-satellite/14/8298/5639")+ assert r.status_code == 503 # the provider's own answer, passed through+ assert len(asked) == tiles.MAX_TILE_TRIES+ assert client.get("/api/settings").json()["usage"] == {}+++def test_tile_proxy_hands_on_what_the_provider_said_about_the_refusal(client, monkeypatch):+ """The day Copernicus' elevation service was down, every radar tile failed+ with a perfectly clear sentence and the map said "try again"."""+ from azimut.engine import tiles++ client.put("/api/settings/keys", json={"mapbox": "pk.abc"})+ body = (+ b"<ows:ExceptionReport><ows:Exception><ows:ExceptionText>"+ b"java.lang.RuntimeException: Could not load necessary digital elevation "+ b"model for orthorectification! Please try again later."+ b"</ows:ExceptionText></ows:Exception></ows:ExceptionReport>"+ )+ asked: list[str] = []++ def refuse(url, **kwargs):+ asked.append(url)+ return httpx.Response(+ 500, content=body, headers={"content-type": "application/xml"},+ request=httpx.Request("GET", url),+ )++ monkeypatch.setattr(tiles, "RETRY_PAUSE", 0)+ _use_tile_upstream(monkeypatch, refuse)++ r = client.get("/api/tiles/mapbox-satellite/14/8298/5639")+ assert r.status_code == 500+ assert r.text == (+ "Could not load necessary digital elevation model for orthorectification! "+ "Please try again later."+ )+ assert r.headers["content-type"].startswith("text/plain")+ # a 500 is the service's verdict on this render, not a hiccup to sit through+ assert len(asked) == 1+ assert client.get("/api/settings").json()["usage"] == {}+++def test_tile_proxy_never_asks_again_for_a_refusal(client, monkeypatch):+ """A 4xx is a verdict about the request: asking again changes nothing and+ spends the allowance twice."""+ client.put("/api/settings/keys", json={"mapbox": "pk.abc"})+ asked = _flaky_upstream(monkeypatch, [400] * 5)++ assert client.get("/api/tiles/mapbox-satellite/14/8298/5639").status_code == 400+ assert len(asked) == 1++def test_tile_proxy_google_remints_session_on_403(client, monkeypatch):client.put("/api/settings/keys", json={"google": "AIza.x"})tokens = iter(["tok-stale", "tok-fresh"])diff --git a/tests/test_tiles.py b/tests/test_tiles.pyindex d9281a0..6b5ea11 100644--- a/tests/test_tiles.py+++ b/tests/test_tiles.py@@ -1,6 +1,7 @@"""Tile math and crop stitching (offline: injected fake tile fetcher)."""import dataclasses+import ioimport mathimport httpx@@ -822,3 +823,126 @@ def test_a_turned_crop_records_the_heading_its_needle_points_at():bearing=42.0, scale_north=True, fetch_tile=fake_fetch,)assert prov["marks"]["north"] == 42.0+++# -- slow providers, and tiles that come late ---------------------------------+++class _Flaky:+ """A client whose tiles fail a few times before arriving."""++ def __init__(self, failures):+ self.failures = list(failures)+ self.asked = []++ def get(self, url, **kwargs):+ self.asked.append(url)+ if self.failures:+ trouble = self.failures.pop(0)+ if isinstance(trouble, Exception):+ raise trouble+ return httpx.Response(trouble, content=b"", request=httpx.Request("GET", url))+ buf = io.BytesIO()+ Image.new("RGB", (16, 16), (10, 120, 10)).save(buf, format="PNG")+ return httpx.Response(+ 200, content=buf.getvalue(), request=httpx.Request("GET", url)+ )+++def test_a_tile_that_timed_out_is_asked_for_again(monkeypatch):+ """One slow tile used to take a whole capture down with it, which is the+ worst way to lose a scene that is only available for this pass."""+ monkeypatch.setattr(tiles, "RETRY_PAUSE", 0)+ client = _Flaky([httpx.ReadTimeout("too slow"), 503])++ tile = tiles._default_fetch(client, "https://example.test/t.png")++ assert tile is not None+ assert len(client.asked) == 3+++def test_a_tile_that_never_comes_still_raises(monkeypatch):+ monkeypatch.setattr(tiles, "RETRY_PAUSE", 0)+ client = _Flaky([httpx.ReadTimeout("too slow")] * tiles.MAX_TILE_TRIES)++ with pytest.raises(httpx.ReadTimeout):+ tiles._default_fetch(client, "https://example.test/t.png")+ assert len(client.asked) == tiles.MAX_TILE_TRIES+++def test_a_refused_tile_is_not_asked_for_again(monkeypatch):+ monkeypatch.setattr(tiles, "RETRY_PAUSE", 0)+ client = _Flaky([400] * 5)++ with pytest.raises(httpx.HTTPStatusError):+ tiles._default_fetch(client, "https://example.test/t.png")+ assert len(client.asked) == 1+++def test_radar_is_given_longer_than_the_shared_ceiling(monkeypatch, tmp_path):+ """Sentinel-1 is drawn from the GRD per request where the optical layer is+ served already baked, so it is the one that outruns the default."""+ monkeypatch.setenv("AZIMUT_HOME", str(tmp_path))+ config.save_settings(+ {+ **config.DEFAULT_SETTINGS,+ "api_keys": {"sentinelhub": "inst-uuid"},+ "sentinel1_layer": "RADAR",+ }+ )+ radar = tiles.get_provider("sentinel1")+ optical = tiles.get_provider("sentinel2")++ assert radar.timeout == tiles.RADAR_TILE_TIMEOUT > tiles.TILE_TIMEOUT+ assert optical.timeout is None # the shared ceiling is enough for a baked tile+ # and one pass of it inherits the ceiling the basemap declared+ assert tiles.get_provider("sentinel1~2026-10-06~151800").timeout == tiles.RADAR_TILE_TIMEOUT+++# -- what a provider says when it refuses -------------------------------------+++# Exactly what Sentinel Hub answered on 2026-10-07, when its elevation service+# was down and every Sentinel-1 render failed with it.+DEM_OUTAGE = b"""<?xml version='1.0' encoding="UTF-8"?>+<ows:ExceptionReport xmlns:ows="http://www.opengis.net/ows/1.1" version="1.0.0" xml:lang="en">+\t<ows:Exception>+\t\t<ows:ExceptionText>java.util.concurrent.ExecutionException: java.util.concurrent.ExecutionException: java.lang.RuntimeException: Could not load necessary digital elevation model for orthorectification! Please try again later or without orthorectification.</ows:ExceptionText>+\t</ows:Exception>+</ows:ExceptionReport>"""+++def test_the_provider_s_own_account_of_a_refusal_is_kept():+ """Without it a Copernicus outage is indistinguishable from a bug here,+ and the analyst goes looking at their own connection."""+ assert tiles.upstream_reason(DEM_OUTAGE) == (+ "Could not load necessary digital elevation model for orthorectification! "+ "Please try again later or without orthorectification."+ )+++def test_a_refusal_that_explains_nothing_says_nothing():+ assert tiles.upstream_reason(b"") == ""+ assert tiles.upstream_reason(b"<html><body>502 Bad Gateway</body></html>") == ""+ assert tiles.upstream_reason(b"\x89PNG\r\n\x1a\n" + b"\xff" * 64) == ""+++def test_a_refusal_never_quotes_back_what_it_was_sent():+ """A message lands in the page, the network log and the screenshot attached+ to a bug report, so the instance id must not travel in one."""+ echoed = (+ "<ows:ExceptionText>java.lang.RuntimeException: Layer not found at "+ "https://services.sentinel-hub.com/ogc/wmts/0f7f2d1e-3c4b-4a5d-8e9f-1a2b3c4d5e6f"+ " for instance 0f7f2d1e-3c4b-4a5d-8e9f-1a2b3c4d5e6f with key pk.eyJhbGciOi"+ "</ows:ExceptionText>"+ ).encode()+ reason = tiles.upstream_reason(echoed)++ assert reason.startswith("Layer not found")+ for secret in ("https://", "0f7f2d1e", "pk.eyJ", "sentinel-hub.com"):+ assert secret not in reason+++def test_a_refusal_is_capped_at_a_sentence():+ flood = ("<ows:ExceptionText>" + "no. " * 500 + "</ows:ExceptionText>").encode()+ assert len(tiles.upstream_reason(flood)) == tiles.REASON_MAX