Skip to content

The OSINT investigator's workbench. One case, one folder, every tool. Local-first: your media and your investigations never leave your machine.

JavaScript

Retry imagery tiles and stabilize browser navigation checks (#111)

OsintMeThat518a4553d4ca
diff --git a/docs/IMAGERY_PROVIDERS.md b/docs/IMAGERY_PROVIDERS.md
index 69016cd..60a3a3f 100644
--- a/docs/IMAGERY_PROVIDERS.md
+++ b/docs/IMAGERY_PROVIDERS.md
@@ -160,6 +160,32 @@ withheld from `all_providers()`, so the basemap vanishes from the selector (the
live map falls back to Esri) until the key changes (verdict cleared) or a test
passes. Settings shows the stored reason inline.
+## Tile retries and failure messages
+
+Three rules, shared by the live proxy and `fetch_crop` so a provider that is
+slow on the map is not quietly fast on the evidence.
+
+**A ceiling per provider.** `TILE_TIMEOUT` is 20s; `Provider.timeout` overrides
+it, and only Sentinel-1 does (45s). Radar is drawn from the GRD through our
+evalscript at request time where the optical layer is served already baked, so
+a cold 512px tile regularly outran the shared ceiling.
+
+**One more try, twice.** `MAX_TILE_TRIES` is 3, for a transport error or a 502
+/503/504, never for a 4xx (a verdict about the request) and never for a 500
+(the service's verdict on this render — on 2026-10-07 every Sentinel-1 tile
+answered 500 for hours, identically). Retries cost nothing on the meter, which
+only ever counts a 2xx. Before this, one slow tile left a hole in the live map
+and took a whole *capture* down with it.
+
+**The provider's own sentence reaches the panel.** Sentinel Hub answers a
+render it could not do with an OGC `ExceptionReport`; `upstream_reason()`
+unwraps it from its Java trace, strips addresses and credentials, caps it at
+240 characters and the proxy returns *that* as the failed tile's body, as
+`text/plain`. The engine hands it to the map as a Blob, so the reason arrives a
+tick after the trouble. Without it, a Copernicus outage is indistinguishable
+from a bug in Azimut — the DEM outage above read as "try again" for an hour.
+Our own sentences still win for 401/403 and 429, which name what to go and do.
+
## Per-provider eco thresholds
`eco_max_zooms` in settings.json lets each tile basemap carry its own eco
@@ -475,6 +501,14 @@ why. One request per probe, on the meter. The first layer that answers is kept a
`sentinel1_layer` in settings.json, carried by the backup, and forgotten when the
instance id changes. It can also be named and checked by hand.
+**Once it is known, it leaves the display list.** The same silence in
+GetCapabilities that makes Find necessary means the instance hands the radar
+layer back among the optical ones, often first, so
+`/api/satellite/sentinel/layers` drops `sentinel1_layer` from what it offers —
+picked as a Sentinel-2 display it would render radar dated and cloud-filtered by
+a calendar that is not its own. Radar is reached through `sentinel1_layer`
+alone; nothing of its road reads that list.
+
- **A pass is a day and a time.** Sentinel-1 passes a place at dawn flying south
and at dusk flying north, so one day can hold two looks from opposite sides.
Radar lookups (`collection=sentinel1`) group granules by pass rather than by
diff --git a/frontend/e2e/command-palette.spec.js b/frontend/e2e/command-palette.spec.js
index 69884c1..ee3a4d2 100644
--- a/frontend/e2e/command-palette.spec.js
+++ b/frontend/e2e/command-palette.spec.js
@@ -39,6 +39,16 @@ async function find(page, term) {
await page.keyboard.press('Control+k');
await expect(search(page)).toBeFocused();
await search(page).fill(term);
+ // Recently added rows may already match the words, before the search answers.
+ await expect(dialog(page).locator('.search')).not.toHaveClass(/busy/);
+}
+
+async function chooseProof(page, name) {
+ await find(page, name);
+ await expect(dialog(page).getByRole('option', { name: new RegExp(name) }))
+ .toHaveAttribute('aria-selected', 'true');
+ await page.keyboard.press('Enter');
+ await expect(dialog(page)).toBeHidden();
}
test('Ctrl+K reads only on opening and reaches a tool with the keyboard', async ({ page }) => {
@@ -102,6 +112,8 @@ test('a case result changes the case and opens its overview', async ({ page }) =
});
async function proofCase(page) {
+ // This path includes the composer's first import and two navigation guards.
+ test.setTimeout(60_000);
const proofs = { 'proof-a': 'Proof A', 'proof-b': 'Proof B' };
const catalog = Object.entries(proofs).map(([name, label]) => ({
id: `${name}-entity`, type: 'proof', label, attrs: { spec: `proofs/.meta/${name}.json` },
@@ -115,10 +127,9 @@ async function proofCase(page) {
if (route.request().method() !== 'GET' || !proofs[name]) return route.fallback();
return route.fulfill({ json: { title: proofs[name], panels: [], pastes: [], shapes: [] } });
});
- await find(page, 'Proof A');
- await dialog(page).getByRole('option', { name: /Proof A/ }).click();
+ await chooseProof(page, 'Proof A');
const title = page.getByRole('textbox', { name: 'Proof name' });
- await expect(title).toHaveValue('Proof A', { timeout: 15_000 });
+ await expect(title).toHaveValue('Proof A');
await title.fill('Proof A revised');
await expect(page.locator('.tool-header .badge', { hasText: 'unsaved' })).toBeVisible();
return { fixture, title };
@@ -126,8 +137,7 @@ async function proofCase(page) {
test('opening another proof retains its existing unsaved-work confirmation', async ({ page }) => {
const { fixture, title } = await proofCase(page);
- await find(page, 'Proof B');
- await dialog(page).getByRole('option', { name: /Proof B/ }).click();
+ await chooseProof(page, 'Proof B');
const ask = page.getByRole('alertdialog', { name: 'Leave this proof?' });
await expect(ask).toBeVisible();
await ask.getByRole('button', { name: 'Keep editing' }).click();
diff --git a/frontend/e2e/detect.spec.js b/frontend/e2e/detect.spec.js
index a0a63f6..ec193ea 100644
--- a/frontend/e2e/detect.spec.js
+++ b/frontend/e2e/detect.spec.js
@@ -908,7 +908,7 @@ test('failed check imagery says why on both maps and can be retried or replaced
await answerBuilder(page);
let failing = true;
await page.route('**/api/tiles/sentinel2*/**', (route) => failing
- ? route.fulfill({ status: 400, contentType: 'application/xml', body: '<ServiceException>Layer not found</ServiceException>' })
+ ? route.fulfill({ status: 400, contentType: 'text/plain', body: 'Layer not found' })
: route.fallback());
await page.getByRole('button', { name: 'Analyzers', exact: true }).click();
await page.getByRole('button', { name: 'New analyzer', exact: true }).click();
@@ -921,7 +921,7 @@ test('failed check imagery says why on both maps and can be retried or replaced
await deck.getByRole('button', { name: 'Place the pins' }).click();
const notices = page.getByRole('status', { name: 'Imagery loading error' });
await expect(notices).toHaveCount(2);
- await expect(notices.first()).toContainText('The imagery service refused this display layer.');
+ await expect(notices.first()).toContainText('Layer not found');
await expect(notices.first()).toContainText('TRUE_COLOR');
await expect(notices.first()).toContainText('2026-09-01');
await expect(notices.last()).toContainText('2026-09-10');
diff --git a/frontend/e2e/graph.spec.js b/frontend/e2e/graph.spec.js
index 632a332..7b087aa 100644
--- a/frontend/e2e/graph.spec.js
+++ b/frontend/e2e/graph.spec.js
@@ -116,6 +116,11 @@ async function inkLost(page) {
*/
async function drawn(page) {
await expect.poll(() => ink(page), { timeout: 5000 }).toBeGreaterThan(0.001);
+ // Existing pixels can belong to the previous camera or selection. Let Konva
+ // paint its queued frame before reading pixels or using its canvas hit regions.
+ await page.evaluate(() => new Promise((resolve) => {
+ requestAnimationFrame(() => requestAnimationFrame(resolve));
+ }));
}
/**
@@ -962,6 +967,7 @@ test('the walk narrows the drawing to itself and what it can reach', async ({ pa
await page.getByRole('button', { name: 'Path to…' }).click();
await page.locator('.panel .link-row', { hasText: '3rd Battalion' }).click();
+ await expect(steps(page)).toHaveCount(2);
await expect.poll(() => ink(page)).not.toBeCloseTo(before, 4);
// And the case comes back whole when the walk is given up.
@@ -1632,7 +1638,7 @@ test('opening a node leaves everything already drawn exactly where it is', async
// Not "roughly the same picture": every node is where it was, so every pixel that
// was drawn is still drawn. What the arrival adds is deliberately not counted.
// Without the drawing being held, this reads 0.96 — the whole case moves.
- expect(await inkLost(page)).toBeLessThan(0.05);
+ await expect.poll(() => inkLost(page)).toBeLessThan(0.05);
});
test('the whole picture is put back by one control, not one per list', async ({ page }) => {
diff --git a/frontend/src/lib/map/basemap.js b/frontend/src/lib/map/basemap.js
index d3797ae..eabb324 100644
--- a/frontend/src/lib/map/basemap.js
+++ b/frontend/src/lib/map/basemap.js
@@ -29,6 +29,8 @@ const DEFAULT_SUBDOMAINS = ['a', 'b', 'c'];
const IMAGERY = 'basemap-imagery';
/** A second picture of the same ground, laid just over the first (`setAlternate`). */
const ALTERNATE = 'basemap-alternate';
+/** Failed tiles after which trouble is reported without waiting for the settle. */
+const TROUBLE_SPEAKS_UP = 4;
/**
* What can be laid over the imagery, in the order they stack — first is
@@ -274,13 +276,33 @@ export function rasterSource(provider, providerId, cell) {
};
}
-export function imageryError(error) {
+/**
+ * A provider's sentence, shown only if it still looks like one.
+ *
+ * The proxy strips addresses and credentials before it sends this on, so the
+ * checks here are the second pair of eyes rather than the first: a page is
+ * where a leaked instance id would end up in a screenshot.
+ */
+export function readableReason(text) {
+ const reason = (text ?? '').trim();
+ if (!reason || reason.length > 240) return '';
+ return /https?:\/\//i.test(reason) ? '' : reason;
+}
+
+export function imageryError(error, failed = 1, reason = '') {
const status = error?.status;
if (status === 404) return 'No imagery is available here for this pass.';
- if (status === 400 || status === 422) return 'The imagery service refused this display layer.';
+ // Ours name what to go and do about it, which the service cannot know. Its
+ // own sentence wins everywhere else, including a refused layer, where what
+ // it objected to is the whole answer.
if (status === 401 || status === 403) return 'The imagery service refused the key; check Settings → Imagery.';
if (status === 429) return 'The imagery allowance is exhausted; check Settings → Imagery.';
- return 'The imagery could not be loaded; try again.';
+ if (reason) return reason;
+ if (status === 400 || status === 422) return 'The imagery service refused this display layer.';
+ // How much of the view is missing is the difference between a hole to pan
+ // past and a picture that never came, so the count is the message.
+ if (failed > 1) return `${failed} tiles could not be loaded; try again.`;
+ return 'A tile could not be loaded; try again.';
}
/**
@@ -353,8 +375,40 @@ export function createBasemaps(engine, hooks = {}) {
return above ? overlayLayers(above)[0] : ceiling();
}
+ // Tiles that failed since the imagery source last settled, by source id.
+ // The report is a verdict on a whole settle rather than on one tile: a lone
+ // hole in a full screen is overzoomed away and worth no panel, and a panel
+ // that outlives the trouble it named is worse than no panel at all.
+ const troubles = new Map();
+
+ function report(source, key) {
+ const trouble = troubles.get(source);
+ troubles.delete(source);
+ const tell = (reason) =>
+ onImageryTrouble({
+ id: key.slice(0, key.lastIndexOf('@')),
+ error: trouble?.error ?? null,
+ failed: trouble?.failed ?? 0,
+ reason,
+ });
+ // The proxy puts the provider's own account of a refusal in the body of
+ // the failed tile, and the engine hands that over as a Blob — so when
+ // there is one to read, the reason arrives a tick after the trouble.
+ const body = trouble?.error?.body;
+ if (body && typeof body.text === 'function') {
+ body.text().then(
+ (text) => tell(readableReason(text)),
+ () => tell(''),
+ );
+ } else {
+ tell('');
+ }
+ }
+
function onSourceData(event) {
if (!event.isSourceLoaded) return;
+ const key = event.sourceId === IMAGERY ? live : event.sourceId === ALTERNATE ? alternate : null;
+ if (key) report(event.sourceId, key);
// every visible tile is in, so what the proxy counted is now final
const billed = event.sourceId === IMAGERY ? metered : event.sourceId === ALTERNATE ? alternateMetered : null;
if (billed) onMeteredTiles(billed);
@@ -364,7 +418,14 @@ export function createBasemaps(engine, hooks = {}) {
function onError(event) {
const source = event?.sourceId;
const key = source === IMAGERY ? live : source === ALTERNATE ? alternate : null;
- if (key) onImageryTrouble({ id: key.slice(0, key.lastIndexOf('@')), error: event.error });
+ if (key) {
+ const trouble = troubles.get(source) ?? { error: null, failed: 0 };
+ troubles.set(source, { error: event.error, failed: trouble.failed + 1 });
+ // A source with nothing left to show may never report itself loaded, so
+ // the settle that would speak for it never comes. Past this many the
+ // trouble is the picture, not a gap in it: say so without waiting.
+ if (trouble.failed + 1 >= TROUBLE_SPEAKS_UP) report(source, key);
+ }
const overlay = source && OVERLAYS.find((entry) => overlaySourceIds(entry).includes(source));
if (overlay) onOverlayTrouble(overlay.id);
}
diff --git a/frontend/src/lib/map/basemap.test.js b/frontend/src/lib/map/basemap.test.js
index 6196e04..68406a0 100644
--- a/frontend/src/lib/map/basemap.test.js
+++ b/frontend/src/lib/map/basemap.test.js
@@ -1,6 +1,14 @@
// @vitest-environment happy-dom
import { describe, expect, it, vi } from 'vitest';
-import { OVERLAY_IDS, imageryError, rasterSource, sourceMaxZoom, tileTemplate, tileUrls } from './basemap.js';
+import {
+ OVERLAY_IDS,
+ imageryError,
+ rasterSource,
+ readableReason,
+ sourceMaxZoom,
+ tileTemplate,
+ tileUrls,
+} from './basemap.js';
/**
* The provider shapes are the ones `/api/satellite/providers` really answers
@@ -55,7 +63,32 @@ it('explains imagery failures without exposing the upstream URL or response', ()
expect(imageryError({ status: 404 })).toContain('No imagery');
expect(imageryError({ status: 403 })).toContain('key');
expect(imageryError({ status: 429 })).toContain('allowance');
- expect(imageryError(new Error('private upstream URL'))).toBe('The imagery could not be loaded; try again.');
+ expect(imageryError(new Error('private upstream URL'))).toBe('A tile could not be loaded; try again.');
+ // how much of the view is missing is the difference between a hole and a blank
+ expect(imageryError(new Error('private upstream URL'), 7)).toBe(
+ '7 tiles could not be loaded; try again.'
+ );
+ // a named failure says why, whatever it took down with it
+ expect(imageryError({ status: 429 }, 12)).toContain('allowance');
+});
+
+it('prefers what the service said to anything we could write for it', () => {
+ const said = 'Could not load necessary digital elevation model.';
+ // without it a Copernicus outage is indistinguishable from a bug in Azimut
+ expect(imageryError({ status: 500 }, 9, said)).toBe(said);
+ expect(imageryError({ status: 400 }, 1, said)).toBe(said);
+ // except where ours names what to go and do, which the service cannot know
+ expect(imageryError({ status: 403 }, 1, said)).toContain('Settings');
+ expect(imageryError({ status: 429 }, 1, said)).toContain('Settings');
+});
+
+it('shows a provider sentence only while it still looks like one', () => {
+ expect(readableReason(' Orthorectification failed. ')).toBe('Orthorectification failed.');
+ expect(readableReason('')).toBe('');
+ expect(readableReason(null)).toBe('');
+ // a second pair of eyes on what the proxy already stripped
+ expect(readableReason('failed for https://services.sentinel-hub.com/ogc/wmts/abc')).toBe('');
+ expect(readableReason('x'.repeat(241))).toBe('');
});
describe('where a provider’s tiles are fetched from', () => {
@@ -451,11 +484,19 @@ describe('the layers on a map', () => {
basemaps.show(SENTINEL, id, 512);
basemaps.setAlternate(SENTINEL, 'sentinel2~SWIR', 512);
const [, onError] = map.calls.on.find(([name]) => name === 'error');
+ const settle = map.calls.on.find(([name]) => name === 'sourcedata')[1];
const error = { status: 400 };
onError({ sourceId: 'basemap-imagery', error });
onError({ sourceId: 'basemap-alternate', error });
onError({ sourceId: 'basemap-roads', error });
- expect(trouble.mock.calls).toEqual([[{ id, error }], [{ id: 'sentinel2~SWIR', error }]]);
+ // one tile is a gap, not a verdict: the view it belongs to speaks first
+ expect(trouble).not.toHaveBeenCalled();
+ settle({ sourceId: 'basemap-imagery', isSourceLoaded: true });
+ settle({ sourceId: 'basemap-alternate', isSourceLoaded: true });
+ expect(trouble.mock.calls).toEqual([
+ [{ id, error, failed: 1, reason: '' }],
+ [{ id: 'sentinel2~SWIR', error, failed: 1, reason: '' }],
+ ]);
const source = map.sources.get('basemap-imagery');
basemaps.show(SENTINEL, id, 512);
expect(map.sources.get('basemap-imagery')).toBe(source);
@@ -465,6 +506,62 @@ describe('the layers on a map', () => {
});
});
+ it('takes back the trouble it reported once a settle comes through clean', async () => {
+ await withStubbedGoogle(async ({ createBasemaps }) => {
+ const map = stubMap();
+ const trouble = vi.fn();
+ const basemaps = createBasemaps(stubEngine(map), { onImageryTrouble: trouble });
+ const id = 'sentinel2~TRUE_COLOR~2026-09-25~2026-09-25';
+ basemaps.show(SENTINEL, id, 512);
+ const [, onError] = map.calls.on.find(([name]) => name === 'error');
+ const settle = map.calls.on.find(([name]) => name === 'sourcedata')[1];
+ onError({ sourceId: 'basemap-imagery', error: { status: 500 } });
+ settle({ sourceId: 'basemap-imagery', isSourceLoaded: true });
+ expect(trouble).toHaveBeenLastCalledWith({ id, error: { status: 500 }, failed: 1, reason: '' });
+ // the analyst pans, the tiles come through: nothing is wrong any more
+ settle({ sourceId: 'basemap-imagery', isSourceLoaded: true });
+ expect(trouble).toHaveBeenLastCalledWith({ id, error: null, failed: 0, reason: '' });
+ });
+ });
+
+ it('passes on what the provider said about the tile it refused', async () => {
+ await withStubbedGoogle(async ({ createBasemaps }) => {
+ const map = stubMap();
+ const trouble = vi.fn();
+ const basemaps = createBasemaps(stubEngine(map), { onImageryTrouble: trouble });
+ const id = 'sentinel1~2026-10-06~151800';
+ basemaps.show(SENTINEL, id, 512);
+ const [, onError] = map.calls.on.find(([name]) => name === 'error');
+ const settle = map.calls.on.find(([name]) => name === 'sourcedata')[1];
+ // what the proxy hands over: the service's own sentence, already stripped
+ const said = 'Could not load necessary digital elevation model.';
+ const error = { status: 500, body: { text: () => Promise.resolve(said) } };
+
+ onError({ sourceId: 'basemap-imagery', error });
+ settle({ sourceId: 'basemap-imagery', isSourceLoaded: true });
+ await Promise.resolve();
+ await Promise.resolve();
+
+ expect(trouble).toHaveBeenLastCalledWith({ id, error, failed: 1, reason: said });
+ });
+ });
+
+ it('says so without waiting when enough tiles fail that nothing will settle', async () => {
+ await withStubbedGoogle(async ({ createBasemaps }) => {
+ const map = stubMap();
+ const trouble = vi.fn();
+ const basemaps = createBasemaps(stubEngine(map), { onImageryTrouble: trouble });
+ const id = 'sentinel2~TRUE_COLOR~2026-09-25~2026-09-25';
+ basemaps.show(SENTINEL, id, 512);
+ const [, onError] = map.calls.on.find(([name]) => name === 'error');
+ const error = { status: 502 };
+ for (const _ of [1, 2, 3]) onError({ sourceId: 'basemap-imagery', error });
+ expect(trouble).not.toHaveBeenCalled();
+ onError({ sourceId: 'basemap-imagery', error });
+ expect(trouble.mock.calls).toEqual([[{ id, error, failed: 4, reason: '' }]]);
+ });
+ });
+
it('adds an overlay once and takes it off when asked', async () => {
await withStubbedGoogle(async ({ createBasemaps }) => {
const map = stubMap();
diff --git a/frontend/src/tools/satellite/MapSurface.svelte b/frontend/src/tools/satellite/MapSurface.svelte
index b8132c5..16b570c 100644
--- a/frontend/src/tools/satellite/MapSurface.svelte
+++ b/frontend/src/tools/satellite/MapSurface.svelte
@@ -207,7 +207,16 @@
onWidgetAuthFailure: onwidgetauthfailure,
onWidgetFailed: onwidgetfailed,
onOverlayTrouble: (id) => onoverlaytrouble(id),
- onImageryTrouble: (failure) => { tileFailure = { id: failure.id, message: imageryError(failure.error) }; },
+ // One report per settle: tiles that came through clear the panel the
+ // ones before them raised, so it never outlives the trouble it named.
+ onImageryTrouble: (failure) => {
+ tileFailure = failure.failed
+ ? {
+ id: failure.id,
+ message: imageryError(failure.error, failure.failed, failure.reason),
+ }
+ : null;
+ },
});
basemaps.setZoomCeiling(zoomCeiling);
showBasemap();
@@ -450,13 +459,20 @@
</div>
<style>
+ /* Clear of the left column, not over it and not under it: the zoom buttons
+ (12px in, 29px wide) and a tool's own rail live there, and whichever wins
+ the stack the analyst loses — either the panel is unreadable or the
+ buttons are unpressable. Reading why the imagery failed and zooming out of
+ it are the same gesture, so both stay available. */
.tile-trouble {
- position: absolute; top: 154px; left: 12px; z-index: 600;
- display: grid; gap: 6px; max-width: min(330px, calc(100% - 24px));
+ position: absolute; top: 154px; left: 53px; z-index: 600;
+ display: grid; gap: 6px; max-width: min(330px, calc(100% - 65px));
padding: 12px; border: 1px solid var(--warn); border-radius: var(--r-sm);
background: var(--bg-1); color: var(--text-1); font-size: var(--fs-xs);
}
- .tile-trouble.right { left: auto; right: 12px; }
+ .tile-trouble.right {
+ left: auto; right: 12px; max-width: min(330px, calc(100% - 24px));
+ }
.tile-trouble span, .tile-trouble p { margin: 0; color: var(--text-2); }
.error-actions { display: flex; flex-wrap: wrap; gap: 7px; }
.map-wrap {
diff --git a/src/azimut/api/satellite.py b/src/azimut/api/satellite.py
index e760006..f71a236 100644
--- a/src/azimut/api/satellite.py
+++ b/src/azimut/api/satellite.py
@@ -67,7 +67,7 @@ def _client() -> httpx.Client:
if _tile_client is None:
_tile_client = httpx.Client(
follow_redirects=True,
- timeout=20,
+ timeout=tiles.TILE_TIMEOUT,
limits=httpx.Limits(max_keepalive_connections=16, max_connections=32),
headers={"User-Agent": tiles.USER_AGENT},
)
@@ -245,13 +245,24 @@ def _offered_layers(found: list[dict[str, Any]]) -> list[dict[str, Any]]:
back to. They carry ``custom`` so a picker can offer them even where the
instance has not been asked — a script on this machine needs no permission
from GetCapabilities to exist — and so the editor knows which rows it owns.
+
+ The Sentinel-1 layer is dropped. Radar lives in the same configuration as
+ the optical layers and GetCapabilities never says which collection a layer
+ reads, so the instance hands it back among the rest; offered here it would
+ be picked as a Sentinel-2 display, dated and cloud-filtered by a calendar
+ that is not its own. Radar is reached through ``sentinel1_layer``, never
+ through this list.
"""
- customs = sentinel.custom_layers(config.load_settings())
+ settings = config.load_settings()
+ customs = sentinel.custom_layers(settings)
+ radar = str(settings.get("sentinel1_layer") or "")
names = {entry["id"] for entry in customs}
kept = [
entry
for entry in found
- if entry.get("id") not in names and not sentinel.is_draft_layer(entry.get("id", ""))
+ if entry.get("id") not in names
+ and entry.get("id") != radar
+ and not sentinel.is_draft_layer(entry.get("id", ""))
]
return kept + [
{"id": entry["id"], "label": entry["label"], "hint": entry["hint"], "custom": True}
@@ -673,7 +684,8 @@ def _serve_tile(
return cached[0], cached[1], {"Cache-Control": "private, max-age=86400"}
upstream: httpx.Response | None = None
- for attempt in (1, 2):
+ for attempt in range(1, tiles.MAX_TILE_TRIES + 1):
+ last = attempt == tiles.MAX_TILE_TRIES
try:
url = tiles.tile_url(tiles.resolve_url(provider), z, x, y, provider.zoom_offset)
except tiles.TileFetchError as exc:
@@ -686,13 +698,21 @@ def _serve_tile(
config.record_provider_status(provider.meter, False, str(exc.__cause__))
raise HTTPException(status_code=502, detail=str(exc)) from exc
try:
- upstream = _client().get(url)
+ upstream = _client().get(url, timeout=provider.timeout or tiles.TILE_TIMEOUT)
except httpx.HTTPError as exc:
- raise HTTPException(status_code=502, detail=f"tile fetch failed: {tiles.upstream_failure(exc)}") from exc
+ # A renderer that misses its window on a cold tile usually answers
+ # the moment after, with its own cache now warm. Asking again beats
+ # a hole the viewer has to notice and press Retry on.
+ if last:
+ raise HTTPException(status_code=502, detail=f"tile fetch failed: {tiles.upstream_failure(exc)}") from exc
+ continue
# a stale Google session token answers 401/403 — re-mint once, transparently
if attempt == 1 and provider.session and upstream.status_code in (401, 403):
google_tiles.invalidate(google_tiles.key_from_url(provider.url))
continue
+ if upstream.status_code in tiles.TRANSIENT_STATUSES and not last:
+ time.sleep(tiles.RETRY_PAUSE)
+ continue
break
assert upstream is not None # the loop always assigns it before breaking
@@ -706,10 +726,13 @@ def _serve_tile(
config.record_provider_status(
provider.meter, False, google_tiles.error_message(upstream)
)
+ # The provider's own sentence, not the bytes it came in: a failed tile
+ # is never drawn, so the body is only ever read as an explanation, and
+ # the raw one carries its address and our credentials with it.
return Response(
- content=upstream.content,
+ content=tiles.upstream_reason(upstream.content).encode("utf-8"),
status_code=upstream.status_code,
- media_type=upstream.headers.get("content-type", "image/png"),
+ media_type="text/plain; charset=utf-8",
)
if provider.meter:
config.record_usage(provider.meter, 1)
diff --git a/src/azimut/engine/tiles.py b/src/azimut/engine/tiles.py
index 6828dcc..36d9398 100644
--- a/src/azimut/engine/tiles.py
+++ b/src/azimut/engine/tiles.py
@@ -12,6 +12,7 @@ from __future__ import annotations
import hashlib
import math
import re
+import time
from concurrent.futures import ThreadPoolExecutor
from dataclasses import dataclass, field, replace
from datetime import datetime, timezone
@@ -32,6 +33,18 @@ SIZE_MAX = 4096 # hard cap on a capture's width/height, in px
MAX_TILES_PER_CROP = (SIZE_MAX // TILE_SIZE) ** 2
USER_AGENT = "Azimut/0.1 (+local OSINT workbench; single-user)"
+# How long one tile may take, and how many times it may be asked for. Both the
+# live proxy and the capture stitcher work to these, so a provider that is slow
+# on the map is not quietly fast on the evidence.
+TILE_TIMEOUT = 20.0
+MAX_TILE_TRIES = 3
+# Upstream answers worth asking again: a gateway hiccup, or a renderer that ran
+# out of time. A 4xx is a verdict about the request and is never retried. A
+# retry costs nothing on the meter — only a 2xx is ever counted.
+TRANSIENT_STATUSES = frozenset({502, 503, 504})
+RETRY_PAUSE = 0.5 # before asking an overloaded service again, in seconds
+RADAR_TILE_TIMEOUT = 45.0 # Sentinel-1, rendered from the GRD per request
+
@dataclass(frozen=True)
class Provider:
@@ -73,6 +86,11 @@ class Provider:
widget: str | None = None
# Provider-specific eco threshold; None uses the global setting.
eco_max_zoom: int | None = None
+ # Seconds one tile may take, when this provider needs longer than
+ # TILE_TIMEOUT. A service that renders on demand rather than serving a tile
+ # it already baked answers in its own time, and the viewer is waiting
+ # either way: a tile that arrives late beats a hole that arrives on time.
+ timeout: float | None = None
# Built-in keyed providers (docs/IMAGERY_PROVIDERS.md): only surfaced from
@@ -279,6 +297,11 @@ def all_providers() -> list[Provider]:
tile_size=512,
zoom_offset=1,
eco_max_zoom=SENTINEL_ECO_MAX_ZOOM,
+ # Radar is drawn from the GRD through our evalscript at request
+ # time, where the optical layer is served from one Sentinel Hub
+ # already baked. A cold 512px tile regularly outruns the shared
+ # ceiling, which is what left holes in the map it was asked for.
+ timeout=RADAR_TILE_TIMEOUT,
)
)
@@ -475,6 +498,42 @@ def is_placeholder_tile(content: bytes) -> bool:
_ADDRESS = re.compile(r"https?://\S+")
+# An instance id, a Mapbox token, a Google key: the three shapes a credential
+# takes in this app, in case one is ever quoted back outside a URL.
+_SECRET = re.compile(
+ r"\b[0-9a-f]{8}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{12}\b|\bpk\.[\w.-]+|\bAIza[\w-]+",
+ re.I,
+)
+_EXCEPTION_TEXT = re.compile(r"<(?:\w+:)?ExceptionText>(.*?)</(?:\w+:)?ExceptionText>", re.S)
+# "java.lang.RuntimeException: " and the chain of wrappers in front of it: the
+# service's plumbing, never the analyst's business.
+_THROWN = re.compile(r"^[\w.$]+(?:Exception|Error):\s*")
+REASON_MAX = 240 # one sentence in a panel, not a stack trace
+
+
+def upstream_reason(content: bytes) -> str:
+ """The provider's own account of a refusal, fit to put in front of someone.
+
+ A service that says why is worth more than any sentence we could write for
+ it: Sentinel Hub answers a render it could not do with an OGC
+ ExceptionReport naming what was missing, and without it a Copernicus outage
+ is indistinguishable from a bug in Azimut. Quoted raw it would be a Java
+ trace, and anything a provider echoes may carry what we sent it, so it is
+ unwrapped, stripped of addresses and credentials, and capped.
+
+ Empty when the body holds no such report — the caller then says what it
+ can from the status alone.
+ """
+ found = _EXCEPTION_TEXT.search(content[:4096].decode("utf-8", "replace"))
+ if not found:
+ return ""
+ reason = " ".join(found.group(1).split())
+ while True:
+ unwrapped = _THROWN.sub("", reason)
+ if unwrapped == reason:
+ break
+ reason = unwrapped
+ return _SECRET.sub("<id>", _ADDRESS.sub("<address>", reason))[:REASON_MAX]
def upstream_failure(exc: BaseException) -> str:
@@ -518,8 +577,24 @@ def tile_url(url_template: str, z: int, x: int, y: int, zoom_offset: int = 0) ->
def _default_fetch(client: httpx.Client, url: str) -> Image.Image | None:
"""Fetch one tile; None for 'no imagery here' (404 or a known placeholder
- tile), raise on other errors."""
- response = client.get(url)
+ tile), raise on other errors.
+
+ A timeout or a gateway answer is asked again rather than raised: one slow
+ tile used to take the whole capture down with it, which is the worst way
+ to lose a scene that was only ever going to be available for this pass.
+ """
+ for attempt in range(1, MAX_TILE_TRIES + 1):
+ last = attempt == MAX_TILE_TRIES
+ try:
+ response = client.get(url)
+ except httpx.HTTPError:
+ if last:
+ raise
+ continue
+ if response.status_code in TRANSIENT_STATUSES and not last:
+ time.sleep(RETRY_PAUSE)
+ continue
+ break
if response.status_code == 404:
return None
response.raise_for_status()
@@ -642,7 +717,9 @@ def fetch_crop(
return tx, ty, fetch(client, url)
with httpx.Client(
- headers={"User-Agent": USER_AGENT}, timeout=20, follow_redirects=True
+ headers={"User-Agent": USER_AGENT},
+ timeout=provider.timeout or TILE_TIMEOUT,
+ follow_redirects=True,
) as client:
with ThreadPoolExecutor(max_workers=6) as pool:
return list(pool.map(lambda xy: grab(*xy), coords))
@@ -683,7 +760,7 @@ def fetch_crop(
try:
filled, parent_served = _overzoom_fill(
resolve_url(provider), gaps, tile_z, native_ts, fetch,
- provider.zoom_offset, out_size=ts,
+ provider.zoom_offset, out_size=ts, timeout=provider.timeout,
)
except TileFetchError:
filled, parent_served = {}, 0
@@ -789,6 +866,7 @@ def _overzoom_fill(
fetch: Callable[[httpx.Client, str], Image.Image | None],
zoom_offset: int = 0,
out_size: int | None = None,
+ timeout: float | None = None,
) -> tuple[dict[tuple[int, int], Image.Image], int]:
"""Best-effort fill for missing tiles from parent-level imagery.
@@ -807,7 +885,9 @@ def _overzoom_fill(
unresolved = list(gaps)
served = 0
with httpx.Client(
- headers={"User-Agent": USER_AGENT}, timeout=20, follow_redirects=True
+ headers={"User-Agent": USER_AGENT},
+ timeout=timeout or TILE_TIMEOUT,
+ follow_redirects=True,
) as client:
for up in range(1, OVERZOOM_LEVELS + 1):
sub = ts >> up # the child's footprint inside its parent, in px
diff --git a/tests/test_sentinel_custom_layers.py b/tests/test_sentinel_custom_layers.py
index 50b7e54..f754a23 100644
--- a/tests/test_sentinel_custom_layers.py
+++ b/tests/test_sentinel_custom_layers.py
@@ -348,6 +348,36 @@ def test_the_editor_is_told_which_layer_is_the_radar_one(client):
assert client.get("/api/satellite/sentinel/custom-layers").json()["radar_layer"] == "RADAR"
+def test_the_radar_layer_is_never_offered_as_a_display(client, monkeypatch):
+ """Radar lives in the same configuration as the optical layers and
+ GetCapabilities does not say which collection a layer reads, so the
+ instance hands it back among the rest — often first. Offered as a display
+ it would be picked, dated and cloud-filtered by a calendar that is not its
+ own; the radar road reads ``sentinel1_layer``, never this list.
+ """
+ config.update_settings(
+ lambda settings: settings.update(
+ api_keys={"sentinelhub": "inst-uuid"}, sentinel1_layer="RADAR"
+ )
+ )
+ monkeypatch.setattr(
+ sentinel,
+ "capabilities_layers",
+ lambda *args, **kw: [
+ {"id": "RADAR", "label": "RADAR", "hint": ""},
+ {"id": "TRUE_COLOR", "label": "True colour", "hint": ""},
+ ],
+ )
+
+ body = client.get("/api/satellite/sentinel/layers?check=true").json()
+ assert body["source"] == "instance"
+ assert [entry["id"] for entry in body["layers"]] == ["TRUE_COLOR"]
+ # so the first entry every picker falls back to is an optical layer again
+ assert body["layers"][0]["id"] == sentinel.DEFAULT_LAYER
+ # and nothing was taken away from radar: it is reached from settings
+ assert config.load_settings()["sentinel1_layer"] == "RADAR"
+
+
def test_the_route_takes_exactly_the_body_the_editor_sends(client):
"""The editor's draft holds more than a layer, and this route forbids what
it does not know. The two lists are written out separately, so they are
diff --git a/tests/test_settings_api.py b/tests/test_settings_api.py
index 7dce754..6449c24 100644
--- a/tests/test_settings_api.py
+++ b/tests/test_settings_api.py
@@ -170,6 +170,117 @@ def test_tile_proxy_unknown_provider_404(client):
assert client.get("/api/tiles/nope/14/0/0").status_code == 404
+def _flaky_upstream(monkeypatch, failures, content=b"JPEGDATA"):
+ """An upstream that fails *failures* times (a status, or an exception to
+ raise) before serving a tile. Returns the list of urls it was asked for."""
+ from azimut.engine import tiles
+
+ monkeypatch.setattr(tiles, "RETRY_PAUSE", 0) # the wait is real, not the test's
+ asked: list[str] = []
+
+ def get(url, **kwargs):
+ asked.append(url)
+ if len(asked) <= len(failures):
+ trouble = failures[len(asked) - 1]
+ if isinstance(trouble, Exception):
+ raise trouble
+ return httpx.Response(
+ trouble, content=b"", headers={"content-type": "text/plain"},
+ request=httpx.Request("GET", url),
+ )
+ return httpx.Response(
+ 200, content=content, headers={"content-type": "image/jpeg"},
+ request=httpx.Request("GET", url),
+ )
+
+ _use_tile_upstream(monkeypatch, get)
+ return asked
+
+
+def test_tile_proxy_asks_again_for_a_tile_that_timed_out(client, monkeypatch):
+ """A renderer that misses its window usually answers the moment after. The
+ tile the analyst asked for beats the hole they would have to notice."""
+ client.put("/api/settings/keys", json={"mapbox": "pk.abc"})
+ asked = _flaky_upstream(monkeypatch, [httpx.ReadTimeout("too slow")])
+
+ r = client.get("/api/tiles/mapbox-satellite/14/8298/5639")
+ assert r.status_code == 200
+ assert r.content == b"JPEGDATA"
+ assert len(asked) == 2
+ # the lost attempt was never served, so it is not on the meter
+ body = client.get("/api/settings").json()
+ assert body["usage"]["mapbox"][body["month"]] == 1
+
+
+def test_tile_proxy_asks_again_after_a_gateway_answer(client, monkeypatch):
+ client.put("/api/settings/keys", json={"mapbox": "pk.abc"})
+ asked = _flaky_upstream(monkeypatch, [503, 502])
+
+ r = client.get("/api/tiles/mapbox-satellite/14/8298/5639")
+ assert r.status_code == 200
+ assert len(asked) == 3 # tiles.MAX_TILE_TRIES, all of them used
+ body = client.get("/api/settings").json()
+ assert body["usage"]["mapbox"][body["month"]] == 1
+
+
+def test_tile_proxy_gives_up_rather_than_asking_forever(client, monkeypatch):
+ from azimut.engine import tiles
+
+ client.put("/api/settings/keys", json={"mapbox": "pk.abc"})
+ asked = _flaky_upstream(monkeypatch, [503] * 5)
+
+ r = client.get("/api/tiles/mapbox-satellite/14/8298/5639")
+ assert r.status_code == 503 # the provider's own answer, passed through
+ assert len(asked) == tiles.MAX_TILE_TRIES
+ assert client.get("/api/settings").json()["usage"] == {}
+
+
+def test_tile_proxy_hands_on_what_the_provider_said_about_the_refusal(client, monkeypatch):
+ """The day Copernicus' elevation service was down, every radar tile failed
+ with a perfectly clear sentence and the map said "try again"."""
+ from azimut.engine import tiles
+
+ client.put("/api/settings/keys", json={"mapbox": "pk.abc"})
+ body = (
+ b"<ows:ExceptionReport><ows:Exception><ows:ExceptionText>"
+ b"java.lang.RuntimeException: Could not load necessary digital elevation "
+ b"model for orthorectification! Please try again later."
+ b"</ows:ExceptionText></ows:Exception></ows:ExceptionReport>"
+ )
+ asked: list[str] = []
+
+ def refuse(url, **kwargs):
+ asked.append(url)
+ return httpx.Response(
+ 500, content=body, headers={"content-type": "application/xml"},
+ request=httpx.Request("GET", url),
+ )
+
+ monkeypatch.setattr(tiles, "RETRY_PAUSE", 0)
+ _use_tile_upstream(monkeypatch, refuse)
+
+ r = client.get("/api/tiles/mapbox-satellite/14/8298/5639")
+ assert r.status_code == 500
+ assert r.text == (
+ "Could not load necessary digital elevation model for orthorectification! "
+ "Please try again later."
+ )
+ assert r.headers["content-type"].startswith("text/plain")
+ # a 500 is the service's verdict on this render, not a hiccup to sit through
+ assert len(asked) == 1
+ assert client.get("/api/settings").json()["usage"] == {}
+
+
+def test_tile_proxy_never_asks_again_for_a_refusal(client, monkeypatch):
+ """A 4xx is a verdict about the request: asking again changes nothing and
+ spends the allowance twice."""
+ client.put("/api/settings/keys", json={"mapbox": "pk.abc"})
+ asked = _flaky_upstream(monkeypatch, [400] * 5)
+
+ assert client.get("/api/tiles/mapbox-satellite/14/8298/5639").status_code == 400
+ assert len(asked) == 1
+
+
def test_tile_proxy_google_remints_session_on_403(client, monkeypatch):
client.put("/api/settings/keys", json={"google": "AIza.x"})
tokens = iter(["tok-stale", "tok-fresh"])
diff --git a/tests/test_tiles.py b/tests/test_tiles.py
index d9281a0..6b5ea11 100644
--- a/tests/test_tiles.py
+++ b/tests/test_tiles.py
@@ -1,6 +1,7 @@
"""Tile math and crop stitching (offline: injected fake tile fetcher)."""
import dataclasses
+import io
import math
import httpx
@@ -822,3 +823,126 @@ def test_a_turned_crop_records_the_heading_its_needle_points_at():
bearing=42.0, scale_north=True, fetch_tile=fake_fetch,
)
assert prov["marks"]["north"] == 42.0
+
+
+# -- slow providers, and tiles that come late ---------------------------------
+
+
+class _Flaky:
+ """A client whose tiles fail a few times before arriving."""
+
+ def __init__(self, failures):
+ self.failures = list(failures)
+ self.asked = []
+
+ def get(self, url, **kwargs):
+ self.asked.append(url)
+ if self.failures:
+ trouble = self.failures.pop(0)
+ if isinstance(trouble, Exception):
+ raise trouble
+ return httpx.Response(trouble, content=b"", request=httpx.Request("GET", url))
+ buf = io.BytesIO()
+ Image.new("RGB", (16, 16), (10, 120, 10)).save(buf, format="PNG")
+ return httpx.Response(
+ 200, content=buf.getvalue(), request=httpx.Request("GET", url)
+ )
+
+
+def test_a_tile_that_timed_out_is_asked_for_again(monkeypatch):
+ """One slow tile used to take a whole capture down with it, which is the
+ worst way to lose a scene that is only available for this pass."""
+ monkeypatch.setattr(tiles, "RETRY_PAUSE", 0)
+ client = _Flaky([httpx.ReadTimeout("too slow"), 503])
+
+ tile = tiles._default_fetch(client, "https://example.test/t.png")
+
+ assert tile is not None
+ assert len(client.asked) == 3
+
+
+def test_a_tile_that_never_comes_still_raises(monkeypatch):
+ monkeypatch.setattr(tiles, "RETRY_PAUSE", 0)
+ client = _Flaky([httpx.ReadTimeout("too slow")] * tiles.MAX_TILE_TRIES)
+
+ with pytest.raises(httpx.ReadTimeout):
+ tiles._default_fetch(client, "https://example.test/t.png")
+ assert len(client.asked) == tiles.MAX_TILE_TRIES
+
+
+def test_a_refused_tile_is_not_asked_for_again(monkeypatch):
+ monkeypatch.setattr(tiles, "RETRY_PAUSE", 0)
+ client = _Flaky([400] * 5)
+
+ with pytest.raises(httpx.HTTPStatusError):
+ tiles._default_fetch(client, "https://example.test/t.png")
+ assert len(client.asked) == 1
+
+
+def test_radar_is_given_longer_than_the_shared_ceiling(monkeypatch, tmp_path):
+ """Sentinel-1 is drawn from the GRD per request where the optical layer is
+ served already baked, so it is the one that outruns the default."""
+ monkeypatch.setenv("AZIMUT_HOME", str(tmp_path))
+ config.save_settings(
+ {
+ **config.DEFAULT_SETTINGS,
+ "api_keys": {"sentinelhub": "inst-uuid"},
+ "sentinel1_layer": "RADAR",
+ }
+ )
+ radar = tiles.get_provider("sentinel1")
+ optical = tiles.get_provider("sentinel2")
+
+ assert radar.timeout == tiles.RADAR_TILE_TIMEOUT > tiles.TILE_TIMEOUT
+ assert optical.timeout is None # the shared ceiling is enough for a baked tile
+ # and one pass of it inherits the ceiling the basemap declared
+ assert tiles.get_provider("sentinel1~2026-10-06~151800").timeout == tiles.RADAR_TILE_TIMEOUT
+
+
+# -- what a provider says when it refuses -------------------------------------
+
+
+# Exactly what Sentinel Hub answered on 2026-10-07, when its elevation service
+# was down and every Sentinel-1 render failed with it.
+DEM_OUTAGE = b"""<?xml version='1.0' encoding="UTF-8"?>
+<ows:ExceptionReport xmlns:ows="http://www.opengis.net/ows/1.1" version="1.0.0" xml:lang="en">
+\t<ows:Exception>
+\t\t<ows:ExceptionText>java.util.concurrent.ExecutionException: java.util.concurrent.ExecutionException: java.lang.RuntimeException: Could not load necessary digital elevation model for orthorectification! Please try again later or without orthorectification.</ows:ExceptionText>
+\t</ows:Exception>
+</ows:ExceptionReport>"""
+
+
+def test_the_provider_s_own_account_of_a_refusal_is_kept():
+ """Without it a Copernicus outage is indistinguishable from a bug here,
+ and the analyst goes looking at their own connection."""
+ assert tiles.upstream_reason(DEM_OUTAGE) == (
+ "Could not load necessary digital elevation model for orthorectification! "
+ "Please try again later or without orthorectification."
+ )
+
+
+def test_a_refusal_that_explains_nothing_says_nothing():
+ assert tiles.upstream_reason(b"") == ""
+ assert tiles.upstream_reason(b"<html><body>502 Bad Gateway</body></html>") == ""
+ assert tiles.upstream_reason(b"\x89PNG\r\n\x1a\n" + b"\xff" * 64) == ""
+
+
+def test_a_refusal_never_quotes_back_what_it_was_sent():
+ """A message lands in the page, the network log and the screenshot attached
+ to a bug report, so the instance id must not travel in one."""
+ echoed = (
+ "<ows:ExceptionText>java.lang.RuntimeException: Layer not found at "
+ "https://services.sentinel-hub.com/ogc/wmts/0f7f2d1e-3c4b-4a5d-8e9f-1a2b3c4d5e6f"
+ " for instance 0f7f2d1e-3c4b-4a5d-8e9f-1a2b3c4d5e6f with key pk.eyJhbGciOi"
+ "</ows:ExceptionText>"
+ ).encode()
+ reason = tiles.upstream_reason(echoed)
+
+ assert reason.startswith("Layer not found")
+ for secret in ("https://", "0f7f2d1e", "pk.eyJ", "sentinel-hub.com"):
+ assert secret not in reason
+
+
+def test_a_refusal_is_capped_at_a_sentence():
+ flood = ("<ows:ExceptionText>" + "no. " * 500 + "</ows:ExceptionText>").encode()
+ assert len(tiles.upstream_reason(flood)) == tiles.REASON_MAX