Skip to content

Qwen3.8-Flash-Next on any consumer hardware: one-click install for Windows / Linux. Strata inference engine, OpenAI/Anthropic API on localhost, optional image input.

C++

Web search: spaced and backed-off requests, a session cache, Bing as the second keyless source, one plain message when every source refuses

Niko1221707eebd1ddcb
diff --git a/apps/host/server/agent/README.md b/apps/host/server/agent/README.md
index 81857dedc..8bfb7f4ae 100644
--- a/apps/host/server/agent/README.md
+++ b/apps/host/server/agent/README.md
@@ -68,7 +68,7 @@ The chat's Mode menu is real Harness behaviour. Each turn the page sends the cha
What the person chose is saved by `tools-settings.mjs` (inside the host's settings file) and shown by `src/components/SettingsTools.tsx` (`GET/PUT /api/agent/tools`, `POST/PATCH/DELETE /api/agent/tools/custom[/id]`). A change reaches the agent from the next message: the profile carries a `toolsSignature`, and the bridge restarts the Harness (after the running turns end) when it differs.
- Tool groups (`tools-catalog.mjs`): each is one switch in plain words and is one or more preset plugins (`withoutDisabled`); `fetch` and `search` are the two halves of the Harness's `tool-web` (both off drops it). Goals off: `/goal` answers that goals are turned off.
-- Web search needs no account. The default is DuckDuckGo's HTML result page (`../web-search-ddg.mjs`): a browser User-Agent, one request at a time at least 1.5 s apart, at most 20 a minute, a 10 minute cache per query, a 15 s limit, the lite page as a fallback, and a plain message when DuckDuckGo asks for a human check ("limiting searches from this computer"). **The results come from DuckDuckGo**; they are not ranked or filtered here. The person may switch to Exa, Perplexity or DeepSeek with their own key (`../web-search-keyed.mjs`; DeepSeek uses the Harness's own provider): the key is saved in the settings file, reaches the agent only as the environment variable `STRATA_SEARCH_KEY`, never the profile file and never the screen (it only learns `keySet`). A keyed service without its key falls back to DuckDuckGo. The profile pins `web.searchProvider` to the chosen id so two usable backends cannot make the choice ambiguous.
+- Web search needs no account. The default is DuckDuckGo's HTML result page (`../web-search-ddg.mjs`), with Bing's plain result page (`../web-search-bing.mjs`) as the second source. Manners, because DuckDuckGo refuses a program that asks fast (a real box: two searches in 1.2 s answered, the third got HTTP 202 and every DuckDuckGo page refused for many minutes): a browser User-Agent and the page's own request shape (Referer, Origin, Sec-Fetch-*), one request at a time per source, 2.5 to 4 s apart (random), at most 8 a minute, a 30 minute session cache per question, a cooldown after a refusal that doubles (1, 2, 4 ... 10 minutes, `Retry-After` honoured) during which that source gets no request, no lite-page follow-up after a refusal, one retry for a timeout or a 5xx, a 15 s limit. When every source refuses, the tool answers one plain sentence (`LIMITED_SENTENCE`: "Web search is limited right now. Try again in a minute, or add a search key (Exa, Perplexity or DeepSeek) in Settings -> Tools -> Web search.") plus an instruction to the model not to retry; the turn guard in `agent-bridge.mjs` (see Turn guard) stops a model that retries anyway. **The results come from DuckDuckGo or Bing**; they are not ranked or filtered here. The person may switch to Exa, Perplexity or DeepSeek with their own key (`../web-search-keyed.mjs`; DeepSeek uses the Harness's own provider): the key is saved in the settings file, reaches the agent only as the environment variable `STRATA_SEARCH_KEY`, never the profile file and never the screen (it only learns `keySet`). A keyed service without its key falls back to DuckDuckGo. The profile pins `web.searchProvider` to the chosen id so two usable backends cannot make the choice ambiguous.
- The browser: Playwright's MCP server (`@playwright/mcp`, staged next to the Harness by `apps/desktop/scripts/stage-harness.mjs`) on the person's own Chrome, Edge or Chromium (`browser-probe.mjs`), `--isolated` so it never sees their profile. Tools appear as `mcp__browser__browser_*`. In Manual, opening a page, clicking, typing, uploading, running a script in the page and closing tabs ask (plain reasons: "Open the web page ...", "Click on the page: ..."); looking (snapshot, screenshot, console, network, waiting, listing tabs) does not. Plan refuses every connected tool. The screen says plainly when no browser, or no tool server, is found.
- The person's own tools: `instructions` (the tool returns their text; look-only, never asks), `command` (runs their command with the assistant's text in the `STRATA_INPUT` environment variable, never spliced into the command line; asks in Manual, refused in Plan) and `server` (a web address or a program, added as `mcp__custom-<name>__*`; asks in Manual). Instruction and command tools are served by `custom-tools-mcp.mjs` (stdio MCP) from a JSON file the settings module rewrites on every change, so a switch in Settings is seen without a restart.
- `strata-gate.mjs` implements the Manual/Plan rules above (`planDecision`, `browserDecision`, the look-only instruction server).
diff --git a/apps/host/server/web-search-bing.mjs b/apps/host/server/web-search-bing.mjs
new file mode 100644
index 000000000..e7d3575eb
--- /dev/null
+++ b/apps/host/server/web-search-bing.mjs
@@ -0,0 +1,68 @@
+/**
+ * The second keyless source for the agent's web search: Bing's plain result page, used only when DuckDuckGo refuses.
+ *
+ * It is read the way a browser reads it (an ordinary User-Agent, the page's own request shape, nothing hidden) and is held to the same
+ * manners as DuckDuckGo (see web-search-ddg.mjs): one request at a time, spaced out, a cooldown after a refusal. Results are not ranked or
+ * filtered here. Pure parsing lives in this file so it can be tested on saved pages.
+ */
+
+const ENTITIES = { amp: '&', lt: '<', gt: '>', quot: '"', apos: "'", nbsp: ' ', '#39': "'", '#x27': "'", '#x2F': '/', '#183': '·' };
+const decode = text => String(text).replace(/&(#x?[0-9a-f]+|[a-z]+);/gi, (whole, key) => {
+ if (key[0] === '#') {
+ const code = key[1].toLowerCase() === 'x' ? parseInt(key.slice(2), 16) : parseInt(key.slice(1), 10);
+ return Number.isFinite(code) && code > 0 && code < 0x110000 ? String.fromCodePoint(code) : whole;
+ }
+ return ENTITIES[key.toLowerCase()] ?? whole;
+});
+const plain = html => decode(String(html).replace(/<[^>]*>/g, '')).replace(/\s+/g, ' ').trim();
+
+export const BING_URL = 'https://www.bing.com/search';
+
+/** Bing wraps every result link in its own redirect (`/ck/a?...&u=a1<base64url of the real address>`). */
+export function bingTarget(href) {
+ const value = decode(href).trim();
+ try {
+ const url = new URL(value, 'https://www.bing.com/');
+ if (/(^|\.)bing\.com$/i.test(url.hostname)) {
+ const u = url.searchParams.get('u');
+ if (u && /^a1/.test(u)) {
+ const target = Buffer.from(u.slice(2).replace(/-/g, '+').replace(/_/g, '/'), 'base64').toString('utf8');
+ const real = new URL(target);
+ return /^https?:$/.test(real.protocol) ? real.href : null;
+ }
+ return null; // Bing's own page (images, maps, "more results")
+ }
+ return /^https?:$/.test(url.protocol) ? url.href : null;
+ } catch { return null; }
+}
+
+/** The result list of the page. Tolerant: a result needs a link and a title; the snippet is optional. */
+export function parseBingResults(html) {
+ const out = [];
+ const seen = new Set();
+ const blocks = String(html).split(/<li\b[^>]*\bclass="[^"]*\bb_algo\b[^"]*"/i).slice(1);
+ for (const block of blocks) {
+ const end = block.search(/<li\b[^>]*\bclass="[^"]*\bb_(?:algo|ans|ad|pag)\b/i);
+ const body = end > 0 ? block.slice(0, end) : block;
+ const h2 = /<h2\b[^>]*>\s*<a\b([^>]*)>([\s\S]*?)<\/a>/i.exec(body);
+ if (!h2) continue;
+ const href = /\bhref="([^"]*)"/i.exec(h2[1])?.[1];
+ const url = href ? bingTarget(href) : null;
+ const title = plain(h2[2]);
+ if (!url || !title || seen.has(url)) continue;
+ const snippetMatch = /<p\b[^>]*class="[^"]*\bb_lineclamp\d*\b[^"]*"[^>]*>([\s\S]*?)<\/p>/i.exec(body)
+ ?? /<div\b[^>]*class="[^"]*\bb_caption\b[^"]*"[^>]*>[\s\S]*?<p\b[^>]*>([\s\S]*?)<\/p>/i.exec(body);
+ const snippet = snippetMatch ? plain(snippetMatch[1]) : '';
+ seen.add(url);
+ out.push({ url, title, ...(snippet ? { snippet } : {}) });
+ }
+ return out;
+}
+
+/** Bing serves a challenge page, or a refusal, instead of results. A page with results is never "blocked". */
+export function bingLooksBlocked(status, html) {
+ if (status === 403 || status === 429 || status === 503) return true;
+ const text = String(html).slice(0, 80000);
+ if (/\bclass="[^"]*\bb_algo\b/.test(text)) return false;
+ return /captcha|unusual traffic|are you a robot|verify you are (?:a )?human|\bcf-chl|challenge-platform/i.test(text);
+}
diff --git a/apps/host/server/web-search-ddg.mjs b/apps/host/server/web-search-ddg.mjs
index e2f02df10..cfe73e4be 100644
--- a/apps/host/server/web-search-ddg.mjs
+++ b/apps/host/server/web-search-ddg.mjs
@@ -1,28 +1,39 @@
/**
- * Web search with no account and no key: DuckDuckGo's own web-page search results.
+ * Web search with no account and no key: DuckDuckGo's own web-page search results, with Bing's plain result page as the second source.
*
* The Harness has three search backends and all of them need a paid service key. This is the fourth: it asks DuckDuckGo's plain
* HTML search page the way a browser does and reads the result list out of the page. It is the default for the agent's
* `web_search` tool; the person can switch to a keyed service in Settings -> Tools.
*
- * Being polite is part of the design:
- * - an ordinary browser User-Agent and the page's own request shape (nothing hidden, nothing faked beyond the User-Agent);
- * - at most one request every 1.5 s and 20 a minute from this program, with the same query answered from a 10 minute cache;
- * - a 15 s time limit, no retries in a loop; when DuckDuckGo asks for a human check or refuses, the answer says so in plain
- * words and the model is told to carry on without search.
- * Results come from DuckDuckGo; they are not ranked or filtered here.
+ * Being polite is part of the design, because DuckDuckGo refuses a program that asks too fast (2026-10-18, a Linux box: two searches in
+ * 1.2 s were answered, the third got HTTP 202 "anomaly" and every DuckDuckGo page then refused for minutes):
+ * - an ordinary browser User-Agent and the page's own request shape (Referer, Origin, Sec-Fetch-*; nothing hidden);
+ * - one request at a time per source, 2.5 to 4 s apart (random, so it does not look like a clock), at most 8 a minute;
+ * - the same question is answered from a session cache (30 minutes), so a model that repeats itself costs nothing;
+ * - when a source refuses (HTTP 202/403/429 or a challenge page) it is left alone for a cooldown that doubles every time it refuses
+ * again (1, 2, 4 ... up to 10 minutes; a Retry-After header is honoured) instead of being asked again, which only extends a block;
+ * - a refused DuckDuckGo page is not followed by the lite page (same refusal, one more strike): the next source is tried instead;
+ * - a timeout or a server error gets one retry after a short pause, then the next source;
+ * - when every source refuses the answer is one plain sentence that also tells the model not to retry (see LIMITED_MESSAGE).
+ * Results come from DuckDuckGo or Bing; they are not ranked or filtered here.
*
* Nothing in this file touches the Harness, so it can be tested alone. `strata-search.mjs` is the plugin that registers it.
*/
+import { BING_URL, bingLooksBlocked, parseBingResults } from './web-search-bing.mjs';
export const DDG_PROVIDER_ID = 'strata-duckduckgo';
const HTML_URL = 'https://html.duckduckgo.com/html/';
const LITE_URL = 'https://lite.duckduckgo.com/lite/';
-const USER_AGENT = 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/124.0 Safari/537.36';
-const CACHE_MS = 10 * 60 * 1000;
-const MIN_GAP_MS = 1500;
-const PER_MINUTE = 20;
+const USER_AGENT = 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/131.0.0.0 Safari/537.36';
+const CACHE_MS = 30 * 60 * 1000;
+const CACHE_SIZE = 120;
+const GAP_MS = 2500;
+const JITTER_MS = 1500;
+const PER_MINUTE = 8;
+const COOLDOWN_MS = 60_000;
+const COOLDOWN_MAX_MS = 10 * 60_000;
+const RETRY_PAUSE_MS = 1500;
const ENTITIES = { amp: '&', lt: '<', gt: '>', quot: '"', apos: "'", nbsp: ' ', '#39': "'", '#x27': "'", '#x2F': '/' };
const decode = text => String(text).replace(/&(#x?[0-9a-f]+|[a-z]+);/gi, (whole, key) => {
@@ -106,83 +117,148 @@ export function looksBlocked(status, html) {
&& !/class="[^"]*\bresult__a\b/.test(html);
}
+
export class SearchUnavailable extends Error {
constructor(message, code = 'WEB_PROVIDER_ERROR', options) { super(message, options); this.name = 'WebError'; this.code = code; }
}
-export const BLOCKED_MESSAGE = 'DuckDuckGo is limiting searches from this computer right now. Try again in a few minutes, or add a search key in Settings, Tools.';
-
-/**
- * @param {{fetch?: typeof fetch, now?: () => number, sleep?: (ms: number) => Promise<void>}} [deps]
- */
-export function createDuckDuckGoSearch({ fetch: fetchImpl = globalThis.fetch, now = Date.now, sleep = ms => new Promise(r => setTimeout(r, ms)) } = {}) {
- const cache = new Map();
- const starts = [];
- let last = 0;
- let chain = Promise.resolve();
+/** The one plain sentence for the person. The words "Settings → Tools → Web search" are where the key goes in the app. */
+export const LIMITED_SENTENCE = 'Web search is limited right now. Try again in a minute, or add a search key (Exa, Perplexity or DeepSeek) in Settings → Tools → Web search.';
+/** What the tool answers: the sentence, then what the model should do (it must not keep searching). */
+export const BLOCKED_MESSAGE = `${LIMITED_SENTENCE} Do not retry this search: go on with what you know and tell the person, in one plain sentence, that web search was limited.`;
+export const UNREACHABLE_MESSAGE = 'The search could not reach the internet. Check the connection and try again. Do not retry this search now: go on with what you know and say that search was unavailable.';
- /** One request at a time, spaced out, and at most PER_MINUTE a minute. */
- function turn(signal) {
- const run = chain.then(async () => {
+/** One request at a time, spaced out with a little randomness, and at most PER_MINUTE a minute. */
+class Pacer {
+ constructor({ now, sleep, random, gap = GAP_MS, jitter = JITTER_MS, perMinute = PER_MINUTE }) {
+ Object.assign(this, { now, sleep, random, gap, jitter, perMinute });
+ this.starts = []; this.last = 0; this.chain = Promise.resolve();
+ }
+ turn(signal) {
+ const run = this.chain.then(async () => {
+ const spacing = this.gap + Math.floor(this.random() * this.jitter);
for (;;) {
- signal?.throwIfAborted();
- const t = now();
- while (starts.length && t - starts[0] > 60_000) starts.shift();
- const wait = Math.max(MIN_GAP_MS - (t - last), starts.length >= PER_MINUTE ? 60_000 - (t - starts[0]) : 0);
+ if (signal?.aborted) throw new SearchUnavailable('The search was cancelled.', 'WEB_ABORTED');
+ const t = this.now();
+ while (this.starts.length && t - this.starts[0] > 60_000) this.starts.shift();
+ const wait = this.last ? Math.max(spacing - (t - this.last), this.starts.length >= this.perMinute ? 60_000 - (t - this.starts[0]) : 0) : 0;
if (wait <= 0) break;
- if (wait > 30_000) throw new SearchUnavailable('Too many searches in a short time. Try again in a minute.', 'WEB_RATE_LIMITED');
- await sleep(wait);
+ if (wait > 30_000) throw new SearchUnavailable(LIMITED_SENTENCE, 'WEB_RATE_LIMITED');
+ await this.sleep(wait);
}
- last = now();
- starts.push(last);
+ this.last = this.now();
+ this.starts.push(this.last);
});
- chain = run.catch(() => {});
+ this.chain = run.catch(() => {});
return run;
}
+}
+
+const REFUSED = Symbol('refused');
+const retryAfterMs = res => {
+ const v = Number(res?.headers?.get?.('retry-after'));
+ return Number.isFinite(v) && v > 0 ? Math.min(v * 1000, COOLDOWN_MAX_MS) : 0;
+};
+const serverError = status => status >= 500 && status < 600;
+const queryKey = query => query.toLowerCase().replace(/\s+/g, ' ').trim();
+
+/**
+ * @param {{fetch?: typeof fetch, now?: () => number, sleep?: (ms: number) => Promise<void>, random?: () => number}} [deps]
+ */
+export function createDuckDuckGoSearch({ fetch: fetchImpl = globalThis.fetch, now = Date.now, sleep = ms => new Promise(r => setTimeout(r, ms)), random = Math.random } = {}) {
+ const cache = new Map();
- async function ask(url, query, signal) {
+ /** A source: its pacing, and what to do after it refuses. */
+ const source = (id, name) => ({ id, name, pacer: new Pacer({ now, sleep, random }), blockedUntil: 0, strikes: 0, refusals: 0 });
+ const ddg = source('duckduckgo', 'DuckDuckGo');
+ const bing = source('bing', 'Bing');
+ const refuse = (s, res) => {
+ s.strikes += 1; s.refusals += 1;
+ s.blockedUntil = now() + Math.max(retryAfterMs(res), Math.min(COOLDOWN_MS * 2 ** (s.strikes - 1), COOLDOWN_MAX_MS));
+ };
+ const accepted = s => { s.strikes = 0; s.blockedUntil = 0; };
+
+ const headers = (origin, extra = {}) => ({
+ 'user-agent': USER_AGENT, accept: 'text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8', 'accept-language': 'en-US,en;q=0.9',
+ 'upgrade-insecure-requests': '1', 'sec-fetch-dest': 'document', 'sec-fetch-mode': 'navigate', 'sec-fetch-site': 'same-origin', 'sec-fetch-user': '?1',
+ ...(origin ? { referer: `${origin}/`, origin } : {}), ...extra,
+ });
+
+ async function get(url, init, signal) {
const timeout = AbortSignal.timeout(15_000);
- const res = await fetchImpl(url, {
- method: 'POST', redirect: 'follow',
- headers: { 'user-agent': USER_AGENT, 'content-type': 'application/x-www-form-urlencoded', accept: 'text/html,application/xhtml+xml', 'accept-language': 'en-US,en;q=0.8' },
- body: new URLSearchParams({ q: query, b: '' }).toString(),
- signal: signal ? AbortSignal.any([signal, timeout]) : timeout,
- });
- const html = await res.text();
- return { status: res.status, html };
+ const res = await fetchImpl(url, { redirect: 'follow', ...init, signal: signal ? AbortSignal.any([signal, timeout]) : timeout });
+ return { res, status: res.status, html: await res.text() };
+ }
+
+ /** One page request with one retry after a pause for a timeout, a network error or a server error. Throws SearchUnavailable only for cancel. */
+ async function ask(s, makeRequest, signal) {
+ let failure;
+ for (let attempt = 0; attempt < 2; attempt++) {
+ await s.pacer.turn(signal);
+ try {
+ const page = await get(...makeRequest(), signal);
+ if (!serverError(page.status)) return page;
+ failure = new Error(`HTTP ${page.status}`);
+ } catch (error) {
+ if (signal?.aborted) throw new SearchUnavailable('The search was cancelled.', 'WEB_ABORTED', { cause: error });
+ failure = error;
+ }
+ if (attempt === 0) await sleep(RETRY_PAUSE_MS + Math.floor(random() * 1000));
+ }
+ return { failure };
+ }
+
+ async function searchDuckDuckGo(query, signal) {
+ const form = new URLSearchParams({ q: query, b: '', kl: 'wt-wt', df: '' }).toString();
+ const post = (url, origin) => () => [url, { method: 'POST', headers: headers(origin, { 'content-type': 'application/x-www-form-urlencoded' }), body: form }];
+ let reachedOnce = false;
+ for (const [url, origin, parse] of [[HTML_URL, 'https://html.duckduckgo.com', parseHtmlResults], [LITE_URL, 'https://lite.duckduckgo.com', parseLiteResults]]) {
+ const page = await ask(ddg, post(url, origin), signal);
+ if (page.failure) return { failure: page.failure };
+ reachedOnce = true;
+ // A refusal on the first page is a refusal for the whole address: the lite page would only add one more strike.
+ if (looksBlocked(page.status, page.html)) { refuse(ddg, page.res); return { refused: true }; }
+ const sources = parse(page.html);
+ if (sources.length) { accepted(ddg); return { sources }; }
+ }
+ if (reachedOnce) accepted(ddg);
+ return { sources: [] };
+ }
+
+ async function searchBing(query, signal) {
+ const url = `${BING_URL}?${new URLSearchParams({ q: query, setlang: 'en' })}`;
+ const page = await ask(bing, () => [url, { method: 'GET', headers: headers(null) }], signal);
+ if (page.failure) return { failure: page.failure };
+ if (bingLooksBlocked(page.status, page.html)) { refuse(bing, page.res); return { refused: true }; }
+ accepted(bing);
+ return { sources: parseBingResults(page.html) };
}
return {
id: DDG_PROVIDER_ID,
available: () => typeof fetchImpl === 'function',
+ /** What happened so far, for tests and for the status line in the app's log. */
+ status: () => ({ duckduckgo: { strikes: ddg.strikes, refusals: ddg.refusals, blockedForMs: Math.max(0, ddg.blockedUntil - now()) }, bing: { strikes: bing.strikes, refusals: bing.refusals, blockedForMs: Math.max(0, bing.blockedUntil - now()) }, cached: cache.size }),
async search(request, signal) {
const query = String(request.query ?? '').trim().slice(0, 400);
if (!query) throw new SearchUnavailable('Type what to search for.', 'WEB_PROVIDER_ERROR');
- const key = query.toLowerCase();
+ const key = queryKey(query);
+ const want = request.maxResults ?? 10;
const hit = cache.get(key);
- if (hit && now() - hit.at < CACHE_MS) return { sources: hit.sources.slice(0, request.maxResults ?? 10), truncated: false };
- let sources = [];
- let blocked = false;
- for (const [url, parse] of [[HTML_URL, parseHtmlResults], [LITE_URL, parseLiteResults]]) {
- await turn(signal);
- let page;
- try { page = await ask(url, query, signal); }
- catch (error) {
- if (signal?.aborted) throw new SearchUnavailable('The search was cancelled.', 'WEB_ABORTED', { cause: error });
- if (error?.name === 'TimeoutError') throw new SearchUnavailable('The search took too long. Try again.', 'WEB_TIMEOUT', { cause: error });
- throw new SearchUnavailable('The search could not reach DuckDuckGo. Check the internet connection and try again.', 'WEB_PROVIDER_ERROR', { cause: error });
- }
- if (looksBlocked(page.status, page.html)) { blocked = true; continue; }
- sources = parse(page.html);
- if (sources.length) { blocked = false; break; }
- }
- if (!sources.length) {
- if (blocked) throw new SearchUnavailable(BLOCKED_MESSAGE, 'WEB_RATE_LIMITED');
- return { sources: [], truncated: false }; // a real search with no results is an answer, not an error
+ if (hit && now() - hit.at < CACHE_MS) return { sources: hit.sources.slice(0, want), truncated: false };
+ let limited = false, unreachable = false;
+ for (const [s, run] of [[ddg, searchDuckDuckGo], [bing, searchBing]]) {
+ if (s.blockedUntil > now()) { limited = true; continue; } // asked again too soon: that only extends a block
+ const result = await run(query, signal);
+ if (result.refused) { limited = true; continue; }
+ if (result.failure) { unreachable = true; continue; }
+ if (!result.sources.length) return { sources: [], truncated: false }; // a real search with no results is an answer, not an error
+ cache.set(key, { at: now(), sources: result.sources });
+ if (cache.size > CACHE_SIZE) cache.delete(cache.keys().next().value);
+ return { sources: result.sources.slice(0, want), truncated: false };
}
- cache.set(key, { at: now(), sources });
- if (cache.size > 60) cache.delete(cache.keys().next().value);
- return { sources: sources.slice(0, request.maxResults ?? 10), truncated: false };
+ if (limited) throw new SearchUnavailable(BLOCKED_MESSAGE, 'WEB_RATE_LIMITED');
+ throw new SearchUnavailable(unreachable ? UNREACHABLE_MESSAGE : BLOCKED_MESSAGE, unreachable ? 'WEB_PROVIDER_ERROR' : 'WEB_RATE_LIMITED');
},
};
}
diff --git a/apps/host/tests/agent-tools-settings.test.mjs b/apps/host/tests/agent-tools-settings.test.mjs
index adc03837d..667abf681 100644
--- a/apps/host/tests/agent-tools-settings.test.mjs
+++ b/apps/host/tests/agent-tools-settings.test.mjs
@@ -186,12 +186,13 @@ test('search: a human check or a refusal is told in plain words, and a real sear
assert.equal(looksBlocked(200, '<div class="anomaly-modal">Unfortunately, bots use DuckDuckGo too</div>'), true);
assert.equal(looksBlocked(200, HTML), false);
const sleeps = [];
- const blocked = createDuckDuckGoSearch({ fetch: async () => new Response('<html>captcha anomaly-modal</html>', { status: 200 }), sleep: async ms => { sleeps.push(ms); } });
+ const tick = () => { let t = 1_000_000; return { now: () => t, sleep: async ms => { sleeps.push(ms); t += ms; } }; };
+ const blocked = createDuckDuckGoSearch({ ...tick(), fetch: async () => new Response('<html>captcha anomaly-modal</html>', { status: 200 }) });
await assert.rejects(blocked.search({ query: 'x' }), e => e.message === BLOCKED_MESSAGE && e.code === 'WEB_RATE_LIMITED');
- const empty = createDuckDuckGoSearch({ fetch: async () => new Response('<html><body>No results.</body></html>', { status: 200 }), sleep: async () => {} });
+ const empty = createDuckDuckGoSearch({ ...tick(), fetch: async () => new Response('<html><body>No results.</body></html>', { status: 200 }) });
assert.deepEqual(await empty.search({ query: 'zzzz' }), { sources: [], truncated: false });
- const offline = createDuckDuckGoSearch({ fetch: async () => { throw new TypeError('fetch failed'); }, sleep: async () => {} });
- await assert.rejects(offline.search({ query: 'x' }), /could not reach DuckDuckGo/);
+ const offline = createDuckDuckGoSearch({ ...tick(), fetch: async () => { throw new TypeError('fetch failed'); } });
+ await assert.rejects(offline.search({ query: 'x' }), /could not reach the internet/);
});
test('search: polite by design: one request at a time, spaced out, a minute limit, and the same query is answered from a short cache', async () => {
diff --git a/apps/host/tests/fixtures/bing-results.html b/apps/host/tests/fixtures/bing-results.html
new file mode 100644
index 000000000..8f624d4f6
--- /dev/null
+++ b/apps/host/tests/fixtures/bing-results.html
@@ -0,0 +1,7 @@
+<!doctype html><html><head><title>strata local llm engine - Search</title></head><body><ol id="b_results">
+<li class="b_algo" data-id iid=SERP.5344><div class="b_tpcn"><a class="tilk" aria-label="github.com" RedirectUrl="" tabindex="-1" target="_blank" href="https://www.bing.com/ck/a?!&amp;&amp;p=742e070f460f31c63a7850b7e2884cb8ee651cb333f2ce0f6a64719eb0a6a8a4JmltdHM9MTc5MjI4MTYwMA&amp;ptn=3&amp;ver=2&amp;hsh=4&amp;fclid=0df69d14-8c8b-65ee-062f-8ae28d666410&amp;u=a1aHR0cHM6Ly9naXRodWIuY29tL05pa28xMjIxL1N0cmF0YQ&amp;ntb=1" h="ID=SERP,5129.1"><div class="tpic"><div class="wr_fav" data-priority="2"><div class="cico siteicon" style="width:32px;height:32px;"><div class="rms_iac" style="height:32px;line-height:32px;width:32px;" data-height="32" data-width="32" data-alt="Global web icon" data-class="rms_img" data-src="https://th.bing.com/th/id/ODLS.A2450BEC-5595-40BA-9F13-D9EC6AB74B9F?w=32&amp;h=32&amp;qlt=90&amp;pcl=fffffa&amp;o=6&amp;pid=1.2"></div></div></div></div><div class="tptxt"><div class="tptt">github.com</div><div class="tpmeta"><div class="b_attribution" tabindex="-1"><cite>https://github.com › Strata</cite></div></div></div></a></div><h2><a target="_blank" href="https://www.bing.com/ck/a?!&amp;&amp;p=742e070f460f31c63a7850b7e2884cb8ee651cb333f2ce0f6a64719eb0a6a8a4JmltdHM9MTc5MjI4MTYwMA&amp;ptn=3&amp;ver=2&amp;hsh=4&amp;fclid=0df69d14-8c8b-65ee-062f-8ae28d666410&amp;u=a1aHR0cHM6Ly9naXRodWIuY29tL05pa28xMjIxL1N0cmF0YQ&amp;ntb=1" h="ID=SERP,5129.2">GitHub - Niko1221/<strong>Strata</strong>: Qwen3.8-Flash-Next on any consumer …</a></h2><div class="b_caption"><p class="b_lineclamp2" data-rslinkclamp-iid=""><span class="news_dt">Sep 24, 2026</span>&nbsp;&#0183;&#32;Strata makes the model fit by sharing the work across your whole PC. Think of a kitchen: the things you use all the …</p></div><div elementtiming=frp.MiddleOfPage aria-hidden="true" style="pointer-events:none;margin-top:-1px;height:1px;width:1px;font-size:1px;position:absolute;top:0;right:0;" tabindex="-1"> &nbsp; </div></li>
+<li class="b_algo" data-id iid=SERP.5345><div class="b_tpcn"><a class="tilk" aria-label="github.com" RedirectUrl="" tabindex="-1" target="_blank" href="https://www.bing.com/ck/a?!&amp;&amp;p=83a355ead65461d915b3a98842374b9104c98dfcc6a84d37dced83063413be2dJmltdHM9MTc5MjI4MTYwMA&amp;ptn=3&amp;ver=2&amp;hsh=4&amp;fclid=0df69d14-8c8b-65ee-062f-8ae28d666410&amp;u=a1aHR0cHM6Ly9naXRodWIuY29tL05pa28xMjIxL1N0cmF0YS9yZWxlYXNlcw&amp;ntb=1" h="ID=SERP,5147.1"><div class="tpic"><div class="wr_fav" data-priority="2"><div class="cico siteicon" style="width:32px;height:32px;"><div class="rms_iac" style="height:32px;line-height:32px;width:32px;" data-height="32" data-width="32" data-alt="Global web icon" data-class="rms_img" data-src="https://th.bing.com/th/id/ODLS.A2450BEC-5595-40BA-9F13-D9EC6AB74B9F?w=32&amp;h=32&amp;qlt=91&amp;pcl=fffffa&amp;o=6&amp;pid=1.2"></div></div></div></div><div class="tptxt"><div class="tptt">github.com</div><div class="tpmeta"><div class="b_attribution" tabindex="-1"><cite>https://github.com › Strata › releases</cite></div></div></div></a></div><h2><a target="_blank" href="https://www.bing.com/ck/a?!&amp;&amp;p=83a355ead65461d915b3a98842374b9104c98dfcc6a84d37dced83063413be2dJmltdHM9MTc5MjI4MTYwMA&amp;ptn=3&amp;ver=2&amp;hsh=4&amp;fclid=0df69d14-8c8b-65ee-062f-8ae28d666410&amp;u=a1aHR0cHM6Ly9naXRodWIuY29tL05pa28xMjIxL1N0cmF0YS9yZWxlYXNlcw&amp;ntb=1" h="ID=SERP,5147.2">Releases &#183; Niko1221/<strong>Strata</strong> - GitHub</a></h2><div class="b_caption"><p class="b_lineclamp2" data-rslinkclamp-iid=""><span class="news_dt">Sep 29, 2026</span>&nbsp;&#0183;&#32;CUDA only, a build option: -DSTRATA_MMQ_TAB=ON with STRATA_MMQ_DIRECT=1 and …</p></div></li>
+<li class="b_algo" data-id iid=SERP.5346><div class="b_tpcn"><a class="tilk" aria-label="strataspa.net" RedirectUrl="" tabindex="-1" target="_blank" href="https://www.bing.com/ck/a?!&amp;&amp;p=ec423c99a1c500049b15008a1c77da5392295e31d9001a6d0ce9ae573da58e90JmltdHM9MTc5MjI4MTYwMA&amp;ptn=3&amp;ver=2&amp;hsh=4&amp;fclid=0df69d14-8c8b-65ee-062f-8ae28d666410&amp;u=a1aHR0cHM6Ly9zdHJhdGFzcGEubmV0Lw&amp;ntb=1" h="ID=SERP,5164.1"><div class="tpic"><div class="wr_fav" data-priority="2"><div class="cico siteicon" style="width:32px;height:32px;"><div class="rms_iac" style="height:32px;line-height:32px;width:32px;" data-height="32" data-width="32" data-alt="Global web icon" data-class="rms_img" data-src="https://th.bing.com/th/id/ODLS.A2450BEC-5595-40BA-9F13-D9EC6AB74B9F?w=32&amp;h=32&amp;qlt=92&amp;pcl=fffffa&amp;o=6&amp;pid=1.2"></div></div></div></div><div class="tptxt"><div class="tptt">strataspa.net</div><div class="tpmeta"><div class="b_attribution" tabindex="-1"><cite>https://strataspa.net</cite></div></div></div></a></div><h2><a target="_blank" href="https://www.bing.com/ck/a?!&amp;&amp;p=ec423c99a1c500049b15008a1c77da5392295e31d9001a6d0ce9ae573da58e90JmltdHM9MTc5MjI4MTYwMA&amp;ptn=3&amp;ver=2&amp;hsh=4&amp;fclid=0df69d14-8c8b-65ee-062f-8ae28d666410&amp;u=a1aHR0cHM6Ly9zdHJhdGFzcGEubmV0Lw&amp;ntb=1" h="ID=SERP,5164.2"><strong>Strata</strong> SpA</a></h2><div class="b_caption"><p class="b_lineclamp2" data-rslinkclamp-iid="">STRATA &#232; una societ&#224; di consulenza professionale specializzata nei servizi di supporto giuridico-amministrativo e nella gestione di …</p></div></li>
+<li class="b_algo" data-id iid=SERP.5347><div class="b_tpcn"><a class="tilk" aria-label="strata.com" RedirectUrl="" tabindex="-1" target="_blank" href="https://www.bing.com/ck/a?!&amp;&amp;p=1dc980cbad2cc71b1b9b3988e8f4602008bc6a4d968de8149c77db25289dcba0JmltdHM9MTc5MjI4MTYwMA&amp;ptn=3&amp;ver=2&amp;hsh=4&amp;fclid=0df69d14-8c8b-65ee-062f-8ae28d666410&amp;u=a1aHR0cHM6Ly93d3cuc3RyYXRhLmNvbS8&amp;ntb=1" h="ID=SERP,5180.1"><div class="tpic"><div class="wr_fav" data-priority="2"><div class="cico siteicon" style="width:32px;height:32px;"><div class="rms_iac" style="height:32px;line-height:32px;width:32px;" data-height="32" data-width="32" data-alt="Global web icon" data-class="rms_img" data-src="https://th.bing.com/th/id/ODLS.A2450BEC-5595-40BA-9F13-D9EC6AB74B9F?w=32&amp;h=32&amp;qlt=93&amp;pcl=fffffa&amp;o=6&amp;pid=1.2"></div></div></div></div><div class="tptxt"><div class="tptt">strata.com</div><div class="tpmeta"><div class="b_attribution" tabindex="-1"><cite>https://www.strata.com</cite></div></div></div></a></div><h2><a target="_blank" href="https://www.bing.com/ck/a?!&amp;&amp;p=1dc980cbad2cc71b1b9b3988e8f4602008bc6a4d968de8149c77db25289dcba0JmltdHM9MTc5MjI4MTYwMA&amp;ptn=3&amp;ver=2&amp;hsh=4&amp;fclid=0df69d14-8c8b-65ee-062f-8ae28d666410&amp;u=a1aHR0cHM6Ly93d3cuc3RyYXRhLmNvbS8&amp;ntb=1" h="ID=SERP,5180.2"><strong>Strata</strong> —3D Modeling Software</a></h2><div class="b_caption"><p class="b_lineclamp2" data-rslinkclamp-iid="">Strata is a 3D/VR/AR Developer and Studio that creates powerful 3D modeling software and solutions. Start a free trial and bring …</p></div></li>
+<li class="b_algo" data-id iid=SERP.5348><div class="b_tpcn"><a class="tilk" aria-label="stratanetwork.com" RedirectUrl="" tabindex="-1" target="_blank" href="https://www.bing.com/ck/a?!&amp;&amp;p=d3ffcc77c74e73bdb811924b6eda6f244b5876d150d8b08dc90500765cc5a782JmltdHM9MTc5MjI4MTYwMA&amp;ptn=3&amp;ver=2&amp;hsh=4&amp;fclid=0df69d14-8c8b-65ee-062f-8ae28d666410&amp;u=a1aHR0cHM6Ly9sb2dpbi5zdHJhdGFuZXR3b3JrLmNvbS9Mb2dpbg&amp;ntb=1" h="ID=SERP,5194.1"><div class="tpic"><div class="wr_fav" data-priority="2"><div class="cico siteicon" style="width:32px;height:32px;"><div class="rms_iac" style="height:32px;line-height:32px;width:32px;" data-height="32" data-width="32" data-alt="Global web icon" data-class="rms_img" data-src="https://th.bing.com/th/id/ODLS.A2450BEC-5595-40BA-9F13-D9EC6AB74B9F?w=32&amp;h=32&amp;qlt=94&amp;pcl=fffffa&amp;o=6&amp;pid=1.2"></div></div></div></div><div class="tptxt"><div class="tptt">stratanetwork.com</div><div class="tpmeta"><div class="b_attribution" tabindex="-1"><cite>https://login.stratanetwork.com › Login</cite></div></div></div></a></div><h2><a target="_blank" href="https://www.bing.com/ck/a?!&amp;&amp;p=d3ffcc77c74e73bdb811924b6eda6f244b5876d150d8b08dc90500765cc5a782JmltdHM9MTc5MjI4MTYwMA&amp;ptn=3&amp;ver=2&amp;hsh=4&amp;fclid=0df69d14-8c8b-65ee-062f-8ae28d666410&amp;u=a1aHR0cHM6Ly9sb2dpbi5zdHJhdGFuZXR3b3JrLmNvbS9Mb2dpbg&amp;ntb=1" h="ID=SERP,5194.2"><strong>Strata</strong> Login</a></h2><div class="b_caption"><p class="b_lineclamp2" data-rslinkclamp-iid="">Strata Login ... Strata Login</p></div></li>
+</ol></body></html>
diff --git a/apps/host/tests/web-search.test.mjs b/apps/host/tests/web-search.test.mjs
new file mode 100644
index 000000000..e34208977
--- /dev/null
+++ b/apps/host/tests/web-search.test.mjs
@@ -0,0 +1,178 @@
+// The agent's keyless web search against a MOCKED, rate-limited DuckDuckGo: spaced requests, a session cache, a cooldown that doubles,
+// Bing as the second source, and one plain sentence (with the place to add a key) when every source refuses.
+import test from 'node:test';
+import assert from 'node:assert/strict';
+import { readFileSync } from 'node:fs';
+import { createDuckDuckGoSearch, BLOCKED_MESSAGE, LIMITED_SENTENCE, UNREACHABLE_MESSAGE } from '../server/web-search-ddg.mjs';
+import { bingTarget, parseBingResults, bingLooksBlocked } from '../server/web-search-bing.mjs';
+
+const BING = readFileSync(new URL('./fixtures/bing-results.html', import.meta.url), 'utf8');
+const DDG = `<div class="result"><h2 class="result__title"><a rel="nofollow" class="result__a" href="//duckduckgo.com/l/?uddg=https%3A%2F%2Fddg.example%2Fa">DDG one</a></h2>
+<a class="result__snippet" href="x">First snippet</a></div>
+<div class="result"><h2 class="result__title"><a rel="nofollow" class="result__a" href="//duckduckgo.com/l/?uddg=https%3A%2F%2Fddg.example%2Fb">DDG two</a></h2></div>`;
+const ANOMALY = '<html><body><div class="anomaly-modal">Unfortunately, bots use DuckDuckGo too.</div></body></html>';
+
+/** A fake clock whose sleep moves time, and a fake web: `script(url, n)` decides each answer. */
+function world(script, { random = () => 0.5 } = {}) {
+ let t = 5_000_000;
+ const calls = [];
+ const fetch = async (url, init) => {
+ const at = t;
+ const call = { url: String(url), method: init?.method ?? 'GET', at, headers: init?.headers ?? {} };
+ calls.push(call);
+ const out = await script(call, calls.length);
+ if (out instanceof Error) throw out;
+ return out;
+ };
+ const search = createDuckDuckGoSearch({ fetch, now: () => t, sleep: async ms => { t += ms; }, random });
+ return { search, calls, advance: ms => { t += ms; }, clock: () => t, hostCalls: host => calls.filter(c => c.url.includes(host)) };
+}
+const page = (body, status = 200, headers = {}) => new Response(body, { status, headers });
+const duck = url => /duckduckgo\.com/.test(url);
+
+test('requests to one source are spaced 2.5 s or more apart, never in a burst', async () => {
+ const w = world(() => page(DDG));
+ for (const q of ['one', 'two', 'three', 'four']) await w.search.search({ query: q });
+ const at = w.calls.map(c => c.at);
+ assert.equal(at.length, 4);
+ for (let i = 1; i < at.length; i++) assert.ok(at[i] - at[i - 1] >= 2500, `gap ${i}: ${at[i] - at[i - 1]} ms`);
+});
+
+test('the request looks like the page\'s own form post: browser headers, Referer, Origin, no hidden tricks', async () => {
+ const w = world(() => page(DDG));
+ await w.search.search({ query: 'strata' });
+ const [c] = w.calls;
+ assert.equal(c.method, 'POST');
+ assert.match(c.headers['user-agent'], /Mozilla\/5\.0 .*Chrome\/\d+/);
+ assert.equal(c.headers.referer, 'https://html.duckduckgo.com/');
+ assert.equal(c.headers.origin, 'https://html.duckduckgo.com');
+ assert.equal(c.headers['sec-fetch-mode'], 'navigate');
+ assert.match(c.headers['accept-language'], /en/);
+});
+
+test('the same question is answered from the session cache, even while DuckDuckGo is refusing', async () => {
+ let refuse = false;
+ const w = world(call => (duck(call.url) ? (refuse ? page(ANOMALY, 202) : page(DDG)) : page(BING)));
+ const a = await w.search.search({ query: 'Strata Engine' });
+ assert.equal(a.sources.length, 2);
+ refuse = true;
+ const b = await w.search.search({ query: 'strata engine' });
+ assert.deepEqual(b.sources, a.sources);
+ assert.equal(w.calls.length, 1, 'no second request');
+ w.advance(20 * 60_000);
+ assert.equal((await w.search.search({ query: 'STRATA ENGINE' })).sources.length, 2, 'still cached after 20 minutes');
+ assert.equal(w.calls.length, 1);
+ w.advance(11 * 60_000);
+ await w.search.search({ query: 'strata engine' });
+ assert.ok(w.calls.length > 1, 'asked again after the cache is 30 minutes old');
+});
+
+test('DuckDuckGo refuses (HTTP 202): Bing answers, DuckDuckGo is left alone for a minute, and its lite page is not tried', async () => {
+ const w = world(call => (duck(call.url) ? page(ANOMALY, 202) : page(BING)));
+ const first = await w.search.search({ query: 'node release notes', maxResults: 3 });
+ assert.equal(first.sources.length, 3);
+ assert.equal(first.sources[0].url, 'https://github.com/Niko1221/Strata');
+ assert.equal(w.hostCalls('html.duckduckgo.com').length, 1);
+ assert.equal(w.hostCalls('lite.duckduckgo.com').length, 0, 'a refusal is not followed by the lite page');
+ assert.equal(w.hostCalls('bing.com').length, 1);
+ // The next question goes straight to Bing: DuckDuckGo is cooling down.
+ await w.search.search({ query: 'another question' });
+ assert.equal(w.hostCalls('duckduckgo.com').length, 1, 'no request to DuckDuckGo during the cooldown');
+ assert.equal(w.hostCalls('bing.com').length, 2);
+ assert.ok(w.search.status().duckduckgo.blockedForMs > 0);
+ // After the cooldown one probe goes out again.
+ w.advance(61_000);
+ await w.search.search({ query: 'a third question' });
+ assert.equal(w.hostCalls('html.duckduckgo.com').length, 2);
+});
+
+test('every source refuses: one plain sentence with the place to add a key; later searches make no request at all', async () => {
+ const w = world(call => page(duck(call.url) ? ANOMALY : '<html>captcha</html>', duck(call.url) ? 202 : 429));
+ const error = await w.search.search({ query: 'hello world' }).catch(e => e);
+ assert.equal(error.code, 'WEB_RATE_LIMITED');
+ assert.equal(error.message, BLOCKED_MESSAGE);
+ assert.ok(error.message.startsWith(LIMITED_SENTENCE));
+ assert.match(LIMITED_SENTENCE, /^Web search is limited right now\. Try again in a minute, or add a search key \(Exa, Perplexity or DeepSeek\) in Settings → Tools → Web search\.$/);
+ assert.match(error.message, /Do not retry this search/, 'the model is told to stop');
+ const used = w.calls.length;
+ assert.equal(used, 2, 'one request per source, nothing more');
+ for (const q of ['again', 'and again', 'once more']) {
+ const e = await w.search.search({ query: q }).catch(x => x);
+ assert.equal(e.code, 'WEB_RATE_LIMITED', q);
+ }
+ assert.equal(w.calls.length, used, 'a model that keeps asking does not extend the block');
+});
+
+test('the cooldown doubles each time a source refuses again, and a Retry-After header is honoured', async () => {
+ const w = world(call => (duck(call.url) ? page(ANOMALY, 202) : page('<html>captcha</html>', 429)));
+ await w.search.search({ query: 'q1' }).catch(() => {});
+ const first = w.search.status().duckduckgo.blockedForMs;
+ assert.ok(first > 55_000 && first <= 60_000, `first cooldown ${first}`);
+ w.advance(61_000);
+ await w.search.search({ query: 'q2' }).catch(() => {});
+ const second = w.search.status().duckduckgo.blockedForMs;
+ assert.ok(second > 115_000 && second <= 120_000, `second cooldown ${second}`);
+ w.advance(121_000);
+ await w.search.search({ query: 'q3' }).catch(() => {});
+ assert.ok(w.search.status().duckduckgo.blockedForMs > 235_000);
+ const r = world(call => (duck(call.url) ? page(ANOMALY, 429, { 'retry-after': '300' }) : page('<html>captcha</html>', 429)));
+ await r.search.search({ query: 'x' }).catch(() => {});
+ assert.ok(r.search.status().duckduckgo.blockedForMs > 295_000, 'Retry-After: 300');
+});
+
+test('a recovered source resets the cooldown', async () => {
+ let refuse = true;
+ const w = world(call => (duck(call.url) ? (refuse ? page(ANOMALY, 202) : page(DDG)) : page('<html>captcha</html>', 429)));
+ await w.search.search({ query: 'a' }).catch(() => {});
+ w.advance(61_000);
+ refuse = false;
+ assert.equal((await w.search.search({ query: 'b' })).sources[0].url, 'https://ddg.example/a');
+ assert.equal(w.search.status().duckduckgo.strikes, 0);
+ assert.equal(w.search.status().duckduckgo.blockedForMs, 0);
+});
+
+test('a timeout or a server error gets one retry; two in a row moves on to the next source; nothing reachable says so', async () => {
+ let n = 0;
+ const flaky = world(call => (duck(call.url) ? (++n === 1 ? page('oops', 503) : page(DDG)) : page(BING)));
+ assert.equal((await flaky.search.search({ query: 'q' })).sources[0].url, 'https://ddg.example/a');
+ assert.equal(flaky.hostCalls('duckduckgo.com').length, 2);
+ const down = world(call => (duck(call.url) ? new TypeError('fetch failed') : page(BING)));
+ const viaBing = await down.search.search({ query: 'q' });
+ assert.equal(viaBing.sources.length, 5, 'DuckDuckGo unreachable twice: Bing answers');
+ const none = world(() => new TypeError('fetch failed'));
+ const e = await none.search.search({ query: 'q' }).catch(x => x);
+ assert.equal(e.message, UNREACHABLE_MESSAGE);
+ assert.notEqual(e.code, 'WEB_RATE_LIMITED');
+});
+
+test('a search with no hits is an answer; a cancelled search is cancelled', async () => {
+ const w = world(() => page('<html><body>No results found.</body></html>'));
+ assert.deepEqual(await w.search.search({ query: 'zzzz qqqq' }), { sources: [], truncated: false });
+ const ctrl = new AbortController(); ctrl.abort();
+ const e = await w.search.search({ query: 'later' }, ctrl.signal).catch(x => x);
+ assert.equal(e.code, 'WEB_ABORTED');
+});
+
+test('too many searches in a minute are held back by the app, not sent to the sources', async () => {
+ const w = world(() => page(DDG));
+ let held = null;
+ for (let i = 0; i < 12 && !held; i++) held = await w.search.search({ query: `question ${i}` }).then(() => null, e => e);
+ assert.ok(held, 'the ninth search in a minute is held back');
+ assert.equal(held.code, 'WEB_RATE_LIMITED');
+ assert.ok(w.hostCalls('duckduckgo.com').length <= 8);
+});
+
+test('Bing: the real result page is read (redirect links decoded, snippets kept); a challenge page is a refusal; ordinary pages are not', () => {
+ const rows = parseBingResults(BING);
+ assert.equal(rows.length, 5);
+ assert.ok(rows.every(r => /^https:\/\//.test(r.url) && !r.url.includes('bing.com')), 'real addresses, not Bing redirects');
+ assert.equal(rows[1].url, 'https://github.com/Niko1221/Strata/releases');
+ assert.ok(rows[0].snippet.length > 20);
+ assert.equal(bingTarget('https://www.bing.com/ck/a?!&&p=1&u=a1aHR0cHM6Ly9leGFtcGxlLmNvbS9hP3g9MQ&ntb=1'), 'https://example.com/a?x=1');
+ assert.equal(bingTarget('https://www.bing.com/images/search?q=x'), null);
+ assert.equal(bingTarget('https://example.org/direct'), 'https://example.org/direct');
+ assert.equal(bingLooksBlocked(429, ''), true);
+ assert.equal(bingLooksBlocked(200, '<html><form>captcha</form></html>'), true);
+ assert.equal(bingLooksBlocked(200, BING), false);
+ assert.deepEqual(parseBingResults('<html>nothing here</html>'), []);
+});