Supercharge your AI agents with data from the web and beyond. Building the library for superintelligence. 🔥
TypeScript aiai-agentsai-crawlerai-scrapingai-searchcrawlerdata-extractionhtml-to-markdownllmmarkdownscraperscrapingweb-crawlerweb-dataweb-data-extractionweb-scraperweb-scrapingweb-searchwebscraping
feat(api): evaluate provider-supplied response guidance rules (#4991)
Rakshith Ramprakash0b25e17057e1
diff --git a/.github/workflows/test-server.yml b/.github/workflows/test-server.ymlindex 322d3ceb9..9526746b8 100644--- a/.github/workflows/test-server.yml+++ b/.github/workflows/test-server.yml@@ -267,6 +267,8 @@ jobs:pnpm exec vitest runsrc/lib/agent-hints.test.tssrc/lib/agent-hints-provider.test.ts+ src/lib/agent-hint-rules.test.ts+ src/lib/agent-hint-signals.test.tssrc/middlewares/__tests__/agent-hints.test.ts--reporter=defaultworking-directory: apps/apidiff --git a/apps/api/.env.example b/apps/api/.env.exampleindex 767e75161..76114a948 100644--- a/apps/api/.env.example+++ b/apps/api/.env.example@@ -183,8 +183,8 @@ X402_ENDPOINT_PRICE_USD=0.01# IMAGE_OCR_ENABLED=true# === External agent hints provider (optional) ===-# POST endpoint that can return additional agent_hints for hint-enabled requests.-# Disabled when unset. See AGENT_HINTS.md for the request/response contract.+# POST endpoint that supplies agent_hints guidance for hint-enabled requests.+# Disabled when unset.# AGENT_HINTS_PROVIDER_URL=# AGENT_HINTS_PROVIDER_SECRET=# AGENT_HINTS_PROVIDER_TIMEOUT_MS=50diff --git a/apps/api/AGENT_HINTS.md b/apps/api/AGENT_HINTS.mddeleted file mode 100644index fe2cb240f..000000000--- a/apps/api/AGENT_HINTS.md+++ /dev/null@@ -1,88 +0,0 @@-# Response guidance for agents--The v2 `POST /search`, `/scrape`, `/parse`, and `/map` routes may return an optional top-level `agent_hints: string[]`. Each response contains at most one result-dependent cross-endpoint suggestion and one low-credit notice. Existing data, errors, warning fields, and HTTP statuses remain unchanged.--Hints are disabled by default. Trusted agent adapters such as the Firecrawl MCP server and CLI can enable them for one request with:--```http-X-Firecrawl-Agent-Hints: true-```--Search and scrape suggestions name the Firecrawl tools (`firecrawl_search`, `firecrawl_scrape`) rather than REST paths, since hints are only requested by the Firecrawl MCP server and CLI. Map, Crawl and Interact suggestions keep their REST paths. Result and redirect URLs inside hints are rendered as JSON-quoted, percent-encoded http(s) hrefs, capped at 200 characters, and labelled by the result's `position` when present.--The header applies to the business request. No request-body schema changes are required. SDKs and other adapters should retain the top-level field when unwrapping `data` and should preserve it on error results. The strings recommend conditional next steps; receiving one does not execute another request or indicate user authorization.--## SDK and adapter delivery--The accompanying JavaScript and Python SDK changes preserve `agent_hints` alongside normal flattened Document, SearchData, and MapData results and on typed errors. Python covers synchronous and asynchronous clients. Markdown and HTML retain their existing content.--Raw HTTP and ordinary SDK callers receive no hints unless their adapter explicitly sets the header above. This change does not add a client-wide SDK hint option. The SDK release must precede upgrading pinned CLI/MCP dependencies for their keyed SDK paths to retain the field.--## Initial rules--- Search excerpts: offer Scrape when a web result has no markdown, HTML, or raw HTML. Inspect actual output per result, not the requested scrape setting. The hint names the excerpt-only results by 1-based position and URL (at most three, then a count of the rest), so the agent can choose which to scrape.-- Empty web search: offer another Search with a broader or alternative query only when a web result collection is explicitly present and empty.-- Search result cluster: for authenticated callers only, when at least four valid HTTP(S) result URLs are present and at least three share one origin representing at least 75% of valid result URLs, offer Map for URL discovery or Crawl for multi-page content. Keyless Search callers cannot use Map or Crawl, so they receive no cluster suggestion.-- Source page 401: when database authentication is enabled and a successful Scrape envelope includes both a page status of 401 and a scrape ID in `metadata.scrapeId` or top-level `scrape_id`, offer Interact using that concrete scrape ID for authentication or page interaction. Self-hosted deployments without database authentication do not receive this suggestion because Interact returns 501 there.-- Source page 404/410: offer Search for a current location or alternative. The suggested query is prefilled as `site:<host>` plus words from the last two path segments of the final URL (file extensions, numeric and opaque IDs dropped), with a placeholder when no usable URL is present. When `metadata.sourceURL` and `metadata.url` differ, the hint names the redirect that led to the dead page. An API cache-miss 404, 429, authentication failure, or timeout does not fire this rule.-- Truncated PDF: when Scrape reports `totalPages > numPages`, offer another Scrape only when its PDF `maxPages` can be raised, capped at the API maximum of 10,000.-- Low credits: when the authoritative billing preflight reports fewer than 100 credits, ask the agent to let the user know they should add more credits. This notice is independent of endpoint success for responses that reach the hint middleware, appears before any cross-endpoint suggestion, and does not replace that suggestion.--Selection reads only request/response fields and billing state already in memory. It makes no additional network/model calls, reads no additional session/database state, and does not scan page text or classify user intent. No hint appears solely to fill an available slot.--Static feedback instructions do not belong in response hints. Adapters that expose a feedback tool should document its contract in the relevant tool descriptions, where the guidance is available before invocation and can reference the adapter's actual feedback tool and identifiers.--## Validation--The focused selector tests exercise excerpt detection, source-page failures, error suppression, the low-credit threshold, and combined hints. Express route fixtures exercise response preservation, opt-in, opt-out, low-credit delivery, and responses without a useful next step. Hosted snips cover a completed scrape, map opt-out, and validation failure through the actual API. The snips use the harness and existing test service; they are not a live paid API smoke test.--## Optional external provider--A deployment can supply additional response guidance from a separate HTTP service. This is disabled unless `AGENT_HINTS_PROVIDER_URL` is set; with it unset, responses are unchanged.--| Variable | Default | Purpose |-| ------------------------------------ | ------------------ | ------------------------------------------------------------------------------------- |-| `AGENT_HINTS_PROVIDER_URL` | unset | Provider endpoint (http or https). Empty means unset. |-| `AGENT_HINTS_PROVIDER_SECRET` | unset | Sent as `Authorization: Bearer <secret>` when set. |-| `AGENT_HINTS_PROVIDER_TIMEOUT_MS` | `50` | Per-lookup timeout. |-| `AGENT_HINTS_PROVIDER_PSEUDONYM_KEY` | per-process random | Keys the keyless `team_id` pseudonym (at least 32 chars). Never sent to the provider. |--For hint-enabled requests with an authenticated team, the API starts a lookup right after authentication and continues without waiting. Provider hints are included only if the lookup has already finished (or is cached) when the response is sent, so the provider never adds latency. An answer that arrives after the response was sent, but within the timeout, is cached for the next request.--Request:--```http-POST {AGENT_HINTS_PROVIDER_URL}-Authorization: Bearer {AGENT_HINTS_PROVIDER_SECRET}-Content-Type: application/json--{- "version": 1,- "team_id": "string (keyless callers: keyless_<64 hex chars>)",- "org_id": "string | null",- "api_key_id": "number | null",- "endpoint": "search | scrape | parse | map",- "surface": "api | mcp | cli",- "keyless": false-}-```--No request or response content, URLs, queries, IP addresses, or API keys are sent. Keyless callers have no account team, so `team_id` is a pseudonym instead: `keyless_` followed by the hex HMAC-SHA256 of the internal keyless team ID, keyed by `AGENT_HINTS_PROVIDER_PSEUDONYM_KEY`. That key is separate from `AGENT_HINTS_PROVIDER_SECRET` and is never sent, so the provider cannot recompute pseudonyms from client addresses. Set it to the same value on every API process for pseudonyms that are stable across processes; when unset, each process uses a random key, so pseudonyms are stable only within a process.--Response (`200` only):--```json-{- "hints": [- { "id": "string, at most 64 chars", "text": "string, at most 500 chars" }- ],- "ttl_seconds": 60-}-```--- Any other status, a timeout, a network error, a body larger than 64 KiB, or a body without a `hints` array means no provider hints, cached for 30 seconds. Requests never fail because of the provider.-- `ttl_seconds` defaults to 60 and is clamped to 0–600. Unknown fields are ignored.-- Each hint is trimmed and has control characters replaced; hints with a missing or oversized `id` or `text` are dropped.-- Provider hints are added only to successful (`success: true`) responses. They follow the deterministic hints and never replace them. Exact duplicates are dropped, at most two provider hints are added, and the total is capped at three.-- Results are cached in memory per process, keyed by team, endpoint, and surface, with one lookup in flight per key and at most 10,000 entries. Different API processes may serve different provider hints until their entries expire.-- Served provider hint IDs are logged with the same `team_id` the provider receives; they are not added to the response. Lookups are counted in `firecrawl_agent_hints_provider_requests_total{outcome="hit|miss|timeout|error|disabled"}`.diff --git a/apps/api/src/config.ts b/apps/api/src/config.tsindex 53282ce92..12be22f40 100644--- a/apps/api/src/config.ts+++ b/apps/api/src/config.ts@@ -90,8 +90,7 @@ const configSchema = z.object({v => (typeof v === "string" && v.trim() === "" ? undefined : v),z.string().trim().optional(),),- // Optional external agent hints provider. Inert unless the URL is set; see- // AGENT_HINTS.md for the request/response contract.+ // Optional external agent hints provider. Inert unless the URL is set.AGENT_HINTS_PROVIDER_URL: emptyStringAsUndefined(z.string()diff --git a/apps/api/src/lib/agent-hint-rules.test.ts b/apps/api/src/lib/agent-hint-rules.test.tsnew file mode 100644index 000000000..a4a08e3d9--- /dev/null+++ b/apps/api/src/lib/agent-hint-rules.test.ts@@ -0,0 +1,253 @@+import {+ evaluateAgentHintRules,+ parseAgentHintRules,+ type AgentHintRule,+} from "./agent-hint-rules";+import type { AgentHintSignalValue } from "./agent-hint-signals";++const signals = (values: Record<string, AgentHintSignalValue>) =>+ new Map(Object.entries(values));++const rule = (+ id: string,+ when: AgentHintRule["when"],+ text: string,+ group?: string,+): AgentHintRule => (group ? { id, group, when, text } : { id, when, text });++describe("agent hint rule matcher", () => {+ it.each([+ ["eq", 404, 404, true],+ ["eq", 404, "404", false],+ ["ne", 404, 410, true],+ ["ne", 404, 404, false],+ ["lt", 99, 100, true],+ ["lt", 100, 100, false],+ ["lte", 100, 100, true],+ ["gt", 4, 3, true],+ ["gt", 3, 3, false],+ ["gte", 0.75, 0.75, true],+ ["gte", 0.74, 0.75, false],+ ["lt", "1", 100, false],+ ] as const)("%s: %p against %p holds = %p", (op, actual, value, expected) => {+ const result = evaluateAgentHintRules(+ [rule("r", [{ signal: "s", op, value } as any], "fired")],+ signals({ s: actual }),+ );+ expect(result).toEqual(expected ? ["fired"] : []);+ });++ it("matches in against a list of scalars", () => {+ const rules = [+ rule("r", [{ signal: "s", op: "in", value: [404, 410] }], "fired"),+ ];+ expect(evaluateAgentHintRules(rules, signals({ s: 410 }))).toEqual([+ "fired",+ ]);+ expect(evaluateAgentHintRules(rules, signals({ s: 401 }))).toEqual([]);+ });++ it("only lets exists hold for absent signals", () => {+ for (const op of ["eq", "ne", "lt", "in"] as const) {+ const value = op === "in" ? [1] : 1;+ expect(+ evaluateAgentHintRules(+ [rule("r", [{ signal: "missing", op, value } as any], "fired")],+ signals({}),+ ),+ ).toEqual([]);+ }+ const exists = (value: boolean) =>+ rule("r", [{ signal: "s", op: "exists", value }], "fired");+ expect(evaluateAgentHintRules([exists(false)], signals({}))).toEqual([+ "fired",+ ]);+ expect(evaluateAgentHintRules([exists(true)], signals({}))).toEqual([]);+ expect(evaluateAgentHintRules([exists(true)], signals({ s: 0 }))).toEqual([+ "fired",+ ]);+ });++ it("never matches list signals with comparison operators", () => {+ expect(+ evaluateAgentHintRules(+ [rule("r", [{ signal: "s", op: "eq", value: "a" }], "fired")],+ signals({ s: ["a"] }),+ ),+ ).toEqual([]);+ });++ it("requires every condition and treats an empty condition list as always", () => {+ const rules = [+ rule(+ "both",+ [+ { signal: "a", op: "eq", value: true },+ { signal: "b", op: "gt", value: 1 },+ ],+ "both",+ ),+ rule("always", [], "always"),+ ];+ expect(evaluateAgentHintRules(rules, signals({ a: true, b: 2 }))).toEqual([+ "both",+ "always",+ ]);+ expect(evaluateAgentHintRules(rules, signals({ a: true, b: 1 }))).toEqual([+ "always",+ ]);+ });++ it("fires only the first matching rule of a group, in rule order", () => {+ const rules = [+ rule("independent", [], "independent"),+ rule("first", [{ signal: "n", op: "gt", value: 5 }], "first", "g"),+ rule("second", [{ signal: "n", op: "gt", value: 1 }], "second", "g"),+ rule("third", [], "third", "g"),+ ];+ expect(evaluateAgentHintRules(rules, signals({ n: 9 }))).toEqual([+ "independent",+ "first",+ ]);+ expect(evaluateAgentHintRules(rules, signals({ n: 2 }))).toEqual([+ "independent",+ "second",+ ]);+ expect(evaluateAgentHintRules(rules, signals({ n: 0 }))).toEqual([+ "independent",+ "third",+ ]);+ });++ it("claims the group even when the matching rule cannot render", () => {+ const rules = [+ rule("needs-id", [], "Use {id}.", "g"),+ rule("fallback", [], "Fallback.", "g"),+ ];+ expect(evaluateAgentHintRules(rules, signals({}))).toEqual([]);+ expect(evaluateAgentHintRules(rules, signals({ id: "x" }))).toEqual([+ "Use x.",+ ]);+ });++ it("fills placeholders and leaves JSON braces alone", () => {+ expect(+ evaluateAgentHintRules(+ [+ rule(+ "r",+ [],+ 'Got {status} at {origin}; try {"url":"{origin}","n":{count}}.',+ ),+ ],+ signals({ status: 404, origin: "https://a.example", count: 3 }),+ ),+ ).toEqual([+ 'Got 404 at https://a.example; try {"url":"https://a.example","n":3}.',+ ]);+ });++ it("renders list placeholders with first and remaining modifiers", () => {+ const list = signals({ items: ["#1", "#2", "#3", "#4", "#5"] });+ expect(+ evaluateAgentHintRules(+ [+ rule(+ "r",+ [],+ "{items:first=3} and {items:remaining=3} more; all: {items}",+ ),+ ],+ list,+ ),+ ).toEqual(["#1, #2, #3 and 2 more; all: #1, #2, #3, #4, #5"]);+ expect(+ evaluateAgentHintRules(+ [rule("r", [], "{items:remaining=9}")],+ signals({ items: ["#1"] }),+ ),+ ).toEqual(["0"]);+ });++ it("emits nothing for list counts above the limit, however many digits", () => {+ const list = signals({ items: ["#1"] });+ for (const text of ["{items:first=101}", "{items:remaining=1000}"]) {+ expect(evaluateAgentHintRules([rule("r", [], text)], list)).toEqual([]);+ }+ expect(+ evaluateAgentHintRules([rule("r", [], "{items:first=100}")], list),+ ).toEqual(["#1"]);+ });++ it("emits nothing for modifiers on scalar signals", () => {+ expect(+ evaluateAgentHintRules([rule("r", [], "{n:first=2}")], signals({ n: 5 })),+ ).toEqual([]);+ });++ it("drops duplicate texts and caps the number of hints", () => {+ const rules = ["a", "a", "b", "c", "d"].map((text, i) =>+ rule(`r${i}`, [], text),+ );+ expect(evaluateAgentHintRules(rules, signals({}))).toEqual(["a", "b", "c"]);+ });+});++describe("agent hint rule parsing", () => {+ it("accepts a valid rule set in order", () => {+ const raw = Array.from({ length: 32 }, (_, i) => ({+ id: `r${i}`,+ when: [],+ text: `Rule ${i}.`,+ }));+ const rules = parseAgentHintRules(raw);+ expect(rules).toHaveLength(32);+ expect(rules![0]).toEqual({ id: "r0", when: [], text: "Rule 0." });+ });++ it("rejects more than 32 rules", () => {+ const raw = Array.from({ length: 33 }, (_, i) => ({+ id: `r${i}`,+ when: [],+ text: "x",+ }));+ expect(parseAgentHintRules(raw)).toBeUndefined();+ });++ it("rejects anything but an array", () => {+ expect(parseAgentHintRules(undefined)).toBeUndefined();+ expect(parseAgentHintRules({ id: "r" })).toBeUndefined();+ });++ it.each([+ [+ "too many conditions",+ Array(17).fill({ signal: "s", op: "eq", value: 1 }),+ ],+ [+ "an oversized in list",+ [{ signal: "s", op: "in", value: Array(33).fill(1) }],+ ],+ ["a non-finite number", [{ signal: "s", op: "gt", value: NaN }]],+ ["an unknown operator", [{ signal: "s", op: "matches", value: "x" }]],+ ])("rejects the whole set when one rule has %s", (_name, when) => {+ expect(+ parseAgentHintRules([+ { id: "ok", when: [], text: "x" },+ { id: "bad", when, text: "x" },+ ]),+ ).toBeUndefined();+ });++ it("accepts 16 conditions", () => {+ expect(+ parseAgentHintRules([+ {+ id: "ok",+ when: Array(16).fill({ signal: "s", op: "eq", value: 1 }),+ text: "x",+ },+ ]),+ ).toHaveLength(1);+ });+});diff --git a/apps/api/src/lib/agent-hint-rules.ts b/apps/api/src/lib/agent-hint-rules.tsnew file mode 100644index 000000000..2cc5ea263--- /dev/null+++ b/apps/api/src/lib/agent-hint-rules.ts@@ -0,0 +1,225 @@+import type {+ AgentHintSignals,+ AgentHintSignalValue,+} from "./agent-hint-signals";++/**+ * Provider-supplied response guidance rules, evaluated locally against+ * computed signals.+ *+ * - Rules are evaluated in order; a rule fires when every condition in `when`+ * holds (an empty `when` always holds).+ * - Rules sharing a `group` are alternatives: the first rule in the group whose+ * conditions hold is the group's rule, and later rules in that group are+ * skipped.+ * - `text` may contain `{signal}` placeholders. List signals also accept+ * `{signal:first=N}` (first N items) and `{signal:remaining=N}` (count of+ * items after the first N), with N at most 100. Lists render joined with+ * ", ". A rule whose text references an absent signal, or uses a modifier it+ * cannot apply, emits nothing.+ * - A condition on an absent signal holds only for `exists` with `false`.+ */+export type AgentHintRuleScalar = string | number | boolean;+export type AgentHintRuleOp =+ | "eq"+ | "ne"+ | "lt"+ | "lte"+ | "gt"+ | "gte"+ | "in"+ | "exists";+export type AgentHintRuleCondition =+ | { signal: string; op: "eq" | "ne"; value: AgentHintRuleScalar }+ | { signal: string; op: "lt" | "lte" | "gt" | "gte"; value: number }+ | { signal: string; op: "in"; value: AgentHintRuleScalar[] }+ | { signal: string; op: "exists"; value: boolean };+export type AgentHintRule = {+ id: string;+ group?: string;+ when: AgentHintRuleCondition[];+ text: string;+};++const MAX_RULES = 32;+const MAX_CONDITIONS = 16;+const MAX_IN_VALUES = 32;+const MAX_NAME_LENGTH = 64;+const MAX_STRING_VALUE_LENGTH = 200;+const MAX_RULE_TEXT_LENGTH = 500;+const MAX_LIST_ARGUMENT = 100;+export const MAX_RULE_HINTS = 3;++const SIGNAL_NAME = /^[a-z][a-z0-9_]*$/;+const PLACEHOLDER = /\{([a-z][a-z0-9_]*)(?::(first|remaining)=(\d+))?\}/g;++function cleanString(value: unknown, maxLength: number): string | undefined {+ if (typeof value !== "string") return undefined;+ const cleaned = value.replace(/\p{Cc}/gu, " ").trim();+ return cleaned && cleaned.length <= maxLength ? cleaned : undefined;+}++function isScalar(value: unknown): value is AgentHintRuleScalar {+ return (+ typeof value === "boolean" ||+ (typeof value === "number" && Number.isFinite(value)) ||+ (typeof value === "string" && value.length <= MAX_STRING_VALUE_LENGTH)+ );+}++function parseCondition(raw: unknown): AgentHintRuleCondition | undefined {+ if (!raw || typeof raw !== "object" || Array.isArray(raw)) return undefined;+ const { signal, op, value } = raw as Record<string, unknown>;+ if (+ typeof signal !== "string" ||+ signal.length > MAX_NAME_LENGTH ||+ !SIGNAL_NAME.test(signal)+ ) {+ return undefined;+ }+ switch (op) {+ case "eq":+ case "ne":+ return isScalar(value) ? { signal, op, value } : undefined;+ case "lt":+ case "lte":+ case "gt":+ case "gte":+ return typeof value === "number" && Number.isFinite(value)+ ? { signal, op, value }+ : undefined;+ case "in":+ return Array.isArray(value) &&+ value.length <= MAX_IN_VALUES &&+ value.every(isScalar)+ ? { signal, op, value }+ : undefined;+ case "exists":+ return typeof value === "boolean" ? { signal, op, value } : undefined;+ default:+ return undefined;+ }+}++function parseRule(raw: unknown): AgentHintRule | undefined {+ if (!raw || typeof raw !== "object" || Array.isArray(raw)) return undefined;+ const rule = raw as Record<string, unknown>;+ const id = cleanString(rule.id, MAX_NAME_LENGTH);+ const text = cleanString(rule.text, MAX_RULE_TEXT_LENGTH);+ const group =+ rule.group === undefined+ ? undefined+ : cleanString(rule.group, MAX_NAME_LENGTH);+ const rawWhen = rule.when === undefined ? [] : rule.when;+ if (+ !id ||+ !text ||+ (rule.group !== undefined && !group) ||+ !Array.isArray(rawWhen) ||+ rawWhen.length > MAX_CONDITIONS+ ) {+ return undefined;+ }+ const when: AgentHintRuleCondition[] = [];+ for (const value of rawWhen) {+ const condition = parseCondition(value);+ if (!condition) return undefined;+ when.push(condition);+ }+ return group ? { id, group, when, text } : { id, when, text };+}++/**+ * Validates a rule set as a whole: returns undefined unless it is an array of+ * at most MAX_RULES rules that are all valid. Dropping individual rules could+ * change which rule in a group applies, so a partially valid set is rejected.+ */+export function parseAgentHintRules(+ values: unknown,+): AgentHintRule[] | undefined {+ if (!Array.isArray(values) || values.length > MAX_RULES) return undefined;+ const rules: AgentHintRule[] = [];+ for (const value of values) {+ const rule = parseRule(value);+ if (!rule) return undefined;+ rules.push(rule);+ }+ return rules;+}++function conditionHolds(+ condition: AgentHintRuleCondition,+ signals: AgentHintSignals,+): boolean {+ const actual: AgentHintSignalValue | undefined = signals.get(+ condition.signal,+ );+ if (condition.op === "exists") {+ return (actual !== undefined) === condition.value;+ }+ if (actual === undefined || Array.isArray(actual)) return false;+ switch (condition.op) {+ case "eq":+ return actual === condition.value;+ case "ne":+ return actual !== condition.value;+ case "in":+ return condition.value.includes(actual);+ case "lt":+ return typeof actual === "number" && actual < condition.value;+ case "lte":+ return typeof actual === "number" && actual <= condition.value;+ case "gt":+ return typeof actual === "number" && actual > condition.value;+ case "gte":+ return typeof actual === "number" && actual >= condition.value;+ }+}++function renderText(+ template: string,+ signals: AgentHintSignals,+): string | undefined {+ let complete = true;+ const rendered = template.replace(+ PLACEHOLDER,+ (_match, name: string, modifier?: string, count?: string) => {+ const value = signals.get(name);+ const n = count === undefined ? undefined : Number(count);+ if (+ value === undefined ||+ (modifier !== undefined && !Array.isArray(value)) ||+ (n !== undefined && n > MAX_LIST_ARGUMENT)+ ) {+ complete = false;+ return "";+ }+ if (!Array.isArray(value)) return String(value);+ if (modifier === "first") return value.slice(0, n).join(", ");+ if (modifier === "remaining") {+ return String(Math.max(0, value.length - (n ?? 0)));+ }+ return value.join(", ");+ },+ );+ return complete ? rendered : undefined;+}++export function evaluateAgentHintRules(+ rules: AgentHintRule[],+ signals: AgentHintSignals,+): string[] {+ const claimedGroups = new Set<string>();+ const hints: string[] = [];+ for (const rule of rules) {+ if (hints.length >= MAX_RULE_HINTS) break;+ if (rule.group !== undefined && claimedGroups.has(rule.group)) continue;+ if (!rule.when.every(condition => conditionHolds(condition, signals))) {+ continue;+ }+ if (rule.group !== undefined) claimedGroups.add(rule.group);+ const text = renderText(rule.text, signals);+ if (text && !hints.includes(text)) hints.push(text);+ }+ return hints;+}diff --git a/apps/api/src/lib/agent-hint-signals.test.ts b/apps/api/src/lib/agent-hint-signals.test.tsnew file mode 100644index 000000000..6b5a0e598--- /dev/null+++ b/apps/api/src/lib/agent-hint-signals.test.ts@@ -0,0 +1,178 @@+import {+ computeAgentHintSignals,+ type AgentHintSignalContext,+} from "./agent-hint-signals";++const signalsFor = (overrides: Partial<AgentHintSignalContext>) =>+ Object.fromEntries(+ computeAgentHintSignals({+ endpoint: "search",+ response: { success: true, data: {} },+ ...overrides,+ }),+ );++describe("agent hint signals", () => {+ it("always reports the endpoint, outcome and capabilities", () => {+ expect(signalsFor({})).toEqual({+ endpoint: "search",+ success: true,+ can_use_map_and_crawl: false,+ can_use_interact: false,+ });+ expect(+ signalsFor({+ endpoint: "map",+ response: { success: false, error: "failed" },+ canUseMapAndCrawl: true,+ canUseInteract: true,+ remainingCredits: 42,+ }),+ ).toEqual({+ endpoint: "map",+ success: false,+ can_use_map_and_crawl: true,+ can_use_interact: true,+ remaining_credits: 42,+ });+ });++ it("omits non-finite remaining credits", () => {+ expect(signalsFor({ remainingCredits: Infinity })).not.toHaveProperty(+ "remaining_credits",+ );+ });++ it("describes the scraped page from metadata", () => {+ expect(+ signalsFor({+ endpoint: "scrape",+ response: {+ success: true,+ data: {+ metadata: {+ statusCode: 410,+ scrapeId: "id with space",+ sourceURL: "https://old.example/a",+ url: "https://new.example/docs/getting-started.html",+ },+ },+ },+ }),+ ).toMatchObject({+ page_status: 410,+ scrape_id: "id%20with%20space",+ page_redirect_from: '"https://old.example/a"',+ page_redirect_to: '"https://new.example/docs/getting-started.html"',+ page_host: "new.example",+ page_path_words: "docs getting started",+ });+ });++ it("falls back to the top-level scrape ID", () => {+ expect(+ signalsFor({+ endpoint: "scrape",+ response: { success: true, scrape_id: "top", data: { metadata: {} } },+ }),+ ).toMatchObject({ scrape_id: "top" });+ });++ it("omits redirect values when the URLs match or are too long to quote", () => {+ const page = (sourceURL: string, url: string) =>+ signalsFor({+ endpoint: "scrape",+ response: { success: true, data: { metadata: { sourceURL, url } } },+ });+ expect(+ page("https://a.example/x", "https://a.example/x"),+ ).not.toHaveProperty("page_redirect_from");+ expect(+ page("https://a.example/x", `https://b.example/${"y".repeat(300)}`),+ ).not.toHaveProperty("page_redirect_from");+ });++ it("omits path words when only opaque ids remain", () => {+ const result = signalsFor({+ endpoint: "scrape",+ response: {+ success: true,+ data: { metadata: { url: "https://x.example/12345" } },+ },+ });+ expect(result).toMatchObject({ page_host: "x.example" });+ expect(result).not.toHaveProperty("page_path_words");+ });++ it("reports document page counts with the requestable remainder", () => {+ expect(+ signalsFor({+ endpoint: "scrape",+ response: {+ success: true,+ data: { metadata: { numPages: 5, totalPages: 12000 } },+ },+ }),+ ).toMatchObject({+ document_pages_returned: 5,+ document_pages_total: 12000,+ document_max_pages: 10000,+ document_pages_requestable: 9995,+ });+ });++ it("never reports a negative requestable page count", () => {+ expect(+ signalsFor({+ endpoint: "scrape",+ response: {+ success: true,+ data: { metadata: { numPages: 12, totalPages: 5 } },+ },+ }),+ ).toMatchObject({ document_pages_requestable: 0 });+ });++ it("summarises web results, excerpts and origins", () => {+ expect(+ signalsFor({+ response: {+ success: true,+ data: {+ web: [+ { url: "https://docs.example.com/a", markdown: "a" },+ { url: "https://docs.example.com/b", position: 7 },+ { url: "https://docs.example.com/c", html: "" },+ { url: "javascript:alert(1)" },+ { description: "no url" },+ ],+ },+ },+ }),+ ).toMatchObject({+ result_count: 5,+ excerpt_count: 2,+ excerpt_share: 0.4,+ excerpt_results: ['#7 "https://docs.example.com/b"', "#4"],+ origin_result_count: 3,+ top_origin: "https://docs.example.com",+ top_origin_count: 3,+ top_origin_share: 1,+ });+ });++ it("reads results from a top-level data array and reports empty sets", () => {+ expect(signalsFor({ response: { success: true, data: [] } })).toMatchObject(+ { result_count: 0, excerpt_count: 0, origin_result_count: 0 },+ );+ expect(+ signalsFor({ response: { success: true, data: { web: [] } } }),+ ).not.toHaveProperty("excerpt_share");+ });++ it("reports no result signals without a web result collection", () => {+ expect(+ signalsFor({ response: { success: true, data: { images: [] } } }),+ ).not.toHaveProperty("result_count");+ });+});diff --git a/apps/api/src/lib/agent-hint-signals.ts b/apps/api/src/lib/agent-hint-signals.tsnew file mode 100644index 000000000..0469c826a--- /dev/null+++ b/apps/api/src/lib/agent-hint-signals.ts@@ -0,0 +1,234 @@+/**+ * Facts about a hint-enabled response, computed locally from the response+ * body and request state already in memory. Provider-supplied rules are+ * matched against these values and may reference them as text placeholders;+ * the values themselves never leave the API.+ */+export type AgentHintEndpoint = "search" | "scrape" | "parse" | "map";++export interface AgentHintSignalContext {+ endpoint: AgentHintEndpoint;+ response: unknown;+ remainingCredits?: number;+ canUseMapAndCrawl?: boolean;+ canUseInteract?: boolean;+}++export type AgentHintSignalValue = string | number | boolean | string[];+export type AgentHintSignals = Map<string, AgentHintSignalValue>;++const PDF_MAX_PAGES_LIMIT = 10000;+const HINT_URL_MAX_CHARS = 200;+const QUERY_PATH_SEGMENTS = 2;++type ObjectValue = Record<string, unknown>;++export function object(value: unknown): ObjectValue {+ return value !== null && typeof value === "object" && !Array.isArray(value)+ ? (value as ObjectValue)+ : {};+}++export function httpUrl(value: unknown): URL | undefined {+ if (typeof value !== "string" || !value) return undefined;+ try {+ const parsed = new URL(value);+ return parsed.protocol === "http:" || parsed.protocol === "https:"+ ? parsed+ : undefined;+ } catch {+ return undefined;+ }+}++/**+ * Result URLs come from third-party pages, so they are rendered only as a+ * parsed http(s) href (whitespace and quotes percent-encoded), length-capped,+ * and JSON-quoted to mark them as data rather than instruction text.+ */+export function quotedUrl(url: URL): string | undefined {+ const href = url.href;+ return href.length <= HINT_URL_MAX_CHARS ? JSON.stringify(href) : undefined;+}++export function resultPosition(item: ObjectValue, index: number): number {+ const position = item.position;+ return typeof position === "number" &&+ Number.isInteger(position) &&+ position > 0+ ? position+ : index + 1;+}++/**+ * Identifier-like path segments carry no searchable meaning: numbers, hex or+ * UUID-style ids, ULIDs, and long mixed alphanumeric tokens such as+ * "W020260806515694454560".+ */+function isOpaqueId(segment: string): boolean {+ if (/^\d+$/.test(segment)) return true;+ if (/^[0-9a-f-]{16,}$/i.test(segment)) return true;+ if (/^[0-9A-HJKMNP-TV-Z]{26}$/i.test(segment)) return true;+ return (+ segment.length >= 12 &&+ /^[A-Za-z0-9_]+$/.test(segment) &&+ (segment.match(/\d/g)?.length ?? 0) >= 4+ );+}++/** Words from the last path segments, e.g. /payments/checkout/migration-from-legacy -> "checkout migration from legacy". */+export function pathWords(url: URL): string {+ const segments = url.pathname+ .split("/")+ .map(segment => {+ try {+ return decodeURIComponent(segment);+ } catch {+ return segment;+ }+ })+ .map(segment => segment.replace(/\.[a-z0-9]{1,5}$/i, ""))+ .filter(segment => segment && !isOpaqueId(segment));+ return segments+ .slice(-QUERY_PATH_SEGMENTS)+ .join(" ")+ .replace(/[-_+.]+/g, " ")+ .replace(/[^\p{L}\p{N} ]+/gu, " ")+ .replace(/\s+/g, " ")+ .trim()+ .slice(0, 80);+}++function finiteNumber(value: unknown): number | undefined {+ return typeof value === "number" && Number.isFinite(value)+ ? value+ : undefined;+}++function setDefined(+ signals: AgentHintSignals,+ name: string,+ value: AgentHintSignalValue | undefined,+): void {+ if (value !== undefined) signals.set(name, value);+}++function addPageSignals(+ signals: AgentHintSignals,+ response: ObjectValue,+ metadata: ObjectValue,+): void {+ if (typeof metadata.statusCode === "number") {+ signals.set("page_status", metadata.statusCode);+ }++ const scrapeId =+ typeof metadata.scrapeId === "string" && metadata.scrapeId+ ? metadata.scrapeId+ : typeof response.scrape_id === "string" && response.scrape_id+ ? response.scrape_id+ : undefined;+ if (scrapeId) signals.set("scrape_id", encodeURIComponent(scrapeId));++ const source = httpUrl(metadata.sourceURL);+ const final = httpUrl(metadata.url);+ if (source && final && source.href !== final.href) {+ const from = quotedUrl(source);+ const to = quotedUrl(final);+ if (from && to) {+ signals.set("page_redirect_from", from);+ signals.set("page_redirect_to", to);+ }+ }+ const page = final ?? source;+ if (page) {+ signals.set("page_host", page.hostname);+ const words = pathWords(page);+ if (words) signals.set("page_path_words", words);+ }++ const returned = finiteNumber(metadata.numPages);+ const total = finiteNumber(metadata.totalPages);+ setDefined(signals, "document_pages_returned", returned);+ setDefined(signals, "document_pages_total", total);+ if (returned !== undefined && total !== undefined) {+ const maxPages = Math.min(total, PDF_MAX_PAGES_LIMIT);+ signals.set("document_max_pages", maxPages);+ signals.set("document_pages_requestable", Math.max(0, maxPages - returned));+ }+}++function addResultSignals(signals: AgentHintSignals, web: unknown[]): void {+ signals.set("result_count", web.length);++ const excerpts: string[] = [];+ const originCounts = new Map<string, number>();+ web.forEach((value, index) => {+ const item = object(value);+ const url = typeof item.url === "string" ? item.url : undefined;+ if (url === undefined) return;+ const parsed = httpUrl(url);+ if (+ item.markdown === undefined &&+ item.html === undefined &&+ item.rawHtml === undefined+ ) {+ const quoted = parsed ? quotedUrl(parsed) : undefined;+ const position = resultPosition(item, index);+ excerpts.push(quoted ? `#${position} ${quoted}` : `#${position}`);+ }+ if (parsed) {+ originCounts.set(+ parsed.origin,+ (originCounts.get(parsed.origin) ?? 0) + 1,+ );+ }+ });++ signals.set("excerpt_count", excerpts.length);+ signals.set("excerpt_results", excerpts);+ if (web.length > 0) {+ signals.set("excerpt_share", excerpts.length / web.length);+ }++ const originResults = [...originCounts.values()].reduce(+ (total, count) => total + count,+ 0,+ );+ signals.set("origin_result_count", originResults);+ const top = [...originCounts.entries()].sort((a, b) => b[1] - a[1])[0];+ if (top) {+ signals.set("top_origin", top[0]);+ signals.set("top_origin_count", top[1]);+ signals.set("top_origin_share", top[1] / originResults);+ }+}++export function computeAgentHintSignals(+ context: AgentHintSignalContext,+): AgentHintSignals {+ const response = object(context.response);+ const data = object(response.data);+ const signals: AgentHintSignals = new Map();++ signals.set("endpoint", context.endpoint);+ signals.set("success", response.success === true);+ signals.set("can_use_map_and_crawl", context.canUseMapAndCrawl === true);+ signals.set("can_use_interact", context.canUseInteract === true);+ setDefined(+ signals,+ "remaining_credits",+ finiteNumber(context.remainingCredits),+ );++ addPageSignals(signals, response, object(data.metadata));++ const web = Array.isArray(response.data)+ ? response.data+ : Array.isArray(data.web)+ ? data.web+ : undefined;+ if (web) addResultSignals(signals, web);++ return signals;+}diff --git a/apps/api/src/lib/agent-hints-provider.test.ts b/apps/api/src/lib/agent-hints-provider.test.tsindex f82720653..f2f0e7553 100644--- a/apps/api/src/lib/agent-hints-provider.test.ts+++ b/apps/api/src/lib/agent-hints-provider.test.ts@@ -110,7 +110,7 @@ describe("external agent hints provider", () => {it("returns an unsettled holder immediately and fills it once the provider answers", async () => {const holder = getProviderHints(context());- expect(holder).toEqual({ settled: false, hints: [] });+ expect(holder).toEqual({ settled: false, hints: [], rules: [] });await settle(holder);expect(holder!.hints).toEqual([{ id: "h1", text: "Provider hint." }]);});@@ -136,7 +136,7 @@ describe("external agent hints provider", () => {].sort(),);expect(body).toEqual({- version: 1,+ version: 2,team_id: "team-contract",org_id: "org-1",api_key_id: 42,@@ -367,6 +367,91 @@ describe("external agent hints provider", () => {]);});+ it("parses a valid provider rule set and cleans rule text", async () => {+ handler = reply(200, {+ hints: [],+ rules: [+ {+ id: "r1",+ group: "g",+ when: [+ { signal: "endpoint", op: "eq", value: "scrape" },+ { signal: "page_status", op: "in", value: [404, 410] },+ ],+ text: " Rule one {page_status}.\n",+ },+ { id: "no-when", text: "Always." },+ ],+ ttl_seconds: 60,+ });+ const holder = getProviderHints(context());+ await settle(holder);+ expect(holder!.rules).toEqual([+ {+ id: "r1",+ group: "g",+ when: [+ { signal: "endpoint", op: "eq", value: "scrape" },+ { signal: "page_status", op: "in", value: [404, 410] },+ ],+ text: "Rule one {page_status}.",+ },+ { id: "no-when", when: [], text: "Always." },+ ]);+ });++ it("treats a missing rules field as no rules", async () => {+ const holder = getProviderHints(context());+ await settle(holder);+ expect(holder!.hints).toEqual([{ id: "h1", text: "Provider hint." }]);+ expect(holder!.rules).toEqual([]);+ });++ it.each([+ ["a non-array rules field", { id: "r1" }],+ [+ "one rule with an unknown operator",+ [+ { id: "ok", when: [], text: "Fine." },+ {+ id: "bad-op",+ when: [{ signal: "endpoint", op: "matches", value: "s" }],+ text: "Bad.",+ },+ ],+ ],+ [+ "one rule with a string threshold",+ [+ {+ id: "bad-value",+ when: [{ signal: "remaining_credits", op: "lt", value: "100" }],+ text: "Bad.",+ },+ ],+ ],+ ["one rule with blank text", [{ id: "blank", when: [], text: " " }]],+ ["a non-object rule", ["not a rule"]],+ ])(+ "treats %s as a failed lookup with a negative cache",+ async (_name, rules) => {+ handler = reply(200, {+ hints: [{ id: "h1", text: "Provider hint." }],+ rules,+ ttl_seconds: 60,+ });+ const before = await counter("error");+ const ctx = context();+ const holder = getProviderHints(ctx);+ await settle(holder);+ expect(holder!.hints).toEqual([]);+ expect(holder!.rules).toEqual([]);+ expect(await counter("error")).toBe(before + 1);+ expect(getProviderHints(ctx)).toBe(holder);+ expect(requests).toHaveLength(1);+ },+ );+it("bounds the cache and evicts the oldest entry first", async () => {const fetchSpy = vi.spyOn(globalThis, "fetch").mockImplementation(async () =>diff --git a/apps/api/src/lib/agent-hints-provider.ts b/apps/api/src/lib/agent-hints-provider.tsindex 096c84ad2..e72aeabd2 100644--- a/apps/api/src/lib/agent-hints-provider.ts+++ b/apps/api/src/lib/agent-hints-provider.ts@@ -2,21 +2,27 @@ import { createHmac, randomBytes } from "node:crypto";import { Counter } from "prom-client";import { z } from "zod";import { config } from "../config";-import type { AgentHintEndpoint } from "./agent-hints";+import { parseAgentHintRules, type AgentHintRule } from "./agent-hint-rules";+import type { AgentHintEndpoint } from "./agent-hint-signals";import type { KeylessSignupSurface } from "./keyless-signup-link";import { logger } from "./logger";/*** Optional external agent hints provider. When AGENT_HINTS_PROVIDER_URL is* set, hint-enabled requests ask a separate HTTP service for additional- * guidance. The lookup starts after auth and is never awaited: the response- * only carries provider hints that have already arrived (or are cached) by- * the time it is sent. Every failure means "no provider hints".+ * guidance: finished hints and response guidance rules. The lookup starts+ * after auth and is never awaited: the response only uses provider data that+ * has already arrived (or is cached) by the time it is sent. Every failure+ * means "no provider hints and no rules".*/export type ProviderHint = { id: string; text: string };/** Shared by every request for the same cache key; read-only for callers. */-export type ProviderHintsHolder = { settled: boolean; hints: ProviderHint[] };+export type ProviderHintsHolder = {+ settled: boolean;+ hints: ProviderHint[];+ rules: AgentHintRule[];+};export type AgentHintsProviderContext = {teamId: string;@@ -37,16 +43,18 @@ const MAX_STORED_HINTS = 10;const MAX_RESPONSE_BYTES = 64 * 1024;const MAX_PROVIDER_HINTS = 2;const MAX_TOTAL_HINTS = 3;+export const AGENT_HINTS_PROVIDER_CONTRACT_VERSION = 2;// Fixed labels only: never put team IDs or provider-supplied values here.export const agentHintsProviderRequestsTotal = new Counter({name: "firecrawl_agent_hints_provider_requests_total",- help: "External agent hints provider lookups by outcome: hit (served from cache or joined an in-flight fetch), miss (fetch started), timeout, error, disabled (no provider configured)",+ help: "External agent hints provider lookups by outcome: hit (served from cache or joined an in-flight fetch), miss (fetch started), timeout, error, disabled (no provider configured); eval_error counts responses whose hint computation failed and were sent without hints",labelNames: ["outcome"],});const responseSchema = z.object({hints: z.array(z.unknown()),+ rules: z.unknown().optional(),ttl_seconds: z.number().finite().optional().catch(undefined),});@@ -116,7 +124,7 @@ function sanitizeHints(values: unknown[]): ProviderHint[] {async function fetchProviderHints(url: string,context: AgentHintsProviderContext,-): Promise<{ hints: ProviderHint[]; ttlMs: number }> {+): Promise<{ hints: ProviderHint[]; rules: AgentHintRule[]; ttlMs: number }> {const secret = config.AGENT_HINTS_PROVIDER_SECRET;try {const response = await fetch(url, {@@ -126,7 +134,7 @@ async function fetchProviderHints(...(secret ? { authorization: `Bearer ${secret}` } : {}),},body: JSON.stringify({- version: 1,+ version: AGENT_HINTS_PROVIDER_CONTRACT_VERSION,team_id: providerTeamId(context.teamId),org_id: context.orgId,api_key_id: context.apiKeyId,@@ -142,12 +150,18 @@ async function fetchProviderHints(}const parsed = responseSchema.safeParse(await readJsonBody(response));if (!parsed.success) throw new Error("Malformed response body");+ const rules =+ parsed.data.rules === undefined+ ? []+ : parseAgentHintRules(parsed.data.rules);+ if (!rules) throw new Error("Malformed rules");const ttlSeconds = Math.min(Math.max(parsed.data.ttl_seconds ?? DEFAULT_TTL_SECONDS, 0),MAX_TTL_SECONDS,);return {hints: sanitizeHints(parsed.data.hints),+ rules,ttlMs: ttlSeconds * 1000,};} catch (error) {@@ -161,7 +175,7 @@ async function fetchProviderHints(timedOut,error: error instanceof Error ? error.message : String(error),});- return { hints: [], ttlMs: NEGATIVE_TTL_MS };+ return { hints: [], rules: [], ttlMs: NEGATIVE_TTL_MS };}}@@ -194,12 +208,13 @@ export function getProviderHints(if (oldest !== undefined) cache.delete(oldest);}const entry: CacheEntry = {- holder: { settled: false, hints: [] },+ holder: { settled: false, hints: [], rules: [] },expiresAt: Infinity,};cache.set(key, entry);- void fetchProviderHints(url, context).then(({ hints, ttlMs }) => {+ void fetchProviderHints(url, context).then(({ hints, rules, ttlMs }) => {entry.holder.hints = hints;+ entry.holder.rules = rules;entry.holder.settled = true;entry.expiresAt = Date.now() + ttlMs;});@@ -207,14 +222,14 @@ export function getProviderHints(}/**- * Deterministic hints first and never dropped, then provider hints that are- * not exact duplicates, up to the provider and total caps.+ * Rule hints first and never dropped, then provider hints that are not exact+ * duplicates, up to the provider and total caps.*/export function mergeAgentHints(- deterministic: string[],+ ruleHints: string[],provider: ProviderHint[],): { hints: string[]; providerHintIds: string[] } {- const hints = [...deterministic];+ const hints = [...ruleHints];const providerHintIds: string[] = [];for (const hint of provider) {if (diff --git a/apps/api/src/lib/agent-hints.ts b/apps/api/src/lib/agent-hints.tsindex 26e9e36ce..5ea09fb88 100644--- a/apps/api/src/lib/agent-hints.ts+++ b/apps/api/src/lib/agent-hints.ts@@ -1,5 +1,15 @@+import {+ httpUrl,+ object,+ pathWords,+ quotedUrl,+ resultPosition,+ type AgentHintEndpoint,+} from "./agent-hint-signals";++export type { AgentHintEndpoint };+/** Deterministic guidance from state already available on the request path. */-export type AgentHintEndpoint = "search" | "scrape" | "parse" | "map";export interface AgentHintContext {endpoint: AgentHintEndpoint;response: unknown;@@ -14,11 +24,6 @@ const SEARCH_CLUSTER_MIN_ORIGIN_RESULTS = 3;const SEARCH_CLUSTER_MIN_SHARE = 0.75;type ObjectValue = Record<string, unknown>;-function object(value: unknown): ObjectValue {- return value !== null && typeof value === "object" && !Array.isArray(value)- ? (value as ObjectValue)- : {};-}function clusteredOrigin(web: unknown[]): string | undefined {const counts = new Map<string, number>();for (const value of web) {@@ -49,41 +54,6 @@ function clusteredOrigin(web: unknown[]): string | undefined {}const EXCERPT_URLS_SHOWN = 3;-const QUERY_PATH_SEGMENTS = 2;--function httpUrl(value: unknown): URL | undefined {- if (typeof value !== "string" || !value) return undefined;- try {- const parsed = new URL(value);- return parsed.protocol === "http:" || parsed.protocol === "https:"- ? parsed- : undefined;- } catch {- return undefined;- }-}--const HINT_URL_MAX_CHARS = 200;--/**- * Result URLs come from third-party pages, so they are rendered only as a- * parsed http(s) href (whitespace and quotes percent-encoded), length-capped,- * and JSON-quoted to mark them as data rather than instruction text.- */-function quotedUrl(url: URL): string | undefined {- const href = url.href;- return href.length <= HINT_URL_MAX_CHARS ? JSON.stringify(href) : undefined;-}--function resultPosition(item: ObjectValue, index: number): number {- const position = item.position;- return typeof position === "number" &&- Number.isInteger(position) &&- position > 0- ? position- : index + 1;-}-/** Names the excerpt-only results by their response position so the agent can pick. */function excerptOnlyHint(web: unknown[]): string | undefined {const excerpts: { position: number; url: string | undefined }[] = [];@@ -119,45 +89,6 @@ function excerptOnlyHint(web: unknown[]): string | undefined {return `${subject} (${listed}${more}). If you need more than an excerpt, use firecrawl_scrape with {"url":"<one of these URLs>","formats":["markdown"]}. Retrieve only needed pages; do not re-scrape results that already contain the required content.`;}-/**- * Identifier-like path segments carry no searchable meaning: numbers, hex or- * UUID-style ids, ULIDs, and long mixed alphanumeric tokens such as- * "W020260806515694454560".- */-function isOpaqueId(segment: string): boolean {- if (/^\d+$/.test(segment)) return true;- if (/^[0-9a-f-]{16,}$/i.test(segment)) return true;- if (/^[0-9A-HJKMNP-TV-Z]{26}$/i.test(segment)) return true;- return (- segment.length >= 12 &&- /^[A-Za-z0-9_]+$/.test(segment) &&- (segment.match(/\d/g)?.length ?? 0) >= 4- );-}--/** Words from the last path segments, e.g. /payments/checkout/migration-from-legacy -> "checkout migration from legacy". */-function pathWords(url: URL): string {- const segments = url.pathname- .split("/")- .map(segment => {- try {- return decodeURIComponent(segment);- } catch {- return segment;- }- })- .map(segment => segment.replace(/\.[a-z0-9]{1,5}$/i, ""))- .filter(segment => segment && !isOpaqueId(segment));- return segments- .slice(-QUERY_PATH_SEGMENTS)- .join(" ")- .replace(/[-_+.]+/g, " ")- .replace(/[^\p{L}\p{N} ]+/gu, " ")- .replace(/\s+/g, " ")- .trim()- .slice(0, 80);-}-function redirectNote(source: URL, final: URL): string {const from = quotedUrl(source);const to = quotedUrl(final);diff --git a/apps/api/src/middlewares/__tests__/agent-hints.test.ts b/apps/api/src/middlewares/__tests__/agent-hints.test.tsindex 8b1dc7da4..b9d5b04f1 100644--- a/apps/api/src/middlewares/__tests__/agent-hints.test.ts+++ b/apps/api/src/middlewares/__tests__/agent-hints.test.ts@@ -1,12 +1,50 @@import express from "express";import request from "supertest";import { config } from "../../config";-import type { ProviderHintsHolder } from "../../lib/agent-hints-provider";+import { evaluateAgentHintRules } from "../../lib/agent-hint-rules";+import { computeAgentHintSignals } from "../../lib/agent-hint-signals";+import { buildAgentHints } from "../../lib/agent-hints";+import {+ agentHintsProviderRequestsTotal,+ getProviderHints,+ type ProviderHintsHolder,+} from "../../lib/agent-hints-provider";import {agentHintsMiddleware,agentHintsProviderMiddleware,} from "../agent-hints";+vi.mock("../../lib/agent-hint-rules", async importOriginal => {+ const actual =+ await importOriginal<typeof import("../../lib/agent-hint-rules")>();+ return {+ ...actual,+ evaluateAgentHintRules: vi.fn(actual.evaluateAgentHintRules),+ };+});+vi.mock("../../lib/agent-hint-signals", async importOriginal => {+ const actual =+ await importOriginal<typeof import("../../lib/agent-hint-signals")>();+ return {+ ...actual,+ computeAgentHintSignals: vi.fn(actual.computeAgentHintSignals),+ };+});+vi.mock("../../lib/agent-hints", async importOriginal => {+ const actual = await importOriginal<typeof import("../../lib/agent-hints")>();+ return { ...actual, buildAgentHints: vi.fn(actual.buildAgentHints) };+});+vi.mock("../../lib/agent-hints-provider", async importOriginal => {+ const actual =+ await importOriginal<typeof import("../../lib/agent-hints-provider")>();+ return { ...actual, getProviderHints: vi.fn(actual.getProviderHints) };+});++async function evalErrors(): Promise<number> {+ const metric = await agentHintsProviderRequestsTotal.get();+ return metric.values.find(v => v.labels.outcome === "eval_error")?.value ?? 0;+}+function appFor({endpoint = "search",@@ -279,7 +317,11 @@ describe("agent hint response middleware with an external provider", () => {it("appends settled provider hints after deterministic hints", async () => {const response = await request(holderAppFor(- { settled: true, hints: [{ id: "p1", text: "Provider hint." }] },+ {+ settled: true,+ hints: [{ id: "p1", text: "Provider hint." }],+ rules: [],+ },EXCERPT_BODY,),)@@ -295,7 +337,11 @@ describe("agent hint response middleware with an external provider", () => {const body = { success: true, data: {} };const response = await request(holderAppFor(- { settled: true, hints: [{ id: "p1", text: "Provider hint." }] },+ {+ settled: true,+ hints: [{ id: "p1", text: "Provider hint." }],+ rules: [],+ },body,),)@@ -309,7 +355,11 @@ describe("agent hint response middleware with an external provider", () => {const body = { success: true, data: {} };const response = await request(holderAppFor(- { settled: false, hints: [{ id: "p1", text: "Provider hint." }] },+ {+ settled: false,+ hints: [{ id: "p1", text: "Provider hint." }],+ rules: [],+ },body,),)@@ -323,7 +373,11 @@ describe("agent hint response middleware with an external provider", () => {const body = { success: false, error: "Bad URL", code: "BAD_REQUEST" };const response = await request(holderAppFor(- { settled: true, hints: [{ id: "p1", text: "Provider hint." }] },+ {+ settled: true,+ hints: [{ id: "p1", text: "Provider hint." }],+ rules: [],+ },body,),)@@ -337,7 +391,11 @@ describe("agent hint response middleware with an external provider", () => {const body = { success: true, data: {} };const response = await request(holderAppFor(- { settled: true, hints: [{ id: "p1", text: "Provider hint." }] },+ {+ settled: true,+ hints: [{ id: "p1", text: "Provider hint." }],+ rules: [],+ },body,),)@@ -347,6 +405,151 @@ describe("agent hint response middleware with an external provider", () => {});});+describe("agent hint response middleware with provider rules", () => {+ const rules = [+ {+ id: "results",+ group: "next",+ when: [+ { signal: "success", op: "eq" as const, value: true },+ { signal: "excerpt_count", op: "gt" as const, value: 0 },+ ],+ text: "Rule: {excerpt_count} of {result_count}.",+ },+ {+ id: "credits",+ when: [{ signal: "remaining_credits", op: "lt" as const, value: 10 }],+ text: "Rule: credits {remaining_credits}.",+ },+ ];++ it("serves rule hints instead of the built-in hints when rules are present", async () => {+ const response = await request(+ holderAppFor(+ {+ settled: true,+ hints: [{ id: "p1", text: "Provider hint." }],+ rules,+ },+ EXCERPT_BODY,+ ),+ )+ .post("/")+ .set("X-Firecrawl-Agent-Hints", "true")+ .send({});+ expect(response.body.agent_hints).toEqual([+ "Rule: 1 of 1.",+ "Provider hint.",+ ]);+ });++ it("evaluates rules on failure envelopes but adds no provider hints there", async () => {+ const app = express();+ app.use(express.json());+ app.post("/", agentHintsMiddleware("scrape"), (req, res) => {+ (req as any).auth = { team_id: "account-team" };+ res.locals.agentCreditsRemaining = 3;+ res.locals.agentHintsProvider = {+ settled: true,+ hints: [{ id: "p1", text: "Provider hint." }],+ rules,+ };+ res.status(400).json({ success: false, error: "Bad request" });+ });+ const response = await request(app)+ .post("/")+ .set("X-Firecrawl-Agent-Hints", "true")+ .send({});+ expect(response.body.agent_hints).toEqual(["Rule: credits 3."]);+ });++ it("uses the built-in hints while the provider has not answered", async () => {+ const response = await request(+ holderAppFor({ settled: false, hints: [], rules: [] }, EXCERPT_BODY),+ )+ .post("/")+ .set("X-Firecrawl-Agent-Hints", "true")+ .send({});+ expect(response.body.agent_hints).toHaveLength(1);+ expect(response.body.agent_hints[0]).toContain("firecrawl_scrape");+ });+});++describe("agent hint failures never change the response", () => {+ const RULES = [+ {+ id: "results",+ when: [{ signal: "excerpt_count", op: "gt" as const, value: 0 }],+ text: "Rule: {excerpt_count}.",+ },+ ];+ const boom = () => {+ throw new Error("boom");+ };++ afterEach(() => {+ vi.mocked(evaluateAgentHintRules).mockClear();+ });++ it.each([+ [+ "the matcher",+ () => vi.mocked(evaluateAgentHintRules).mockImplementationOnce(boom),+ RULES,+ ],+ [+ "signal computation",+ () => vi.mocked(computeAgentHintSignals).mockImplementationOnce(boom),+ RULES,+ ],+ [+ "the built-in hints",+ () => vi.mocked(buildAgentHints).mockImplementationOnce(boom),+ [],+ ],+ ])(+ "sends the original body with status 200 when %s throws",+ async (_name, arm, rules) => {+ arm();+ const before = await evalErrors();+ const response = await request(+ holderAppFor({ settled: true, hints: [], rules }, EXCERPT_BODY),+ )+ .post("/")+ .set("X-Firecrawl-Agent-Hints", "true")+ .send({});+ expect(response.statusCode).toBe(200);+ expect(response.body).toEqual(EXCERPT_BODY);+ expect(await evalErrors()).toBe(before + 1);+ },+ );++ it("still responds when the provider lookup cannot start", async () => {+ vi.mocked(getProviderHints).mockImplementationOnce(boom);+ const app = express();+ app.use(express.json());+ app.post(+ "/",+ agentHintsMiddleware("search"),+ (req, _res, next) => {+ (req as any).auth = { team_id: "account-team" };+ next();+ },+ agentHintsProviderMiddleware("search"),+ (_req, res) => {+ res.json(EXCERPT_BODY);+ },+ );+ const response = await request(app)+ .post("/")+ .set("X-Firecrawl-Agent-Hints", "true")+ .send({});+ expect(response.statusCode).toBe(200);+ expect(response.body.agent_hints).toHaveLength(1);+ expect(response.body.agent_hints[0]).toContain("firecrawl_scrape");+ });+});+describe("agent hints provider middleware", () => {const original = {url: config.AGENT_HINTS_PROVIDER_URL,@@ -416,6 +619,50 @@ describe("agent hints provider middleware", () => {expect(fetchSpy).not.toHaveBeenCalled();});+ it("falls back to the built-in hints when the provider sends a malformed rule set", async () => {+ vi.spyOn(globalThis, "fetch").mockImplementation(+ async () =>+ new Response(+ JSON.stringify({+ hints: [{ id: "p1", text: "Provider hint." }],+ rules: [+ { id: "ok", when: [], text: "Rule hint." },+ { id: "bad", when: [{ signal: "s", op: "nope" }], text: "x" },+ ],+ ttl_seconds: 60,+ }),+ { status: 200 },+ ),+ );+ const body = { success: true, data: { metadata: { statusCode: 404 } } };+ const app = express();+ app.use(express.json());+ app.post(+ "/",+ agentHintsMiddleware("scrape"),+ (req, _res, next) => {+ (req as any).auth = { team_id: `team-${++teamCounter}` };+ next();+ },+ agentHintsProviderMiddleware("scrape"),+ async (_req, res) => {+ await vi.waitFor(() =>+ expect(res.locals.agentHintsProvider?.settled).toBe(true),+ );+ res.json(body);+ },+ );+ const response = await request(app)+ .post("/")+ .set("X-Firecrawl-Agent-Hints", "true")+ .send({});+ expect(response.statusCode).toBe(200);+ expect(response.body.agent_hints).toHaveLength(1);+ expect(response.body.agent_hints[0]).toContain("firecrawl_search");+ expect(JSON.stringify(response.body)).not.toContain("Rule hint.");+ expect(JSON.stringify(response.body)).not.toContain("Provider hint.");+ });+it("does not call the provider without the opt-in header", async () => {const fetchSpy = mockProvider();await request(providerAppFor({ team_id: `team-${++teamCounter}` }))@@ -449,7 +696,7 @@ describe("agent hints provider middleware", () => {expect(fetchSpy).toHaveBeenCalledTimes(1);const body = String(fetchSpy.mock.calls[0][1]?.body);expect(JSON.parse(body)).toEqual({- version: 1,+ version: 2,team_id: teamId,org_id: "org-1",api_key_id: 7,diff --git a/apps/api/src/middlewares/agent-hints.ts b/apps/api/src/middlewares/agent-hints.tsindex 5f88f70ae..0e8b6632f 100644--- a/apps/api/src/middlewares/agent-hints.ts+++ b/apps/api/src/middlewares/agent-hints.ts@@ -1,8 +1,14 @@-import type { Request, RequestHandler } from "express";+import type { Request, RequestHandler, Response } from "express";import { config } from "../config";import type { RequestWithMaybeAuth } from "../controllers/v1/types";-import { buildAgentHints, type AgentHintEndpoint } from "../lib/agent-hints";+import { evaluateAgentHintRules } from "../lib/agent-hint-rules";import {+ computeAgentHintSignals,+ type AgentHintEndpoint,+} from "../lib/agent-hint-signals";+import { buildAgentHints } from "../lib/agent-hints";+import {+ agentHintsProviderRequestsTotal,getProviderHints,mergeAgentHints,providerTeamId,@@ -15,7 +21,45 @@ function agentHintsRequested(req: Request): boolean {return req.get("X-Firecrawl-Agent-Hints")?.trim().toLowerCase() === "true";}-/** Only registered on business POST routes, never feedback or polling routes. */+function agentHintsFor(+ req: Request,+ res: Response,+ endpoint: AgentHintEndpoint,+ body: Record<string, unknown>,+): string[] {+ const teamId = (req as RequestWithMaybeAuth).auth?.team_id;+ const provider: ProviderHintsHolder | undefined =+ res.locals.agentHintsProvider;+ const context = {+ endpoint,+ response: body,+ remainingCredits: res.locals.agentCreditsRemaining,+ canUseMapAndCrawl: !!teamId && !teamId.startsWith("preview_keyless_"),+ canUseInteract: config.USE_DB_AUTHENTICATION === true,+ };+ const rules = provider?.settled ? provider.rules : [];+ const { hints, providerHintIds } = mergeAgentHints(+ rules.length > 0+ ? evaluateAgentHintRules(rules, computeAgentHintSignals(context))+ : buildAgentHints(context),+ provider?.settled && body.success === true ? provider.hints : [],+ );+ if (providerHintIds.length > 0) {+ logger.info("Served external agent hints", {+ module: "agent-hints-provider",+ endpoint,+ team_id: teamId ? providerTeamId(teamId) : undefined,+ hint_ids: providerHintIds,+ });+ }+ return hints;+}++/**+ * Only registered on business POST routes, never feedback or polling routes.+ * Hint computation must never change the response otherwise: any error sends+ * the original body unchanged.+ */export function agentHintsMiddleware(endpoint: AgentHintEndpoint,): RequestHandler {@@ -25,31 +69,19 @@ export function agentHintsMiddleware(res.json = function (body) {if (!body || typeof body !== "object" || Array.isArray(body))return json.call(this, body);- const teamId = (req as RequestWithMaybeAuth).auth?.team_id;- const provider: ProviderHintsHolder | undefined =- res.locals.agentHintsProvider;- const { hints, providerHintIds } = mergeAgentHints(- buildAgentHints({+ let output = body;+ try {+ const hints = agentHintsFor(req, res, endpoint, body);+ if (hints.length > 0) output = { ...body, agent_hints: hints };+ } catch (error) {+ agentHintsProviderRequestsTotal.inc({ outcome: "eval_error" });+ logger.warn("Agent hints computation failed", {+ module: "agent-hints",endpoint,- response: body,- remainingCredits: res.locals.agentCreditsRemaining,- canUseMapAndCrawl: !!teamId && !teamId.startsWith("preview_keyless_"),- canUseInteract: config.USE_DB_AUTHENTICATION === true,- }),- provider?.settled && body.success === true ? provider.hints : [],- );- if (providerHintIds.length > 0) {- logger.info("Served external agent hints", {- module: "agent-hints-provider",- endpoint,- team_id: teamId ? providerTeamId(teamId) : undefined,- hint_ids: providerHintIds,+ error: error instanceof Error ? error.message : String(error),});}- return json.call(- this,- hints.length > 0 ? { ...body, agent_hints: hints } : body,- );+ return json.call(this, output);};next();};@@ -68,17 +100,25 @@ export function agentHintsProviderMiddleware(const teamId = authReq.auth?.team_id;if (!teamId) return next();const apiKeyId = authReq.acuc?.api_key_id;- res.locals.agentHintsProvider = getProviderHints({- teamId,- orgId: authReq.auth?.org_id ?? null,- apiKeyId:- typeof apiKeyId === "number" && Number.isFinite(apiKeyId)- ? apiKeyId- : null,- endpoint,- surface: keylessSignupSurface(req),- keyless: teamId.startsWith("preview_keyless_"),- });+ try {+ res.locals.agentHintsProvider = getProviderHints({+ teamId,+ orgId: authReq.auth?.org_id ?? null,+ apiKeyId:+ typeof apiKeyId === "number" && Number.isFinite(apiKeyId)+ ? apiKeyId+ : null,+ endpoint,+ surface: keylessSignupSurface(req),+ keyless: teamId.startsWith("preview_keyless_"),+ });+ } catch (error) {+ logger.warn("Agent hints provider lookup could not start", {+ module: "agent-hints-provider",+ endpoint,+ error: error instanceof Error ? error.message : String(error),+ });+ }next();};}