Skip to content

Supercharge your AI agents with data from the web and beyond. Building the library for superintelligence. 🔥

chore: improve extractor system prompt (#3715)

tomsideguide2671aeb346b8
diff --git a/apps/api/src/lib/deterministicJson/llm/prompts.ts b/apps/api/src/lib/deterministicJson/llm/prompts.ts
index 4414790c6..0aed99466 100644
--- a/apps/api/src/lib/deterministicJson/llm/prompts.ts
+++ b/apps/api/src/lib/deterministicJson/llm/prompts.ts
@@ -35,35 +35,81 @@ Return only JSON of the form {"snippets": ["...", "..."]} - no markdown fences,
// Phase 2: write the reusable extractor. Kept generic on purpose - the sandbox
// (a browser-like DOM with no network) enforces the environment, so the prompt
// describes intent, not an inventory of available globals.
-const EXTRACTOR_SYSTEM = `You write ONE reusable JavaScript function that extracts structured data from a web page.
+const EXTRACTOR_SYSTEM = `You write one reusable JavaScript function:
async function extract(doc, askLlm) { ... }
-- \`doc\` is a standard DOM Document: querySelector / querySelectorAll, document.evaluate (XPath), textContent, getAttribute, etc.
-- \`askLlm(prompt)\` returns a string; \`askLlm(prompt, jsonSchema)\` returns a value of that shape. It can fail and return null, so guard its result.
+- \`doc\` is a standard DOM Document: querySelector, querySelectorAll, document.evaluate, textContent, getAttribute, etc.
+- \`askLlm(prompt)\` returns a string.
+- \`askLlm(prompt, jsonSchema)\` returns a value of that schema shape. It may return null, so guard its result.
-This function is generated once from ONE sample page, cached, then re-run on OTHER pages of the same shape. Therefore:
+Return contract:
+- Return only the function source, beginning with \`async function extract\`.
+- The returned data must match the target JSON Schema exactly: required fields, correct types, and no extra keys.
+- Every returned value must be read from the runtime DOM or from askLlm at runtime.
+- Never return a literal value copied from the sample page. Literals are allowed only for selectors, attribute names, regexes, enum values, error messages, and returned object keys.
+- Do not use network, storage, timers, randomness, browser navigation, mutation observers, external libraries, or DOM mutation.
-- Read every value from the DOM at runtime. NEVER return a literal you saw in the sample page - the values differ on other pages. Literals are only allowed in selectors, attribute names, regexes, and the returned object's keys.
-- The returned value must match the target JSON Schema exactly: include required fields, use the correct types, and do not add extra keys.
-- For required schema fields or fields central to the task, throw if the expected anchor/container is missing.
-- Return null or [] only when the surrounding container was found and the specific optional value is genuinely absent. Do not treat a missing primary container as optional absence.
-- Do NOT hide breakage. Use one primary extraction strategy per value. Do not silently fall back to broader whole-page scraping when it fails. If you handle known explicit variants, branch on a specific container or page-state check, and throw when none match.
-- Anchor every value to a specific element. Never fish for a value by matching a pattern (e.g. a price, a date, an email) over doc.body.innerText or the whole page - you will match an unrelated occurrence and return a confident, wrong value.
+Failure behavior:
+- If a required container or required value is missing, throw an Error with a specific message.
+- Return null, "", or [] only when the correct surrounding container was found and that specific optional value is genuinely absent.
+- Never fabricate a schema-shaped fallback object after extraction fails.
+- Do not use try/catch to swallow extraction errors. Let failures throw, or throw a clearer Error.
+- Use one primary extraction strategy per value. Explicit branches for known page variants are okay; throw if none match.
+
+Extraction strategy:
+- Prefer specific structured sources: semantic DOM, labels, headings, tables, attributes, JSON scripts, inline state blobs, or nearby containers from the provided HTML snippets.
+- Anchor values to specific elements when the page has meaningful structure.
+- Broad page-text reads such as doc.body.innerText or doc.body.textContent are allowed only as the primary strategy when the page itself is raw JSON/plaintext, or when the requested value only exists in rendered body text.
+- Regex over broad page text is allowed only when no more specific DOM container or structured source exists. Do not use broad regex as a fallback after a selector fails.
+- If data is JSON in a specific element, JSON.parse that element's textContent and read fields directly.
+- If the whole page is JSON/plaintext, parse or search the exact available body text.
+- If JSON is surrounded by non-JSON text and there is no better container, extracting the JSON substring with regex is acceptable as the primary strategy.
+- Never ask askLlm to parse JSON you can parse deterministically.
- For arrays/lists, first select the repeated item/container elements, then extract each field relative to that item. Do not query the whole document for per-item fields inside the loop.
-- When a value is present, coerce it to the schema's type (e.g. strip non-numerics for numbers). Keep dates as the page displays them unless the task asks for a specific format.
-- Do not use network, storage, timers, randomness, browser navigation, mutation observers, external libraries, or DOM mutation.
+- When a value is present, coerce it to the schema's type. Keep dates as displayed unless the task asks for a specific format.
Selectors:
-- Anchor on stable, meaningful things: labels, semantic attributes (itemprop, role, data-*), heading text. Avoid fragile position (:nth-child, sibling chains).
-- Prefer descendant combinators over child combinators (\`>\`); an unseen wrapper element breaks \`>\`. A selector that matches 0 elements is usually too strict, not proof the data is absent.
-- Read URLs from href/src, not from link text.
-- Only standard CSS selectors work. To match by visible text, select structurally then filter in JS, e.g. \`[...doc.querySelectorAll('tr')].find(r => r.textContent.includes('Total'))\`.
+- Prefer stable, meaningful anchors: labels, headings, semantic attributes, itemprop, role, data-*, table headers, and nearby static text.
+- Avoid fragile position selectors such as :nth-child and long sibling chains.
+- Prefer descendant combinators over child combinators (\`>\`), because unseen wrappers often break direct-child selectors.
+- Read URLs from href/src attributes, not from link text.
+- CSS selectors cannot match visible text directly. To match text, select candidate elements then filter in JS, e.g. \`[...doc.querySelectorAll("tr")].find(r => r.textContent.includes("Total"))\`.
askLlm:
-- If the data is already structured (a JSON body, a <script type="application/ld+json">, an inline state blob), JSON.parse it and read the field directly. Never ask askLlm to parse JSON you could parse deterministically.
-- Use it for what selectors and parsing cannot do - summarize, classify, translate, disambiguate by meaning. It transforms the text you pass it (no outside lookups like currency conversion; returns null when the text lacks the answer), so pass the DOM text it needs - for a summary, the article body, not a bare title or label. If a field has a fixed value set, include it in the prompt.
-- When you call askLlm once per item over a list, issue the calls concurrently with Promise.all rather than awaiting one at a time.
+- Use askLlm only for semantic transformations selectors cannot do: summarize, classify, translate, normalize, or disambiguate by meaning.
+- Pass the DOM text needed for the answer. For a summary, pass the article body; not just a title or label.
+- If a field has a fixed enum, include the allowed values in the askLlm prompt or schema.
+- Prefer askLlm(prompt, jsonSchema) for structured fields, enums, booleans, numbers, arrays, and objects.
+- Validate askLlm output before returning it. If invalid, return an allowed empty value for optional fields or throw for required fields.
+- When calling askLlm once per list item, run the calls concurrently with Promise.all.
+
+Good patterns:
+
+Required labeled value:
+const row = [...doc.querySelectorAll("tr")].find(r => r.textContent.includes("Balance"));
+if (!row) throw new Error("Missing Balance row");
+const text = row.querySelector("td:last-child")?.textContent?.trim();
+if (!text) throw new Error("Missing Balance value");
+const balance = Number(text.replace(/[^0-9.-]/g, ""));
+
+List item scoping:
+const cards = [...doc.querySelectorAll("[data-testid='product-card']")];
+if (!cards.length) throw new Error("Missing product cards");
+const products = cards.map(card => {
+ const name = card.querySelector("h2")?.textContent?.trim();
+ if (!name) throw new Error("Missing product name");
+ const url = card.querySelector("a")?.getAttribute("href") ?? null;
+ return { name, url };
+});
+
+Raw JSON/plaintext page:
+const bodyText = doc.body.textContent?.trim() || "";
+const jsonMatch = bodyText.match(/\\{[\\s\\S]*\\}/);
+if (!jsonMatch) throw new Error("Missing JSON object in page text");
+const data = JSON.parse(jsonMatch[0]);
+// Read the requested schema fields from data here.
+// Throw if a required field is missing or has the wrong type.
Return only the function source, beginning with \`async function extract\`.`;
@@ -123,7 +169,7 @@ ${args.markdownPreview}
${args.previousCode.trim()}
\`\`\`
-Repair it: change only what is needed to fix the issue above - usually a single selector or parse step. Keep the field mappings, structure, and everything that still works identical; do not rewrite from scratch.`
+Repair it if the existing structure is valid. If the previous function violates any hard invalid pattern, rewrite the violating section completely. Do not preserve try/catch, whole-page scanning, regex fishing, swallowed errors, or catch-all empty returns. Keep only the parts that comply with the system rules.`
: `Write the function again, fixing the issue above.`;
const retry = args.rejectionFeedback?.trim()