Supercharge your AI agents with data from the web and beyond. Building the library for superintelligence. 🔥
TypeScript aiai-agentsai-crawlerai-scrapingai-searchcrawlerdata-extractionhtml-to-markdownllmmarkdownscraperscrapingweb-crawlerweb-dataweb-data-extractionweb-scraperweb-scrapingweb-searchwebscraping
chore: improve extractor system prompt (#3715)
tomsideguide2671aeb346b8
diff --git a/apps/api/src/lib/deterministicJson/llm/prompts.ts b/apps/api/src/lib/deterministicJson/llm/prompts.tsindex 4414790c6..0aed99466 100644--- a/apps/api/src/lib/deterministicJson/llm/prompts.ts+++ b/apps/api/src/lib/deterministicJson/llm/prompts.ts@@ -35,35 +35,81 @@ Return only JSON of the form {"snippets": ["...", "..."]} - no markdown fences,// Phase 2: write the reusable extractor. Kept generic on purpose - the sandbox// (a browser-like DOM with no network) enforces the environment, so the prompt// describes intent, not an inventory of available globals.-const EXTRACTOR_SYSTEM = `You write ONE reusable JavaScript function that extracts structured data from a web page.+const EXTRACTOR_SYSTEM = `You write one reusable JavaScript function:async function extract(doc, askLlm) { ... }-- \`doc\` is a standard DOM Document: querySelector / querySelectorAll, document.evaluate (XPath), textContent, getAttribute, etc.-- \`askLlm(prompt)\` returns a string; \`askLlm(prompt, jsonSchema)\` returns a value of that shape. It can fail and return null, so guard its result.+- \`doc\` is a standard DOM Document: querySelector, querySelectorAll, document.evaluate, textContent, getAttribute, etc.+- \`askLlm(prompt)\` returns a string.+- \`askLlm(prompt, jsonSchema)\` returns a value of that schema shape. It may return null, so guard its result.-This function is generated once from ONE sample page, cached, then re-run on OTHER pages of the same shape. Therefore:+Return contract:+- Return only the function source, beginning with \`async function extract\`.+- The returned data must match the target JSON Schema exactly: required fields, correct types, and no extra keys.+- Every returned value must be read from the runtime DOM or from askLlm at runtime.+- Never return a literal value copied from the sample page. Literals are allowed only for selectors, attribute names, regexes, enum values, error messages, and returned object keys.+- Do not use network, storage, timers, randomness, browser navigation, mutation observers, external libraries, or DOM mutation.-- Read every value from the DOM at runtime. NEVER return a literal you saw in the sample page - the values differ on other pages. Literals are only allowed in selectors, attribute names, regexes, and the returned object's keys.-- The returned value must match the target JSON Schema exactly: include required fields, use the correct types, and do not add extra keys.-- For required schema fields or fields central to the task, throw if the expected anchor/container is missing.-- Return null or [] only when the surrounding container was found and the specific optional value is genuinely absent. Do not treat a missing primary container as optional absence.-- Do NOT hide breakage. Use one primary extraction strategy per value. Do not silently fall back to broader whole-page scraping when it fails. If you handle known explicit variants, branch on a specific container or page-state check, and throw when none match.-- Anchor every value to a specific element. Never fish for a value by matching a pattern (e.g. a price, a date, an email) over doc.body.innerText or the whole page - you will match an unrelated occurrence and return a confident, wrong value.+Failure behavior:+- If a required container or required value is missing, throw an Error with a specific message.+- Return null, "", or [] only when the correct surrounding container was found and that specific optional value is genuinely absent.+- Never fabricate a schema-shaped fallback object after extraction fails.+- Do not use try/catch to swallow extraction errors. Let failures throw, or throw a clearer Error.+- Use one primary extraction strategy per value. Explicit branches for known page variants are okay; throw if none match.++Extraction strategy:+- Prefer specific structured sources: semantic DOM, labels, headings, tables, attributes, JSON scripts, inline state blobs, or nearby containers from the provided HTML snippets.+- Anchor values to specific elements when the page has meaningful structure.+- Broad page-text reads such as doc.body.innerText or doc.body.textContent are allowed only as the primary strategy when the page itself is raw JSON/plaintext, or when the requested value only exists in rendered body text.+- Regex over broad page text is allowed only when no more specific DOM container or structured source exists. Do not use broad regex as a fallback after a selector fails.+- If data is JSON in a specific element, JSON.parse that element's textContent and read fields directly.+- If the whole page is JSON/plaintext, parse or search the exact available body text.+- If JSON is surrounded by non-JSON text and there is no better container, extracting the JSON substring with regex is acceptable as the primary strategy.+- Never ask askLlm to parse JSON you can parse deterministically.- For arrays/lists, first select the repeated item/container elements, then extract each field relative to that item. Do not query the whole document for per-item fields inside the loop.-- When a value is present, coerce it to the schema's type (e.g. strip non-numerics for numbers). Keep dates as the page displays them unless the task asks for a specific format.-- Do not use network, storage, timers, randomness, browser navigation, mutation observers, external libraries, or DOM mutation.+- When a value is present, coerce it to the schema's type. Keep dates as displayed unless the task asks for a specific format.Selectors:-- Anchor on stable, meaningful things: labels, semantic attributes (itemprop, role, data-*), heading text. Avoid fragile position (:nth-child, sibling chains).-- Prefer descendant combinators over child combinators (\`>\`); an unseen wrapper element breaks \`>\`. A selector that matches 0 elements is usually too strict, not proof the data is absent.-- Read URLs from href/src, not from link text.-- Only standard CSS selectors work. To match by visible text, select structurally then filter in JS, e.g. \`[...doc.querySelectorAll('tr')].find(r => r.textContent.includes('Total'))\`.+- Prefer stable, meaningful anchors: labels, headings, semantic attributes, itemprop, role, data-*, table headers, and nearby static text.+- Avoid fragile position selectors such as :nth-child and long sibling chains.+- Prefer descendant combinators over child combinators (\`>\`), because unseen wrappers often break direct-child selectors.+- Read URLs from href/src attributes, not from link text.+- CSS selectors cannot match visible text directly. To match text, select candidate elements then filter in JS, e.g. \`[...doc.querySelectorAll("tr")].find(r => r.textContent.includes("Total"))\`.askLlm:-- If the data is already structured (a JSON body, a <script type="application/ld+json">, an inline state blob), JSON.parse it and read the field directly. Never ask askLlm to parse JSON you could parse deterministically.-- Use it for what selectors and parsing cannot do - summarize, classify, translate, disambiguate by meaning. It transforms the text you pass it (no outside lookups like currency conversion; returns null when the text lacks the answer), so pass the DOM text it needs - for a summary, the article body, not a bare title or label. If a field has a fixed value set, include it in the prompt.-- When you call askLlm once per item over a list, issue the calls concurrently with Promise.all rather than awaiting one at a time.+- Use askLlm only for semantic transformations selectors cannot do: summarize, classify, translate, normalize, or disambiguate by meaning.+- Pass the DOM text needed for the answer. For a summary, pass the article body; not just a title or label.+- If a field has a fixed enum, include the allowed values in the askLlm prompt or schema.+- Prefer askLlm(prompt, jsonSchema) for structured fields, enums, booleans, numbers, arrays, and objects.+- Validate askLlm output before returning it. If invalid, return an allowed empty value for optional fields or throw for required fields.+- When calling askLlm once per list item, run the calls concurrently with Promise.all.++Good patterns:++Required labeled value:+const row = [...doc.querySelectorAll("tr")].find(r => r.textContent.includes("Balance"));+if (!row) throw new Error("Missing Balance row");+const text = row.querySelector("td:last-child")?.textContent?.trim();+if (!text) throw new Error("Missing Balance value");+const balance = Number(text.replace(/[^0-9.-]/g, ""));++List item scoping:+const cards = [...doc.querySelectorAll("[data-testid='product-card']")];+if (!cards.length) throw new Error("Missing product cards");+const products = cards.map(card => {+ const name = card.querySelector("h2")?.textContent?.trim();+ if (!name) throw new Error("Missing product name");+ const url = card.querySelector("a")?.getAttribute("href") ?? null;+ return { name, url };+});++Raw JSON/plaintext page:+const bodyText = doc.body.textContent?.trim() || "";+const jsonMatch = bodyText.match(/\\{[\\s\\S]*\\}/);+if (!jsonMatch) throw new Error("Missing JSON object in page text");+const data = JSON.parse(jsonMatch[0]);+// Read the requested schema fields from data here.+// Throw if a required field is missing or has the wrong type.Return only the function source, beginning with \`async function extract\`.`;@@ -123,7 +169,7 @@ ${args.markdownPreview}${args.previousCode.trim()}\`\`\`-Repair it: change only what is needed to fix the issue above - usually a single selector or parse step. Keep the field mappings, structure, and everything that still works identical; do not rewrite from scratch.`+Repair it if the existing structure is valid. If the previous function violates any hard invalid pattern, rewrite the violating section completely. Do not preserve try/catch, whole-page scanning, regex fishing, swallowed errors, or catch-all empty returns. Keep only the parts that comply with the system rules.`: `Write the function again, fixing the issue above.`;const retry = args.rejectionFeedback?.trim()