Open-source agentic AI Search Execution Platform. Track AI visibility, find opportunities, and let agents turn signals into prioritized actions and drafted fixes across ChatGPT, Claude, Gemini, Google AI Overviews, Google AI Mode, Perplexity, Grok, and Microsoft Copilot.
TypeScript aeoai-agentai-searchanswer-engine-optimizationbrand-monitoringchatgptclaudecopilotdockergeminigeogoogle-ai-overviewsgrokmcpnextjsopen-sourceperplexityseosupabasetypescript
feat(action-center): read the competitor pages AI answers cite and say what they have that the brand's page doesn't (learning C1) (#1718)
Cihan38953cd20bde
diff --git a/docs/guides/action-center.mdx b/docs/guides/action-center.mdxindex 0fe0bf9d8..12b5731bd 100644--- a/docs/guides/action-center.mdx+++ b/docs/guides/action-center.mdx@@ -110,6 +110,8 @@ A task the AI writes without its **URL**, its **Change**, **Done when** or **HowSome tasks that only read data, such as checking a page's traffic in your connected Analytics, can be carried out by Ansvisor itself overnight. The action's History says so. Nothing that would change something outside Ansvisor runs without your approval: such a task waits as **Waiting Approval**.+**Analyze competitor content** is one of the tasks Ansvisor carries out itself. Overnight, it reads the pages AI answers cited for the action's prompts in the last two weeks: up to three, your competitors' pages first, then pages cited in answers that don't name your brand. It reads your own page cited there too, and runs Site Audit's checks on them all (structure, FAQ, schema, statistics, dates, length). The task then says which checks at least half of their pages pass that yours doesn't; it takes at least two of their pages to compare. Where the action names no prompts, it reads its topic's prompts, or for an action about competitors or sources, the prompts you track. Pages are read for free from their public address, at most 12 a day for a brand, never from a site whose robots.txt keeps crawlers out, and only what the checks found is kept, never the page's text. When there is nothing to compare (no prompts, no page cited, or none could be read), the task stays open for you with the reason, and Ansvisor tries again a week later.+### Assigning workAn action with no one on it belongs to **Everyone** in the workspace until someone takes it. Pick a teammate in the action's **Assignee** field, and set a **Due date** if it has one. The **Assignee** filter on the Actions tab shows one person's actions, or **Everyone** for the unassigned ones.diff --git a/server/src/config/action-engine.js b/server/src/config/action-engine.jsindex 5a5c71110..c597e62d6 100644--- a/server/src/config/action-engine.js+++ b/server/src/config/action-engine.js@@ -414,6 +414,26 @@ export const ENGINE_THRESHOLDS = Object.freeze({*/finalReadingDay: 30,}),++ /**+ * What the "Analyze competitor content" task reads (learning C1): the+ * pages cited in the brand's tracked answers for the action's prompts over+ * the last `answersDays`, at most `perAction` competitor pages a run, at+ * most `perBrandPerDay` new reads a day for the brand (its own page+ * included; a run needing more than are left waits for the next night),+ * and a page read within `freshDays` reused rather than read again. Read+ * free, never through a paid scraper. Fewer than `minPages` competitor+ * pages read are compared with nothing; a run that finds nothing to+ * compare leaves the task to a person and tries again after `retryDays`.+ */+ competitorPages: Object.freeze({+ answersDays: 14,+ perAction: 3,+ perBrandPerDay: 12,+ freshDays: 14,+ minPages: 2,+ retryDays: 7,+ }),});/**diff --git a/server/src/lib/action-center/basket-metrics.js b/server/src/lib/action-center/basket-metrics.jsindex 581652c1d..63f11c2a8 100644--- a/server/src/lib/action-center/basket-metrics.js+++ b/server/src/lib/action-center/basket-metrics.js@@ -118,17 +118,23 @@ export async function basketScope(action) {if (!promptIds.length && opp?.prompt_id) promptIds = [opp.prompt_id];}if (!promptIds.length && payload.topicId) {- const { data: prompts, error } = await supabaseAdmin- .from('prompts')- .select('id, prompt_sets!inner(brand_id)')- .eq('topic_id', payload.topicId)- .eq('prompt_sets.brand_id', action.brand_id);- if (error) throw new Error(error.message);- promptIds = (prompts || []).map((p) => p.id);+ promptIds = await topicPromptIds(action.brand_id, [payload.topicId]);}return { promptIds, pages: scopePages(payload) };}+/** The prompts the brand's topics have now, by id. */+export async function topicPromptIds(brandId, topicIds) {+ if (!topicIds.length) return [];+ const { data: prompts, error } = await supabaseAdmin+ .from('prompts')+ .select('id, prompt_sets!inner(brand_id)')+ .eq('prompt_sets.brand_id', brandId)+ .in('topic_id', topicIds);+ if (error) throw new Error(error.message);+ return (prompts || []).map((p) => p.id);+}+/*** The pages an action is measured on: the ones it names, and the addresses* its drafts went live at (publishedPages by draft, CE-013), each once.diff --git a/server/src/lib/action-center/competitor-pages.js b/server/src/lib/action-center/competitor-pages.jsnew file mode 100644index 000000000..251cb36d7--- /dev/null+++ b/server/src/lib/action-center/competitor-pages.js@@ -0,0 +1,352 @@+/**+ * What the pages AI answers cite have (learning C1).+ *+ * "Analyze competitor content" was a step for a person: nothing looked at the+ * pages AI engines cite instead of the brand's. This reads them: the pages+ * cited in the brand's own tracked answers for the action's prompts lately —+ * its competitors' pages, then pages cited in answers that don't name the+ * brand — and the brand's own page cited there. An action that names no+ * prompts reads its topics' prompts, else the brand's tracked ones. Each page+ * is fetched free (safe-fetch: public addresses only, never a paid scraper),+ * unless its site's robots.txt shuts the reader out, and run through Site+ * Audit's own rule checks that read the page alone — structure, FAQ, schema,+ * statistics, dates, length. What the checks found is kept+ * (cited_page_features, 00369), never the page's text, and nothing is+ * summarised by AI. The result is the checks at least half of the competitor+ * pages pass that the brand's page doesn't.+ *+ * Only addresses the brand's own answers cited are read — never one a request+ * names — at most `perBrandPerDay` new reads a day for a brand, and a page+ * read (or found unreadable) in the last `freshDays` isn't read again.+ */++import supabaseAdmin from '../../config/supabase.js';+import { resolve } from '../../config/action-engine.js';+import { fetchPage } from '../safe-fetch.js';+import { extractHostname } from '../response-parser.js';+import { opportunityKey } from '../audit/opportunities.js';+import { robotsGroups } from '../audit/signals/trust.js';+import { logger } from '../logger.js';+import { topicPromptIds } from './basket-metrics.js';+import { promptIdsOf } from './priority.js';+import { FEATURE_CHECKS, featureGaps, pageFeatures } from './page-features.js';++// The page checks live in page-features.js, which reads no database; kept+// here for the imports that name this module.+export {+ FEATURE_CHECKS,+ MIN_PAGE_CHARACTERS,+ featureGaps,+ pageContext,+ pageFeatures,+} from './page-features.js';++const { competitorPages: C } = resolve();++const DAY_MS = 86_400_000;+const READ_TIMEOUT_MS = 8_000;+/** Answers read for the action's prompts, newest first. */+const ANSWERS_READ = 300;+/** Prompts whose answers are read, at most. */+const PROMPTS_READ = 50;+/** safe-fetch's User-Agent, as robots.txt names a crawler. */+const ROBOTS_AGENT = 'ansvisorbot';+/** The most of a robots.txt read (RFC 9309 asks crawlers to read 500 KiB). */+const ROBOTS_MAX_BYTES = 512 * 1024;++/**+ * A run that leaves its task as it was (execution/run.js reads `retryLater`):+ * the brand's reads for the day can't cover it, so it waits for the next+ * night; or it found nothing to compare, so the task stays a person's, shows+ * `output` (why), and isn't tried again for `retryDays`.+ */+export class ReadLaterError extends Error {+ /**+ * @param {string} message+ * @param {{ output?: object | null, retryDays?: number }} [later]+ */+ constructor(message, { output = null, retryDays = 0 } = {}) {+ super(message);+ this.retryLater = true;+ this.output = output;+ this.retryDays = retryDays;+ }+}++const hostOf = (value) => extractHostname(/^https?:\/\//i.test(value) ? value : `https://${value}`);+const onDomain = (host, domain) => host === domain || host.endsWith(`.${domain}`);++/**+ * The pages to read from the answers: the competitors' most cited first, then+ * the most cited in answers that don't name the brand, at most `perAction`;+ * and the brand's own page most cited there.+ *+ * @param {Array<{ mention_count?: number, citations?: Array<{ url?: string }> }>} answers+ * @param {{ ownDomains: string[], competitorDomains: string[], perAction?: number }} options+ */+export function citedPages(answers, { ownDomains, competitorDomains, perAction = C.perAction }) {+ const own = ownDomains.map(hostOf).filter(Boolean);+ const rivals = competitorDomains.map(hostOf).filter(Boolean);+ const tally = { own: new Map(), rival: new Map(), other: new Map() };+ for (const answer of answers) {+ const seen = new Set();+ for (const cite of answer.citations ?? []) {+ const url = typeof cite?.url === 'string' ? cite.url : null;+ const host = url ? hostOf(url) : null;+ const key = url ? opportunityKey(url) : null;+ if (!host || !key || seen.has(key) || !/^https?:\/\//i.test(url)) continue;+ seen.add(key);+ const group = own.some((d) => onDomain(host, d))+ ? tally.own+ : rivals.some((d) => onDomain(host, d))+ ? tally.rival+ : (answer.mention_count ?? 0) === 0+ ? tally.other+ : null;+ if (!group) continue;+ const known = group.get(key) ?? { key, url, domain: host, count: 0 };+ known.count += 1;+ group.set(key, known);+ }+ }+ const ranked = (map) =>+ [...map.values()].sort((a, b) => b.count - a.count || a.key.localeCompare(b.key));+ const competitors = [...ranked(tally.rival), ...ranked(tally.other)].slice(0, perAction);+ return { competitors, yours: ranked(tally.own)[0] ?? null };+}++const passing = (checks) => FEATURE_CHECKS.filter((key) => checks?.[key] === 'pass');++/**+ * Whether a site's robots.txt shuts this reader out of the whole site: the+ * groups naming it, else those for every crawler ('*'), disallow "/" and+ * none allows it back. Rules for single paths aren't read (robotsGroups).+ */+export function robotsDisallow(robotsTxt) {+ const groups = robotsGroups(robotsTxt);+ const own = groups.filter((group) => group.agents.includes(ROBOTS_AGENT));+ const apply = own.length > 0 ? own : groups.filter((group) => group.agents.includes('*'));+ return apply.some((group) => group.disallowAll) && !apply.some((group) => group.allowAll);+}++/** An HTML page served where robots.txt should be: no robots.txt. */+const isHtml = (body) => /^\s*(<!doctype\s+html|<html[\s>])/i.test(body ?? '');++/**+ * Whether the reader may read pages on the site at `url`. No robots.txt (a+ * 4xx) lets it in; one that can't be read (a server error, asked to wait,+ * no answer) keeps it out, as RFC 9309 asks.+ */+async function robotsAllow(url, read) {+ try {+ const robots = await read(new URL('/robots.txt', url).href, {+ timeoutMs: READ_TIMEOUT_MS,+ // Crawlers read it whatever its type (S3 serves it as octet-stream).+ types: /(?:)/,+ accept: 'text/plain',+ maxBytes: ROBOTS_MAX_BYTES,+ truncate: true,+ });+ return isHtml(robots.html) || !robotsDisallow(robots.html);+ } catch (err) {+ return err?.code === 'http_error' && err.status < 500 && err.status !== 429;+ }+}++/**+ * The prompts whose answers are read: the ones the action names; else its+ * topics' prompts; else (an action about competitors or sources) the brand's+ * tracked prompts. At most PROMPTS_READ.+ */+export async function scopePromptIds(brandId, payload) {+ const named = promptIdsOf([payload]);+ if (named.length > 0) return named.slice(0, PROMPTS_READ);+ if ((payload?.entity ?? payload?.targetEntity) === 'topics') {+ const listed = Array.isArray(payload.targetIds)+ ? payload.targetIds+ : (Array.isArray(payload.targets) ? payload.targets : []).map((target) => target?.id);+ const topicIds = [...new Set([payload.topicId, ...listed])].filter(+ (id) => typeof id === 'string' && id,+ );+ return (await topicPromptIds(brandId, topicIds)).slice(0, PROMPTS_READ);+ }+ const { data, error } = await supabaseAdmin+ .from('prompts')+ .select('id, prompt_sets!inner(brand_id)')+ .eq('prompt_sets.brand_id', brandId)+ .eq('is_active', true)+ .order('created_at', { ascending: true })+ .limit(PROMPTS_READ);+ if (error) throw new Error(error.message);+ return (data ?? []).map((row) => row.id);+}++/**+ * The "Analyze competitor content" task's result: the competitor pages read,+ * the brand's, and the checks the first pass that the second doesn't.+ *+ * It throws ReadLaterError when it can't give one: the day's reads left+ * can't cover the pages (the next night), or there is nothing to compare —+ * no prompts, no page cited, none readable (`retryDays`, the reason as the+ * task's output).+ *+ * @param {{ brandId: string, payload: object, now?: Date, read?: typeof fetchPage }} input+ * read: the page reader, for tests+ */+export async function analyzeCompetitorPages({+ brandId,+ payload,+ now = new Date(),+ read = fetchPage,+}) {+ const nothing = (reason, message, found = {}) =>+ new ReadLaterError(message, {+ output: {+ competitorPages: [],+ yourPage: null,+ gaps: [],+ unreadable: 0,+ ...found,+ reason,+ },+ retryDays: C.retryDays,+ });++ const promptIds = await scopePromptIds(brandId, payload);+ if (promptIds.length === 0) throw nothing('no_prompts', 'no prompts to read the answers of');++ const since = new Date(now.getTime() - C.answersDays * DAY_MS).toISOString();+ const [answersRes, domainsRes, competitorsRes] = await Promise.all([+ supabaseAdmin+ .from('prompt_results')+ .select('mention_count, citations')+ .eq('brand_id', brandId)+ .neq('platform', 'chatgpt-shopping')+ .in('prompt_id', promptIds)+ .gte('created_at', since)+ .order('created_at', { ascending: false })+ .limit(ANSWERS_READ),+ supabaseAdmin.from('brand_domains').select('domain').eq('brand_id', brandId),+ supabaseAdmin.from('competitors').select('domain').eq('brand_id', brandId),+ ]);+ for (const res of [answersRes, domainsRes, competitorsRes]) {+ if (res.error) throw new Error(res.error.message);+ }+ const picked = citedPages(answersRes.data ?? [], {+ ownDomains: (domainsRes.data ?? []).map((row) => row.domain).filter(Boolean),+ competitorDomains: (competitorsRes.data ?? []).map((row) => row.domain).filter(Boolean),+ });+ if (picked.competitors.length === 0) {+ throw nothing('none_cited', 'no competitor page cited for these prompts lately');+ }++ // What was read lately is reused, a page found unreadable too.+ const pages = [...picked.competitors, ...(picked.yours ? [{ ...picked.yours, own: true }] : [])];+ const [knownRes, todayRes] = await Promise.all([+ supabaseAdmin+ .from('cited_page_features')+ .select('page_key, features, read_at')+ .eq('brand_id', brandId)+ .in(+ 'page_key',+ pages.map((page) => page.key),+ ),+ supabaseAdmin+ .from('cited_page_features')+ .select('id', { count: 'exact', head: true })+ .eq('brand_id', brandId)+ .gte('read_at', `${now.toISOString().slice(0, 10)}T00:00:00.000Z`),+ ]);+ if (knownRes.error) throw new Error(knownRes.error.message);+ if (todayRes.error) throw new Error(todayRes.error.message);+ const freshSince = now.getTime() - C.freshDays * DAY_MS;+ const fresh = new Map(+ (knownRes.data ?? [])+ .filter((row) => Date.parse(row.read_at) >= freshSince)+ .map((row) => [row.page_key, row.features ?? {}]),+ );+ // All of an action's pages are read the same night, or none: a run that+ // read some would compare a part, and its task wouldn't run again.+ const toRead = pages.filter((page) => !fresh.has(page.key)).length;+ if (toRead > C.perBrandPerDay - (todayRes.count ?? 0)) {+ throw new ReadLaterError("the brand's page reads for today can't cover this action's");+ }++ const allowed = new Map();+ for (const page of pages) {+ if (fresh.has(page.key)) {+ const known = fresh.get(page.key);+ if (!known.unreadable) page.features = known;+ continue;+ }+ let features = null;+ let disallowed = false;+ try {+ const origin = new URL(page.url).origin;+ if (!allowed.has(origin)) allowed.set(origin, await robotsAllow(page.url, read));+ disallowed = !allowed.get(origin);+ if (!disallowed) {+ const fetched = await read(page.url, { timeoutMs: READ_TIMEOUT_MS });+ features = await pageFeatures(fetched.url, fetched.html, now.getTime());+ }+ } catch {+ // A page that blocks the reader, or isn't one: unreadable, not a failure.+ }+ if (features) page.features = features;+ // An unreadable page is kept too: it counts against the day's reads and+ // isn't tried again while it would be fresh.+ const { error } = await supabaseAdmin.from('cited_page_features').upsert(+ {+ brand_id: brandId,+ page_key: page.key,+ url: page.url,+ domain: page.domain,+ own: Boolean(page.own),+ features: features ?? { unreadable: true, ...(disallowed ? { disallowed: true } : {}) },+ read_at: now.toISOString(),+ },+ { onConflict: 'brand_id,page_key' },+ );+ if (error) {+ logger.warn({ err: error, brandId, page: page.key }, '[competitor-pages] read not kept');+ }+ }++ const unreadable = pages.filter((page) => !page.features).length;+ const competitorPages = pages+ .filter((page) => !page.own && page.features)+ .map((page) => ({+ url: page.url,+ domain: page.domain,+ cited: page.count,+ words: page.features.words ?? null,+ checks: page.features.checks ?? {},+ }));+ const own = pages.find((page) => page.own);+ const yourPageStatus = !own ? 'not_cited' : own.features ? 'read' : 'unreadable';+ if (competitorPages.length === 0) {+ throw nothing('unreadable', 'no competitor page cited could be read', {+ yourPageStatus,+ unreadable,+ });+ }+ const yourPage = own?.features+ ? {+ url: own.url,+ cited: own.count,+ words: own.features.words ?? null,+ checks: own.features.checks ?? {},+ }+ : null;+ // The checks each passes, by key: what the task shows.+ const shown = ({ checks, ...page }) => ({ ...page, passes: passing(checks) });+ return {+ competitorPages: competitorPages.map(shown),+ yourPage: yourPage ? shown(yourPage) : null,+ yourPageStatus,+ gaps: featureGaps(competitorPages, yourPage),+ // Fewer pages than this, and nothing is compared (featureGaps).+ compared: competitorPages.length >= C.minPages,+ unreadable,+ };+}diff --git a/server/src/lib/action-center/competitor-pages.test.js b/server/src/lib/action-center/competitor-pages.test.jsnew file mode 100644index 000000000..66a468ea5--- /dev/null+++ b/server/src/lib/action-center/competitor-pages.test.js@@ -0,0 +1,415 @@+import { beforeEach, describe, expect, it, vi } from 'vitest';++/**+ * What the pages AI answers cite have (learning C1): which prompts and pages+ * are read, what Site Audit's checks find on them, what the brand's page+ * lacks, the daily cap on reads, robots.txt, and the runs that find nothing.+ */++let tables;+let writes;+let queries;+let todayReads;+let upsertError;++vi.mock('../../config/supabase.js', () => ({+ default: {+ from: (table) => {+ const op = { table, head: false, filters: [] };+ queries.push(op);+ const chain = {+ select: (_cols, opts) => ((op.head = opts?.head ?? false), chain),+ eq: (column, value) => (op.filters.push(['eq', column, value]), chain),+ neq: () => chain,+ in: (column, value) => (op.filters.push(['in', column, value]), chain),+ gte: () => chain,+ order: () => chain,+ limit: () => chain,+ upsert: async (row) => (writes.push(row), { error: upsertError }),+ then: (resolve) =>+ resolve(+ op.head+ ? { count: todayReads, error: null }+ : { data: tables[table] ?? [], error: null },+ ),+ };+ return chain;+ },+ },+}));+const warn = vi.fn();+vi.mock('../logger.js', () => ({ logger: { warn: (...a) => warn(...a), error: vi.fn() } }));++const {+ FEATURE_CHECKS,+ ReadLaterError,+ analyzeCompetitorPages,+ citedPages,+ featureGaps,+ pageFeatures,+ robotsDisallow,+ scopePromptIds,+} = await import('./competitor-pages.js');+const pure = await import('./page-features.js');++const NOW = new Date('2026-10-28T03:00:00Z');++/** Enough words that what was read is a page, not a wall. */+const PROSE = `<p>${'A kettle boils water quickly and safely for tea. '.repeat(14)}</p>`;+const FAQ_PAGE = `<html><head><script type="application/ld+json">${JSON.stringify({+ '@type': 'FAQPage',+ mainEntity: [{ '@type': 'Question' }, { '@type': 'Question' }],+})}</script></head><body><h1>Best kettles</h1><h2>Which kettle?</h2>${PROSE}+<table><thead><tr><th>Kettle</th></tr></thead><tbody><tr><td>a</td></tr></tbody></table></body></html>`;+const PLAIN_PAGE = `<html><body><h1>Kettles</h1>${PROSE}</body></html>`;+const WALL_PAGE = '<html><body><p>We use cookies. Accept to continue.</p></body></html>';++/** A page that isn't there, as safe-fetch says it. */+const notFound = () =>+ Object.assign(new Error('The page answered 404'), { code: 'http_error', status: 404 });++const isRobots = (url) => url.endsWith('/robots.txt');++beforeEach(() => {+ writes = [];+ queries = [];+ todayReads = 0;+ upsertError = null;+ warn.mockClear();+ tables = {+ prompt_results: [+ {+ mention_count: 0,+ citations: [+ { url: 'https://rival.com/kettles' },+ { url: 'https://rival.com/kettles' },+ { url: 'https://blog.example/best' },+ { url: 'https://acme.com/kettles' },+ ],+ },+ {+ mention_count: 2,+ citations: [{ url: 'https://rival.com/kettles' }, { url: 'https://news.example/x' }],+ },+ ],+ brand_domains: [{ domain: 'acme.com' }],+ competitors: [{ domain: 'www.rival.com' }],+ cited_page_features: [],+ prompts: [],+ };+});++describe('page checks', () => {+ // page-features.js (tested there) reads no database, so the brief and+ // Improve load it without Supabase; they stay importable from here.+ it('are the ones page-features.js gives', () => {+ expect(FEATURE_CHECKS).toBe(pure.FEATURE_CHECKS);+ expect(featureGaps).toBe(pure.featureGaps);+ expect(pageFeatures).toBe(pure.pageFeatures);+ });+});++describe('citedPages', () => {+ it("takes the competitors' pages first, then pages cited where the brand isn't named, and the brand's own", () => {+ const { competitors, yours } = citedPages(tables.prompt_results, {+ ownDomains: ['acme.com'],+ competitorDomains: ['www.rival.com'],+ });+ // A page cited twice in one answer counts once; news.example was cited+ // in an answer that names the brand, so it isn't a page it loses to.+ expect(competitors.map((page) => [page.url, page.count])).toEqual([+ ['https://rival.com/kettles', 2],+ ['https://blog.example/best', 1],+ ]);+ expect(yours).toMatchObject({ url: 'https://acme.com/kettles', count: 1 });+ });++ it('reads at most a few pages an action', () => {+ expect(+ citedPages(tables.prompt_results, {+ ownDomains: [],+ competitorDomains: ['rival.com'],+ perAction: 1,+ }).competitors,+ ).toHaveLength(1);+ });+});++describe('robotsDisallow', () => {+ it('keeps out of a site whose robots.txt shuts out this reader or every crawler', () => {+ expect(robotsDisallow('User-agent: *\nDisallow: /')).toBe(true);+ expect(robotsDisallow('User-agent: AnsvisorBot\nDisallow: /')).toBe(true);+ // Its own group wins over everyone's.+ expect(robotsDisallow('User-agent: *\nDisallow: /\n\nUser-agent: AnsvisorBot\nAllow: /')).toBe(+ false,+ );+ expect(robotsDisallow('User-agent: *\nDisallow: /\nAllow: /')).toBe(false);+ });++ it('reads a site that shuts out others, a path, or nothing', () => {+ expect(robotsDisallow('User-agent: GPTBot\nDisallow: /')).toBe(false);+ expect(robotsDisallow('User-agent: *\nDisallow: /private/')).toBe(false);+ expect(robotsDisallow('')).toBe(false);+ });+});++describe('scopePromptIds', () => {+ const promptQuery = () => queries.find((q) => q.table === 'prompts');++ it('takes the prompts the action names', async () => {+ expect(await scopePromptIds('b1', { promptIds: ['p1', 'p2'] })).toEqual(['p1', 'p2']);+ expect(promptQuery()).toBeUndefined();+ });++ it("takes a topic action's prompts from its topics", async () => {+ tables.prompts = [{ id: 'p3' }, { id: 'p4' }];+ // As C04 and C09 send it: topic targets, no prompts.+ const payload = {+ targetEntity: 'topics',+ targetCount: 2,+ targets: [{ id: 't1', label: 'Kettles' }],+ targetIds: ['t1', 't2'],+ topicName: 'Kettles',+ };+ expect(await scopePromptIds('b1', payload)).toEqual(['p3', 'p4']);+ expect(promptQuery().filters).toContainEqual(['in', 'topic_id', ['t1', 't2']]);+ expect(promptQuery().filters).toContainEqual(['eq', 'prompt_sets.brand_id', 'b1']);+ });++ it("takes the brand's tracked prompts for an action about competitors or sources", async () => {+ tables.prompts = [{ id: 'p5' }];+ for (const targetEntity of ['competitors', 'sources']) {+ queries = [];+ const payload = { targetEntity, targetCount: 1, targets: [{ id: 'x', label: 'Rival' }] };+ expect(await scopePromptIds('b1', payload)).toEqual(['p5']);+ expect(promptQuery().filters).toEqual(+ expect.arrayContaining([+ ['eq', 'prompt_sets.brand_id', 'b1'],+ ['eq', 'is_active', true],+ ]),+ );+ }+ });+});++describe('analyzeCompetitorPages', () => {+ const payload = { promptIds: ['p1'] };+ const read = vi.fn(async (url) => {+ if (isRobots(url)) throw notFound();+ return { url, html: url.startsWith('https://acme.com') ? PLAIN_PAGE : FAQ_PAGE };+ });+ const pagesRead = (reader = read) =>+ reader.mock.calls.map(([url]) => url).filter((url) => !isRobots(url));+ const run = (over = {}) =>+ analyzeCompetitorPages({ brandId: 'b1', payload, now: NOW, read, ...over });+ /** What a run that throws threw. */+ const thrown = (promise) =>+ promise.then(+ () => null,+ (err) => err,+ );++ beforeEach(() => {+ read.mockClear();+ });++ it("reads the cited pages, keeps their features only, and says what the brand's page lacks", async () => {+ const result = await run();++ expect(pagesRead()).toEqual([+ 'https://rival.com/kettles',+ 'https://blog.example/best',+ 'https://acme.com/kettles',+ ]);+ expect(writes.map((row) => [row.page_key, row.own])).toEqual([+ ['rival.com/kettles', false],+ ['blog.example/best', false],+ ['acme.com/kettles', true],+ ]);+ // Features, never the page's text.+ expect(JSON.stringify(writes)).not.toContain('Which kettle?');+ expect(result.competitorPages.map((page) => page.url)).toEqual([+ 'https://rival.com/kettles',+ 'https://blog.example/best',+ ]);+ expect(result.competitorPages[0].passes).toContain('faq-schema');+ expect(result.yourPage.passes).not.toContain('faq-schema');+ expect(result.gaps.map((gap) => gap.key)).toEqual(expect.arrayContaining(['faq-schema']));+ expect(result).toMatchObject({ yourPageStatus: 'read', compared: true, unreadable: 0 });+ expect(result).not.toHaveProperty('capped');+ });++ it("reads the brand's tracked prompts' answers for an action naming none (C01)", async () => {+ tables.prompts = [{ id: 'p7' }, { id: 'p8' }];+ const c01 = {+ targetEntity: 'competitors',+ targetCount: 1,+ targets: [{ id: 'rival', label: 'Rival' }],+ competitorNames: ['Rival'],+ };++ const result = await run({ payload: c01 });++ const answers = queries.find((q) => q.table === 'prompt_results');+ expect(answers.filters).toContainEqual(['in', 'prompt_id', ['p7', 'p8']]);+ expect(result.competitorPages).toHaveLength(2);+ });++ it("reads a topic action's prompts' answers", async () => {+ tables.prompts = [{ id: 'p9' }];+ const result = await run({+ payload: { targetEntity: 'topics', targets: [{ id: 't1', label: 'Kettles' }] },+ });+ expect(queries.find((q) => q.table === 'prompt_results').filters).toContainEqual([+ 'in',+ 'prompt_id',+ ['p9'],+ ]);+ expect(result.gaps.length).toBeGreaterThan(0);+ });++ it('reuses a page read lately rather than reading it again', async () => {+ tables.cited_page_features = [+ {+ page_key: 'rival.com/kettles',+ read_at: '2026-10-20T00:00:00Z',+ features: { words: 900, checks: { 'faq-schema': 'pass' } },+ },+ ];+ const result = await run();+ expect(pagesRead()).not.toContain('https://rival.com/kettles');+ expect(result.competitorPages[0]).toMatchObject({ words: 900, passes: ['faq-schema'] });+ });++ it("reads all of an action's pages the same night or none, and waits for the next", async () => {+ // Three pages to read, two reads left: none is read.+ todayReads = 10;+ const err = await thrown(run());+ expect(err).toBeInstanceOf(ReadLaterError);+ expect(err).toMatchObject({ retryLater: true, output: null, retryDays: 0 });+ expect(read).not.toHaveBeenCalled();+ expect(writes).toEqual([]);++ // Three left: all three.+ todayReads = 9;+ const result = await run();+ expect(pagesRead()).toHaveLength(3);+ expect(result.yourPageStatus).toBe('read');++ // A page read lately needs no read of the day's.+ read.mockClear();+ tables.cited_page_features = [+ {+ page_key: 'acme.com/kettles',+ read_at: '2026-10-27T00:00:00Z',+ features: { words: 400, checks: {} },+ },+ ];+ todayReads = 10;+ await expect(run()).resolves.toMatchObject({ yourPageStatus: 'read' });+ expect(pagesRead()).toHaveLength(2);+ });++ it("keeps a page it couldn't read, so it counts and isn't read again while fresh", async () => {+ const reader = vi.fn(async (url) => {+ if (isRobots(url)) throw notFound();+ if (url === 'https://blog.example/best') throw new Error('blocked');+ return { url, html: url.startsWith('https://acme.com') ? PLAIN_PAGE : FAQ_PAGE };+ });+ const result = await run({ read: reader });++ expect(writes.find((row) => row.page_key === 'blog.example/best').features).toEqual({+ unreadable: true,+ });+ expect(result).toMatchObject({ unreadable: 1, compared: false, gaps: [] });++ // Read as unreadable on the next run, and not read again.+ reader.mockClear();+ tables.cited_page_features = writes.map((row) => ({ ...row }));+ writes = [];+ const again = await run({ read: reader });+ expect(pagesRead(reader)).toEqual([]);+ expect(again).toMatchObject({ unreadable: 1, competitorPages: [expect.any(Object)] });+ });++ it("says the brand's page couldn't be read rather than that it wasn't cited", async () => {+ const reader = vi.fn(async (url) => {+ if (isRobots(url)) throw notFound();+ return { url, html: url.startsWith('https://acme.com') ? WALL_PAGE : FAQ_PAGE };+ });+ const result = await run({ read: reader });+ expect(result).toMatchObject({ yourPage: null, yourPageStatus: 'unreadable', unreadable: 1 });+ expect(writes.find((row) => row.own).features).toEqual({ unreadable: true });++ tables.prompt_results[0].citations = tables.prompt_results[0].citations.filter(+ (cite) => !cite.url.startsWith('https://acme.com'),+ );+ writes = [];+ expect(await run()).toMatchObject({ yourPage: null, yourPageStatus: 'not_cited' });+ });++ it("doesn't read a site whose robots.txt keeps it out", async () => {+ const reader = vi.fn(async (url) => {+ if (url === 'https://blog.example/robots.txt') {+ return { url, html: 'User-agent: *\nDisallow: /' };+ }+ if (isRobots(url)) throw notFound();+ return { url, html: FAQ_PAGE };+ });+ const result = await run({ read: reader });+ expect(pagesRead(reader)).not.toContain('https://blog.example/best');+ expect(writes.find((row) => row.page_key === 'blog.example/best').features).toEqual({+ unreadable: true,+ disallowed: true,+ });+ expect(result.unreadable).toBe(1);+ });++ it("stays out of a site whose robots.txt can't be read, and reads one with none", async () => {+ const reader = vi.fn(async (url) => {+ if (url === 'https://blog.example/robots.txt') {+ throw Object.assign(new Error('The page answered 503'), {+ code: 'http_error',+ status: 503,+ });+ }+ if (url === 'https://rival.com/robots.txt')+ return { url, html: '<!doctype html><p>Home</p>' };+ if (isRobots(url)) throw notFound();+ return { url, html: FAQ_PAGE };+ });+ await run({ read: reader });+ expect(pagesRead(reader)).toEqual(['https://rival.com/kettles', 'https://acme.com/kettles']);+ });++ it("logs a read it couldn't keep without calling the page unreadable", async () => {+ upsertError = { message: 'write failed' };+ const result = await run();+ expect(result).toMatchObject({ unreadable: 0, compared: true });+ expect(warn).toHaveBeenCalledTimes(3);+ });++ it('leaves the task to a person, with why, when there is nothing to compare', async () => {+ const failing = vi.fn(async () => {+ throw new Error('blocked');+ });+ const unreadable = await thrown(run({ read: failing }));+ expect(unreadable).toBeInstanceOf(ReadLaterError);+ expect(unreadable.retryDays).toBe(7);+ expect(unreadable.output).toMatchObject({+ competitorPages: [],+ gaps: [],+ unreadable: 3,+ yourPageStatus: 'unreadable',+ reason: 'unreadable',+ });++ tables.prompt_results = [];+ const none = await thrown(run());+ expect(none).toMatchObject({ retryDays: 7, output: { reason: 'none_cited' } });++ // An action naming no prompts, for a brand tracking none.+ const noPrompts = await thrown(run({ payload: { targetEntity: 'competitors', targets: [] } }));+ expect(noPrompts).toMatchObject({ retryDays: 7, output: { reason: 'no_prompts' } });+ });+});diff --git a/server/src/lib/action-center/execution/run.js b/server/src/lib/action-center/execution/run.jsindex b8dc03559..8e512a76f 100644--- a/server/src/lib/action-center/execution/run.js+++ b/server/src/lib/action-center/execution/run.js@@ -41,6 +41,8 @@ import { resolve } from '../../../config/action-engine.js';const { noise } = resolve();+const DAY_MS = 86_400_000;+/** A task can only be carried out from the statuses that mean "not done". */const RUNNABLE_STATUSES = ['todo', 'in_progress', 'waiting_approval'];@@ -226,6 +228,39 @@ export async function runTask(taskId, { now = new Date(), sources } = {}) {logger.info({ taskId, tool: tool.id }, '[task-run] completed');return { status: 'succeeded', output };} catch (err) {+ // A tool that can't go on (learning C1) leaves the task as it was: its+ // reads for the day spent, for the next night; or nothing found to work+ // on, the task a person's, with why as its output, and the nightly sweep+ // passing it over for `retryDays`.+ if (err?.retryLater) {+ const retryAfter =+ err.retryDays > 0 ? new Date(now.getTime() + err.retryDays * DAY_MS).toISOString() : null;+ await record({+ task_id: task.id,+ action_id: action.id,+ brand_id: action.brand_id,+ tool_id: tool.id,+ tool_version: tool.version,+ status: 'blocked',+ input,+ output: err.output ?? null,+ error: err.message,+ started_at: startedAt.toISOString(),+ finished_at: new Date().toISOString(),+ });+ if (err.output || retryAfter) {+ const { error } = await supabaseAdmin+ .from('action_tasks')+ .update({+ output: { ...(err.output ?? {}), ...(retryAfter ? { retryAfter } : {}) },+ updated_at: new Date().toISOString(),+ })+ .eq('id', task.id);+ if (error) logger.error({ err: error, taskId }, '[task-run] could not keep why');+ }+ logger.info({ taskId, tool: tool.id, reason: err.message, retryAfter }, '[task-run] later');+ return { status: 'blocked', reason: err.message, ...(retryAfter ? { retryAfter } : {}) };+ }// The action is deliberately untouched. One step failing is one step to// do by hand, not a recovery plan cancelled.await record({@@ -272,14 +307,18 @@ export async function sweepTaskExecution({ now = new Date() } = {}) {// On the cloud, only for organizations still on a plan: a tool reads// outside services on the organization's behalf. Filtered in the query, so// the old tasks of an organization whose plan ended can't fill the read.+ // `retry_after`: a tool that found nothing to work on asked to be passed+ // over until then (runTask).let query = supabaseAdmin.from('action_tasks');query = isCloud()? query.select(- 'id, task_key, action_id, actions!inner(status, brands!inner(organizations!inner(subscription_status)))',+ 'id, task_key, action_id, retry_after:output->>retryAfter, actions!inner(status, brands!inner(organizations!inner(subscription_status)))',).in('actions.brands.organizations.subscription_status', ACTIVE_SUBSCRIPTION_STATUSES)- : query.select('id, task_key, action_id, actions!inner(status)');+ : query.select(+ 'id, task_key, action_id, retry_after:output->>retryAfter, actions!inner(status)',+ );const { data: rows, error } = await query.in('status', RUNNABLE_STATUSES).not('actions.status', 'in', '("completed","dismissed")')@@ -289,13 +328,17 @@ export async function sweepTaskExecution({ now = new Date() } = {}) {// Filter here rather than in the query: which tasks a tool implements is// code, not a column, and duplicating it into SQL is how the two drift.- const runnable = (rows ?? []).filter((row) => getTask(row.task_key)?.tool);+ // One waiting to be tried again doesn't take a place under the ceiling.+ const withTool = (rows ?? []).filter((row) => getTask(row.task_key)?.tool);+ const waiting = (row) => row.retry_after && Date.parse(row.retry_after) > now.getTime();+ const runnable = withTool.filter((row) => !waiting(row));const attempts = runnable.slice(0, maxTaskRunsPerNight);const summary = {succeeded: 0,failed: 0,blocked: 0,+ waitingToRetry: withTool.length - runnable.length,skippedForCeiling: runnable.length - attempts.length,};diff --git a/server/src/lib/action-center/execution/run.test.js b/server/src/lib/action-center/execution/run.test.jsindex d5d7bbed5..ba50f05c7 100644--- a/server/src/lib/action-center/execution/run.test.js+++ b/server/src/lib/action-center/execution/run.test.js@@ -15,6 +15,7 @@ import { ENGINE_THRESHOLDS } from '../../../config/action-engine.js';const { maxTaskRunsPerNight } = ENGINE_THRESHOLDS.noise;import { TOOLS } from './tools.js';+import { ReadLaterError } from '../competitor-pages.js';const TASK = 'task-1';const ACTION = 'action-1';@@ -162,6 +163,48 @@ describe('runTask', () => {expect(run.row.error).toContain('timeout');});+ /** A tool whose reads for the day are spent (learning C1) tries again the next night. */+ it('leaves the task as it was when its tool can only go on tomorrow', async () => {+ rpc.mockRejectedValue(Object.assign(new Error('reads for today spent'), { retryLater: true }));+ const writes = mockDb({ task: task(), action });++ const result = await runTask(TASK, { now: NOW, sources: ALL });++ expect(result).toEqual({ status: 'blocked', reason: 'reads for today spent' });+ expect(writes.updated.some((w) => w.table === 'action_tasks')).toBe(false);+ expect(writes.inserted.find((w) => w.table === 'action_task_runs').row).toMatchObject({+ status: 'blocked',+ error: 'reads for today spent',+ });+ expect(writes.inserted.some((w) => w.table === 'action_events')).toBe(false);+ });++ /**+ * A tool with nothing to work on (learning C1: no page cited) doesn't+ * complete its task: the task stays a person's, shows why, and the sweep+ * passes it over for a week.+ */+ it('leaves the task to a person, with why, when its tool found nothing to work on', async () => {+ const why = { competitorPages: [], gaps: [], reason: 'none_cited' };+ rpc.mockRejectedValue(new ReadLaterError('no page cited', { output: why, retryDays: 7 }));+ const writes = mockDb({ task: task(), action });++ const result = await runTask(TASK, { now: NOW, sources: ALL });++ const retryAfter = '2026-10-02T06:00:00.000Z';+ expect(result).toEqual({ status: 'blocked', reason: 'no page cited', retryAfter });+ const [kept, ...more] = writes.updated.filter((w) => w.table === 'action_tasks');+ expect(more).toEqual([]);+ expect(kept.patch).not.toHaveProperty('status');+ expect(kept.patch.output).toEqual({ ...why, retryAfter });+ expect(writes.inserted.find((w) => w.table === 'action_task_runs').row).toMatchObject({+ status: 'blocked',+ output: why,+ error: 'no page cited',+ });+ expect(writes.inserted.some((w) => w.table === 'action_events')).toBe(false);+ });+it('records a blocked attempt rather than passing over it in silence', async () => {const writes = mockDb({ task: task({ task_key: 'update_content' }), action });@@ -408,6 +451,28 @@ describe('sweepTaskExecution', () => {expect(summary.succeeded + summary.failed + summary.blocked).toBe(maxTaskRunsPerNight);});+ it('passes over a task waiting to be tried again, without it taking a place', async () => {+ const waiting = {+ task_key: 'validate',+ action_id: ACTION,+ retry_after: '2026-10-01T00:00:00Z',+ };+ const due = Array.from({ length: maxTaskRunsPerNight }, (_, i) => ({+ id: `t${i}`,+ task_key: 'validate',+ action_id: ACTION,+ // Its week is over: tried again.+ ...(i === 0 ? { retry_after: '2026-09-24T00:00:00Z' } : {}),+ }));+ mockSweep([{ id: 'w1', ...waiting }, { id: 'w2', ...waiting }, ...due]);++ const summary = await sweepTaskExecution({ now: NOW });++ expect(summary.waitingToRetry).toBe(2);+ expect(summary.skippedForCeiling).toBe(0);+ expect(summary.succeeded + summary.failed + summary.blocked).toBe(maxTaskRunsPerNight);+ });+it('carries on past a task that throws unexpectedly', async () => {mockSweep([{ id: 't1', task_key: 'validate', action_id: ACTION },@@ -523,7 +588,9 @@ describe('sweepTaskExecution', () => {try {const asked = mockPlanSweep(lapsedThenPaying());await sweepTaskExecution({ now: NOW });- expect(asked.select).toBe('id, task_key, action_id, actions!inner(status)');+ expect(asked.select).toBe(+ 'id, task_key, action_id, retry_after:output->>retryAfter, actions!inner(status)',+ );expect(asked.filters.map(([column]) => column)).not.toContain('actions.brands.organizations.subscription_status',);diff --git a/server/src/lib/action-center/execution/tools.js b/server/src/lib/action-center/execution/tools.jsindex 19fb62c7f..8e25e1f45 100644--- a/server/src/lib/action-center/execution/tools.js+++ b/server/src/lib/action-center/execution/tools.js@@ -33,6 +33,7 @@import supabaseAdmin from '../../../config/supabase.js';import { runGaReport } from '../../composio.js';import { resolve } from '../../../config/action-engine.js';+import { analyzeCompetitorPages } from '../competitor-pages.js';const DAY_MS = 86_400_000;@@ -50,6 +51,22 @@ function entityIdFor(organizationId) {const tool = (id, spec) => Object.freeze({ id, version: 1, writesExternally: false, ...spec });export const TOOLS = Object.freeze({+ /**+ * What the competitor pages AI answers cite for the action's prompts have+ * that the brand's page doesn't (learning C1): read free from the public+ * pages its own answers cited, through Site Audit's rule checks. Reading a+ * public page changes nothing, so no approval.+ */+ competitor_page_features: tool('competitor_page_features', {+ source: 'competitors',+ description:+ "What the competitor pages AI answers cite for the action's prompts have that the brand's page doesn't.",++ async run({ brandId, payload, now }) {+ return analyzeCompetitorPages({ brandId, payload, now });+ },+ }),+/*** The AI Visibility Score over the validation window and the one before it* — what `validate` asks for, read from the daily rollups rather than thediff --git a/server/src/lib/action-center/page-features.js b/server/src/lib/action-center/page-features.jsnew file mode 100644index 000000000..be7bb570e--- /dev/null+++ b/server/src/lib/action-center/page-features.js@@ -0,0 +1,135 @@+/**+ * What Site Audit's checks find on one page read, and what competitor pages+ * have that the brand's doesn't (learning C1): the part of+ * competitor-pages.js that reads no database, so the brief (C2), Improve+ * (C2) and their tests load it without the supabase client, whose config+ * exits when the SUPABASE_* env is absent (as in CI).+ */++import * as cheerio from 'cheerio';+import { resolve } from '../../config/action-engine.js';+import { signalRegistry } from '../audit/signals/index.js';+import { describeSignal } from '../audit/rubric.js';++const { competitorPages: C } = resolve();++/**+ * Less text than this, and what was read is a wall (a consent or sign-in+ * page, a page drawn by script), not the page: as content-cited-length.js's+ * CITED_MIN_CHARACTERS.+ */+export const MIN_PAGE_CHARACTERS = 500;++/**+ * The checks compared: Site Audit's that read the page alone, never its+ * site's robots.txt or llms.txt, and that the page's author can act on.+ */+export const FEATURE_CHECKS = Object.freeze([+ 'length',+ 'structural-depth',+ 'h2-coverage',+ 'lists',+ 'tables',+ 'faq-schema',+ 'json-ld-presence',+ 'statistic-density',+ 'quotations',+ 'citation-density',+ 'outbound-authority-links',+ 'author-byline',+ 'date-markup',+ 'freshness',+ 'readability',+]);++/** Elements whose text isn't the page's words (audit/context.js removes them). */+const NOT_TEXT = new Set(['script', 'style', 'noscript', 'template', 'svg']);++/**+ * The body's text without NOT_TEXT's elements, in one walk. Removing them+ * first, as audit/context.js does, splices each out of its parent's+ * children: tens of thousands of empty <style>s in a page AI cites would+ * hold the server for a minute.+ */+export function bodyText($) {+ const start = $('body')[0] ?? $.root()[0];+ const out = [];+ const stack = [start];+ while (stack.length) {+ const node = stack.pop();+ if (node.type === 'text') {+ out.push(node.data);+ continue;+ }+ if (node.type === 'cdata' && node !== start) {+ for (const child of node.children ?? []) if (child.type === 'text') out.push(child.data);+ continue;+ }+ if (node !== start && NOT_TEXT.has(node.name)) continue;+ const children = node.children ?? [];+ for (let i = children.length - 1; i >= 0; i -= 1) stack.push(children[i]);+ }+ return out.join('');+}++/** What Site Audit's checks read, built from a page fetched here. */+export function pageContext(url, html, now = Date.now()) {+ const $ = cheerio.load(html);+ // Read without changing `$`, so it keeps its JSON-LD blocks (audit/context.js).+ const text = bodyText($).replace(/\s+/g, ' ').trim();+ const u = new URL(url);+ return {+ url,+ origin: `${u.protocol}//${u.host}`,+ protocol: u.protocol.replace(':', ''),+ statusCode: 200,+ html,+ htmlBytes: Buffer.byteLength(html, 'utf8'),+ $,+ text,+ wordCount: text ? text.split(/\s+/).length : 0,+ robotsTxt: null,+ llmsTxt: null,+ query: null,+ now,+ };+}++/**+ * A page's features: its words and each compared check's status; null when+ * what was read is a wall rather than the page (MIN_PAGE_CHARACTERS).+ */+export async function pageFeatures(url, html, now = Date.now()) {+ const ctx = pageContext(url, html, now);+ if (ctx.text.length < MIN_PAGE_CHARACTERS) return null;+ const checks = {};+ for (const signal of signalRegistry) {+ if (!FEATURE_CHECKS.includes(signal.key)) continue;+ try {+ checks[signal.key] = (await signal.evaluate(ctx)).status;+ } catch {+ checks[signal.key] = 'na';+ }+ }+ return { words: ctx.wordCount, checks };+}++/**+ * The checks at least half of the competitor pages pass that the brand's+ * page doesn't, the most passed first; every such check when no page of the+ * brand's was read. None from fewer than `minPages` pages: one page isn't+ * what "their pages" have.+ */+export function featureGaps(competitorPages, yourPage, { minPages = C.minPages } = {}) {+ const read = competitorPages.filter((page) => page.checks);+ if (read.length < Math.max(1, minPages)) return [];+ const needed = Math.ceil(read.length / 2);+ return FEATURE_CHECKS.map((key) => ({+ key,+ label: describeSignal(key)?.label ?? key,+ competitors: read.filter((page) => page.checks[key] === 'pass').length,+ of: read.length,+ }))+ .filter((gap) => gap.competitors >= needed && yourPage?.checks?.[gap.key] !== 'pass')+ .sort((a, b) => b.competitors - a.competitors);+}diff --git a/server/src/lib/action-center/page-features.test.js b/server/src/lib/action-center/page-features.test.jsnew file mode 100644index 000000000..5376044c2--- /dev/null+++ b/server/src/lib/action-center/page-features.test.js@@ -0,0 +1,102 @@+import { describe, expect, it } from 'vitest';+import * as cheerio from 'cheerio';+import { FEATURE_CHECKS, bodyText, featureGaps, pageFeatures } from './page-features.js';+import { signalRegistry } from '../audit/signals/index.js';++/**+ * What Site Audit's checks find on a page read, and what competitor pages+ * have that the brand's doesn't (learning C1). No supabase mock on purpose:+ * this module has to load without the SUPABASE_* env, as in CI, for the+ * brief and Improve that read it.+ */++/** Enough words that what was read is a page, not a wall. */+const PROSE = `<p>${'A kettle boils water quickly and safely for tea. '.repeat(14)}</p>`;+const FAQ_PAGE = `<html><head><script type="application/ld+json">${JSON.stringify({+ '@type': 'FAQPage',+ mainEntity: [{ '@type': 'Question' }, { '@type': 'Question' }],+})}</script></head><body><h1>Best kettles</h1><h2>Which kettle?</h2>${PROSE}+<table><thead><tr><th>Kettle</th></tr></thead><tbody><tr><td>a</td></tr></tbody></table></body></html>`;+const WALL_PAGE = '<html><body><p>We use cookies. Accept to continue.</p></body></html>';++describe('FEATURE_CHECKS', () => {+ it("are Site Audit checks, none of them reading the site's robots.txt or llms.txt", () => {+ const keys = new Set(signalRegistry.map((signal) => signal.key));+ expect(FEATURE_CHECKS.filter((key) => !keys.has(key))).toEqual([]);+ expect(FEATURE_CHECKS).not.toContain('ai-bot-access');+ expect(FEATURE_CHECKS).not.toContain('llms-txt-presence');+ });+});++describe('pageFeatures', () => {+ it("runs Site Audit's checks over a page read here", async () => {+ const { words, checks } = await pageFeatures('https://rival.com/kettles', FAQ_PAGE);+ expect(words).toBeGreaterThan(0);+ expect(checks['faq-schema']).toBe('pass');+ expect(checks.tables).toBe('pass');+ expect(Object.keys(checks).sort()).toEqual([...FEATURE_CHECKS].sort());+ });++ it('reads a consent wall or an empty page as no page at all', async () => {+ expect(await pageFeatures('https://rival.com/kettles', WALL_PAGE)).toBeNull();+ expect(+ await pageFeatures('https://rival.com/kettles', '<html><body></body></html>'),+ ).toBeNull();+ });+});++describe('featureGaps', () => {+ const page = (passing) => ({+ checks: Object.fromEntries(+ FEATURE_CHECKS.map((key) => [key, passing.includes(key) ? 'pass' : 'fail']),+ ),+ });++ it("names the checks at least half the competitor pages pass that the brand's page doesn't", () => {+ const gaps = featureGaps(+ [page(['faq-schema', 'tables']), page(['faq-schema']), page(['lists'])],+ page(['tables']),+ );+ expect(gaps.map((gap) => [gap.key, gap.competitors, gap.of])).toEqual([['faq-schema', 2, 3]]);+ expect(gaps[0].label).toEqual(expect.any(String));+ });++ it('names every such check when no page of the brand was read', () => {+ expect(+ featureGaps([page(['tables']), page(['tables', 'lists'])], null).map((gap) => gap.key),+ ).toEqual(['tables', 'lists']);+ });++ it('compares nothing from fewer than two competitor pages', () => {+ expect(featureGaps([page(['tables'])], null)).toEqual([]);+ expect(featureGaps([page(['tables'])], page([]))).toEqual([]);+ expect(featureGaps([], null)).toEqual([]);+ });+});++describe('reading a page AI cites, whatever it holds', () => {+ it("reads the body's words as Site Audit does, its scripts, styles and drawings left out", () => {+ const html =+ '<html><head><title>T</title><style>a{}</style></head><body><h1>Hi</h1><script>x=1</script>' ++ '<p>a <b>b</b> c</p><svg><text>no</text></svg><noscript>ns</noscript>' ++ '<template><p>tp</p></template>tail</body></html>';+ const removed = cheerio.load(html);+ removed('script, style, noscript, template, svg').remove();+ expect(bodyText(cheerio.load(html))).toBe(removed('body').text());+ expect(bodyText(cheerio.load(html))).toBe('Hia b ctail');+ });++ it('reads a page of tens of thousands of empty styles in a second, not a minute', async () => {+ const html = `<html><body>${PROSE}${'<style></style>'.repeat(100_000)}</body></html>`;+ const started = Date.now();+ expect(await pageFeatures('https://example.com/a', html)).not.toBeNull();+ expect(Date.now() - started).toBeLessThan(5_000);+ });++ it('reads a page of tens of thousands of empty list items in seconds, not a minute', async () => {+ const html = `<html><body>${PROSE}${'<li></li>'.repeat(40_000)}</body></html>`;+ const started = Date.now();+ expect(await pageFeatures('https://example.com/a', html)).not.toBeNull();+ expect(Date.now() - started).toBeLessThan(5_000);+ });+});diff --git a/server/src/lib/action-center/tasks/registry.js b/server/src/lib/action-center/tasks/registry.jsindex 8870ffcc8..98c36a310 100644--- a/server/src/lib/action-center/tasks/registry.js+++ b/server/src/lib/action-center/tasks/registry.js@@ -147,6 +147,8 @@ export const TASKS = Object.freeze({analyze_competitor_content: agent('analyze_competitor_content', {requires: ['competitors'],outputs: ['findings'],+ // Read free from the pages the brand's answers cite (learning C1).+ tool: 'competitor_page_features',}),analyze_ai_traffic: system('analyze_ai_traffic', {requires: ['ai_traffic'],diff --git a/server/src/lib/audit/signals/readability.js b/server/src/lib/audit/signals/readability.jsindex 4f6736fd0..885a87296 100644--- a/server/src/lib/audit/signals/readability.js+++ b/server/src/lib/audit/signals/readability.js@@ -59,6 +59,38 @@ function inlineText(el) {return out;}+/** The elements read as blocks, by name. */+const BLOCKS = new Set('h1 h2 h3 p li blockquote div section article figcaption dd dt'.split(' '));++/**+ * The body's block elements in document order, as+ * `$('body').find('h1, h2, …')` gives them, in one walk: that find's+ * context step is quadratic in the elements it starts from, and a page of+ * tens of thousands of short paragraphs held the server for a minute.+ */+function blockElements($) {+ const found = [];+ const body = $('body')[0];+ if (!body) return found;+ const stack = [...(body.children ?? [])].reverse();+ while (stack.length) {+ const node = stack.pop();+ if (node.type !== 'tag' && node.type !== 'script' && node.type !== 'style') continue;+ if (BLOCKS.has(node.name.toLowerCase())) found.push(node);+ const children = node.children ?? [];+ for (let i = children.length - 1; i >= 0; i -= 1) stack.push(children[i]);+ }+ return found;+}++/** Whether an element sits inside one of these, by name (a selector per element costs more than the walk). */+function insideOf(el, names) {+ for (let up = el.parent; up && up.type === 'tag'; up = up.parent) {+ if (names.includes(up.name.toLowerCase())) return true;+ }+ return false;+}+/*** The page's headings and prose, in order: a quote once, each list item on* its own, a table's cells not at all. Text a page builder leaves loose in@@ -71,23 +103,21 @@ export function readingBlocks(html) {const $ = cheerio.load(html ?? '');$('script, style, noscript, template, svg').remove();const blocks = [];- $('body')- .find('h1, h2, h3, p, li, blockquote, div, section, article, figcaption, dd, dt')- .each((_, el) => {- const tag = el.name.toLowerCase();- if (['div', 'section', 'article', 'figcaption', 'dd', 'dt'].includes(tag)) {- // Inside a block read whole, or a table, it is read with it, or not at all.- if ($(el).parents('p, li, blockquote, td, th').length) return;- const loose = inlineText(el).replace(/\s+/g, ' ').trim();- if (loose) blocks.push({ tag: 'p', text: loose });- return;- }- // A paragraph in a list item, a quote or a cell is read with it, or not at all.- if (tag === 'p' && $(el).closest('li, blockquote, td, th').length) return;- if (tag === 'li' && $(el).closest('td, th').length) return;- const text = ownText(el).replace(/\s+/g, ' ').trim();- if (text) blocks.push({ tag, text });- });+ blockElements($).forEach((el) => {+ const tag = el.name.toLowerCase();+ if (['div', 'section', 'article', 'figcaption', 'dd', 'dt'].includes(tag)) {+ // Inside a block read whole, or a table, it is read with it, or not at all.+ if (insideOf(el, ['p', 'li', 'blockquote', 'td', 'th'])) return;+ const loose = inlineText(el).replace(/\s+/g, ' ').trim();+ if (loose) blocks.push({ tag: 'p', text: loose });+ return;+ }+ // A paragraph in a list item, a quote or a cell is read with it, or not at all.+ if (tag === 'p' && insideOf(el, ['li', 'blockquote', 'td', 'th'])) return;+ if (tag === 'li' && insideOf(el, ['td', 'th'])) return;+ const text = ownText(el).replace(/\s+/g, ' ').trim();+ if (text) blocks.push({ tag, text });+ });return blocks;}diff --git a/server/src/lib/audit/signals/readability.test.js b/server/src/lib/audit/signals/readability.test.jsindex db7547a3f..892d7cad7 100644--- a/server/src/lib/audit/signals/readability.test.js+++ b/server/src/lib/audit/signals/readability.test.js@@ -90,4 +90,11 @@ describe('readability as the editor reads it', () => {expect(fleschBand(70.4)).toEqual({ status: 'pass', score: 1 });expect(fleschBand(49.6)).toEqual({ status: 'pass', score: 1 });});++ it('reads a page of tens of thousands of paragraphs in one pass, not a minute', () => {+ const html = '<p>One short line here.</p>'.repeat(40_000);+ const started = Date.now();+ expect(readingBlocks(html)).toHaveLength(40_000);+ expect(Date.now() - started).toBeLessThan(3_000);+ });});diff --git a/supabase/migrations/00369_cited_page_features.sql b/supabase/migrations/00369_cited_page_features.sqlnew file mode 100644index 000000000..cde8485af--- /dev/null+++ b/supabase/migrations/00369_cited_page_features.sql@@ -0,0 +1,46 @@+-- ============================================================================+-- 00369: what the pages AI answers cite have (learning C1)+-- ============================================================================+-- The Action Center's "Analyze competitor content" task was a step for a+-- person: nothing read the competitor pages AI engines cite instead of the+-- brand. The task now reads them (server action-center/competitor-pages.js):+-- the public pages cited in the brand's own tracked answers for the action's+-- prompts, and the brand's own page cited there, fetched free (safe-fetch,+-- never a paid scraper) and run through Site Audit's own rule checks+-- (structure, FAQ, schema, statistics, dates, length). Only what the checks+-- found is kept here, never the page's text, and nothing is summarised by AI.+--+-- One row per brand and page: read again after two weeks, and a day's reads+-- counted against the brand's daily cap. The server writes and reads it+-- (service role); members see what it found on the task.++create table if not exists public.cited_page_features (+ id uuid primary key default gen_random_uuid(),+ brand_id uuid not null references public.brands(id) on delete cascade,+ -- The page as Site Audit keys it: host and path, lower case.+ page_key text not null,+ url text not null,+ domain text not null,+ -- The brand's own page, cited for the same prompts.+ own boolean not null default false,+ -- { words, checks: { "<check key>": "pass" | "warn" | "fail" | "na" } }, or+ -- { unreadable: true[, disallowed: true] } for a page that couldn't be read+ -- (it blocks readers, a wall, robots.txt): kept so it counts against the+ -- day's reads and isn't tried again while fresh.+ features jsonb not null default '{}'::jsonb,+ read_at timestamptz not null default now(),+ unique (brand_id, page_key)+);++create index if not exists cited_page_features_brand_read_idx+ on public.cited_page_features (brand_id, read_at desc);++-- The server's alone: no member reads it directly.+alter table public.cited_page_features enable row level security;+revoke all on table public.cited_page_features from anon, authenticated;+grant all on table public.cited_page_features to service_role;++comment on table public.cited_page_features is+ 'Learning C1 (00369): Site Audit rule checks of the pages cited in a brand''s tracked answers for an action''s prompts (competitors'' and its own), read free (safe-fetch). Features only, never page text. Server only.';++notify pgrst, 'reload schema';diff --git a/supabase/schema.sql b/supabase/schema.sqlindex 1e37d1430..6d7fadbb2 100644--- a/supabase/schema.sql+++ b/supabase/schema.sql@@ -42713,6 +42713,56 @@ comment on function public.action_scorecard(uuid, timestamptz) isnotify pgrst, 'reload schema';+-- ─────────────────────────────────────────────────────────────────────────+-- migrations/00369_cited_page_features.sql+-- ─────────────────────────────────────────────────────────────────────────+-- ============================================================================+-- 00369: what the pages AI answers cite have (learning C1)+-- ============================================================================+-- The Action Center's "Analyze competitor content" task was a step for a+-- person: nothing read the competitor pages AI engines cite instead of the+-- brand. The task now reads them (server action-center/competitor-pages.js):+-- the public pages cited in the brand's own tracked answers for the action's+-- prompts, and the brand's own page cited there, fetched free (safe-fetch,+-- never a paid scraper) and run through Site Audit's own rule checks+-- (structure, FAQ, schema, statistics, dates, length). Only what the checks+-- found is kept here, never the page's text, and nothing is summarised by AI.+--+-- One row per brand and page: read again after two weeks, and a day's reads+-- counted against the brand's daily cap. The server writes and reads it+-- (service role); members see what it found on the task.++create table if not exists public.cited_page_features (+ id uuid primary key default gen_random_uuid(),+ brand_id uuid not null references public.brands(id) on delete cascade,+ -- The page as Site Audit keys it: host and path, lower case.+ page_key text not null,+ url text not null,+ domain text not null,+ -- The brand's own page, cited for the same prompts.+ own boolean not null default false,+ -- { words, checks: { "<check key>": "pass" | "warn" | "fail" | "na" } }, or+ -- { unreadable: true[, disallowed: true] } for a page that couldn't be read+ -- (it blocks readers, a wall, robots.txt): kept so it counts against the+ -- day's reads and isn't tried again while fresh.+ features jsonb not null default '{}'::jsonb,+ read_at timestamptz not null default now(),+ unique (brand_id, page_key)+);++create index if not exists cited_page_features_brand_read_idx+ on public.cited_page_features (brand_id, read_at desc);++-- The server's alone: no member reads it directly.+alter table public.cited_page_features enable row level security;+revoke all on table public.cited_page_features from anon, authenticated;+grant all on table public.cited_page_features to service_role;++comment on table public.cited_page_features is+ 'Learning C1 (00369): Site Audit rule checks of the pages cited in a brand''s tracked answers for an action''s prompts (competitors'' and its own), read free (safe-fetch). Features only, never page text. Server only.';++notify pgrst, 'reload schema';+-- ─────────────────────────────────────────────────────────────────────────-- migrations/00370_prompt_demand.sql-- ─────────────────────────────────────────────────────────────────────────