Skip to content

Open-source agentic AI Search Execution Platform. Track AI visibility, find opportunities, and let agents turn signals into prioritized actions and drafted fixes across ChatGPT, Claude, Gemini, Google AI Overviews, Google AI Mode, Perplexity, Grok, and Microsoft Copilot.

feat(action-center): report detector thresholds against one brand's action outcomes (#1757)

Cihan03a83b4ab71c
diff --git a/docs/guides/action-center.mdx b/docs/guides/action-center.mdx
index b0728afa5..68ddc9942 100644
--- a/docs/guides/action-center.mdx
+++ b/docs/guides/action-center.mdx
@@ -205,6 +205,12 @@ Switch History from **Actions** to **Scorecard** to see what each kind of action
When an action is closed, its condition can open a new action after a two-week rest if it is still firing. The old one stays in History as the record of that round of work.
+### Detector thresholds (maintainers)
+
+The detectors' thresholds are first settings, not calibrations. To see how they did at a brand, export its closed actions with `server/src/scripts/export-detector-outcomes.sql` (psql, read-only, with the brand's id as `brand_id`; it stops without one), writing outside the repository, such as `> ~/outcomes.jsonl`, and read them with `node src/scripts/calibrate-detector-thresholds.js ~/outcomes.jsonl` from `server/`. Each action is read by its signals as they were when it opened, from its baseline; older actions without one are left out and counted. The report shows, per signal kind with 20 or more closed actions, how its actions turned out and why they were dismissed, by the signal's strength in quarters, in the unit of the kind's threshold (a topic's coverage, a relative fall, a competitor's multiple). Where the weakest quarter was mostly dismissed as not relevant or wrong data, or improves less often than the rest, it suggests a threshold in that unit; when strength doesn't spread enough to compare, it says so. A kind with no definition gets its distribution only. A suggestion is a starting point; nothing changes until you edit the threshold the report names.
+
+Run it on one brand (Ansvisor's own, or your own self-hosted data). Running it across customers' brands is not supported; it waits on a cross-brand learning decision and a privacy policy update.
+
## Where actions come from
Most actions are generated from signals each night. You can also send work to the Action Center from other pages. A sent action has no detection signals; its evidence is what you sent, shown on its Overview tab.
diff --git a/server/src/lib/detector-calibration.js b/server/src/lib/detector-calibration.js
new file mode 100644
index 000000000..c94b7917c
--- /dev/null
+++ b/server/src/lib/detector-calibration.js
@@ -0,0 +1,708 @@
+/**
+ * Detector thresholds against what their actions did (learning C6).
+ *
+ * The Action Center's detectors (config/action-engine.js) were set as first
+ * settings, conservative, "not calibrations". This reads, for one brand, the
+ * actions they opened that are over (completed or dismissed): per signal
+ * kind, how the actions turned out (improved, mixed, no meaningful change,
+ * declined), how many were dismissed and why, and the same by the signal's
+ * strength when the action opened, in quarters.
+ *
+ * Strength is read in the unit of the kind's threshold (KIND_DEFINITIONS):
+ * a topic's coverage for `topic_uncovered`, the relative fall for
+ * `topic_drop`, the competitor's multiple for `competitor_leads`. Where the
+ * weakest quarter mostly ends dismissed as not relevant or wrong data, or
+ * improves at most half as often as the stronger actions, it suggests a
+ * threshold at that quarter's edge, in that unit. A kind with no definition
+ * gets its distribution only, never a suggestion. A suggestion is a starting
+ * point for a maintainer, never applied.
+ *
+ * It reads rows exported for one brand (scripts/export-detector-outcomes.sql)
+ * and gives back aggregates only: nothing of an action (its id, its brand)
+ * is in what it returns. Nothing here reads a database or the network.
+ */
+
+import { ENGINE_THRESHOLDS } from '../config/action-engine.js';
+
+/** Fewer actions than this for a kind, and nothing is said of it. */
+export const MIN_ACTIONS = 20;
+/** Fewer actions than this on either side of the comparison, and it isn't made. */
+const MIN_BUCKET = 5;
+/** Below this share improved, over enough measured actions, a detector rarely leads anywhere. */
+const QUIET_IMPROVED = 0.1;
+
+/** The outcomes that say how an action turned out (the rest are unmeasured). */
+export const MEASURED = ['improved', 'mixed', 'no_meaningful_change', 'declined'];
+export const DISMISS_REASONS = ['not_relevant', 'already_done', 'wrong_data', 'not_now'];
+/** The reasons that say the detector shouldn't have fired; `already_done`
+ * and `not_now` say nothing about the threshold. */
+export const REJECT_REASONS = ['not_relevant', 'wrong_data'];
+
+const num = (value) => {
+ const n = typeof value === 'string' && value.trim() !== '' ? Number(value) : value;
+ return typeof n === 'number' && Number.isFinite(n) ? n : null;
+};
+
+// ─── Strength per kind ──────────────────────────────────────────────────────
+
+/*
+ * How each strength is read from a baseline signal's `currentValue` and
+ * `changeValue`. Where one needs the value before, it is current − change:
+ * the detectors it's used for store their change as current minus previous.
+ */
+const before = (current, change) => (current === null || change === null ? null : current - change);
+const VALUE = { formula: 'current', strength: (current) => current };
+const PERCENT = {
+ formula: 'current ÷ 100',
+ strength: (current) => (current === null ? null : current / 100),
+};
+const RISE = { formula: 'change', strength: (_, change) => change };
+const GAP = { formula: '−change', strength: (_, change) => (change === null ? null : -change) };
+const GAP_PERCENT = {
+ formula: '−change ÷ 100',
+ strength: (_, change) => (change === null ? null : -change / 100),
+};
+const FALL = {
+ formula: '−change ÷ (current − change)',
+ strength: (current, change) => {
+ const previous = before(current, change);
+ return previous > 0 ? -change / previous : null;
+ },
+};
+const GROWTH = {
+ formula: 'change ÷ (current − change)',
+ strength: (current, change) => {
+ const previous = before(current, change);
+ return previous > 0 ? change / previous : null;
+ },
+};
+const MULTIPLE = {
+ formula: 'current ÷ (current − change)',
+ strength: (current, change) => {
+ const previous = before(current, change);
+ return previous > 0 ? current / previous : null;
+ },
+};
+const SHARE_OF_CURRENT = {
+ formula: 'change ÷ current',
+ strength: (current, change) => (current > 0 && change !== null ? change / current : null),
+};
+
+const CONFIG = 'config/action-engine.js';
+
+/** A threshold in config/action-engine.js, by its path there, at its value now. */
+const config = (path) => {
+ const [group, key] = path.split('.');
+ return { name: path, file: CONFIG, value: ENGINE_THRESHOLDS[group][key] };
+};
+
+function define(measure, read, stronger, unit, threshold) {
+ return Object.freeze({
+ measure,
+ formula: read.formula,
+ strength: read.strength,
+ stronger,
+ unit,
+ threshold: Object.freeze(threshold),
+ });
+}
+
+/**
+ * What a kind's strength is, in its threshold's unit: the value read from
+ * the baseline, whether higher or lower is stronger (past the threshold),
+ * the unit, and the threshold that decides it. Read from the detectors:
+ * lib/pulse/metrics.js and lib/signals/record.js for the original kinds,
+ * lib/signals/library/classify.js and detect.js for the library's. A kind
+ * whose stored value isn't what its threshold judges (a count of pages
+ * found, say) has no definition.
+ */
+export const KIND_DEFINITIONS = Object.freeze({
+ // Original detectors (detection.*).
+ sharp_drop: define(
+ 'relative fall of visibility',
+ FALL,
+ 'higher',
+ 'share of the level before (0–1)',
+ config('detection.visibilityDropRatio'),
+ ),
+ visibility_slipping: define(
+ 'relative fall of visibility',
+ FALL,
+ 'higher',
+ 'share of the level before (0–1)',
+ config('detection.slippingMinRatio'),
+ ),
+ platform_gap: define(
+ 'gap between the best and worst platform',
+ GAP,
+ 'higher',
+ 'points',
+ config('detection.platformGapPoints'),
+ ),
+ competitor_citation_gap: define(
+ "competitor's lead in cited answers",
+ GAP,
+ 'higher',
+ 'cited answers',
+ config('detection.citationGapMinAbsolute'),
+ ),
+ competitor_surge: define(
+ "competitor's rise",
+ RISE,
+ 'higher',
+ 'points',
+ config('detection.competitorSurgePoints'),
+ ),
+ prompt_gain: define("prompt's rise", RISE, 'higher', 'points', config('detection.moverMinGain')),
+ page_opportunity: define(
+ "page's sessions",
+ VALUE,
+ 'higher',
+ 'sessions',
+ config('detection.pageMinSessions'),
+ ),
+ uncited_mentions: define(
+ 'prompts mentioned but never cited',
+ VALUE,
+ 'higher',
+ 'prompts',
+ config('detection.uncitedMinPrompts'),
+ ),
+
+ // Prompts (library.*; classify.js groupPrompts).
+ ...Object.fromEntries(
+ [
+ 'prompt_visibility_gap',
+ 'high_demand_gap',
+ 'uncovered_demand',
+ 'prompt_unmapped',
+ 'content_underperforming',
+ ].map((kind) => [
+ kind,
+ define('prompts in the group', VALUE, 'higher', 'prompts', config('library.promptGroupMin')),
+ ]),
+ ),
+ ...Object.fromEntries(
+ [
+ ['lost_mentions', 2],
+ ['high_value_prompt_lost', 1],
+ ['citation_at_risk', 2],
+ ].map(([kind, value]) => [
+ kind,
+ define('prompts in the group', VALUE, 'higher', 'prompts', {
+ name: `groupPrompts minimum.${kind}`,
+ file: 'lib/signals/library/classify.js',
+ value,
+ }),
+ ]),
+ ),
+
+ // Topics: coverage is stored in percent, its thresholds are shares.
+ topic_uncovered: define(
+ 'coverage',
+ PERCENT,
+ 'lower',
+ 'share of tracked prompts (0–1)',
+ config('library.topicUncoveredCoverage'),
+ ),
+ topic_gap: define(
+ 'coverage',
+ PERCENT,
+ 'lower',
+ 'share of tracked prompts (0–1)',
+ config('library.topicPartialCoverage'),
+ ),
+ topic_slipping: define(
+ 'relative fall of coverage',
+ FALL,
+ 'higher',
+ 'share of the level before (0–1)',
+ config('library.slipRatio'),
+ ),
+ topic_drop: define(
+ 'relative fall of coverage',
+ FALL,
+ 'higher',
+ 'share of the level before (0–1)',
+ config('library.dropRatio'),
+ ),
+
+ // Platforms and regions.
+ platform_slipping: define(
+ 'relative fall of the mention rate',
+ FALL,
+ 'higher',
+ 'share of the level before (0–1)',
+ config('library.slipRatio'),
+ ),
+ platform_drop: define(
+ 'relative fall of the mention rate',
+ FALL,
+ 'higher',
+ 'share of the level before (0–1)',
+ config('library.dropRatio'),
+ ),
+ country_gap: define(
+ 'gap between the best and worst region',
+ GAP_PERCENT,
+ 'higher',
+ 'mention rate (0–1)',
+ config('library.countryGapPoints'),
+ ),
+
+ // Competitors.
+ competitor_leads: define(
+ "competitor's mentions per brand mention",
+ MULTIPLE,
+ 'higher',
+ 'multiple',
+ config('library.competitorLeadMultiple'),
+ ),
+ competitor_momentum: define(
+ "competitor's mention growth",
+ GROWTH,
+ 'higher',
+ 'share of the mentions before (0–1)',
+ config('library.competitorMomentumRatio'),
+ ),
+ competitor_topic_lead: define(
+ 'tracked prompts in the topic',
+ VALUE,
+ 'higher',
+ 'prompts',
+ config('library.topicMinPrompts'),
+ ),
+ competitor_uncovered_topic: define(
+ 'tracked prompts in the topic',
+ VALUE,
+ 'higher',
+ 'prompts',
+ config('library.topicMinPrompts'),
+ ),
+
+ // Citation sources: current is the answers citing it, change those citing
+ // it with only a competitor.
+ competitor_winning_source: define(
+ 'share of its answers with only a competitor',
+ SHARE_OF_CURRENT,
+ 'higher',
+ 'share of answers (0–1)',
+ config('library.competitorSourceShare'),
+ ),
+ ...Object.fromEntries(
+ ['competitor_cited_source', 'third_party_presence_gap', 'third_party_citation_gap'].map(
+ (kind) => [
+ kind,
+ define(
+ 'answers citing the source',
+ VALUE,
+ 'higher',
+ 'answers',
+ config('library.sourceMinResults'),
+ ),
+ ],
+ ),
+ ),
+ authority_citation_gap: define(
+ 'answers citing the source',
+ VALUE,
+ 'higher',
+ 'answers',
+ config('library.authorityMinResults'),
+ ),
+
+ // Fan-outs.
+ fanout_gap: define(
+ 'queries in the group',
+ VALUE,
+ 'higher',
+ 'queries',
+ config('library.fanoutGroupMin'),
+ ),
+ competitor_fanout_gap: define(
+ 'queries in the group',
+ VALUE,
+ 'higher',
+ 'queries',
+ config('library.fanoutGroupMin'),
+ ),
+
+ // AI traffic: the site's sessions or visits, before and now.
+ ai_traffic_drop: define(
+ 'relative fall of AI sessions',
+ FALL,
+ 'higher',
+ 'share of the level before (0–1)',
+ config('library.aiTrafficDropRatio'),
+ ),
+ ai_visits_weekly_drop: define(
+ 'relative fall of AI visits',
+ FALL,
+ 'higher',
+ 'share of the level before (0–1)',
+ config('library.aiVisitsWeeklyDropRatio'),
+ ),
+ cross_channel_risk: define(
+ 'relative fall of AI sessions',
+ FALL,
+ 'higher',
+ 'share of the level before (0–1)',
+ config('library.aiTrafficSlipRatio'),
+ ),
+ visibility_traffic_loss: define(
+ 'relative fall of AI sessions',
+ FALL,
+ 'higher',
+ 'share of the level before (0–1)',
+ { name: 'crossChannel trafficFall', file: 'lib/signals/library/detect.js', value: 0.2 },
+ ),
+});
+
+/**
+ * A baseline signal's strength: by its kind's definition, else (with no
+ * definition) the size of its change, else of its value, and the column used.
+ */
+export function strengthOf(kind, current, change) {
+ const definition = KIND_DEFINITIONS[kind];
+ if (definition) return { strength: num(definition.strength(current, change)), from: null };
+ if (change !== null) return { strength: Math.abs(change), from: 'changeValue' };
+ if (current !== null) return { strength: Math.abs(current), from: 'currentValue' };
+ return { strength: null, from: null };
+}
+
+/** Higher is stronger once this multiplies a strength. */
+const signOf = (kind) => (KIND_DEFINITIONS[kind]?.stronger === 'lower' ? -1 : 1);
+
+// ─── Rows ───────────────────────────────────────────────────────────────────
+
+/**
+ * The export's JSON lines, as rows; a line that isn't one is counted, never
+ * kept. A row is one action and one signal of its baseline. The export's
+ * last line counts the closed actions with no baseline of their signals,
+ * left out (`withoutBaseline`; null when the export had no such line).
+ *
+ * @returns {{ rows: Array<{ action: string, kind: string, current: number | null,
+ * change: number | null, strength: number | null, strengthFrom: string | null,
+ * status: string, outcome: string | null, dismissReason: string | null }>,
+ * skipped: number, withoutBaseline: number | null }}
+ */
+export function parseRows(text) {
+ const rows = [];
+ let skipped = 0;
+ let withoutBaseline = null;
+ for (const line of String(text ?? '')
+ .replace(/^\uFEFF/, '')
+ .split(/\r?\n/)) {
+ if (!line.trim()) continue;
+ let row;
+ try {
+ row = JSON.parse(line);
+ } catch {
+ skipped += 1;
+ continue;
+ }
+ const left = num(row?.withoutBaseline);
+ if (row && row.action === undefined && left !== null) {
+ withoutBaseline = (withoutBaseline ?? 0) + left;
+ continue;
+ }
+ if (!row || typeof row.action !== 'string' || typeof row.kind !== 'string' || !row.kind) {
+ skipped += 1;
+ continue;
+ }
+ const current = num(row.current);
+ const change = num(row.change);
+ const { strength, from } = strengthOf(row.kind, current, change);
+ rows.push({
+ action: row.action,
+ kind: row.kind,
+ current,
+ change,
+ strength,
+ strengthFrom: from,
+ status: typeof row.status === 'string' ? row.status : '',
+ outcome: typeof row.outcome === 'string' ? row.outcome : null,
+ dismissReason: typeof row.dismissReason === 'string' ? row.dismissReason : null,
+ });
+ }
+ return { rows, skipped, withoutBaseline };
+}
+
+/** One row per action and signal kind: its strongest signal of that kind. */
+export function actionsByKind(rows) {
+ const kinds = new Map();
+ const score = (row) => (row.strength === null ? -Infinity : row.strength * signOf(row.kind));
+ for (const row of rows) {
+ const byAction = kinds.get(row.kind) ?? new Map();
+ const known = byAction.get(row.action);
+ if (!known || score(row) > score(known)) byAction.set(row.action, row);
+ kinds.set(row.kind, byAction);
+ }
+ return new Map([...kinds].map(([kind, byAction]) => [kind, [...byAction.values()]]));
+}
+
+// ─── The report ─────────────────────────────────────────────────────────────
+
+/** The value at a share of sorted values (nearest rank). */
+function quantile(sorted, share) {
+ return sorted[Math.min(sorted.length - 1, Math.max(0, Math.ceil(share * sorted.length) - 1))];
+}
+
+/** How a set of actions turned out: results are a completed action's, as on the Scorecard. */
+function tally(actions) {
+ const measured = actions.filter((a) => a.status === 'completed' && MEASURED.includes(a.outcome));
+ const improved = measured.filter((a) => a.outcome === 'improved').length;
+ const dismissed = actions.filter((a) => a.status === 'dismissed');
+ const rejected = dismissed.filter((a) => REJECT_REASONS.includes(a.dismissReason)).length;
+ return {
+ actions: actions.length,
+ measured: measured.length,
+ improved,
+ dismissed: dismissed.length,
+ rejected,
+ improvedRate: measured.length ? improved / measured.length : null,
+ dismissRate: actions.length ? dismissed.length / actions.length : null,
+ rejectRate: actions.length ? rejected / actions.length : null,
+ };
+}
+
+/** A definition as the report gives it back: everything but the function. */
+function describe({ strength: _, ...definition }) {
+ return definition;
+}
+
+/**
+ * The report, per kind with enough actions: its outcomes, dismissals and
+ * their reasons, its strength quarters (weakest first; actions with no
+ * strength are outside them), its weakest quarter against every action
+ * stronger than that quarter's edge (`rest`), and a verdict:
+ * - `raise`: the weakest quarter was dismissed as not relevant or wrong
+ * data at least half the time, more often than the rest (`rejected`), or,
+ * with 5 measured on each side, improved at most half as often
+ * (`improves_less`); `suggestion` is that quarter's edge, in the unit of
+ * the kind's threshold;
+ * - `keep`: the weakest quarter does about as well as the rest;
+ * - `unclear`: under 5 actions on either side: strength doesn't spread
+ * enough to compare;
+ * - `no_definition`: the kind's strength has no threshold to compare with;
+ * its distribution only.
+ * `quiet` is apart from the verdict: under 10% of its measured actions
+ * improved. A kind with fewer actions is listed as insufficient, with its count.
+ */
+export function calibrate(rows, { min = MIN_ACTIONS } = {}) {
+ const kinds = [];
+ const insufficient = [];
+ for (const [kind, actions] of [...actionsByKind(rows)].sort(([a], [b]) => a.localeCompare(b))) {
+ if (actions.length < min) {
+ insufficient.push({ kind, actions: actions.length });
+ continue;
+ }
+ const definition = KIND_DEFINITIONS[kind] ?? null;
+ const sign = signOf(kind);
+ const completed = actions.filter((a) => a.status === 'completed');
+ const outcomes = Object.fromEntries(
+ [...MEASURED, 'not_measurable', 'pending_measurement'].map((o) => [
+ o,
+ completed.filter((a) => a.outcome === o).length,
+ ]),
+ );
+ const dismissReasons = Object.fromEntries(
+ DISMISS_REASONS.map((r) => [r, actions.filter((a) => a.dismissReason === r).length]),
+ );
+
+ // Scores run weakest to strongest whichever way the kind's strength goes.
+ const scores = actions
+ .filter((a) => a.strength !== null)
+ .map((a) => a.strength * sign)
+ .sort((a, b) => a - b);
+ const cuts = scores.length ? [0.25, 0.5, 0.75].map((share) => quantile(scores, share)) : [];
+ const quarterOf = (action) => {
+ if (action.strength === null) return -1;
+ const quarter = cuts.findIndex((cut) => action.strength * sign <= cut);
+ return quarter === -1 ? 3 : quarter;
+ };
+ const buckets = cuts.length
+ ? [0, 1, 2, 3].map((q) => ({
+ // Back in the strength's own unit; `|| 0` keeps -0 out.
+ edge: (q < 3 ? cuts[q] : scores[scores.length - 1]) * sign || 0,
+ ...tally(actions.filter((a) => quarterOf(a) === q)),
+ }))
+ : [];
+ const weakest = buckets[0] ?? null;
+ const rest = weakest ? tally(actions.filter((a) => quarterOf(a) > 0)) : null;
+
+ const reasons = [];
+ const comparable = Boolean(
+ weakest && weakest.actions >= MIN_BUCKET && rest.actions >= MIN_BUCKET,
+ );
+ if (definition && comparable) {
+ if (weakest.rejectRate >= 0.5 && weakest.rejectRate > rest.rejectRate) {
+ reasons.push('rejected');
+ }
+ if (
+ weakest.measured >= MIN_BUCKET &&
+ rest.measured >= MIN_BUCKET &&
+ rest.improvedRate > 0 &&
+ weakest.improvedRate <= rest.improvedRate / 2
+ ) {
+ reasons.push('improves_less');
+ }
+ }
+ const verdict = !definition
+ ? 'no_definition'
+ : !comparable
+ ? 'unclear'
+ : reasons.length
+ ? 'raise'
+ : 'keep';
+
+ const overall = tally(actions);
+ const strengthFrom = definition
+ ? null
+ : {
+ changeValue: actions.filter((a) => a.strengthFrom === 'changeValue').length,
+ currentValue: actions.filter((a) => a.strengthFrom === 'currentValue').length,
+ };
+ kinds.push({
+ kind,
+ definition: definition ? describe(definition) : null,
+ strengthFrom,
+ ...overall,
+ unrated: actions.length - scores.length,
+ outcomes,
+ dismissReasons,
+ buckets,
+ rest,
+ verdict,
+ reasons,
+ suggestion: verdict === 'raise' ? weakest.edge : null,
+ quiet: overall.measured >= min && (overall.improvedRate ?? 0) < QUIET_IMPROVED,
+ });
+ }
+ return { kinds, insufficient };
+}
+
+const pct = (rate) => (rate === null ? '–' : `${Math.round(rate * 100)}%`);
+const fmt = (value) => String(Math.round(value * 1000) / 1000);
+
+const REASON_TEXT = {
+ rejected: 'was mostly dismissed as not relevant or wrong data',
+ improves_less: 'improves less often than the rest',
+};
+
+function strengthLine(k) {
+ const d = k.definition;
+ if (!d) {
+ const used = Object.entries(k.strengthFrom)
+ .filter(([, count]) => count > 0)
+ .map(([column, count]) => `${column} for ${count}`);
+ return (
+ 'No threshold definition for this kind: strength is the size of changeValue, else of ' +
+ `currentValue${used.length ? ` (${used.join(', ')})` : ''}. No suggestion is made.`
+ );
+ }
+ const shared = Object.entries(KIND_DEFINITIONS)
+ .filter(([kind, other]) => kind !== k.kind && other.threshold.name === d.threshold.name)
+ .map(([kind]) => kind);
+ return (
+ `Strength: ${d.measure} = ${d.formula}, in ${d.unit}; ${d.stronger} is stronger. ` +
+ `Threshold: ${d.threshold.name} in ${d.threshold.file}, now ${fmt(d.threshold.value)}.` +
+ (shared.length ? ` Also used by ${shared.join(', ')}.` : '')
+ );
+}
+
+function verdictLine(k) {
+ const d = k.definition;
+ switch (k.verdict) {
+ case 'raise':
+ return (
+ `Suggestion: consider ${d.measure} ${d.stronger === 'lower' ? 'below' : 'above'} ` +
+ `${fmt(k.suggestion)} (now ${fmt(d.threshold.value)}): its weakest quarter ` +
+ `${k.reasons.map((r) => REASON_TEXT[r]).join(' and ')}.`
+ );
+ case 'keep':
+ return 'Keep: its weakest quarter does about as well as the rest.';
+ case 'unclear':
+ return "Unclear: strength doesn't spread enough to compare its weakest quarter with the rest.";
+ default:
+ return null;
+ }
+}
+
+/** The report as Markdown, for a maintainer to read. */
+export function renderReport(
+ { kinds, insufficient },
+ { skipped = 0, min = MIN_ACTIONS, withoutBaseline = null } = {},
+) {
+ const lines = [
+ '# Detector thresholds against outcomes (one brand)',
+ '',
+ "Each action is read by its signals' strength when it opened, as its baseline recorded it.",
+ '',
+ ];
+ if (kinds.length === 0) {
+ lines.push(`No signal kind has ${min} or more closed actions yet.`, '');
+ }
+ for (const k of kinds) {
+ lines.push(`## ${k.kind}`, '');
+ lines.push(
+ `${k.actions} closed actions · ${k.measured} measured · improved ${pct(k.improvedRate)} · dismissed ${pct(k.dismissRate)}`,
+ '',
+ strengthLine(k),
+ '',
+ );
+ if (k.buckets.length) {
+ const edge = k.definition?.stronger === 'lower' ? 'down to' : 'up to';
+ lines.push(
+ `| Strength ${edge} | Actions | Measured | Improved | Dismissed | Not relevant or wrong data |`,
+ '| --- | --- | --- | --- | --- | --- |',
+ );
+ for (const b of k.buckets) {
+ lines.push(
+ `| ${fmt(b.edge)} | ${b.actions} | ${b.measured} | ${pct(b.improvedRate)} | ${pct(b.dismissRate)} | ${pct(b.rejectRate)} |`,
+ );
+ }
+ const w = k.buckets[0];
+ lines.push(
+ '',
+ `Weakest quarter against the rest (${w.actions} vs ${k.rest.actions} actions): ` +
+ `improved ${pct(w.improvedRate)} vs ${pct(k.rest.improvedRate)} ` +
+ `(of ${w.measured} vs ${k.rest.measured} measured), ` +
+ `not relevant or wrong data ${pct(w.rejectRate)} vs ${pct(k.rest.rejectRate)}.`,
+ );
+ } else {
+ lines.push('No action has a strength to put in quarters.');
+ }
+ if (k.unrated > 0) {
+ lines.push('', `${k.unrated} action(s) had no strength and are outside the quarters.`);
+ }
+ const reasons = Object.entries(k.dismissReasons)
+ .filter(([, count]) => count > 0)
+ .map(([reason, count]) => `${reason} ${count}`);
+ if (reasons.length) lines.push('', `Dismissed because: ${reasons.join(', ')}`);
+ const verdict = verdictLine(k);
+ if (verdict) lines.push('', verdict);
+ if (k.quiet) {
+ lines.push(
+ '',
+ 'Note: under 10% of its measured actions improved; check what the detector asks for.',
+ );
+ }
+ lines.push('');
+ }
+ if (insufficient.length) {
+ lines.push(
+ `Too few closed actions to say (under ${min}): ` +
+ insufficient.map((k) => `${k.kind} (${k.actions})`).join(', '),
+ '',
+ );
+ }
+ if (withoutBaseline !== null) {
+ lines.push(
+ `${withoutBaseline} closed action(s) with linked signals had no baseline of them and were left out.`,
+ '',
+ );
+ }
+ if (skipped) lines.push(`${skipped} line(s) of the export weren't rows and were skipped.`, '');
+ return lines.join('\n');
+}
diff --git a/server/src/lib/detector-calibration.test.js b/server/src/lib/detector-calibration.test.js
new file mode 100644
index 000000000..1642677d9
--- /dev/null
+++ b/server/src/lib/detector-calibration.test.js
@@ -0,0 +1,492 @@
+import { describe, expect, it } from 'vitest';
+import { ENGINE_THRESHOLDS } from '../config/action-engine.js';
+import { LIBRARY_KINDS } from './signals/library/kinds.js';
+import {
+ KIND_DEFINITIONS,
+ actionsByKind,
+ calibrate,
+ parseRows,
+ renderReport,
+ strengthOf,
+} from './detector-calibration.js';
+
+/** Detector thresholds against what their actions did (learning C6). */
+
+/** A kind whose strength is its value, higher stronger: prompts, at least 3. */
+const K = 'uncited_mentions';
+
+/**
+ * One export line per action n, from `from` to `to`: its baseline values,
+ * from `values(n)` (the value n by default), and how it ended, from `end(n)`.
+ */
+function lines(kind, from, to, end, values = (n) => ({ current: n })) {
+ const out = [];
+ for (let n = from; n <= to; n += 1) {
+ const [status, outcome = 'pending_measurement', dismissReason = null] = end(n);
+ out.push(
+ JSON.stringify({
+ action: `${kind}-${n}`,
+ kind,
+ ...values(n),
+ status,
+ outcome,
+ dismissReason,
+ }),
+ );
+ }
+ return out;
+}
+const rowsOf = (...groups) => parseRows(groups.flat().join('\n')).rows;
+const kindOf = (report, kind) => report.kinds.find((k) => k.kind === kind);
+const improved = () => ['completed', 'improved'];
+const rejected = (reason = 'not_relevant') => ['dismissed', 'pending_measurement', reason];
+
+describe('parseRows', () => {
+ it('reads baseline values, numbers given as text too, and counts lines that are not rows', () => {
+ const { rows, skipped, withoutBaseline } = parseRows(
+ [
+ '{"action":"a1","kind":"topic_drop","current":"35","change":"-15","status":"completed","outcome":"improved"}',
+ '{"action":"a2","kind":"k","change":null,"current":"0.4","status":"dismissed","outcome":"pending_measurement","dismissReason":"not_now"}',
+ '{"action":"a3","kind":"k","change":-12,"current":3,"status":"completed","outcome":"declined"}',
+ '{"action":"a4","kind":"topic_drop","status":"completed"}',
+ '',
+ 'not json',
+ '{"action":"a5"}',
+ '{"kind":"topic_drop"}',
+ '{"withoutBaseline":"4"}',
+ ].join('\r\n'),
+ );
+ expect(skipped).toBe(3);
+ expect(withoutBaseline).toBe(4);
+ expect(rows).toEqual([
+ {
+ action: 'a1',
+ kind: 'topic_drop',
+ current: 35,
+ change: -15,
+ strength: 0.3,
+ strengthFrom: null,
+ status: 'completed',
+ outcome: 'improved',
+ dismissReason: null,
+ },
+ {
+ action: 'a2',
+ kind: 'k',
+ current: 0.4,
+ change: null,
+ strength: 0.4,
+ strengthFrom: 'currentValue',
+ status: 'dismissed',
+ outcome: 'pending_measurement',
+ dismissReason: 'not_now',
+ },
+ {
+ action: 'a3',
+ kind: 'k',
+ current: 3,
+ change: -12,
+ strength: 12,
+ strengthFrom: 'changeValue',
+ status: 'completed',
+ outcome: 'declined',
+ dismissReason: null,
+ },
+ {
+ action: 'a4',
+ kind: 'topic_drop',
+ current: null,
+ change: null,
+ strength: null,
+ strengthFrom: null,
+ status: 'completed',
+ outcome: null,
+ dismissReason: null,
+ },
+ ]);
+ });
+
+ it('has no count of actions without a baseline when the export has no such line', () => {
+ expect(parseRows('{"action":"a1","kind":"k","current":1}').withoutBaseline).toBeNull();
+ });
+});
+
+describe('strengthOf', () => {
+ it.each([
+ ['topic_uncovered', 15, null, 0.15],
+ ['topic_drop', 35, -15, 0.3],
+ ['sharp_drop', 14, -6, 0.3],
+ ['platform_gap', 8, -22.5, 22.5],
+ ['country_gap', 12, -18, 0.18],
+ ['competitor_leads', 30, 10, 1.5],
+ ['competitor_leads', 30, 30, null],
+ ['competitor_momentum', 15, 5, 0.5],
+ ['competitor_winning_source', 40, 24, 0.6],
+ ['competitor_surge', 40, 16, 16],
+ ['uncited_mentions', 7, null, 7],
+ ['topic_drop', null, -15, null],
+ ])('reads %s (current %s, change %s) as %s in its unit', (kind, current, change, strength) => {
+ const read = strengthOf(kind, current, change).strength;
+ if (strength === null) expect(read).toBeNull();
+ else expect(read).toBeCloseTo(strength);
+ });
+
+ it('reads a kind with no definition by the size of its change, else its value', () => {
+ expect(strengthOf('audit_low_score', 9, -3)).toEqual({ strength: 3, from: 'changeValue' });
+ expect(strengthOf('audit_low_score', 9, null)).toEqual({ strength: 9, from: 'currentValue' });
+ expect(strengthOf('audit_low_score', null, null)).toEqual({ strength: null, from: null });
+ });
+});
+
+describe('KIND_DEFINITIONS', () => {
+ it("names each config threshold by its path and today's value", () => {
+ for (const [kind, d] of Object.entries(KIND_DEFINITIONS)) {
+ if (d.threshold.file !== 'config/action-engine.js') continue;
+ const [group, key] = d.threshold.name.split('.');
+ expect(ENGINE_THRESHOLDS[group][key], kind).toBe(d.threshold.value);
+ }
+ });
+
+ it("defines library kinds and the original detectors' kinds that keep their measure", () => {
+ expect(
+ Object.keys(KIND_DEFINITIONS)
+ .filter((kind) => !(kind in LIBRARY_KINDS))
+ .sort(),
+ ).toEqual([
+ 'competitor_citation_gap',
+ 'competitor_surge',
+ 'page_opportunity',
+ 'platform_gap',
+ 'prompt_gain',
+ 'sharp_drop',
+ 'uncited_mentions',
+ 'visibility_slipping',
+ ]);
+ });
+});
+
+describe('actionsByKind', () => {
+ it("keeps one row per action and kind, its strongest signal's", () => {
+ const rows = rowsOf([
+ '{"action":"a1","kind":"k","change":2,"status":"completed"}',
+ '{"action":"a1","kind":"k","change":-9,"status":"completed"}',
+ '{"action":"a1","kind":"other","change":1,"status":"completed"}',
+ '{"action":"a2","kind":"k","change":1,"status":"dismissed"}',
+ // Lower coverage is the stronger signal.
+ '{"action":"a3","kind":"topic_uncovered","current":15,"status":"completed"}',
+ '{"action":"a3","kind":"topic_uncovered","current":5,"status":"completed"}',
+ ]);
+ const byKind = actionsByKind(rows);
+ expect(byKind.get('k').map((r) => [r.action, r.strength])).toEqual([
+ ['a1', 9],
+ ['a2', 1],
+ ]);
+ expect(byKind.get('other')).toHaveLength(1);
+ expect(byKind.get('topic_uncovered').map((r) => r.strength)).toEqual([0.05]);
+ });
+});
+
+describe('calibrate', () => {
+ it('says nothing of a kind with fewer than 20 actions', () => {
+ const report = calibrate(rowsOf(lines(K, 1, 19, improved)));
+ expect(report.kinds).toEqual([]);
+ expect(report.insufficient).toEqual([{ kind: K, actions: 19 }]);
+ });
+
+ it('suggests a threshold, in its unit, where the weakest quarter was dismissed as not relevant', () => {
+ const report = calibrate(rowsOf(lines(K, 1, 20, (n) => (n <= 5 ? rejected() : improved()))));
+ const k = kindOf(report, K);
+ expect(k).toMatchObject({
+ actions: 20,
+ measured: 15,
+ improved: 15,
+ dismissed: 5,
+ rejected: 5,
+ verdict: 'raise',
+ reasons: ['rejected'],
+ suggestion: 5,
+ quiet: false,
+ });
+ expect(k.definition).toMatchObject({
+ unit: 'prompts',
+ stronger: 'higher',
+ threshold: { name: 'detection.uncitedMinPrompts', value: 3 },
+ });
+ expect(k.dismissReasons).toEqual({
+ not_relevant: 5,
+ already_done: 0,
+ wrong_data: 0,
+ not_now: 0,
+ });
+ expect(k.buckets.map((b) => [b.edge, b.actions])).toEqual([
+ [5, 5],
+ [10, 5],
+ [15, 5],
+ [20, 5],
+ ]);
+ expect(k.rest).toMatchObject({ actions: 15, rejected: 0 });
+ });
+
+ it("doesn't take already done or not now as the detector being wrong", () => {
+ for (const reason of ['already_done', 'not_now']) {
+ const report = calibrate(
+ rowsOf(lines(K, 1, 20, (n) => (n <= 5 ? rejected(reason) : improved()))),
+ );
+ expect(kindOf(report, K)).toMatchObject({ dismissed: 5, rejected: 0, verdict: 'keep' });
+ }
+ });
+
+ it('raises at exactly half the weakest quarter rejected, not below', () => {
+ // 24 actions: the weakest quarter is 1–6.
+ const half = calibrate(
+ rowsOf(lines(K, 1, 24, (n) => (n <= 3 ? rejected('wrong_data') : improved()))),
+ );
+ expect(kindOf(half, K)).toMatchObject({ verdict: 'raise', reasons: ['rejected'] });
+ expect(kindOf(half, K).buckets[0]).toMatchObject({ actions: 6, rejectRate: 0.5 });
+
+ const under = calibrate(
+ rowsOf(lines(K, 1, 24, (n) => (n <= 2 ? rejected('wrong_data') : improved()))),
+ );
+ expect(kindOf(under, K)).toMatchObject({ verdict: 'keep', suggestion: null });
+ });
+
+ it('raises where the weakest quarter improves exactly half as often, not more', () => {
+ const half = calibrate(
+ rowsOf(lines(K, 1, 24, (n) => ['completed', n <= 3 || n > 6 ? 'improved' : 'declined'])),
+ );
+ expect(kindOf(half, K)).toMatchObject({
+ verdict: 'raise',
+ reasons: ['improves_less'],
+ suggestion: 6,
+ });
+ expect(kindOf(half, K).buckets[0].improvedRate).toBe(0.5);
+ expect(kindOf(half, K).rest.improvedRate).toBe(1);
+
+ const over = calibrate(
+ rowsOf(lines(K, 1, 24, (n) => ['completed', n <= 4 || n > 6 ? 'improved' : 'declined'])),
+ );
+ expect(kindOf(over, K)).toMatchObject({ verdict: 'keep' });
+ });
+
+ it("doesn't compare improvement with fewer than 5 measured on a side", () => {
+ // The weakest quarter: one measured, and it declined; the rest all improved.
+ const weak = calibrate(
+ rowsOf(
+ lines(K, 1, 20, (n) =>
+ n === 1 ? ['completed', 'declined'] : n <= 5 ? rejected('not_now') : improved(),
+ ),
+ ),
+ );
+ expect(kindOf(weak, K)).toMatchObject({ verdict: 'keep', reasons: [] });
+ expect(kindOf(weak, K).buckets[0].measured).toBe(1);
+
+ // The rest: four measured, all improved; the weakest quarter's five declined.
+ const rest = calibrate(
+ rowsOf(
+ lines(K, 1, 20, (n) =>
+ n <= 5 ? ['completed', 'declined'] : n <= 9 ? improved() : rejected('already_done'),
+ ),
+ ),
+ );
+ expect(kindOf(rest, K)).toMatchObject({ verdict: 'keep', reasons: [] });
+ expect(kindOf(rest, K).rest.measured).toBe(4);
+ });
+
+ it('compares the weakest quarter with every stronger action when ties empty the strongest', () => {
+ // 16–20 tie with 15 at the top: the strongest quarter is empty.
+ const report = calibrate(
+ rowsOf(
+ lines(
+ K,
+ 1,
+ 20,
+ (n) => (n <= 5 ? rejected() : improved()),
+ (n) => ({ current: Math.min(n, 15) }),
+ ),
+ ),
+ );
+ const k = kindOf(report, K);
+ expect(k.buckets.map((b) => b.actions)).toEqual([5, 5, 10, 0]);
+ expect(k.rest.actions).toBe(15);
+ expect(k).toMatchObject({ verdict: 'raise', suggestion: 5 });
+ });
+
+ it("is unclear when strength doesn't spread enough to compare", () => {
+ const even = calibrate(
+ rowsOf(
+ lines(
+ K,
+ 1,
+ 20,
+ (n) => (n <= 15 ? rejected() : improved()),
+ () => ({ current: 3 }),
+ ),
+ ),
+ );
+ expect(kindOf(even, K)).toMatchObject({ verdict: 'unclear', suggestion: null });
+ expect(kindOf(even, K).rest.actions).toBe(0);
+
+ // Three actions above a tie of seventeen.
+ const tied = calibrate(
+ rowsOf(
+ lines(
+ K,
+ 1,
+ 20,
+ (n) => (n <= 10 ? rejected() : improved()),
+ (n) => ({ current: n <= 17 ? 3 : n }),
+ ),
+ ),
+ );
+ expect(kindOf(tied, K)).toMatchObject({ verdict: 'unclear', suggestion: null });
+ expect(renderReport(tied)).toContain("strength doesn't spread enough to compare");
+ });
+
+ it('leaves actions with no strength out of the quarters, not out of the totals', () => {
+ const report = calibrate(
+ rowsOf(
+ lines(
+ K,
+ 1,
+ 24,
+ (n) => (n <= 4 || n > 20 ? rejected() : improved()),
+ (n) => ({ current: n <= 20 ? n : null }),
+ ),
+ ),
+ );
+ const k = kindOf(report, K);
+ expect(k).toMatchObject({ actions: 24, dismissed: 8, unrated: 4 });
+ expect(k.buckets.reduce((sum, b) => sum + b.actions, 0)).toBe(20);
+ expect(k.buckets[0]).toMatchObject({ edge: 5, actions: 5, rejected: 4 });
+ expect(k.rest).toMatchObject({ actions: 15, rejected: 0 });
+ expect(renderReport(report)).toContain('4 action(s) had no strength');
+ });
+
+ it('suggests a lower coverage where lower is stronger', () => {
+ // Coverage 0–19%: the weakest quarter is the highest, 15–19%.
+ const report = calibrate(
+ rowsOf(
+ lines(
+ 'topic_uncovered',
+ 1,
+ 20,
+ (n) => (n > 15 ? rejected() : improved()),
+ (n) => ({ current: n - 1, change: 0 }),
+ ),
+ ),
+ );
+ const k = kindOf(report, 'topic_uncovered');
+ expect(k.buckets.map((b) => b.edge)).toEqual([0.15, 0.1, 0.05, 0]);
+ expect(k).toMatchObject({ verdict: 'raise', suggestion: 0.15 });
+ const text = renderReport(report);
+ expect(text).toContain('consider coverage below 0.15 (now 0.2)');
+ expect(text).toContain('| Strength down to |');
+ expect(text).toContain('library.topicUncoveredCoverage in config/action-engine.js, now 0.2');
+ });
+
+ it('gives a kind with no definition its distribution only, and the column it read', () => {
+ const report = calibrate(rowsOf(lines('k', 1, 20, (n) => (n <= 5 ? rejected() : improved()))));
+ const k = kindOf(report, 'k');
+ expect(k).toMatchObject({
+ definition: null,
+ verdict: 'no_definition',
+ reasons: [],
+ suggestion: null,
+ strengthFrom: { changeValue: 0, currentValue: 20 },
+ });
+ expect(k.buckets).toHaveLength(4);
+ const text = renderReport(report);
+ expect(text).toContain('No threshold definition for this kind');
+ expect(text).toContain('currentValue for 20');
+ expect(text).not.toContain('Suggestion');
+ });
+
+ it('notes a quiet kind beside a suggestion', () => {
+ // 40 actions: the weakest quarter (1–10) dismissed as not relevant; of
+ // the 30 measured, one improved.
+ const report = calibrate(
+ rowsOf(
+ lines(K, 1, 40, (n) =>
+ n <= 10 ? rejected() : ['completed', n === 40 ? 'improved' : 'declined'],
+ ),
+ ),
+ );
+ expect(kindOf(report, K)).toMatchObject({ verdict: 'raise', suggestion: 10, quiet: true });
+ const text = renderReport(report);
+ expect(text).toContain('Suggestion: consider');
+ expect(text).toContain('Note: under 10% of its measured actions improved');
+ });
+
+ it('calls a kind quiet when under 10% of its results improved, evenly', () => {
+ const report = calibrate(
+ rowsOf(lines(K, 1, 20, (n) => ['completed', n === 3 ? 'improved' : 'declined'])),
+ );
+ expect(kindOf(report, K)).toMatchObject({ verdict: 'keep', quiet: true, suggestion: null });
+ expect(kindOf(report, K).outcomes).toMatchObject({ improved: 1, declined: 19 });
+ });
+
+ it('keeps a threshold when the weakest quarter does as well as the rest', () => {
+ const report = calibrate(
+ rowsOf(lines(K, 1, 20, (n) => ['completed', n % 2 ? 'improved' : 'declined'])),
+ );
+ expect(kindOf(report, K)).toMatchObject({
+ verdict: 'keep',
+ improvedRate: 0.5,
+ quiet: false,
+ });
+ expect(renderReport(report)).toContain('Keep: its weakest quarter does about as well');
+ });
+
+ it('counts a result only for a completed action, as the Scorecard does', () => {
+ const report = calibrate(
+ rowsOf(
+ lines(K, 1, 20, (n) => (n <= 10 ? ['dismissed', 'improved'] : ['completed', 'declined'])),
+ ),
+ );
+ expect(kindOf(report, K)).toMatchObject({ measured: 10, improved: 0, dismissed: 10 });
+ });
+
+ it('takes a lower minimum', () => {
+ const report = calibrate(rowsOf(lines(K, 1, 8, improved)), { min: 5 });
+ expect(report.kinds.map((k) => k.kind)).toEqual([K]);
+ });
+});
+
+describe('renderReport', () => {
+ it("prints aggregates, the suggestion and what was left out, never an action's id", () => {
+ const rows = rowsOf(
+ lines(K, 1, 20, (n) => (n <= 5 ? rejected() : improved())),
+ lines('topic_gap', 1, 4, improved),
+ );
+ const text = renderReport(calibrate(rows), { skipped: 2, withoutBaseline: 3 });
+ expect(text).toContain(`## ${K}`);
+ expect(text).toContain(
+ 'Suggestion: consider prompts mentioned but never cited above 5 (now 3): its weakest quarter was mostly dismissed as not relevant or wrong data.',
+ );
+ expect(text).toContain('Threshold: detection.uncitedMinPrompts in config/action-engine.js');
+ expect(text).toContain('Dismissed because: not_relevant 5');
+ expect(text).toContain('Too few closed actions to say (under 20): topic_gap (4)');
+ expect(text).toContain('3 closed action(s) with linked signals had no baseline');
+ expect(text).toContain("2 line(s) of the export weren't rows");
+ expect(text).not.toMatch(new RegExp(`${K}-\\d`));
+ expect(JSON.stringify(calibrate(rows))).not.toMatch(new RegExp(`${K}-\\d`));
+ });
+
+ it('softens the improvement reason and names the kinds sharing a threshold', () => {
+ const rows = rowsOf(
+ lines(
+ 'topic_slipping',
+ 1,
+ 20,
+ (n) => ['completed', n <= 5 ? 'declined' : 'improved'],
+ // From 50% coverage, down n points: a fall of n / 50.
+ (n) => ({ current: 50 - n, change: -n }),
+ ),
+ );
+ const text = renderReport(calibrate(rows));
+ expect(text).toContain(
+ 'consider relative fall of coverage above 0.1 (now 0.1): its weakest quarter improves less often than the rest.',
+ );
+ expect(text).toContain('Also used by platform_slipping.');
+ });
+});
diff --git a/server/src/scripts/calibrate-detector-thresholds.js b/server/src/scripts/calibrate-detector-thresholds.js
new file mode 100644
index 000000000..837b31c95
--- /dev/null
+++ b/server/src/scripts/calibrate-detector-thresholds.js
@@ -0,0 +1,69 @@
+/**
+ * Detector thresholds against outcomes (learning C6): reads one brand's
+ * closed actions, exported by src/scripts/export-detector-outcomes.sql, and
+ * prints, per signal kind, how its actions turned out and were dismissed,
+ * by the strength of the signals that opened them in quarters, with a
+ * suggested threshold, in its own unit, where the weakest quarter rarely
+ * leads anywhere. See src/lib/detector-calibration.js for how strength is
+ * read per kind and how a suggestion is made: it is a starting point for the
+ * maintainer, never applied by itself.
+ *
+ * Run it on one brand (Ansvisor's own, or your own self-hosted data).
+ * Running it across customers' brands is not supported; it waits on the
+ * cross-brand learning decision (C5) and a privacy policy update.
+ *
+ * Run from server/, with the export kept outside the repository:
+ * node src/scripts/calibrate-detector-thresholds.js ~/outcomes.jsonl [--min 20] [--out ~/report.json]
+ *
+ * --min the fewest closed actions a signal kind is judged on (20; 5 at least)
+ * --out also write the report as JSON
+ *
+ * Reads no database and no network, and prints and writes aggregates only:
+ * no action's id. Exits 1 when the file can't be read or no row could be.
+ * Isn't run in CI.
+ */
+import { readFileSync, writeFileSync } from 'fs';
+import { MIN_ACTIONS, calibrate, parseRows, renderReport } from '../lib/detector-calibration.js';
+
+const USAGE =
+ 'Usage: node src/scripts/calibrate-detector-thresholds.js <outcomes.jsonl> [--min 20] [--out <report.json>]';
+const args = process.argv.slice(2);
+const fail = (message) => {
+ console.error(message);
+ process.exit(1);
+};
+/** An option's value; a flag given with none, or with another flag after it, is an error. */
+const option = (name) => {
+ const at = args.indexOf(name);
+ if (at === -1) return null;
+ const value = args[at + 1];
+ if (value === undefined || value.startsWith('--')) fail(`${name} takes a value.\n${USAGE}`);
+ return value;
+};
+const file = args[0] && !args[0].startsWith('--') ? args[0] : null;
+if (!file) fail(USAGE);
+const minArg = option('--min');
+const min = minArg === null ? MIN_ACTIONS : Number(minArg);
+if (!Number.isInteger(min) || min < 5) fail('--min takes a whole number of actions, 5 at least.');
+const out = option('--out');
+
+let text;
+try {
+ text = readFileSync(file, 'utf8');
+} catch (err) {
+ fail(`Can't read ${file}: ${err.code === 'ENOENT' ? 'no such file' : err.message}.`);
+}
+const { rows, skipped, withoutBaseline } = parseRows(text);
+if (!rows.length) {
+ fail(
+ withoutBaseline
+ ? `No row could be read: the ${withoutBaseline} closed action(s) had no baseline of their signals.`
+ : 'No row could be read.',
+ );
+}
+const report = calibrate(rows, { min });
+console.log(renderReport(report, { skipped, min, withoutBaseline }));
+if (out) {
+ writeFileSync(out, `${JSON.stringify({ ...report, skipped, withoutBaseline }, null, 2)}\n`);
+ console.log(`Report written to ${out}.`);
+}
diff --git a/server/src/scripts/export-detector-outcomes.sql b/server/src/scripts/export-detector-outcomes.sql
new file mode 100644
index 000000000..47086028f
--- /dev/null
+++ b/server/src/scripts/export-detector-outcomes.sql
@@ -0,0 +1,61 @@
+-- Detector thresholds against outcomes (learning C6): exports one brand's
+-- closed actions (completed or dismissed) with the signals that opened them,
+-- one JSON object a line, for src/scripts/calibrate-detector-thresholds.js.
+--
+-- The signals are read from the action's baseline (actions.baseline.signals,
+-- lib/action-center/generate.js): their values when it opened. A signal's own
+-- row is no record of that: its values are rewritten every night it stays
+-- open, and an action takes signals as evidence after it opened too. Closed
+-- actions with linked signals but no baseline of them (older ones) are left
+-- out, and counted on the last line.
+--
+-- Run it on one brand (Ansvisor's own, or your own self-hosted data).
+-- Running it across customers' brands is not supported; it waits on the
+-- cross-brand learning decision (C5) and a privacy policy update.
+--
+-- Run from server/, with psql (it only reads), writing outside the repository:
+-- psql "<database url>" -X -q -A -t -v brand_id=<brand uuid> \
+-- -f src/scripts/export-detector-outcomes.sql > ~/outcomes.jsonl
+--
+-- brand_id is required; without it the script stops with an error. A line
+-- holds the action's id (so each action counts once per signal kind), the
+-- signal's kind and values, and how the action ended: no payload, prompt,
+-- URL or competitor.
+
+\set ON_ERROR_STOP on
+set default_transaction_read_only = on;
+
+\if :{?brand_id}
+\else
+ \warn 'Usage: psql "<database url>" -X -q -A -t -v brand_id=<brand uuid> -f src/scripts/export-detector-outcomes.sql > ~/outcomes.jsonl'
+ do $$ begin raise exception 'brand_id is required: the export reads one brand only'; end $$;
+\endif
+
+select json_build_object(
+ 'action', a.id,
+ 'kind', b->>'kind',
+ 'current', b->'currentValue',
+ 'change', b->'changeValue',
+ 'status', a.status,
+ 'outcome', a.outcome,
+ 'dismissReason', a.dismiss_reason
+ )
+ from public.actions a
+ cross join lateral jsonb_array_elements(
+ case when jsonb_typeof(a.baseline->'signals') = 'array'
+ then a.baseline->'signals' else '[]'::jsonb end
+ ) b
+ where a.brand_id = :'brand_id'::uuid
+ and a.deleted_at is null
+ and a.status in ('completed', 'dismissed')
+ order by a.created_at, a.id, b->>'kind';
+
+select json_build_object('withoutBaseline', count(*))
+ from public.actions a
+ where a.brand_id = :'brand_id'::uuid
+ and a.deleted_at is null
+ and a.status in ('completed', 'dismissed')
+ and exists (select 1 from public.action_signals l where l.action_id = a.id)
+ and case when jsonb_typeof(a.baseline->'signals') = 'array'
+ then jsonb_array_length(a.baseline->'signals')
+ else 0 end = 0;