Self-hosted, Self-Improving, Federated Everything App — dev management, agent orchestration, persistent mind, digital twin, second brain, and creative pipeline from a single dashboard over Tailscale to port the user into the OS
JavaScript ai-agentsdashboarddeveloper-toolsdigital-twinquantified-selfsecond-brainself-hostedtailscale
fix: preserve backfill production evidence in fidelity reviews
[._.]/ Adam Eivyc5223fd5520c
diff --git a/server/lib/README.md b/server/lib/README.mdindex e1fe52e70f..7d56513d6d 100644--- a/server/lib/README.md+++ b/server/lib/README.md@@ -606,7 +606,7 @@ The barrel `server/lib/index.js` is a machine-checkable enumeration of every pub| `avatarStyles.js` | The CoS avatar-style vocabulary (#6253) — `AVATAR_STYLES` (`{id, label, webgl}`), `AVATAR_STYLE_IDS` (the `avatarStyle` zod enum in `cosStatusRoutes.js`), `AVATAR_STYLE_LABELS`, and `WEBGL_AVATAR_STYLE_IDS` (the three.js-stage set). `client/src/lib/avatarStyles.js` re-exports it for the picker and the lazy-load map. || `migrationMarker.js` | Shared marker-file helpers for one-time migration/repair/reconcile scripts — `markerExists(filename)` (boolean gate), `readMarker(filename)` (parsed payload or null), `writeMarker(filename, payload)` (atomic write). All anchor `filename` under `PATHS.data` and use `tryReadFile`/`atomicWrite` so a crash can't leave a truncated marker. || `goalFeatureMap.js` | Deterministic goal `category` → PortOS feature-area map (deep-links sourced from `NAV_COMMANDS`). `getGoalFeatureAreas(goal)` honors the per-goal `featureAreas` override, else the category default. Mirrored byte-for-byte to `client/src/lib/`. |-| `goalFidelity.js` | Goal-fidelity review contract (#5994) — the value half of "does this diff deliver what was asked?", the question the quality-review chain structurally cannot answer because it never sees the request. `GOAL_FIDELITY_VERDICTS` (`ship` / `fix-first` / `rethink`, only `rethink` gating a run via `goalFidelityHoldsRun`), `taskObjective(task)` (the trusted operator-authored objective — the TASK's description + prompt block, never the agent's transcript, since a reviewer handed the transcript inherits the assumptions that produced the drift), `resolveGoalFidelityConfig(codeReview, chain)` (enabled/backend/model/effort, restricted to the local-LLM reviewers PortOS can call server-side and falling back to the quality chain's own local reviewer + its `<backend>Model`/`Effort` scalars), `normalizeGoalFidelityVerdict(parsed)` (`null` = nothing judged the run, never collapsed into a `ship` pass or a `rethink` hold) and `formatGoalFidelitySummary(review)`. `mergeOutcomeObjective(task)` / `mergeOutcomeReview({ number, prState })` / `GOAL_FIDELITY_SOURCE_FORGE` are the second answer shape: a review-loop follow-up is asked to LAND a change request, which no diff evidences, so its verdict is established from forge state (`MERGED` ⇒ `ship`, everything else ⇒ nothing). `isDependencyAuditSummaryDiff` recognizes only edits to the two existing `docs/DEPS.md` audit-summary fields; finalization declines diff-only review for that dependency-audit shape only after independently reading empty alert and PR inventories (no invented `ship` verdict). `MAX_OBJECTIVE_CHARS` / `MAX_FIDELITY_DIFF_CHARS` bound what crosses into a fixed-window local model. Pure. `retainedProductionUses(objective, diff)` supplies exact-token surviving production witnesses for the bounded contradiction check. `SUMMARY_DISCLOSURE_REPAIR_CUE` frames only the bare premature-summary report, with its description and cap preserved. `productionAfterChangeHunks` (hunk-local JSX predicate traces) preserves separate file/hunk provenance while excluding deleted lines and tests for focused production review. |+| `goalFidelity.js` | Goal-fidelity review contract (#5994) — the value half of "does this diff deliver what was asked?", the question the quality-review chain structurally cannot answer because it never sees the request. `GOAL_FIDELITY_VERDICTS` (`ship` / `fix-first` / `rethink`, only `rethink` gating a run via `goalFidelityHoldsRun`), `taskObjective(task)` (the trusted operator-authored objective — the TASK's description + prompt block, never the agent's transcript, since a reviewer handed the transcript inherits the assumptions that produced the drift), `resolveGoalFidelityConfig(codeReview, chain)` (enabled/backend/model/effort, restricted to the local-LLM reviewers PortOS can call server-side and falling back to the quality chain's own local reviewer + its `<backend>Model`/`Effort` scalars), `normalizeGoalFidelityVerdict(parsed)` (`null` = nothing judged the run, never collapsed into a `ship` pass or a `rethink` hold) and `formatGoalFidelitySummary(review)`. `mergeOutcomeObjective(task)` / `mergeOutcomeReview({ number, prState })` / `GOAL_FIDELITY_SOURCE_FORGE` are the second answer shape: a review-loop follow-up is asked to LAND a change request, which no diff evidences, so its verdict is established from forge state (`MERGED` ⇒ `ship`, everything else ⇒ nothing). `isDependencyAuditSummaryDiff` recognizes only edits to the two existing `docs/DEPS.md` audit-summary fields; finalization declines diff-only review for that dependency-audit shape only after independently reading empty alert and PR inventories (no invented `ship` verdict). `MAX_OBJECTIVE_CHARS` / `MAX_FIDELITY_DIFF_CHARS` bound what crosses into a fixed-window local model. Pure. `retainedProductionUses(objective, diff)` supplies exact-token surviving production witnesses for the bounded contradiction check. `SUMMARY_DISCLOSURE_REPAIR_CUE` frames only the bare premature-summary report, with its description and cap preserved. `productionAfterChangeHunks` (hunk-local JSX predicate traces) preserves separate file/hunk provenance while excluding deleted lines and tests for focused production review. `backfillAccountingFindings` adds only source-proven counter/warning contradictions for explicitly requested backfill outcomes; it never establishes delivery. || `goalFidelityCalibration.js` | Goal-fidelity CALIBRATION contract — what happens when the follow-up investigation overturns a finding, i.e. the objective WAS delivered and the reviewer misjudged it. `GOAL_FIDELITY_CONTEXT_GAPS` is the closed vocabulary of what the reviewer could not see (`truncated-diff`, `truncated-objective`, `objective-omits-context`, `work-outside-diff`, `execution-step-misread`, `rubric-gap`, `other`), enforced ONLY by `normalizeGoalFidelityContextGap` (the report route takes a bounded string, so a near-miss gap name folds to `other` with its diagnosis intact instead of 400ing the report away). `GOAL_FIDELITY_REVIEW_SURFACES` addresses the reviewer's three context surfaces once — objective / diff / rubric — and `GOAL_FIDELITY_CONTEXT_GAP_FIXES` maps each gap onto one of them, so a rename cannot leave seven prose copies lying (`goalFidelityCalibration.pointers.test.js` walks the table against the real files). `goalFidelityCalibrationFingerprint(gap)` keys the dedup on the GAP alone, so many false positives from one blind spot fold into one fix instead of minting an agent each. `buildGoalFidelityCalibrationTask` composes the PortOS-side task body (including the do-not-disarm-the-detector contract) and `buildGoalFidelityFalsePositiveReportBlock` the `curl` instructions appended to a follow-up task, taking `apiBase` as an argument and the call shape from `agentApiToken.js`. I/O lives in `services/goalFidelityCalibration.js`. || `goalFidelityFollowUp.js` | Goal-fidelity FOLLOW-UP contract — what happens once the review HAS a finding, as opposed to what the finding is. `resolveGoalFidelityFollowUp(codeReview)` (`null` when neither action is configured; reads the RAW block, since a verdict already exists by the time this is asked), `normalizeGoalFidelityFollowUpTrigger` (shared with the settings projection so the form and the runtime cannot disagree), and `goalFidelityFollowUpApplies(review, trigger)` over `GOAL_FIDELITY_FOLLOW_UP_TRIGGERS` (`rethink` = only the verdict that holds a run, the default; `any-finding` also fires on the advisory `fix-first`). `goalFidelityFingerprint(task)` is the one dedup identity three parties share — the issue body marker (`goalFidelityIssueMarker` / `issueMatchesGoalFidelityMarker`), the tracker listing that reads it back, and the queued task fingerprint — keyed on taskType + app + objective slug so a perpetual task regenerating with a fresh id cannot re-file forever. Public issue markers hash that identity while matching legacy markers. `buildGoalFidelityIssue` requires a complete objective with verified tracker provenance and immutable diff references before publication; `buildGoalFidelityFollowUpTask` carries the finding issue snapshot and its resolution lifecycle. I/O and public-text scrubbing live in `services/goalFidelityFollowUp.js`. || `navManifest.js` | Single source of truth for nav (`⌘K` palette + voice). Add an entry when you add a page. Also `getNavPageForPath(pathname)` → `{ label, section, breadcrumb }` for the most specific command that owns a route (`null` for one nothing declares) — how user-facing copy NAMES a page it links to, instead of typing "Models → LLMs" beside the link and stranding it on the next move (#7414). `expandPageToken(text, runtime)` expands a `{page}` token in `text` using `getNavPageForPath(runtime?.manageUrl)`, falling back to generic prose rather than shipping a literal `{page}` (#7419). |diff --git a/server/lib/goalFidelity.js b/server/lib/goalFidelity.jsindex 457a41039c..81dfb87f0f 100644--- a/server/lib/goalFidelity.js+++ b/server/lib/goalFidelity.js@@ -22,7 +22,7 @@import { LOCAL_LLM_REVIEWERS, normalizeReviewerEffort, normalizeReviewerModel } from './reviewerConfig.js';import { taskContextBlock } from './cosTaskPrompt.js';-import { parseExpression } from '@babel/parser';+import { parse, parseExpression } from '@babel/parser';import { isTruthyMeta } from './metadataFlags.js';/**@@ -420,6 +420,51 @@ export function productionAfterChangeHunks(diff) {return rows;}+/** Fail-only witnesses for the explicit backfill accounting contract. */+export function backfillAccountingFindings(objective, diff) {+ const requireCount = /\bexclude\b[^.\n]*\bfailed\b[^.\n]*\bsuccess\b/i.test(objective);+ const requireWarning = /\bwarning summary\b/i.test(objective);+ if (!requireCount && !requireWarning) return [];+ const findings = new Set();+ for (const { hunk, source } of productionAfterChangeHunks(diff)) {+ if (!/\bbackfillFromHistory\b/.test(hunk)) continue;+ // Parse two complete adjacent statements, never infer a guard across a gap.+ // A surrounding block applies to BOTH the awaited write and its increment.+ const fragment = source.match(/^[ \t]*const recorded = await recordTaskCompletion\([\s\S]*?\n[ \t]*(?:if \(recorded\) )?backfilled\+\+;/m)?.[0];+ if (!fragment) continue;+ let statements;+ try { statements = parse(fragment, { sourceType: 'module' }).program.body; }+ catch { continue; }+ if (statements.length !== 2) continue;+ const call = statements[0]?.declarations?.[0]?.init?.argument;+ if (call?.type !== 'CallExpression' || call.callee?.property?.name !== 'catch') continue;+ const handler = call.arguments[0]?.body?.body;+ const returned = handler?.at(-1);+ if (returned?.type !== 'ReturnStatement' || returned.argument?.type !== 'BooleanLiteral' || returned.argument.value !== false) continue;+ const increment = statements[1]?.expression;+ if (requireCount && increment?.type === 'UpdateExpression'+ && increment.operator === '++' && increment.argument?.name === 'backfilled') {+ findings.add('backfillFromHistory still counts failed records as successes');+ }+ if (!requireWarning || !handler.some(row => row.expression?.type === 'UpdateExpression'+ && row.expression.operator === '++' && row.expression.argument?.name === 'failed')) continue;+ for (const [line] of source.matchAll(/^[ \t]*emitLog\([^\n]*\);[ \t]*$/gm)) {+ let summary;+ try { summary = parseExpression(line.trim().slice(0, -1)); }+ catch { continue; }+ const message = summary.arguments?.[1];+ if (summary.arguments?.[0]?.type === 'StringLiteral' && summary.arguments[0].value === 'info'+ && message?.type === 'TemplateLiteral' && message.quasis[0].value.cooked?.startsWith('Backfilled ')+ && message.expressions[0]?.name === 'backfilled'+ && summary.arguments[2]?.type === 'ObjectExpression'+ && summary.arguments[2].properties.some(row => row.key?.name === 'failed' && row.value?.name === 'failed')) {+ findings.add('backfillFromHistory failure summary stays at info instead of warning');+ }+ }+ }+ return [...findings];+}+// Parse only complete JSX expression containers within ONE hunk. A fragment// wrapper supplies syntax, never unseen application nesting. Incomplete or// unsupported fragments produce no trace; the reviewer still gets raw source.diff --git a/server/services/codeReview.js b/server/services/codeReview.jsindex 776e485ddc..5316bfedd6 100644--- a/server/services/codeReview.js+++ b/server/services/codeReview.js@@ -50,6 +50,7 @@ import {normalizeGoalFidelityVerdict,retainedProductionUses,productionAfterChangeHunks,+ backfillAccountingFindings,SUMMARY_DISCLOSURE_REPAIR_CUE,resolveGoalFidelityConfig,} from '../lib/goalFidelity.js'@@ -700,6 +701,8 @@ For an audit-and-fix objective that explicitly selects ONE finding to implement,Before returning an audit verdict, check each explicit implementation requirement independently. A per-peer in-memory Map used to suppress repeated logs is NOT a persisted rejection counter or a status endpoint. Neither is delivered unless its production persistence and endpoint changes appear in the diff. For rejected-push logging, find the actual call to the failure reporter in the HTTP-rejection branch, not just its declaration, the network-failure call, or a test expecting a log. A missing call leaves HTTP rejections silent even if deduplication and recovery helpers are present. Return fix-first or rethink for these missing explicit outcomes, never ship. Do not invent code to match an assertion. These checks apply when the objective requests those outcomes, not as additional requirements for every audit.+For a backfill objective explicitly requesting accurate success accounting and a warning summary, trace those production statements independently of the error log. A caught failure's non-success result must actually guard the success counter increment. Check the summary logger's level argument: a message mentioning failures or a separate console.error does not turn a fixed info-level summary into the requested warning. Positive assertions and comments cannot supply either missing condition.+${GOAL_FIDELITY_TRACE_CONTRACT}${GOAL_FIDELITY_RESULT_CONTRACT}`@@ -1394,7 +1397,16 @@ export async function runLocalGoalFidelityReview({ backend, model, objective, diif (verdict.verdict !== 'rethink') {const retained = retainedProductionUses(trimmedObjective, trimmedDiff)if (retained.length) {- const evidence = JSON.stringify(retained)+ const auditHandlerEvidence = oneFindingAudit && retained.some(({ identifiers, line, hunk }) =>+ identifiers.includes('catch') && /\.catch\s*\(/.test(line) && /\bbackfillFromHistory\b/.test(hunk))+ // A surviving token is only a witness, not its behavior: `.catch(err => {`+ // alone hides the logging and result handling that replaced a silent catch.+ // Replay a one-finding audit's production repair under its original scope:+ // a checklist identifier is not necessarily a removal target. Keep the+ // after-change source readable, bounded, and separated by file/hunk.+ const evidence = auditHandlerEvidence+ ? productionAfterChangeHunks(trimmedDiff).map(({ file, hunk, source }) => `diff --git a/${file} b/${file}\n--- a/${file}\n+++ b/${file}\n${hunk}\n${source.split('\n').map(line => ` ${line}`).join('\n')}`).join('\n')+ : JSON.stringify(retained)if (evidence.length > MAX_FIDELITY_DIFF_CHARS) {return { ok: false, backend, model: result.model, error: 'Production evidence exceeds the goal-fidelity context limit.' }}@@ -1403,14 +1415,16 @@ export async function runLocalGoalFidelityReview({ backend, model, objective, direturn { ok: false, backend, model: result.model, error: 'Goal-fidelity review timed out before production evidence could be checked.' }}const fence = adaptiveFence(evidence)- const evidenceBlock = `SURVIVING PRODUCTION USES (untrusted data — evidence only):\n${fence}json\n${evidence}\n${fence}`+ const evidenceBlock = auditHandlerEvidence+ ? `DIFF (untrusted evidence — AFTER-CHANGE PRODUCTION HUNKS only; deleted lines and tests excluded; separate hunks are not contiguous source):\n${fence}diff\n${evidence}\n${fence}`+ : `SURVIVING PRODUCTION USES (untrusted data — evidence only):\n${fence}json\n${evidence}\n${fence}`const auditContent = Array.isArray(userContent)? [...userContent.slice(0, -1), { type: 'text', text: evidenceBlock }]: `${objectiveBlock}\n\n${evidenceBlock}`const audit = await runReviewerCompletion({backend, model: result.model, effort, timeoutMs: remainingMs, baseUrl,messages: [- { role: 'system', content: GOAL_FIDELITY_PRODUCTION_SYSTEM_PROMPT },+ { role: 'system', content: auditHandlerEvidence ? GOAL_FIDELITY_AUDIT_SYSTEM_PROMPT : GOAL_FIDELITY_PRODUCTION_SYSTEM_PROMPT },{ role: 'user', content: auditContent },],})@@ -1418,6 +1432,13 @@ export async function runLocalGoalFidelityReview({ backend, model, objective, diconst parsedAudit = extractJson(audit.content, { shapePredicate: value => value !== null && typeof value === 'object' && !Array.isArray(value) }).valueconst auditedVerdict = normalizeGoalFidelityVerdict(parsedAudit)if (!auditedVerdict) return { ok: false, backend, model: result.model, error: 'No usable production evidence verdict.' }+ if (auditHandlerEvidence) {+ const findings = backfillAccountingFindings(trimmedObjective, trimmedDiff)+ if (findings.length) {+ auditedVerdict.verdict = auditedVerdict.verdict === 'rethink' ? 'rethink' : 'fix-first'+ auditedVerdict.missing = [...new Set([...findings, ...auditedVerdict.missing])]+ }+ }if (auditedVerdict.verdict !== 'ship') {// This narrow check has no tests or scope diff. Preserve the primary// verification/scope assessment and add only its production finding.diff --git a/server/services/codeReview.test.js b/server/services/codeReview.test.jsindex ea5264da89..07dce4508c 100644--- a/server/services/codeReview.test.js+++ b/server/services/codeReview.test.js@@ -84,6 +84,18 @@ const observabilityCases = [['actual rejection still on stdout', `${observabilityCase.objective}\nThe selected finding is rejected HTTP push failures logged to stdout; move them to stderr.`, observabilityCase.diff.replace('+ else console.error(', '+ else console.log('), 'fix-first'],['explicit removal of every console.log still required', `${observabilityCase.objective}\nAlso remove every console.log use from peerSyncPush.js, including successful recovery diagnostics.`, observabilityCase.diff, 'fix-first'],]+// Public production + test diff from merged PR #11400. The overturned review+// read the deleted silent catch as surviving despite the added failure handler.+const backfillObservabilityCase = JSON.parse(readFileSync(new URL('../test/fixtures/goal-fidelity-backfill-observability.json', import.meta.url), 'utf8'))+const backfillRequirement = 'Audit task-learning observability and implement ONE finding: repair the `catch` in backfillFromHistory silently swallowing recordTaskCompletion failures and counting them as successes. Log each failure with agent/task IDs, exclude failed records from the success total, and emit a warning summary with the failure count.'+const backfillObservabilityCases = [+ ['removed silent catch replaced by contextual failure reporting', backfillObservabilityCase.objective, backfillObservabilityCase.diff, 'ship'],+ ['complete repair with explicit accounting and warning requirements', backfillRequirement, backfillObservabilityCase.diff, 'ship'],+ ['regression test without production repair', backfillRequirement, backfillObservabilityCase.diff.split(/(?=^diff --git )/m).find(part => part.startsWith('diff --git a/server/services/taskLearning/lifecycle.test.js')), 'rethink'],+ ['failure details discarded despite positive regression assertions', backfillRequirement, backfillObservabilityCase.diff.replace(/^\+ console\.error\(.*$/m, '+ void err;'), 'fix-first'],+ ['failed records still counted despite positive regression assertions', backfillRequirement, backfillObservabilityCase.diff.replace('+ if (recorded) backfilled++;', '+ backfilled++;'), 'fix-first'],+ ['failure summary still reports info despite positive regression assertions', backfillRequirement, backfillObservabilityCase.diff.replace("+ emitLog(failed ? 'warn' : 'info',", "+ emitLog('info',"), 'fix-first'],+]const processManifestCase = JSON.parse(readFileSync(new URL('../test/fixtures/goal-fidelity-process-manifest.json', import.meta.url), 'utf8'))const manifestDiffParts = processManifestCase.diff.split(/(?=^diff --git )/m).filter(Boolean)const manifestCases = [@@ -1445,6 +1457,72 @@ describe('codeReview helpers', () => {.toMatchObject({ ok: true, verdict: 'rethink' })})+ it('sends the surviving backfill catch with its after-change handler and result checks', async () => {+ global.fetch = vi.fn(async (_url, init) => {+ const { messages } = JSON.parse(init.body)+ if (messages[1].content.includes('AFTER-CHANGE PRODUCTION HUNKS only')) {+ const source = messages[1].content.match(/DIFF[^\n]*\n(`+)diff\n([\s\S]*)\n\1$/)[2]+ expect(source).toContain('+++ b/server/services/taskLearning/lifecycle.js')+ expect(source).toContain('@@ -105,11 +106,19 @@')+ expect(source.match(/^@@ /gm)).toHaveLength(2)+ expect(source).toContain('let failed = 0;')+ expect(source).toContain('failed++;')+ expect(source).toContain('console.error(`❌ 📚 TaskLearning: Backfill failed to record agent ${agent.id} (task ${agent.taskId}): ${err.message}`)')+ expect(source).toContain('return false;')+ expect(source).toContain('if (recorded) backfilled++;')+ expect(source).toContain("emitLog(failed ? 'warn' : 'info'")+ expect(source).toContain('{ backfilled, failed }')+ expect(messages[1].content).not.toContain('.catch(() => {})')+ expect(messages[1].content).not.toContain('lifecycle.test.js')+ expect(messages[1].content).not.toContain('toHaveBeenCalledWith')+ expect(messages[1].content).toContain('separate hunks are not contiguous source')+ expect(messages[0].content).toContain('Search categories are candidate-discovery guidance')+ expect(messages[0].content).toContain('Check the summary logger\'s level argument')+ expect(messages[0].content).toContain('Positive assertions and comments cannot supply either missing condition')+ } else {+ expect(messages[1].content).toContain(backfillObservabilityCase.diff.trim())+ }+ return completion(shipVerdict)+ })+ // Transport/context regression only; actual judgements use the opt-in+ // local model below, including test-only and broken-production controls.+ expect(await runLocalGoalFidelityReview({ backend: 'ollama', model: 'example-model', ...backfillObservabilityCase }))+ .toMatchObject({ ok: true, verdict: 'ship', missing: [], unrequested: [] })+ expect(global.fetch).toHaveBeenCalledTimes(2)+ })++ it.each([+ backfillObservabilityCases[1],+ backfillObservabilityCases[4],+ backfillObservabilityCases[5],+ ])('never lets model approval waive explicit backfill accounting: %s', async (_name, objective, diff, expected) => {+ global.fetch = vi.fn().mockResolvedValue(completion(shipVerdict))+ const result = await runLocalGoalFidelityReview({ backend: 'ollama', model: 'example-model', objective, diff })+ expect(result).toMatchObject({ ok: true, verdict: expected })+ if (expected === 'ship') expect(result.missing).toEqual([])+ else expect(result.missing.join(' ')).toMatch(/backfillFromHistory.*(?:count|summary)/)+ expect(global.fetch).toHaveBeenCalledTimes(2)+ })++ // Actual judgement pin: explicitly opt-in so ordinary CI makes zero provider calls.+ it.runIf(Boolean(process.env.GOAL_FIDELITY_EVAL_MODEL)).each(backfillObservabilityCases)('judges the backfill catch replacement with a real local model: %s', async (_name, objective, diff, expected) => {+ global.fetch = nativeFetch+ const result = await runLocalGoalFidelityReview({+ backend: 'ollama', model: process.env.GOAL_FIDELITY_EVAL_MODEL,+ objective: taskObjective({ description: objective }), diff, timeoutMs: 180_000,+ })+ console.log(`🔍 Backfill evaluation ${_name}: ${JSON.stringify(result)}`)+ expect(result.ok, result.error).toBe(true)+ if (expected === 'ship') expect(result, JSON.stringify(result)).toMatchObject({ verdict: 'ship', missing: [], unrequested: [] })+ else {+ expect(['fix-first', 'rethink'], JSON.stringify(result)).toContain(result.verdict)+ expect(expected === 'rethink' ? result.missing.length + result.unrequested.length : result.missing.length, JSON.stringify(result)).toBeGreaterThan(0)+ }+ if (expected === 'rethink') expect(result.verdict).toBe('rethink')+ if (_name.startsWith('failed records still counted')) expect(result.missing.join(' ')).toMatch(/count|success/i)+ if (_name.startsWith('failure summary still reports info')) expect(result.missing.join(' ')).toMatch(/warn|summary|level/i)+ }, 190_000)+// Actual judgement pin: explicitly opt-in so ordinary CI makes zero provider calls.it.runIf(Boolean(process.env.GOAL_FIDELITY_EVAL_MODEL)).each(observabilityCases)('judges the one-finding observability audit with a real local model: %s', async (_name, objective, diff, expected) => {global.fetch = nativeFetchdiff --git a/server/test/fixtures/goal-fidelity-backfill-observability.json b/server/test/fixtures/goal-fidelity-backfill-observability.jsonnew file mode 100644index 0000000000..469d39a6b1--- /dev/null+++ b/server/test/fixtures/goal-fidelity-backfill-observability.json@@ -0,0 +1,5 @@+{+ "source": "Public merged PR #11400, base 53d2074eb3, reviewed head 84050218d09cdabc3769489e4c57e61bb243bf86. Reconstructed one-finding audit objective; no live instance data.",+ "objective": "[Improvement: PortOS] Logging and observability audit\n\nAudit how PortOS logs events, reports runtime failures, and surfaces\ndiagnostics.\n\nRepository: <repo-root>\n\n## Mode: implement the highest-value fix\n\nThis banner OVERRIDES any later instruction to file issues, leave source unchanged, or skip commits. Pick ONE coherent, high-value finding from the mission below and implement it this run. Do not boil the ocean.\n\n1. **Scan broadly, rank candidates, then pick a bounded slice** containing the highest-impact actionable finding. Follow the discovery contract below before choosing.\n2. **Read the actual code** and the project's `AGENTS.md` / conventions. Honor documented non-issues.\n3. **Implement the fix** — the smallest change that actually solves the concrete problem. If the only obstacle was a design choice, make the call and state it.\n4. **Verify** with the project's tests (or a focused new test when the path is untested and a silent break would cost data, money, or quota).\n5. **Commit** following the repo's conventions. Do not bundle unrelated cleanup.\n\nDo not create tracker issues in fix mode, including for deferred findings. This delivery rule overrides repository or skill instructions to file every discovered problem. Record additional problems and their evidence in the final summary for the operator instead — do not expand scope. If nothing in the slice is worth changing, say so and stop without a drive-by refactor.\n\n## Repository-wide discovery, worst offender first, and quality assessment\n\nThis contract overrides narrower slice-selection or ranking advice in the mission, including customized prompts. Preserve the selected file-issues/fix mode and documented non-issues.\n1. Inventory all first-party source roots with git ls-files or rg --files, excluding vendored/generated/build/dependency output. Do a cheap repository-wide signal scan BEFORE choosing where to investigate. Do not deep-read the entire repository.\n2. Category search: Inventory critical workflows and their failure/status signals; rank invisible data loss, silent failures and operations that cannot be diagnosed.\n3. Rank at least the top five candidates (or all if fewer), with paths, measured signals, impact, reach, confidence and why higher candidates win. Validate the top candidates through callers, tests and relevant history before filing or fixing. Severity and user impact outrank ease, small scope and recency. Churn and fan-in break ties; inactivity alone never dismisses a severe offender. If the raw worst is exempt, already filed or not actionable, report why and continue down the ranking. Existing unresolved issues still count against quality. Never pick a nit while a verified major problem remains actionable.\n4. Keep investigation bounded after this scan. Report scanned roots, tools/commands, candidate ranking, exclusions, reviewed paths, unreviewed inventory and stopping reason. If tools or budget prevent broad discovery, report partial coverage, never claim a global worst or a clean repository.\n5. Assess the PRE-FIX codebase for this category on 0–100 (higher is healthier): 90–100 no material defect found after broad evidence; 70–89 localized moderate debt; 40–69 significant recurring or widespread problems; 10–39 severe/core-workflow defects; 0–9 pervasive critical failure. Score the category, not the agent's performance or issue count. Explain the score using severity, prevalence and concrete evidence. Do not award points merely for filing or fixing this run. Use score:null for unavailable or inapplicable assessment; zero is a real score.\n6. Score the worst finding's severity separately on 1–10: 1–3 minor localized impact, 4–6 material recurring cost, 7–8 major workflow/reliability impact, 9–10 critical data/security/availability failure. Use 0 only when no material finding was verified. List this severity for every finding in the narrative.\n7. Include exactly one single-line QUALITY_AUDIT_JSON: {...} in your completion sentinel summary AND final response, with no secrets or personal data. Replace the example values with observed evidence. The category is fixed to this task; do not rate other categories. Keep ordinary completion/PR instructions and summaries too.\nQUALITY_AUDIT_JSON: {\"version\":1,\"category\":\"observability\",\"score\":null,\"worstSeverity\":0,\"coverage\":\"unavailable\",\"confidence\":\"low\",\"summary\":\"Explain the assessment and strongest evidence\",\"scannedFiles\":0,\"totalFiles\":0}\nAllowed coverage: broad, partial, unavailable, not-applicable. Allowed confidence: low, medium, high. scannedFiles counts files actually included in the category signal scan; totalFiles is the eligible inventory. Broad requires the whole inventory to be scanned (deep review remains bounded). Partial assessments are excluded from the overall score. For partial coverage, use score:null when the inspected evidence cannot support a category score; explain why rather than inventing one. A numeric partial score is provisional. If unavailable or not-applicable, score must be null and explain why. Even a zero-finding audit returns this assessment.\n\nHunt specifically for:\n\n- **Silent failure swallowing** — a `catch` that discards the error with no\n log and no telemetry, so the failure is invisible in production.\n- **Log noise** — a polling loop, socket frame handler, or hot render path\n emitting a line every tick, burying the errors that matter.\n- **Missing error context** — an error logged with only `err.message`, with no\n task id, record id, route, or stack to reproduce from.\n- **Wrong level** — a fatal runtime error logged through the equivalent of\n `console.log` instead of the project's error path.\n- **Uninstrumented workflows** — a multi-step background pipeline or agent\n transition with no progress logging, so a stuck job looks identical to a slow\n one.\n- **Lost correlation** — a request, job, or message id that is not carried\n across a service, queue, or thread boundary, so one failure's log lines cannot\n be joined together.\n- **Nothing to alert on** — a critical path (request latency and error rate,\n queue depth or consumer lag, job success and duration, data freshness) with no\n metric or countable signal, so a degradation is only discovered by a user.\n- **Health signals that do not reflect health** — a status or health endpoint\n that reports OK while a dependency the service needs is down, or that never\n reports degraded states.\n- **No trail for consequential actions** — destructive, administrative, or\n permission-changing operations with no record of who did what and when,\n where the domain needs one.\n\nMatch the project's scale: a single-process tool needs good logs and a truthful\nstatus view, not a tracing stack. For each finding, name the catch block,\nuninstrumented step, or missing signal and state the operational blind spot it\ncreates: what breaks, and how long before anyone notices.",+ "diff": "diff --git a/server/services/taskLearning/lifecycle.js b/server/services/taskLearning/lifecycle.js\nindex 98640e3d6a..ca76a9375b 100644\n--- a/server/services/taskLearning/lifecycle.js\n+++ b/server/services/taskLearning/lifecycle.js\n@@ -92,6 +92,7 @@ export async function backfillFromHistory() {\n const agents = await getAgents();\n \n let backfilled = 0;\n+ let failed = 0;\n for (const agent of agents) {\n // Same rule as the live listener above, and the reason it is repeated rather\n // than assumed: the backfill re-reads the ARCHIVE, where every relaunch the\n@@ -105,11 +106,19 @@ export async function backfillFromHistory() {\n metadata: agent.metadata\n };\n \n- await recordTaskCompletion(withoutStaleCoordinatorVerdict(agent, task), task).catch(() => {});\n- backfilled++;\n+ // Count only records that actually landed, and name the failing agent/task\n+ // so a partial backfill is diagnosable instead of reporting a full count.\n+ const recorded = await recordTaskCompletion(withoutStaleCoordinatorVerdict(agent, task), task)\n+ .then(result => result !== null) // null = deliberately skipped, nothing written\n+ .catch(err => {\n+ failed++;\n+ console.error(`❌ 📚 TaskLearning: Backfill failed to record agent ${agent.id} (task ${agent.taskId}): ${err.message}`);\n+ return false;\n+ });\n+ if (recorded) backfilled++;\n }\n }\n \n- emitLog('info', `Backfilled ${backfilled} completed tasks into learning system`, { backfilled }, '📚 TaskLearning');\n+ emitLog(failed ? 'warn' : 'info', `Backfilled ${backfilled} completed tasks into learning system${failed ? ` (${failed} failed)` : ''}`, { backfilled, failed }, '📚 TaskLearning');\n return backfilled;\n }\ndiff --git a/server/services/taskLearning/lifecycle.test.js b/server/services/taskLearning/lifecycle.test.js\nindex 2992cda4fe..81519d47fd 100644\n--- a/server/services/taskLearning/lifecycle.test.js\n+++ b/server/services/taskLearning/lifecycle.test.js\n@@ -168,3 +168,17 @@ describe('agent:completed listener — a resumed record is a continuation, not a\n expect(observeAgentChurn).not.toHaveBeenCalled();\n });\n });\n+\n+describe('backfillFromHistory — failed records are reported, not counted', () => {\n+ beforeEach(() => vi.clearAllMocks());\n+\n+ it('logs the failing agent and returns only the records that landed', async () => {\n+ agentsStore.list = [completed('ok-task', 'ux', true), { ...completed('bad-task', 'ux', true), id: 'agent-bad' }];\n+ recordTaskCompletion.mockImplementationOnce(async () => {}).mockImplementationOnce(async () => { throw new Error('disk full'); });\n+ const errorSpy = vi.spyOn(console, 'error').mockImplementation(() => {});\n+\n+ expect(await backfillFromHistory()).toBe(1);\n+ expect(errorSpy).toHaveBeenCalledWith(expect.stringContaining('agent-bad (task bad-task): disk full'));\n+ errorSpy.mockRestore();\n+ });\n+});\n"+}