Skip to content

Self-hosted, Self-Improving, Federated Everything App — dev management, agent orchestration, persistent mind, digital twin, second brain, and creative pipeline from a single dashboard over Tailscale to port the user into the OS

fix: goal-fidelity rubric credits prompt-text behavior fixes plus version bump for agent-stall diagnosis objectives

[._.]/ Adam Eivya854196557d5
diff --git a/server/services/codeReview.js b/server/services/codeReview.js
index f196c2c0a6..7800a8655b 100644
--- a/server/services/codeReview.js
+++ b/server/services/codeReview.js
@@ -723,6 +723,8 @@ Judge the production behavior before treating a regression test as the behavior
For an objective that asks to move or reorder an existing UI section, a diff may show the move as a complete added block and a complete deleted block. When the same complete named section, with materially unchanged contents, is added at the requested position and its former placement is removed, judge the pair as one relocation that satisfies the requested order. Do not call the insertion an unrequested addition or the removal a missing section. This rule applies only when the diff establishes that matched move: an added duplicate whose old copy remains, a different section, or independent content changes must still be judged on their own; matching headings alone do not prove a relocation.
+For an objective to diagnose why an autonomous agent run stalled or failed and fix it, where the agent behavior is defined by a task prompt template (for example a \`taskPromptDefaults/prompts.js\` entry with a \`PROMPT_VERSIONS\` bump in \`versions.js\`), the prompt text IS the production behavior: an added prompt section telling the agent to stay with the work until the stated completion condition, plus the version bump, delivers a behavior fix without any runtime-code change, and a run id or commit cited in the version note is the diagnosis evidence. Do not list missing runtime code merely because the fix lives in prompt text. A separate hunk repairing a CI failure that blocks that same delivery (for example renaming an unused export a dead-exports check rejects) is part of the objective, not unrequested. This applies only when the diff shows the prompt-section addition and the version bump; a prompt edit with no version bump, or a diff with neither prompt text nor a CI repair, remains incomplete.
+
For an objective to add buttons or links that send an item to other pages (for example a card action that opens an image or video generator), judge the rendered UI, not only the helper that builds the destinations. A helper that returns route strings (such as a sendLinks object) is supporting code, not the buttons. A control can be rendered data-driven: a JSX array of [key, Icon, label] tuples mapped to one <Link to={links[key]}> per entry, so no separate literal element exists per button. When the diff ADDS such a mapped array or per-entry Link/button whose keys match the helper's returned keys and whose targets are the requested destinations, count each requested destination as delivered; do not report the buttons missing because the JSX has no hand-written element per destination. This exempts only a rendering path the diff actually adds: a helper with no added JSX consumer, a tuple key the helper never returns, or a requested destination absent from both still counts as missing.
For an objective to add a newly launched AI model to the provider catalog, presets, pricing, or model-comparison seed, judge the sourced facts the diff supplies: the model id in the provider/preset lists, its published prices, and a migration for existing installs when the diff changes shipped provider data. A model-comparison row whose quality, latency, or throughput fields are null (or whose benchmark scores are absent) is NOT missing work when the diff or commit states the vendor has not published those figures; recording an unpublished benchmark would be fabrication. Count a null benchmark as missing only when the objective explicitly asks for benchmarks and the diff offers no statement that none are published. This exempts only unpublished benchmark metrics for a new model; a missing model id, price, preset entry, or required migration remains missing.
diff --git a/server/services/codeReview.test.js b/server/services/codeReview.test.js
index 822efc765d..57bae62e97 100644
--- a/server/services/codeReview.test.js
+++ b/server/services/codeReview.test.js
@@ -1427,6 +1427,33 @@ describe('codeReview helpers', () => {
expect(helperOnly).toMatchObject({ ok: true, verdict: 'fix-first', missing: ['rendered send buttons'] })
})
+ it('credits a prompt-text behavior fix plus its version bump and CI repair for an agent-stall diagnosis objective', async () => {
+ const objective = 'Figure out what was blocking agent-example from completing the release this run and fix it'
+ const promptDiff = [
+ 'diff --git a/server/services/taskPromptDefaults/prompts.js b/server/services/taskPromptDefaults/prompts.js',
+ '+## Stay with the release PR',
+ '+Do not stop while CI is pending; wait until the PR is MERGED.',
+ 'diff --git a/server/services/taskPromptDefaults/versions.js b/server/services/taskPromptDefaults/versions.js',
+ "+ 'do-release': 18, // v18: agent-example stopped at CI pending",
+ 'diff --git a/server/services/prWatcher.js b/server/services/prWatcher.js',
+ '-export function unusedHelper() {}',
+ '+function unusedHelper() {}',
+ ].join('\n')
+ global.fetch = vi.fn(async (_url, init) => {
+ const request = JSON.parse(init.body)
+ const creditsPromptFix = request.messages[0].content.includes('the prompt text IS the production behavior')
+ && request.messages[0].content.includes('a prompt edit with no version bump')
+ const hasBump = request.messages[1].content.includes("'do-release': 18")
+ return mockJsonResponse({ choices: [{ message: { content: JSON.stringify(creditsPromptFix && hasBump
+ ? { verdict: 'ship', missing: [], unrequested: [], evidence: 'Prompt section and version bump deliver the behavior fix.' }
+ : { verdict: 'fix-first', missing: ['runtime code'], unrequested: ['prWatcher change'], evidence: 'No runtime change.' }) } }] })
+ })
+ const full = await runLocalGoalFidelityReview({ backend: 'ollama', model: 'example-model', objective, diff: promptDiff })
+ const noBump = await runLocalGoalFidelityReview({ backend: 'ollama', model: 'example-model', objective, diff: promptDiff.replace("'do-release': 18", "'do-release': 17") })
+ expect(full).toMatchObject({ ok: true, verdict: 'ship', missing: [], unrequested: [] })
+ expect(noBump).toMatchObject({ ok: true, verdict: 'fix-first' })
+ })
+
// Transport/contract regression; the opt-in real-model test below checks judgement.
it.each(manifestCases)('preserves the bounded manifest calibration and verdict: %s', async (_name, diff, verdict) => {
global.fetch = vi.fn().mockImplementationOnce(async (_url, init) => {