Skip to content

The ultimate harness for coding agents and beyond. All your agents. All your machines. One command center. Start with code, then follow your curiosity and build across disciplines: CAD, circuits, robots, games and music.

Jev Sheets: keep practice and adopted questions in the project (#218)

Deed1c3e0f8bc28
diff --git a/store/agents/jev-sheets/AGENTS.md b/store/agents/jev-sheets/AGENTS.md
index b28922e3b..e4efe24d9 100644
--- a/store/agents/jev-sheets/AGENTS.md
+++ b/store/agents/jev-sheets/AGENTS.md
@@ -47,16 +47,18 @@ second server.
pass. A selected sheet row can be pinned into it. The person can inspect the paired answers,
mark a preference and keep their evidence in `.harness/question-trials/<id>/`. Read its
`review.md`, `trial.json` and `column.json` when continuing their work. After they choose a
- wording, retain it in `sheet.json`, save and read again. Only the changed column is asked again.
+ wording, **Use on whole sheet** retains it in `sheet.json`; read the updated file before editing.
+ Only the changed column is asked again.
Question Lab deliberately samples hard rows and/or a spread; it does not estimate population
accuracy. Its original and candidate are asked again together, so the original answer can differ
from an older cached sheet cell. Human preferences are separate from model probabilities and are
not ground-truth labels. Inspect each row's provider/model in `trial.json`; never write findings
from an offline stand-in or an incomplete trial. A changed answer or higher confidence does not
-establish better wording. **Try on whole sheet** adds a separate runtime column; it does not edit
-`sheet.json`. Retain an accepted header there, keeping the person's source file and context. Kept
-packets include sampled text and metadata, remain readable after source deletion and are immutable.
+establish better wording. **Use on whole sheet** saves a separate question in
+`sheet.json`, keeping the original, source file and context. Read that file before editing; a
+`questionTrial` field records the kept trial that supplied a question and prevents duplicate adoption.
+Kept packets include sampled text and metadata, remain readable after source deletion and are immutable.
Use a new trial for a new evaluation or review. Do not edit the generated packet or its answers.
If the questions could not be answered (no key, the viewer is not running, errors), say so plainly.
@@ -159,8 +161,9 @@ folder in the workspace, so the person's folder stays tidy.
- **`rows`**: only for a made-up sheet you write yourself (1 to 10,000, each with a non-empty `text`,
optional `id`, plain fields, `group`, and `truth` labels that are never sent to Jev).
-What a person does in the pane (typed columns, row edits, sort, filters, the review line) is not
-saved to `sheet.json`. If they like a column they typed, add it to `columns` for them.
+Question Lab's **Use on whole sheet** saves its chosen question to `sheet.json`. Other pane changes
+(typed columns, row edits, sort, filters, the review line) are not saved to `sheet.json`. If they like
+a column they typed, add it to `columns` for them.
## The header grammar
@@ -231,12 +234,26 @@ a full pass takes most of a minute.
- Their data stays in the workspace. The rows go to the Jev API to be answered and nowhere
else. Never send it anywhere else, and never copy more of it into the chat than you need.
+## Offline practice and missing credit
+
+If the person requests offline practice, set `"offline": true` in **sheet.json** alongside the
+fictional rows and questions. The existing pane reloads it and uses the word-matching stand-in for
+both sheet cells and Question Lab, even when a live key is saved. Verify `sheet.offline: true` and
+`sheet.client: "mock"` in the verdict. Create the requested sheet in the pane; do not substitute a
+scratch script or a table in chat for the interactive workflow. Do not change credentials, edit the
+installed package, or start a second server. Practice results are not live model evidence.
+
+A provider error is a failed fill, not a completed sheet. Read the error and explain it. Offer
+offline practice when credit is unavailable; never silently switch real analysis to a stand-in.
+Only set `"offline": false` when the person asks to resume live use. Existing cells are recomputed
+on a mode change. Requests already in flight can finish, but their answers do not enter the new sheet.
+
## Without a key
-`"client": "mock"` in the verdict means there is no Jev key, and an offline stand-in that only
-matches words is answering. On a person's own data its answers are not good enough to act on, and
-the pane says so in a yellow bar. Do not write findings from them. Tell the person to paste a key
-into the **Jev · live mind** panel in the pane (an OpenRouter key from `openrouter.ai/keys` takes
+`"client": "mock"` in the verdict means offline practice is active or there is no Jev key. A
+word-matching stand-in is answering. Its answers are not good enough to act on, and the pane labels
+them as practice. Do not write findings from them. For requested real analysis without a key, tell
+the person to paste a key into the **Jev · live mind** panel in the pane (an OpenRouter key from `openrouter.ai/keys` takes
about a minute). The key is saved on their machine in `~/.config/typesafe/credentials`, and every
cell is then asked again. Never ask them to paste a key into the chat.
@@ -253,7 +270,8 @@ cell is then asked again. Never ask them to paste a key into the chat.
- The person's own data is in the sheet (or they chose the sample, knowing it is made up).
- `sheet.json` passes `toolchain/check.mjs`. Every choice option has a meaning. `context` is set.
-- The verdict shows your save was loaded, a live `client`, and every cell filled.
+- The verdict shows your save was loaded and every cell filled, with a live `client` for real analysis
+ or explicit offline practice when requested. A practice run does not meet the live-analysis checks.
- You read `answers.csv`, opened the rows behind the main numbers, and wrote `findings.md`.
- You told the person, in plain words: the three findings that matter, how sure Jev was, and the
next question worth asking.
diff --git a/store/agents/jev-sheets/README.md b/store/agents/jev-sheets/README.md
index 492e88f58..31e432c6c 100644
--- a/store/agents/jev-sheets/README.md
+++ b/store/agents/jev-sheets/README.md
@@ -33,6 +33,12 @@ in your project folder. Its rows go to the Jev API to be answered, and nowhere e
The pane opens on a small made-up sample so there is something to try in the first ten seconds. It
is labelled as made up, and one click takes you back to it.
+For an offline workshop, ask the agent to create an **offline practice** sheet. It saves
+`"offline": true` in `sheet.json`; cells and Question Lab use the word-matching stand-in even if
+a provider key is saved. The pane labels the practice mode, and it survives reopening the project.
+Ask the agent to go live when ready. Changing modes recomputes answers; practice answers and kept
+trials are never evidence of live model quality. Provider failures show **Answers unavailable**.
+
## Ask
| You type | It becomes |
@@ -80,11 +86,15 @@ frozen rows, exact wire questions, raw answers, per-call provider/model/usage, y
The packet includes your sampled data; keep it where you would keep the original file. No API
keys are included. Reopen it from **Kept & open trials**, even after the source file is gone.
-**Try on whole sheet** adds the candidate as a separate column, preserving the original and reusing
-the trial's answers when the connected route and requested model still match. It refuses a trial
-whose rows, context or original question changed. This follows the sheet's existing pane-only
-column behavior: ask the agent to retain your chosen header in `sheet.json` across viewer restarts.
-Keeping a trial itself is durable. Open trials last only for the current viewer process.
+**Use on whole sheet** saves the candidate as a separate question in `sheet.json`, preserving the
+original question and your data. It reuses trial answers when the connected route and requested
+model still match, then fills the remaining rows. The chosen wording survives reset and viewer
+restart; repeated clicks or reopening the kept trial do not add duplicate questions. New viewer
+sessions ask the saved questions again rather than treating old answers as current.
+
+If the rows, context or original question changed, start a new trial. A broken or conflicting
+project edit leaves the current sheet and kept trial available for retry. Open trials last only
+for the current viewer process; kept trials remain in the project.
One trial runs at a time, at most four row requests in flight. Cancellation stops further rows;
requests already in flight can finish, including the client's normal transient-error retries.
diff --git a/store/agents/jev-sheets/skills/sheets/SKILL.md b/store/agents/jev-sheets/skills/sheets/SKILL.md
index 0f206f122..70351476c 100644
--- a/store/agents/jev-sheets/skills/sheets/SKILL.md
+++ b/store/agents/jev-sheets/skills/sheets/SKILL.md
@@ -5,6 +5,12 @@ row. The person brings a file they could never read in full. The craft is wordin
answers are right and the confidence is honest, then turning the answers into findings a person can
act on. A made-up sheet is only the fallback when they have no file at hand.
+For requested offline practice, write the fictional rows and `"offline": true` into `sheet.json`.
+The existing pane and Question Lab then use their stand-in even with a saved key. Check the verdict
+for `sheet.offline: true` and `sheet.client: "mock"`. Keep the exercise in the pane; do not replace it
+with an external script or chat-only answer table. Preserve credentials and the installed package.
+Practice comparisons are not model-quality evidence. Switch back only when the person asks to go live.
+
## How one row is judged
For each row the viewer makes ONE call. The state is the row (its `text` plus plain fields like
@@ -72,9 +78,11 @@ rows, questions, probabilities and per-call provider/model in `trial.json`; `col
the tested header. Keep human preferences separate from truth labels. This deliberate sample
cannot estimate whole-sheet accuracy, and greater confidence does not establish correctness.
`mock` answers are only a practice flow. Failed or cancelled trials retain their status and are
-not complete comparisons. After the person chooses a wording, add it to the original
-`sheet.json`; the pane's **Try on whole sheet** is a temporary extra column, like other pane-added
-columns. Do not replace their source file with the packet's `sheet.json`: that file contains only
+not complete comparisons. After the person chooses a wording, **Use on whole sheet** saves a
+separate question in the original
+`sheet.json`. Read that file before editing so you retain the chosen wording and its `questionTrial`
+identifier. Other pane-added columns remain temporary. Do not replace their source file with the
+packet's `sheet.json`: that file contains only
the frozen sample and belongs in a separate workspace if they want to rerun it.
Keep their data in the workspace. Quote only what the report needs.
diff --git a/store/agents/jev-sheets/test/PLAYTEST-2026-09-22.md b/store/agents/jev-sheets/test/PLAYTEST-2026-09-22.md
new file mode 100644
index 000000000..581466e67
--- /dev/null
+++ b/store/agents/jev-sheets/test/PLAYTEST-2026-09-22.md
@@ -0,0 +1,36 @@
+# Jev Sheets user playtest — 2026-09-22
+
+Draft PR: [#218](https://github.com/autonomous-ai/openharness/pull/218). No merge.
+Branch: `codex/jev-sheets-kept-questions`.
+
+## Completed
+
+- Created a fresh project through the native Harness Store and submitted an original twelve-ticket
+ offline workshop prompt to its embedded coding agent. The baseline exposed an unwanted live
+ provider call, HTTP 402 and misleading completion state. These are fixed on this branch.
+- Used the actual generated sheet in the patched viewer in Chrome: compare urgency wording,
+ mark a row preference, write a note, keep the trial, adopt the wording, restart the viewer and
+ reopen. Both questions, the note, offline mode and **Saved in project** persisted.
+- Installed this branch and created another native project, `jev-sheets-2026-09-22-13-20`
+ (tab **Hosting support tickets practice sheet**). Its original prompt produced twelve fictional
+ tickets, `offline:true`, two questions, 24/24 filled cells and `client:mock`. The exercise stayed
+ in the pane; no script or chat substitute was authored.
+- All 70 Node tests, package conformance, doctor and syntax checks passed. Tests cover persistent
+ adoption, stale input, idempotence, offline/live transitions and a loopback HTTP 402 provider.
+
+## Resume here
+
+The native rerun's interactive Question Lab review, adoption, follow-up agent revision and
+reopen are unfinished. Use the existing project above. Compare
+`Production is down or customers cannot complete checkout?`, add a preference and review note,
+keep and adopt it, then ask the agent to continue from the saved review without changing the
+original data or question. Reopen and verify both questions and the review packet survive.
+The test operator should own preview controls; tell the embedded agent to edit files and inspect
+artifacts without controlling the desktop or browser.
+
+The completed Chrome packet is `6987de23-0862-45c4-b187-c8b1daa1b241` in the local
+`jev-app-practice-jkuvp4w5` workspace. Scratch baselines and exact hashes live in this branch's
+`.scratch/native-rerun-baseline/`. These are local resume aids, not committed fixtures.
+
+The configured live provider was out of credit. Offline results verify interaction and persistence,
+not Jev answer quality. Do not describe these checks as a complete native acceptance pass.
diff --git a/store/agents/jev-sheets/test/kept-column.test.mjs b/store/agents/jev-sheets/test/kept-column.test.mjs
new file mode 100644
index 000000000..c957531c3
--- /dev/null
+++ b/store/agents/jev-sheets/test/kept-column.test.mjs
@@ -0,0 +1,85 @@
+import { test } from 'node:test'
+import assert from 'node:assert/strict'
+import { mkdtempSync, readFileSync, writeFileSync, rmSync, symlinkSync, readdirSync } from 'node:fs'
+import { join } from 'node:path'
+import { tmpdir } from 'node:os'
+import { keepTrialColumn } from '../viewer/kept-column.mjs'
+import { normalizeSheet, parseHeader } from '../viewer/grammar.mjs'
+import { trialHash } from '../viewer/question-lab.mjs'
+import { loadSource } from '../viewer/source.mjs'
+
+function fixture(t, overrides = {}) {
+ const workspace = mkdtempSync(join(tmpdir(), 'jev-kept-question-'))
+ t.after(() => rmSync(workspace, { recursive: true, force: true }))
+ const file = join(workspace, 'sheet.json')
+ const raw = { title: 'Fictional support', context: 'Support tickets', demo: false, columns: ['Urgent?'], rows: [{ id: 'r1', text: 'Please help now', channel: 'email' }], custom: { keep: 'this note' }, ...overrides }
+ if (raw.source) writeFileSync(join(workspace, raw.source), 'message,channel\nPlease help now,email\n')
+ writeFileSync(file, JSON.stringify(raw))
+ const source = raw.source ? loadSource(workspace, raw.source) : null
+ const sheet = normalizeSheet(source ? { ...raw, rows: source.rows } : raw)
+ const rows = sheet.rows.map((r, i) => ({ id: r.id, n: i + 1, text: r.text, meta: r.meta }))
+ const original = { ...sheet.columns[0], source: 'file' }
+ const trial = { id: '7bd60d19-b1aa-4fe1-abd0-742889aa2df2', original, candidate: parseHeader('Urgent: low < high').column,
+ datasetSha: trialHash({ context: sheet.context, source: source?.info.name || null, rows }) }
+ return { workspace, file, raw, trial, columns: [original], read: () => JSON.parse(readFileSync(file)) }
+}
+
+test('adoption preserves unrelated edits and data, disambiguates the original and is idempotent', (t) => {
+ const v = fixture(t)
+ const edited = { ...v.raw, title: 'Agent updated title', suggestions: ['Refund?'] }
+ writeFileSync(v.file, JSON.stringify(edited))
+ const result = keepTrialColumn(v.workspace, v.trial, v.columns)
+ assert.equal(result.id, 'urgent_v2')
+ const saved = v.read()
+ assert.deepEqual({ ...saved, columns: saved.columns.slice(0, 1) }, edited)
+ assert.equal(saved.columns[1].questionTrial, v.trial.id)
+ const bytes = readFileSync(v.file, 'utf8')
+ assert.equal(keepTrialColumn(v.workspace, v.trial, v.columns).alreadyApplied, true)
+ assert.equal(readFileSync(v.file, 'utf8'), bytes)
+ assert.deepEqual(readdirSync(v.workspace), ['sheet.json'])
+})
+
+for (const change of ['rows', 'context', 'original', 'invalid']) test(`an unobserved ${change} edit cannot be overwritten by adoption`, (t) => {
+ const v = fixture(t)
+ const raw = v.raw
+ if (change === 'rows') raw.rows[0].text = 'Different message'
+ if (change === 'context') raw.context = 'Different purpose'
+ if (change === 'original') raw.columns[0] = { id: 'urgent', header: 'Refund?' }
+ const bytes = change === 'invalid' ? '{unfinished' : JSON.stringify(raw)
+ writeFileSync(v.file, bytes)
+ assert.throws(() => keepTrialColumn(v.workspace, v.trial, v.columns))
+ assert.equal(readFileSync(v.file, 'utf8'), bytes)
+ assert.deepEqual(readdirSync(v.workspace), ['sheet.json'])
+})
+
+test('a changed imported file is detected before its watcher notification', (t) => {
+ const v = fixture(t, { source: 'tickets.csv', rows: [] })
+ writeFileSync(join(v.workspace, 'tickets.csv'), 'message,channel\nNew message,email\n')
+ const bytes = readFileSync(v.file, 'utf8')
+ assert.throws(() => keepTrialColumn(v.workspace, v.trial, v.columns), /rows or context changed/)
+ assert.equal(readFileSync(v.file, 'utf8'), bytes)
+})
+
+test('a question edited after adoption is not silently replaced or duplicated', (t) => {
+ const v = fixture(t)
+ keepTrialColumn(v.workspace, v.trial, v.columns)
+ const saved = v.read()
+ saved.columns[1].header = 'Needs a refund?'
+ writeFileSync(v.file, JSON.stringify(saved))
+ assert.throws(() => keepTrialColumn(v.workspace, v.trial, v.columns), /saved question was edited/)
+ assert.deepEqual(v.read(), saved)
+})
+
+test('full projects and linked sheet files are unchanged on failure', (t) => {
+ const v = fixture(t)
+ const full = { ...v.raw, columns: ['Urgent?', ...Array.from({ length: 11 }, (_, i) => `Question ${i}?`)] }
+ writeFileSync(v.file, JSON.stringify(full))
+ assert.throws(() => keepTrialColumn(v.workspace, v.trial, v.columns), /12 columns/)
+ assert.deepEqual(v.read(), full)
+ const target = join(v.workspace, 'original.json')
+ writeFileSync(target, JSON.stringify(v.raw))
+ rmSync(v.file)
+ symlinkSync(target, v.file)
+ assert.throws(() => keepTrialColumn(v.workspace, v.trial, v.columns), /regular sheet.json/)
+ assert.deepEqual(JSON.parse(readFileSync(target)), v.raw)
+})
diff --git a/store/agents/jev-sheets/test/practice.test.mjs b/store/agents/jev-sheets/test/practice.test.mjs
new file mode 100644
index 000000000..4ac890075
--- /dev/null
+++ b/store/agents/jev-sheets/test/practice.test.mjs
@@ -0,0 +1,151 @@
+import { test } from 'node:test'
+import assert from 'node:assert/strict'
+import { createServer } from 'node:http'
+import { mkdtempSync, readFileSync, writeFileSync, rmSync } from 'node:fs'
+import { tmpdir } from 'node:os'
+import { join } from 'node:path'
+import { startSheetsViewer } from '../viewer/viewer.mjs'
+
+const sleep = (ms) => new Promise((resolve) => setTimeout(resolve, ms))
+const sheet = {
+ title: 'Practice workshop', description: 'Fictional support messages', context: 'Hosting support',
+ demo: false, concurrency: 1, columns: [{ id: 'urgent', header: 'Urgent?' }],
+ rows: [{ id: 'a', text: 'Our production site is down.' }, { id: 'b', text: 'Please send a receipt, no rush.' }]
+}
+async function until(read, predicate) {
+ for (let i = 0; i < 100; i++) {
+ const value = await read()
+ if (predicate(value)) return value
+ await sleep(30)
+ }
+ assert.fail('Timed out waiting for practice state')
+}
+async function fixture(t, offline, status = 200) {
+ const workspace = mkdtempSync(join(tmpdir(), 'jev-practice-'))
+ const file = join(workspace, 'sheet.json')
+ const requests = []
+ const provider = createServer(async (req, res) => {
+ let body = ''
+ for await (const part of req) body += part
+ const input = JSON.parse(body)
+ requests.push(input)
+ res.writeHead(status, { 'content-type': 'application/json' })
+ res.end(JSON.stringify(status === 200 ? {
+ answers: Object.fromEntries(Object.keys(input.questions).map((id) => [id, { noul: 0.99 }])),
+ usage: { input_tokens: 1 }, model: 'local-test-model'
+ } : { error: 'fixture out of credit' }))
+ })
+ await new Promise((resolve) => provider.listen(0, '127.0.0.1', resolve))
+ const changes = {
+ JEV_OFFLINE: '0', JEV_LIVE_TESTS: '1', TYPESAFE_API_KEY: 'local-test-key',
+ TYPESAFE_API_URL: `http://127.0.0.1:${provider.address().port}`,
+ TYPESAFE_CREDENTIALS: join(workspace, 'no-credentials')
+ }
+ const prior = Object.fromEntries(Object.keys(changes).map((key) => [key, process.env[key]]))
+ Object.assign(process.env, changes)
+ const originalFetch = globalThis.fetch
+ globalThis.fetch = (input, ...options) => {
+ assert.equal(new URL(typeof input === 'string' ? input : input.url).hostname, '127.0.0.1', 'tests must never contact a real provider')
+ return originalFetch(input, ...options)
+ }
+ let viewer
+ t.after(async () => {
+ await viewer?.close()
+ provider.closeAllConnections()
+ await new Promise((resolve) => provider.close(resolve))
+ globalThis.fetch = originalFetch
+ for (const [key, value] of Object.entries(prior)) {
+ if (value === undefined) delete process.env[key]
+ else process.env[key] = value
+ }
+ rmSync(workspace, { recursive: true, force: true })
+ })
+ writeFileSync(file, JSON.stringify({ ...sheet, offline }))
+ viewer = await startSheetsViewer({ workspace, port: 0, paceMs: 0 })
+ const state = async () => (await fetch(viewer.url + '/state')).json()
+ const ctl = async (cmd, body = {}) => (await (await fetch(viewer.url + '/control', {
+ method: 'POST', body: JSON.stringify({ cmd, ...body })
+ })).json())
+ const verdict = async () => {
+ try { return JSON.parse(readFileSync(join(workspace, '.harness/verdict.json'), 'utf8')) } catch { return {} }
+ }
+ return { workspace, file, requests, state, ctl, viewer, verdict }
+}
+
+test('project practice fills cells and Question Lab offline despite a configured live provider; saved packets reopen offline', async (t) => {
+ const v = await fixture(t, true, 402)
+ const s = await until(v.state, (s) => s.stats.cellsFilled === 2)
+ assert.equal(s.client, 'mock')
+ assert.equal(s.offline, true)
+ assert.equal(v.requests.length, 0)
+ const token = s.questionLabToken
+ const preview = await v.ctl('labPreview', { token, column: 'urgent', count: 10 })
+ assert.equal(preview.draft.offline, true)
+ const start = await v.ctl('labStart', { token, draft: preview.draft.id, header: 'Production is down?' })
+ const trial = (await until(() => v.ctl('labGet', { token, id: start.trial.id }), (r) => r.trial.status === 'complete')).trial
+ assert.deepEqual(trial.summary.providers, ['mock'])
+ await v.ctl('labKeep', { token, id: trial.id })
+ const packet = JSON.parse(readFileSync(join(v.workspace, '.harness/question-trials', trial.id, 'sheet.json')))
+ assert.equal(packet.offline, true)
+ assert.equal(v.requests.length, 0)
+ const saved = await v.ctl('labApply', { token, id: trial.id })
+ assert.equal(saved.ok, true, saved.error)
+ assert.equal(JSON.parse(readFileSync(v.file)).offline, true)
+ const report = await until(v.verdict, (r) => r.ready)
+ assert.equal(report.sheet.offline, true)
+ assert.match(report.summary, /Offline practice/)
+})
+
+test('switching practice and live modes recomputes cells rather than reusing answers from the other route', async (t) => {
+ const v = await fixture(t, false)
+ await until(v.state, (s) => s.stats.cellsFilled === 2)
+ assert.equal(v.requests.length, 2)
+ writeFileSync(v.file, JSON.stringify({ ...sheet, offline: true }))
+ let s = await until(v.state, (s) => s.offline && s.stats.cellsFilled === 2)
+ assert.equal(s.client, 'mock')
+ assert.notEqual(s.cells.a.urgent.p, 0.99)
+ assert.equal(v.requests.length, 2)
+ writeFileSync(v.file, JSON.stringify({ ...sheet, offline: false }))
+ s = await until(v.state, (s) => !s.offline && s.stats.cellsFilled === 2)
+ assert.equal(v.requests.length, 4)
+ assert.equal(s.client, 'typesafe')
+ assert.equal(s.cells.a.urgent.p, 0.99)
+})
+
+test('a provider error writes a failed fill promptly, and switching to practice recovers without credentials changes', async (t) => {
+ const v = await fixture(t, false, 402)
+ const failed = await until(v.verdict, (r) => r.phases?.[1]?.state === 'failed')
+ assert.equal(failed.ready, false)
+ assert.equal(failed.phases[1].name, 'Answers unavailable')
+ assert.match(failed.summary, /out of credit/)
+ assert.equal(failed.findings.find((f) => f.kind === 'jev').severity, 'error')
+ writeFileSync(v.file, JSON.stringify({ ...sheet, offline: true }))
+ const s = await until(v.state, (s) => s.offline && s.stats.cellsFilled === 2)
+ assert.equal(s.jevError, null)
+ await until(v.verdict, (r) => r.ready && r.sheet.offline)
+ assert.equal(process.env.TYPESAFE_API_KEY, 'local-test-key')
+})
+
+test('an invalid practice flag is reported without sending rows to the configured provider', async (t) => {
+ const v = await fixture(t, 'true')
+ const s = await until(v.state, (s) => !!s.error)
+ assert.match(s.error, /offline must be true or false/)
+ assert.equal(s.client, 'mock')
+ assert.equal(v.requests.length, 0)
+})
+
+test('an offline preview stays offline after the project switches, and cannot be applied as live evidence', async (t) => {
+ const v = await fixture(t, true)
+ const s = await until(v.state, (s) => s.stats.cellsFilled === 2)
+ const token = s.questionLabToken
+ const { draft } = await v.ctl('labPreview', { token, column: 'urgent', count: 10 })
+ writeFileSync(v.file, JSON.stringify({ ...sheet, offline: false }))
+ await until(v.state, (s) => !s.offline && s.stats.cellsFilled === 2)
+ const { trial: started } = await v.ctl('labStart', { token, draft: draft.id, header: 'Production is down?' })
+ const { trial } = await until(() => v.ctl('labGet', { token, id: started.id }), (r) => r.trial.status === 'complete')
+ assert.deepEqual(trial.summary.providers, ['mock'])
+ assert.equal(trial.stale, true)
+ assert.equal(v.requests.length, 2, 'only the sheet made live calls; the frozen practice trial did not')
+ await v.ctl('labKeep', { token, id: trial.id })
+ assert.equal((await v.ctl('labApply', { token, id: trial.id })).ok, false)
+})
diff --git a/store/agents/jev-sheets/test/question-lab.test.mjs b/store/agents/jev-sheets/test/question-lab.test.mjs
index afcd20623..470e9f54f 100644
--- a/store/agents/jev-sheets/test/question-lab.test.mjs
+++ b/store/agents/jev-sheets/test/question-lab.test.mjs
@@ -574,10 +574,10 @@ test('real sheet integration caches candidate results, keeps the original, expor
rows: data.rows.map((r) => ({ id: r.id, text: r.text, ...r.meta }))
})
)
- const viewer = await startSheetsViewer({ workspace: ws, autostart: false })
+ let viewer = await startSheetsViewer({ workspace: ws, autostart: false })
try {
const state = async () => (await fetch(viewer.url + '/state')).json()
- const token = (await state()).questionLabToken
+ let token = (await state()).questionLabToken
const ctl = async (cmd, body = {}) =>
(
await fetch(viewer.url + '/control', {
@@ -615,8 +615,8 @@ test('real sheet integration caches candidate results, keeps the original, expor
)
assert.equal(
JSON.parse(readFileSync(join(ws, 'sheet.json'))).columns.length,
- 1,
- 'try-on-sheet follows existing pane-only semantics'
+ 2,
+ 'the chosen question is saved alongside the original'
)
assert.equal(
(await fetch(viewer.url + `/download/question-trial-${t.id}.zip`)).status,
@@ -626,6 +626,15 @@ test('real sheet integration caches candidate results, keeps the original, expor
await (await fetch(viewer.url + '/download/answers.csv')).text(),
/Urgent.*Urgent/s
)
+ assert.equal(applied.applied.persisted, true)
+ await ctl('reset')
+ assert.equal((await state()).columns.length, 2, 'an immediate reset retains the question')
+ await viewer.close()
+ viewer = await startSheetsViewer({ workspace: ws, autostart: false })
+ token = (await state()).questionLabToken
+ assert.equal((await state()).columns.length, 2, 'the question survives a process restart')
+ assert.equal((await ctl('labApply', { id: t.id })).applied.alreadyApplied, true)
+ assert.equal(JSON.parse(readFileSync(join(ws, 'sheet.json'))).columns.length, 2)
await ctl('editRow', { id: 'r12', text: 'Different message' })
assert.equal((await ctl('labApply', { id: t.id })).ok, false)
assert.equal(
diff --git a/store/agents/jev-sheets/toolchain/check.mjs b/store/agents/jev-sheets/toolchain/check.mjs
index e367bf72c..8ac6b9731 100644
--- a/store/agents/jev-sheets/toolchain/check.mjs
+++ b/store/agents/jev-sheets/toolchain/check.mjs
@@ -23,6 +23,8 @@ try {
if (!sheet || typeof sheet !== 'object' || Array.isArray(sheet)) { console.log('error sheet.json must be a JSON object'); process.exit(1) }
if (!sheet.title || typeof sheet.title !== 'string') warn('no title')
+if (sheet.offline !== undefined && typeof sheet.offline !== 'boolean') error('offline must be true or false')
+if (sheet.offline === true) console.log('info offline practice: sheet and Question Lab use the stand-in, even with a saved key')
// Rows the agent wrote are made up and must say so. Rows from the person's own file are what they are.
if (sheet.source === undefined && (!sheet.description || !/made.?up|synthetic|fictional|not real/i.test(String(sheet.description)))) warn('say in "description" that the data is made up')
diff --git a/store/agents/jev-sheets/toolchain/doctor.sh b/store/agents/jev-sheets/toolchain/doctor.sh
index 58afb420e..5c4ba146b 100755
--- a/store/agents/jev-sheets/toolchain/doctor.sh
+++ b/store/agents/jev-sheets/toolchain/doctor.sh
@@ -7,7 +7,7 @@ if ! node -e "process.exit(Number(process.versions.node.split('.')[0]) >= 18 ? 0
fi
echo "ok node $(node -v)"
cd "$(dirname "$0")/.."
-for file in viewer/questions.mjs viewer/question-lab.mjs viewer/question-lab-ui.mjs viewer/trial-zip.mjs; do
+for file in viewer/questions.mjs viewer/question-lab.mjs viewer/question-lab-ui.mjs viewer/trial-zip.mjs viewer/kept-column.mjs; do
node --check "$file"
done
echo "ok Question Lab modules"
diff --git a/store/agents/jev-sheets/viewer/grammar.mjs b/store/agents/jev-sheets/viewer/grammar.mjs
index 4bf84c394..355872dce 100644
--- a/store/agents/jev-sheets/viewer/grammar.mjs
+++ b/store/agents/jev-sheets/viewer/grammar.mjs
@@ -129,13 +129,14 @@ export function normalizeSheet(raw) {
if (!parsed.ok) { errors.push(`column ${i + 1}: ${parsed.error}`); return }
if (colIds.has(parsed.column.id)) { errors.push(`column ${i + 1}: id "${parsed.column.id}" is used twice`); return }
colIds.add(parsed.column.id)
- columns.push(parsed.column)
+ columns.push({ ...parsed.column, ...(typeof c?.questionTrial === 'string' && /^[0-9a-f-]{36}$/i.test(c.questionTrial) ? { questionTrial: c.questionTrial } : {}) })
})
let reviewBelow = Number(src.reviewBelow ?? 0.65)
if (!Number.isFinite(reviewBelow) || reviewBelow < 0 || reviewBelow > 1) { errors.push('reviewBelow must be between 0 and 1'); reviewBelow = 0.65 }
let concurrency = Math.round(Number(src.concurrency ?? 8))
if (!Number.isFinite(concurrency) || concurrency < 1 || concurrency > 32) { errors.push('concurrency must be 1 to 32'); concurrency = 8 }
+ if (src.offline !== undefined && typeof src.offline !== 'boolean') errors.push('offline must be true or false')
return {
title: typeof src.title === 'string' && src.title.trim() ? src.title.trim().slice(0, 120) : 'Untitled sheet',
@@ -144,6 +145,8 @@ export function normalizeSheet(raw) {
textLabel: typeof src.textLabel === 'string' && src.textLabel.trim() ? src.textLabel.trim().slice(0, 40) : 'Text',
suggestions: Array.isArray(src.suggestions) ? src.suggestions.filter((s) => typeof s === 'string' && parseHeader(s).ok).slice(0, 24) : [],
demo: src.demo !== false,
+ // An invalid practice setting must not accidentally send rows to a live provider.
+ offline: src.offline !== undefined && src.offline !== false,
reviewBelow, concurrency, rows, columns, errors,
}
}
diff --git a/store/agents/jev-sheets/viewer/index.html b/store/agents/jev-sheets/viewer/index.html
index e4496eb7d..221f9efb3 100644
--- a/store/agents/jev-sheets/viewer/index.html
+++ b/store/agents/jev-sheets/viewer/index.html
@@ -25,7 +25,7 @@
<div class="stat"><label>Jev columns</label><span id="sCols">0</span></div>
<div class="stat"><label>cells filled</label><span><b id="sCells">0</b><i id="sCellsOf"></i></span></div>
<div class="stat" id="rateStat"><label>cells / s</label><span id="sRate">0</span></div>
- <div class="stat"><label>cost so far</label><span id="sCost">$0</span></div>
+ <div class="stat"><label id="costLabel">cost so far</label><span id="sCost">$0</span></div>
<div class="stat" id="flagStat"><label>for review</label><span id="sFlag">0</span></div>
</div>
<div class="controls">
diff --git a/store/agents/jev-sheets/viewer/kept-column.mjs b/store/agents/jev-sheets/viewer/kept-column.mjs
new file mode 100644
index 000000000..ef90c3dac
--- /dev/null
+++ b/store/agents/jev-sheets/viewer/kept-column.mjs
@@ -0,0 +1,60 @@
+import { lstatSync, readFileSync, writeFileSync, renameSync, rmSync } from 'node:fs'
+import { join } from 'node:path'
+import { randomUUID } from 'node:crypto'
+import { normalizeSheet, parseHeader, columnKey, LIMITS } from './grammar.mjs'
+import { loadSource } from './source.mjs'
+import { trialHash } from './question-lab.mjs'
+
+// Read the actual project, not the watcher's last notification. An agent can save
+// between the last preview and this click. Merge only a new question into that file.
+export function keepTrialColumn(workspace, trial, liveColumns) {
+ const file = join(workspace, 'sheet.json')
+ const stat = lstatSync(file)
+ if (!stat.isFile() || stat.isSymbolicLink())
+ throw Error('Keep questions in a regular sheet.json inside this project.')
+ const before = readFileSync(file, 'utf8')
+ const raw = JSON.parse(before)
+ let expanded = raw, source = null
+ if (typeof raw?.source === 'string' && raw.source.trim()) {
+ const loaded = loadSource(workspace, raw.source.trim(), { textColumn: raw.textColumn, limit: LIMITS.maxRows })
+ if (loaded.error) throw Error(loaded.error)
+ source = loaded.info.name
+ expanded = { ...raw, rows: [...(Array.isArray(raw.rows) ? raw.rows : []), ...loaded.rows] }
+ }
+ const normalized = normalizeSheet(expanded)
+ if (normalized.errors.length) throw Error(`Fix sheet.json before keeping a question: ${normalized.errors[0]}`)
+ if (normalized.offline !== !!trial.offline)
+ throw Error('The project switched between practice and live mode. Preview a new trial before keeping its wording.')
+ const rows = normalized.rows.map((r, i) => ({ id: r.id, n: i + 1, text: r.text, meta: r.meta }))
+ if (trialHash({ context: normalized.context, source, rows }) !== trial.datasetSha)
+ throw Error('The project rows or context changed. Preview a new trial before keeping its wording.')
+ const original = normalized.columns.find((c) => c.id === trial.original.id)
+ if ((original && columnKey(original) !== columnKey(trial.original)) || (!original && trial.original.source === 'file'))
+ throw Error('The original question changed in sheet.json. Preview a new trial before keeping its wording.')
+
+ const candidate = parseHeader(trial.candidate.header)
+ if (!candidate.ok) throw Error(candidate.error)
+ const saved = (raw.columns || []).find((c) => c?.questionTrial === trial.id)
+ if (saved) {
+ const parsed = parseHeader(saved.header, saved.id)
+ if (!parsed.ok || columnKey(parsed.column) !== columnKey(candidate.column))
+ throw Error('This saved question was edited. Keep that edit and start a new trial.')
+ return { raw, id: parsed.column.id, alreadyApplied: true }
+ }
+ const occupied = new Set([...normalized.columns, ...liveColumns].map((c) => c.id))
+ if (occupied.size >= LIMITS.maxColumns)
+ throw Error('This sheet has 12 columns. Remove one from sheet.json before keeping another.')
+ let id = candidate.column.id, suffix = 2
+ while (occupied.has(id)) id = `${candidate.column.id.slice(0, 24)}_v${suffix++}`
+ const next = { ...raw, columns: [...(raw.columns || []), { id, header: candidate.column.header, questionTrial: trial.id }] }
+ const temporary = join(workspace, `.sheet-questions-${randomUUID()}.tmp`)
+ try {
+ writeFileSync(temporary, JSON.stringify(next, null, 2) + '\n', { flag: 'wx', mode: stat.mode & 0o777 })
+ if (lstatSync(file).isSymbolicLink() || readFileSync(file, 'utf8') !== before)
+ throw Error('sheet.json changed while saving. Try again after the latest edit loads.')
+ renameSync(temporary, file)
+ } finally {
+ rmSync(temporary, { force: true })
+ }
+ return { raw: next, id, alreadyApplied: false }
+}
diff --git a/store/agents/jev-sheets/viewer/question-lab-ui.mjs b/store/agents/jev-sheets/viewer/question-lab-ui.mjs
index bf7e6e9bc..90110d06d 100644
--- a/store/agents/jev-sheets/viewer/question-lab-ui.mjs
+++ b/store/agents/jev-sheets/viewer/question-lab-ui.mjs
@@ -26,7 +26,7 @@ export function initQuestionLab({ post, getState, getSelected }) {
</section>
<section id="qlRun" class="ql-run" hidden>
<div class="ql-question-pair"><div><span>ORIGINAL</span><p id="qlOldHeader"></p></div><div><span>NEW WORDING</span><p id="qlNewHeader"></p></div></div>
- <div class="ql-summary"><div id="qlNumbers"></div><div class="ql-actions"><button id="qlCancel">Cancel remaining calls</button><button id="qlKeep" class="primary">Keep this trial</button><a id="qlDownload" class="ql-button" hidden>Download trial.zip</a><button id="qlApply" hidden>Try on whole sheet</button></div></div>
+ <div class="ql-summary"><div id="qlNumbers"></div><div class="ql-actions"><button id="qlCancel">Cancel remaining calls</button><button id="qlKeep" class="primary">Keep this trial</button><a id="qlDownload" class="ql-button" hidden>Download trial.zip</a><button id="qlApply" hidden>Use on whole sheet</button></div></div>
<p id="qlInterpret" class="ql-caption"></p>
</section>
<section id="qlHistoryList" class="ql-history" hidden></section>
@@ -111,9 +111,10 @@ export function initQuestionLab({ post, getState, getSelected }) {
$('qlCancel').disabled = busy || !!trial?.cancelRequested
$('qlChanged').disabled = busy
$('qlApply').hidden = !trial?.kept || trial.status !== 'complete'
- $('qlApply').disabled = busy || !!trial?.stale
- $('qlApply').textContent =
- `Try on whole sheet · ${getState()?.rows.length ?? trial?.population ?? 0} rows`
+ const saved = trial && getState()?.columns.some((c) => c.questionTrial === trial.id && c.header === trial.candidate.header)
+ $('qlApply').disabled = busy || !!trial?.stale || !!saved
+ $('qlApply').textContent = saved ? 'Saved in project' :
+ `Use on whole sheet · ${getState()?.rows.length ?? trial?.population ?? 0} rows`
$('qlDownload').hidden = !trial?.kept
$('qlKeep').hidden = !!trial?.kept
}
@@ -666,9 +667,8 @@ export function initQuestionLab({ post, getState, getSelected }) {
const res = await call('labApply', { id: trial.id })
if (!res.ok) throw new Error(res.error)
message(
- `Added a separate column; ${res.applied.reused} trial rows reused. It will fill as the sheet runs. This column lasts in this pane; ask your agent to retain the wording in sheet.json.`
+ `Question saved in your project; ${res.applied.reused} trial rows reused. The original question is unchanged. Your chosen wording will still be here when you reopen the sheet.`
)
- $('qlApply').textContent = 'Column added'
})
)
$('qlHistory').addEventListener('click', () => guard(showHistory))
diff --git a/store/agents/jev-sheets/viewer/question-lab.mjs b/store/agents/jev-sheets/viewer/question-lab.mjs
index a0c806631..fae0f7432 100644
--- a/store/agents/jev-sheets/viewer/question-lab.mjs
+++ b/store/agents/jev-sheets/viewer/question-lab.mjs
@@ -210,6 +210,7 @@ function filesFor(trial) {
description,
context: trial.context,
demo: false,
+ offline: !!trial.offline || trial.summary.providers.includes('mock'),
columns: [
{ id: 'original', header: trial.original.header },
{ id: 'candidate', header: trial.candidate.header }
@@ -298,7 +299,7 @@ function filesFor(trial) {
'comparison.csv':
lines.map((line) => line.map(csv).join(',')).join('\r\n') + '\r\n',
'review.md': report.join('\n'),
- 'README.md': `# Question Lab trial\n\n${description}\n\nThis folder contains the exact sampled text and metadata. Keep it wherever you would keep the original data. No API keys are included.\n\n- trial.json: exact questions sent, raw answers, per-call provider/model/usage, frozen rows and your notes.\n- comparison.csv: answers and preferences, safe to open as a spreadsheet.\n- review.md: readable review.\n- sheet.json: the frozen sample with both headers. Copy it into a separate Jev Sheets workspace to ask the questions again. Live runs can produce different answers. With no key, the viewer explicitly uses its offline stand-in.\n- column.json: the candidate column definition; an agent can add it to the original sheet.json after you decide.\n- checksums.json: SHA-256 hashes of the files in this packet.\n\nA trial does not overwrite the source file. “Try on whole sheet” adds a separate runtime column; like other pane-added columns, it lasts until reset or viewer restart. Ask your agent to retain a chosen header in sheet.json. Kept trials survive restart.\n\nSelected ${trial.rows.length} of ${trial.population} rows using ${trial.selection.method}. ${mock ? 'OFFLINE STAND-IN / INCOMPLETE: these results are not model validation or research findings.' : 'No automatic claim of accuracy is made.'}\n`
+ 'README.md': `# Question Lab trial\n\n${description}\n\nThis folder contains the exact sampled text and metadata. Keep it wherever you would keep the original data. No API keys are included.\n\n- trial.json: exact questions sent, raw answers, per-call provider/model/usage, frozen rows and your notes.\n- comparison.csv: answers and preferences, safe to open as a spreadsheet.\n- review.md: readable review.\n- sheet.json: the frozen sample with both headers. Copy it into a separate Jev Sheets workspace to ask the questions again. Live runs can produce different answers. With no key, the viewer explicitly uses its offline stand-in.\n- column.json: the candidate column definition; an agent can add it to the original sheet.json after you decide.\n- checksums.json: SHA-256 hashes of the files in this packet.\n\nKeeping a trial does not change the source file. “Use on whole sheet” saves the chosen wording as a separate question in the original sheet.json, preserving the original question and data. Repeated use of the same kept trial does not duplicate the question. The chosen wording and kept trial survive restart; answers are recomputed in a new viewer session.\n\nSelected ${trial.rows.length} of ${trial.population} rows using ${trial.selection.method}. ${mock ? 'OFFLINE STAND-IN / INCOMPLETE: these results are not model validation or research findings.' : 'No automatic claim of accuracy is made.'}\n`
}
}
@@ -325,6 +326,7 @@ export function createQuestionLab({
column = now.columns.find((c) => c.id === trial.original.id)
return (
!!now.invalid ||
+ !!now.offline !== !!trial.offline ||
now.dataSha !== trial.datasetSha ||
!column ||
columnKey(column) !== columnKey(trial.original)
@@ -410,6 +412,7 @@ export function createQuestionLab({
try {
const result = await evaluatePair({
state: rowState(row),
+ key: trial.offline ? '' : undefined,
questions,
mock,
salt: 1,
@@ -543,6 +546,7 @@ export function createQuestionLab({
title: now.title,
source: now.source,
context: now.context,
+ offline: !!now.offline,
original: clone(column),
rows,
datasetSha: now.dataSha,
diff --git a/store/agents/jev-sheets/viewer/studio.js b/store/agents/jev-sheets/viewer/studio.js
index 8cdfc8366..acff0a1e3 100644
--- a/store/agents/jev-sheets/viewer/studio.js
+++ b/store/agents/jev-sheets/viewer/studio.js
@@ -290,8 +290,15 @@ function applyShared(v) {
if (!draggingReview) { $('reviewRange').value = S.reviewBelow; $('reviewVal').textContent = S.reviewBelow.toFixed(2) }
if (v.ghost) applyGhost(v.ghost)
if (v.client) S.client = v.client
- // Own data with no key: say plainly that the stand-in is not the real model.
- $('standin').classList.toggle('hidden', !(S.own && S.client === 'mock'))
+ if (typeof v.offline === 'boolean') S.offline = v.offline
+ // The shared HUD describes machine-wide credentials. A practice project deliberately ignores them.
+ document.querySelector('[data-jev-hud]').hidden = !!S.offline
+ $('costLabel').textContent = S.client === 'mock' ? 'est. live cost' : 'cost so far'
+ // Label practice for sample rows too, including when a live key is present on this machine.
+ $('standin').classList.toggle('hidden', S.client !== 'mock')
+ $('standin').textContent = S.offline
+ ? 'Offline practice. A word-matching stand-in fills these cells and Question Lab. No new live model calls are made. These answers do not establish model quality.'
+ : 'These are not real model answers. An offline stand-in is matching words. Connect a key in the Jev · live mind panel to ask the real model.'
renderFilterChip()
dirty.top = dirty.hist = dirty.find = true
}
diff --git a/store/agents/jev-sheets/viewer/viewer.mjs b/store/agents/jev-sheets/viewer/viewer.mjs
index d0b645fff..933753630 100644
--- a/store/agents/jev-sheets/viewer/viewer.mjs
+++ b/store/agents/jev-sheets/viewer/viewer.mjs
@@ -17,6 +17,7 @@ import { writeFileSync, renameSync, readFileSync, existsSync, rmSync } from 'nod
import { extname } from 'node:path'
import { questionForColumn } from './questions.mjs'
import { createQuestionLab, trialHash } from './question-lab.mjs'
+import { keepTrialColumn } from './kept-column.mjs'
const HERE = dirname(fileURLToPath(import.meta.url))
const MARKER = 'sheet.json'
@@ -34,7 +35,7 @@ export async function startSheetsViewer({ workspace, port = 0, autostart = true,
// Every `let` lives here, above the first call that could touch it.
let server = null, watcher = null
let sourceWatcher = null, sourceFile = null, sourceInfo = null, sourceError = null, sourceTimer = null // the person's own file, if sheet.json names one
- let sheet = normalizeSheet({ rows: [] }), configError = null, jevError = null
+ let sheet = normalizeSheet({ rows: [] }), configError = null, jevError = null, lastGoodConfig = null
let rows = [], rowById = new Map(), columns = []
let cells = new Map() // row id -> Map(column id -> cell)
let cache = new Map() // column key -> Map(row state -> answer)
@@ -51,7 +52,7 @@ export async function startSheetsViewer({ workspace, port = 0, autostart = true,
const rng = mulberry32(20260919)
// Which live route is active (typesafe, cloudflare, openrouter), or null for the offline stand-in.
// Asked each time, because a key can arrive in the credentials file while the viewer runs.
- const liveRoute = () => resolveCredentials()?.provider ?? null
+ const liveRoute = () => sheet.offline ? null : resolveCredentials()?.provider ?? null
const picker = createPicker()
const colById = (id) => columns.find((c) => c.id === id)
const mock = sheetMock(colById)
@@ -186,7 +187,7 @@ export async function startSheetsViewer({ workspace, port = 0, autostart = true,
}
}
function lightView() {
- return { rev, running, reviewBelow, reviewOnly, sort, filter, order, error: configError, jevError, client: liveRoute() ?? 'mock', ghost: ghostView(), ...tally() }
+ return { rev, running, reviewBelow, reviewOnly, sort, filter, order, error: configError, jevError, offline: sheet.offline, client: liveRoute() ?? 'mock', ghost: ghostView(), ...tally() }
}
function fullState() {
const cellsOut = {}
@@ -204,7 +205,7 @@ export async function startSheetsViewer({ workspace, port = 0, autostart = true,
concurrency: sheet.concurrency, limits: LIMITS,
questionLabToken: lab.token,
suggestions: sheet.suggestions.length ? sheet.suggestions : FALLBACK_SUGGESTIONS,
- columns: columns.map((c) => ({ id: c.id, header: c.header, name: c.name, type: c.type, options: c.options, descriptions: c.descriptions, levels: c.levels, bare: !!c.bare, source: c.source, kind: describeColumn(c) })),
+ columns: columns.map((c) => ({ id: c.id, header: c.header, name: c.name, type: c.type, options: c.options, descriptions: c.descriptions, levels: c.levels, bare: !!c.bare, source: c.source, questionTrial: c.questionTrial, kind: describeColumn(c) })),
rows: rows.map((r, i) => ({ id: r.id, n: i + 1, text: r.text, meta: r.meta, group: r.group, edited: !!r.edited, labelled: !!r.truth })),
cells: cellsOut,
...lightView(),
@@ -231,7 +232,7 @@ export async function startSheetsViewer({ workspace, port = 0, autostart = true,
const accLine = columns.filter((c) => colStats[c.id].acc != null).map((c) => `${c.name} ${pct(colStats[c.id].acc)}`).join(', ')
const findings = []
if (configError) findings.push({ severity: 'error', kind: 'sheet', ref: MARKER, message: configError })
- if (jevError) findings.push({ severity: 'warning', kind: 'jev', message: jevError })
+ if (jevError) findings.push({ severity: 'error', kind: 'jev', message: jevError })
const report = []
for (const c of columns) {
const s = colStats[c.id]
@@ -250,15 +251,15 @@ export async function startSheetsViewer({ workspace, port = 0, autostart = true,
if (g) findings.push({ severity: 'info', kind: 'confidence', message: `Average confidence by row group: ${g}.` })
const full = stats.cellsTotal > 0 && stats.cellsFilled === stats.cellsTotal
return {
- ready: !configError && full,
- summary: clean(configError ? `sheet.json needs a fix: ${configError}` : `${sheet.title}: ${stats.rows} rows x ${stats.columns} Jev columns, ${stats.cellsFilled}/${stats.cellsTotal} cells, ${stats.flagged} under ${reviewBelow}${accLine ? `. Right: ${accLine}` : ''}`).slice(0, 200),
+ ready: !configError && !jevError && full,
+ summary: clean(configError ? `sheet.json needs a fix: ${configError}` : jevError ? `Answers unavailable: ${jevError}` : `${sheet.offline ? 'Offline practice · ' : ''}${sheet.title}: ${stats.rows} rows x ${stats.columns} Jev columns, ${stats.cellsFilled}/${stats.cellsTotal} cells, ${stats.flagged} under ${reviewBelow}${accLine ? `. Right: ${accLine}` : ''}`).slice(0, 200),
findings, artifact: MARKER, answersFile: ANSWERS,
phases: [
{ id: 'load', name: 'Sheet loaded', state: configError ? 'failed' : 'done' },
- { id: 'fill', name: 'Cells filled', state: full ? 'done' : stats.cellsTotal ? 'active' : 'pending' },
+ { id: 'fill', name: jevError ? 'Answers unavailable' : full ? 'Cells filled' : 'Filling cells', state: jevError ? 'failed' : full ? 'done' : stats.cellsTotal ? 'active' : 'pending' },
{ id: 'review', name: 'Review', state: !full ? 'pending' : stats.flagged ? 'active' : 'done' },
],
- sheet: { client: liveRoute() ?? 'mock', loadedAt, source: sourceInfo?.name ?? null, reviewBelow, rows: stats.rows, cellsFilled: stats.cellsFilled, cellsTotal: stats.cellsTotal, flagged: stats.flagged, costUsd: stats.costUsd, groups, columns: report },
+ sheet: { client: liveRoute() ?? 'mock', offline: sheet.offline, loadedAt, source: sourceInfo?.name ?? null, reviewBelow, rows: stats.rows, cellsFilled: stats.cellsFilled, cellsTotal: stats.cellsTotal, flagged: stats.flagged, costUsd: stats.costUsd, groups, columns: report },
}
}
function shown(col, a) {
@@ -327,7 +328,7 @@ export async function startSheetsViewer({ workspace, port = 0, autostart = true,
// so the wave is visible. `tick` and `drain` skip the pacing.
if (!fast && !liveRoute() && paceMs > 0) await sleep(paceMs * (0.7 + 0.6 * rng()))
const questions = Object.fromEntries(need.map((c) => [c.id, questionFor(c)]))
- const res = await evaluate({ state, questions, salt: 1, mock, model: process.env.JEV_MODEL || 'jev-latest' })
+ const res = await evaluate({ state, questions, key: sheet.offline ? '' : undefined, salt: 1, mock, model: process.env.JEV_MODEL || 'jev-latest' })
if (stopped || myEpoch !== epoch || rowById.get(row.id) !== row || JSON.stringify(rowState(row)) !== key) return
const real = Number(res.usage?.input_tokens)
const tokens = Number.isFinite(real) && real > 0 ? real : Math.ceil((key.length + JSON.stringify(toWire(questions)).length) / 4)
@@ -347,10 +348,12 @@ export async function startSheetsViewer({ workspace, port = 0, autostart = true,
burst.cells += n; burst.ms = now - burst.startedAt; burst.rate = burst.cells / Math.max(0.03, burst.ms / 1000)
if (!hasMissing(row)) todo.delete(row.id)
} catch (e) {
+ if (stopped || myEpoch !== epoch) return
counters.errors++
jevError = clean(e?.message ?? e)
row.retryAt = Date.now() + 5000 // leave the cell waiting, show the error, try again in a while (see pump)
pushView()
+ verdictSoon()
} finally {
inflight.delete(row)
}
@@ -437,7 +440,16 @@ export async function startSheetsViewer({ workspace, port = 0, autostart = true,
const problems = [sourceError, ...next.errors].filter(Boolean).slice(0, 3).join('; ')
if (!next.rows.length && rows.length && !fresh) { configError = `${MARKER}: ${problems || 'needs at least one row with text'}. Showing the last good sheet`; pushView(); verdictSoon(); return }
configError = problems ? `${MARKER}: ${problems}` : null
+ const modeChanged = sheet.offline !== next.offline
sheet = next
+ if (modeChanged) {
+ cache = new Map()
+ patches = []
+ jevError = null
+ counters = { calls: 0, cacheHits: 0, tokens: 0, costUsd: 0, errors: 0, computed: 0 }
+ burst = { active: false, startedAt: 0, cells: 0, rate: 0, ms: 0 }
+ }
+ lastGoodConfig = raw
loadedAt = new Date().toISOString()
epoch++
rows = next.rows.map((r) => ({ ...r, edited: false, retryAt: 0 }))
@@ -480,7 +492,7 @@ export async function startSheetsViewer({ workspace, port = 0, autostart = true,
if (!cur.source && !existsSync(join(workspace, SAMPLE_BACKUP))) writeFileSync(join(workspace, SAMPLE_BACKUP), JSON.stringify(cur, null, 2) + '\n')
} catch { /* no sample worth keeping */ }
const title = file.replace(/\.[^.]+$/, '').replace(/[-_]+/g, ' ').trim() || 'My file'
- const next = { title, description: `Rows from ${file}, the person's own file.`, source: file, textLabel: got.info.textColumn, demo: false, concurrency: 16, columns: [], suggestions: OWN_SUGGESTIONS }
+ const next = { title, description: `Rows from ${file}, the person's own file.`, source: file, textLabel: got.info.textColumn, offline: sheet.offline, demo: false, concurrency: 16, columns: [], suggestions: OWN_SUGGESTIONS }
writeMarker(next)
filter = null; sort = null; reviewOnly = false
applySheet(next, true)
@@ -493,6 +505,7 @@ export async function startSheetsViewer({ workspace, port = 0, autostart = true,
try { sample = JSON.parse(readFileSync(f, 'utf8')); if (sample && !sample.source) break; sample = null } catch { sample = null }
}
if (!sample) return { ok: false, error: 'the made-up sample is not on this machine' }
+ sample.offline = sheet.offline
writeMarker(sample)
filter = null; sort = null; reviewOnly = false
applySheet(sample, true)
@@ -505,7 +518,7 @@ export async function startSheetsViewer({ workspace, port = 0, autostart = true,
burst = { active: false, startedAt: 0, cells: 0, rate: 0, ms: 0 }
patches = []
ghost = { ...ghost, phase: 'idle', header: '', pausedUntil: 0, nextAt: Date.now() + 5000, cursor: 0 }
- applySheet(watcher.get(), true)
+ applySheet(lastGoodConfig ?? watcher.get(), true)
if (watcher.error()) { configError = `${watcher.error()}. Showing the last good sheet`; pushView() } // still broken on disk
}
@@ -580,7 +593,6 @@ export async function startSheetsViewer({ workspace, port = 0, autostart = true,
// Question Lab owns only frozen trials; adding a tested header uses the existing sheet/cache.
let labSnapshotRev = -1, labData = null
- const labApplied = new Map()
function labSnapshot(withConfidence = false) {
if (labSnapshotRev !== rev) {
const frozenRows = rows.map((r, i) => ({ id: r.id, n: i + 1, text: r.text, meta: { ...r.meta } }))
@@ -591,20 +603,15 @@ export async function startSheetsViewer({ workspace, port = 0, autostart = true,
const cell = cellOf(row.id, col.id)
return cell ? [[row.id, cell.conf]] : []
}))])) : null
- return { ...labData, title: sheet.title, source: sourceInfo?.name || null, context: sheet.context, columns, order, confidences, invalid: !!configError }
+ return { ...labData, offline: sheet.offline, title: sheet.title, source: sourceInfo?.name || null, context: sheet.context, columns, order, confidences, invalid: !!configError }
}
const lab = createQuestionLab({ workspace, snapshot: labSnapshot,
+ // Switching a project to practice also prevents any remaining trial work from making live calls.
+ evaluatePair: (options) => evaluate({ ...options, key: sheet.offline || options.key === '' ? '' : undefined }),
notify: (data) => server?.broadcast(data, 'trial'),
apply: (trial) => {
- const prior = labApplied.get(trial.id)
- if (prior && columns.includes(prior.column)) return { ...prior.result, alreadyApplied: true }
- if (columns.length >= LIMITS.maxColumns) return { ok: false, error: 'This sheet has 12 columns. Remove one before trying another.' }
- // The original stays visible, including when both versions have the same name.
- const parsed = parseHeader(trial.candidate.header)
- let id = parsed.column.id, suffix = 2
- while (colById(id)) id = `${parsed.column.id.slice(0, 24)}_v${suffix++}`
- const col = { ...parsed.column, id, source: 'pane', addedAt: Date.now() }
- columns.push(col)
+ const saved = keepTrialColumn(workspace, trial, columns)
+ const col = { ...parseHeader(trial.candidate.header).column, id: saved.id }
// Reuse only answers from this exact question/data and the currently connected route/model.
const route = liveRoute() || 'mock', model = process.env.JEV_MODEL || 'jev-latest'
let reused = 0
@@ -613,10 +620,9 @@ export async function startSheetsViewer({ workspace, port = 0, autostart = true,
cacheFor(col).set(JSON.stringify(rowState(row)), { answer: row.candidate.answer, latencyMs: row.provenance.latencyMs, tokens: 0 })
reused++
}
- const result = { id, rows: rows.length, reused, persisted: false }
- labApplied.set(trial.id, { column: col, result })
- touch(); refill(); changed()
- return result
+ touch()
+ applySheet(saved.raw, false)
+ return { id: saved.id, rows: rows.length, reused, persisted: true, alreadyApplied: saved.alreadyApplied }
}
})