Skip to content

armgit / kiln

Public Imported from GitHub
0

Build and revise procedural 3D assets with your coding agent. Local MCP server, CLI and TypeScript engine with rendered review, editable source and GLB export.

Read model capabilities on OpenCode 2.2 in the vision gate (#344)

Matthew Kissingerab3acce03687
diff --git a/CHANGELOG.md b/CHANGELOG.md
index 2bdefc8e..7285677e 100644
--- a/CHANGELOG.md
+++ b/CHANGELOG.md
@@ -51,6 +51,11 @@ does not publish a package.
sheet views are unchanged. A request that switches projection builds one more program per material.
Both changes alter render-service source, so its build identity changes and a running service of
the previous build is reported as incompatible rather than joined.
+- `scripts/check-vision.mjs` works on OpenCode 2.x. OpenCode 2.2 rejects `opencode models --verbose`,
+ so the gate reported every model as unverified. On 2.x it now reads the catalogue from
+ `opencode api --standalone GET /api/model` and its `capabilities.input`; a 1.x install still uses
+ `--verbose`. The docs say the script covers OpenCode models only. `docs/harnesses.md` records that OpenCode 2.2.0 delivers MCP images to the provider in
+ Code Mode and with `codemode: false`.
- `KILN_EVALUATOR_TIMEOUT_MS` now works above 60,000, as documented. Only the host read it: the worker
kept its own 60-second timer, so a build that yielded to the event loop after that failed with
`DEADLINE_EXCEEDED` at any limit, usually once the abandoned build had run to its end. The worker now
diff --git a/docs/dispatch.md b/docs/dispatch.md
index 50e39690..a0de65da 100644
--- a/docs/dispatch.md
+++ b/docs/dispatch.md
@@ -6,7 +6,7 @@ The repository's dispatcher runs a coding-agent harness in a temporary authoring
node scripts/dispatch-asset.mjs --harness opencode --model PROVIDER/MODEL --name water-tower "A steel water tower on a riveted lattice frame"
```
-Use a model available through your configured provider. Confirm image input support with `node scripts/check-vision.mjs PROVIDER/MODEL`, then check that the harness actually forwards render images. Provider metadata alone does not verify the whole path.
+Use a model available through your configured provider. For an OpenCode model, confirm image input support with `node scripts/check-vision.mjs PROVIDER/MODEL`, which reads the installed OpenCode's model catalogue; for other harnesses, use the provider's model documentation. Then check that the harness actually forwards render images. Provider metadata alone does not verify the whole path.
Run `bun run smoke:harness -- --harness opencode` before a batch. A quota or authentication failure is not a measure of asset quality. Inspect the retained transcript and build result when a run fails.
diff --git a/docs/harnesses.md b/docs/harnesses.md
index 71ee90e5..3d127dba 100644
--- a/docs/harnesses.md
+++ b/docs/harnesses.md
@@ -24,6 +24,11 @@ as text before inference. Do not treat those MCP results as visual review or
increase text limits to carry base64. See the
[reproduction and supported route](https://github.com/instruktlabs/kiln/blob/main/docs/reviews/2026-09-22-cline-image-qualification.md).
OpenCode 2.0.14 preserves standard MCP images in captured provider requests.
+OpenCode 2.2.0 does too, both in default Code Mode (`execute` calling
+`tools.kiln_workspace.kiln_render`) and with per-server `codemode: false`. A loopback
+Responses API mock received the Kiln PNG as `input_image` inside `function_call_output`,
+byte-identical in both modes. In 2.2.0 `--format json` output the image is at
+`part.state.metadata.content`; there is no `state.attachments` field.
Hermes 0.21.4 preserves MCP images as cached files; its vision tool can load them
for the main model when that route supports images. Activated Cline VS Code 4.1.20
tests preserve MCP images in Legacy and reproduce the CLI defect in Next/SDK.
@@ -348,8 +353,8 @@ node scripts/observe-mcp.mjs
process runs the engine over whatever program came back -- the child's own claim of success is
the one piece of evidence that proves nothing.
-Before trusting a new model id, confirm image support with
-`node scripts/check-vision.mjs PROVIDER/MODEL`, then confirm the harness actually forwards
+Before trusting a new OpenCode model id, confirm image support with
+`node scripts/check-vision.mjs PROVIDER/MODEL` (it reads OpenCode's model catalogue), then confirm the harness actually forwards
the images. Provider metadata alone verifies neither.
### Reading a result honestly
diff --git a/scripts/check-vision.mjs b/scripts/check-vision.mjs
index 88aaaf22..d552f782 100644
--- a/scripts/check-vision.mjs
+++ b/scripts/check-vision.mjs
@@ -13,9 +13,13 @@
//
// node scripts/check-vision.mjs opencode-go/kimi-k3 opencode-go/glm-5.3
//
-// Exits non-zero and names the offenders. OpenCode answers from a local cache,
-// so the whole check costs about two seconds however many models you pass.
+// Exits non-zero and names the offenders. On OpenCode 2.x it reads the whole
+// catalogue once from `GET /api/model` on a private server. Older 1.x installs
+// print capabilities with `opencode models <provider> --verbose` instead. Both
+// answer from a local cache in a few seconds.
import { spawnSync } from 'node:child_process';
+import { resolve } from 'node:path';
+import { fileURLToPath } from 'node:url';
/** Pull top-level JSON objects out of `opencode models --verbose` output. */
function jsonBlocks(text) {
@@ -42,53 +46,115 @@ function jsonBlocks(text) {
return out;
}
-function capabilities(provider) {
- const r = spawnSync('opencode', ['models', provider, '--verbose'], {
+function runOpencode(args) {
+ return spawnSync('opencode', args, {
encoding: 'utf8',
shell: process.platform === 'win32',
+ timeout: 120_000,
});
- const map = new Map();
- for (const o of jsonBlocks(r.stdout ?? '')) {
- if (o.id) map.set(o.id, o.capabilities ?? {});
+}
+
+/** OpenCode 2.x catalogue: `{ data: [{ providerID, id, capabilities }] }`, grouped by provider. */
+function catalogueByProvider(stdout) {
+ const byProvider = new Map();
+ let data;
+ try {
+ data = JSON.parse(stdout).data;
+ } catch {
+ return byProvider;
+ }
+ if (!Array.isArray(data)) return byProvider;
+ for (const m of data) {
+ if (!m?.providerID || !m.id) continue;
+ if (!byProvider.has(m.providerID)) byProvider.set(m.providerID, new Map());
+ byProvider.get(m.providerID).set(m.id, m.capabilities ?? {});
}
- return map;
+ return byProvider;
}
-const models = process.argv.slice(2);
-if (!models.length) {
- console.error('usage: check-vision.mjs <provider/model> [<provider/model> ...]');
- process.exit(2);
+/** The major version `opencode --version` reports, or null when it reports nothing usable. */
+function majorVersion(run) {
+ const r = run(['--version']);
+ const match = r.status === 0 ? /(\d+)\.\d+/.exec(r.stdout ?? '') : null;
+ return match ? Number(match[1]) : null;
}
-const byProvider = new Map();
-for (const m of models) {
- const slash = m.indexOf('/');
- const provider = slash > 0 ? m.slice(0, slash) : 'opencode';
- if (!byProvider.has(provider)) byProvider.set(provider, capabilities(provider));
+/**
+ * Model capabilities per provider, as `Map<provider, Map<modelId, capabilities>>`.
+ * OpenCode 2.x is the primary path: one catalogue request serves every provider.
+ * The version picks the route, because 1.x has no `api` command and 2.x has no
+ * `--verbose`.
+ */
+export function loadCapabilities(providers, run = runOpencode) {
+ const out = new Map();
+ const major = majorVersion(run);
+ if (major === null) {
+ for (const provider of providers) out.set(provider, new Map());
+ return out;
+ }
+ if (major >= 2) {
+ const r = run(['api', '--standalone', 'GET', '/api/model']);
+ const catalogue = r.status === 0 ? catalogueByProvider(r.stdout ?? '') : new Map();
+ for (const provider of providers) out.set(provider, catalogue.get(provider) ?? new Map());
+ return out;
+ }
+ for (const provider of providers) {
+ const legacy = run(['models', provider, '--verbose']);
+ const map = new Map();
+ for (const o of legacy.status === 0 ? jsonBlocks(legacy.stdout ?? '') : []) {
+ if (o.id) map.set(o.id, o.capabilities ?? {});
+ }
+ out.set(provider, map);
+ }
+ return out;
}
-const blind = [];
-const unknown = [];
-for (const m of models) {
+/** 2.x lists input modalities as an array; 1.x has an `input` object and `attachment`. */
+export function seesImages(caps) {
+ if (!caps) return false;
+ if (Array.isArray(caps.input)) return caps.input.includes('image');
+ if (caps.input && typeof caps.input === 'object' && 'image' in caps.input) {
+ return caps.input.image === true;
+ }
+ return caps.attachment === true;
+}
+
+function splitModel(m) {
const slash = m.indexOf('/');
- const provider = slash > 0 ? m.slice(0, slash) : 'opencode';
- const id = slash > 0 ? m.slice(slash + 1) : m;
- const caps = byProvider.get(provider)?.get(id);
- if (!caps) unknown.push(m);
- else if (caps.attachment !== true) blind.push(m);
- const state = !caps
- ? 'not listed by this provider'
- : caps.attachment === true
- ? 'sees images'
- : 'TEXT ONLY';
- console.log(` ${m.padEnd(44)} ${state}`);
+ return slash > 0
+ ? { provider: m.slice(0, slash), id: m.slice(slash + 1) }
+ : { provider: 'opencode', id: m };
}
-if (blind.length || unknown.length) {
- console.error('');
- if (blind.length)
- console.error(`refusing to dispatch ${blind.length} text-only model(s): ${blind.join(', ')}`);
- if (unknown.length) console.error(`could not verify: ${unknown.join(', ')}`);
- process.exit(1);
+function main(models) {
+ if (!models.length) {
+ console.error('usage: check-vision.mjs <provider/model> [<provider/model> ...]');
+ process.exit(2);
+ }
+
+ const byProvider = loadCapabilities([...new Set(models.map((m) => splitModel(m).provider))]);
+
+ const blind = [];
+ const unknown = [];
+ for (const m of models) {
+ const { provider, id } = splitModel(m);
+ const caps = byProvider.get(provider)?.get(id);
+ const sees = seesImages(caps);
+ if (!caps) unknown.push(m);
+ else if (!sees) blind.push(m);
+ const state = !caps ? 'not listed by this provider' : sees ? 'sees images' : 'TEXT ONLY';
+ console.log(` ${m.padEnd(44)} ${state}`);
+ }
+
+ if (blind.length || unknown.length) {
+ console.error('');
+ if (blind.length)
+ console.error(`refusing to dispatch ${blind.length} text-only model(s): ${blind.join(', ')}`);
+ if (unknown.length) console.error(`could not verify: ${unknown.join(', ')}`);
+ process.exit(1);
+ }
+ console.log(`all ${models.length} models accept image input`);
}
-console.log(`all ${models.length} models accept image input`);
+
+const isMain = process.argv[1] && resolve(process.argv[1]) === fileURLToPath(import.meta.url);
+if (isMain) main(process.argv.slice(2));
diff --git a/scripts/check-vision.test.mjs b/scripts/check-vision.test.mjs
new file mode 100644
index 00000000..bf0ba159
--- /dev/null
+++ b/scripts/check-vision.test.mjs
@@ -0,0 +1,124 @@
+/**
+ * The vision gate's own test.
+ *
+ * `check-vision.mjs` reads model capabilities from the installed OpenCode. OpenCode 2.x
+ * serves the catalogue at `GET /api/model` and rejects the 1.x
+ * `opencode models <provider> --verbose`. 2.x is the primary path; 1.x is kept for older
+ * installs. A gate that can no longer read capabilities reports every model as
+ * unverified, so both shapes are pinned here with a fake runner. Nothing spawns
+ * OpenCode or touches the network.
+ */
+import { expect, test } from 'bun:test';
+import { spawnSync } from 'node:child_process';
+import { fileURLToPath } from 'node:url';
+import { loadCapabilities, seesImages } from './check-vision.mjs';
+
+const script = fileURLToPath(new URL('./check-vision.mjs', import.meta.url));
+
+/** Trimmed from a real OpenCode 2.2.0 `opencode api --standalone GET /api/model` reply. */
+const CATALOGUE_V2 = JSON.stringify({
+ location: { directory: 'C:\\work' },
+ data: [
+ {
+ id: 'glm-5.3-flash',
+ modelID: 'glm-5.3-flash',
+ providerID: 'opencode-go',
+ capabilities: { tools: true, input: ['text', 'image', 'video'], output: ['text'] },
+ },
+ {
+ id: 'glm-5.3',
+ modelID: 'glm-5.3',
+ providerID: 'opencode-go',
+ capabilities: { tools: true, input: ['text'], output: ['text'] },
+ },
+ {
+ id: 'liquid/lfm-3-1.7b',
+ modelID: 'liquid/lfm-3-1.7b',
+ providerID: 'openrouter',
+ capabilities: { tools: true, input: ['text', 'image'], output: ['text'] },
+ },
+ ],
+});
+
+/** The 1.x `--verbose` shape: an id line, then a pretty-printed JSON block per model. */
+const VERBOSE_V1 = `opencode-go/kimi-k3
+{
+ "id": "kimi-k3",
+ "providerID": "opencode-go",
+ "capabilities": { "attachment": true, "toolcall": true }
+}
+opencode-go/glm-5.3
+{
+ "id": "glm-5.3",
+ "providerID": "opencode-go",
+ "capabilities": { "attachment": false, "toolcall": true }
+}
+`;
+
+const V2 = { status: 0, stdout: 'opencode v2.2.0\n', stderr: '' };
+const V1 = { status: 0, stdout: '1.18.30\n', stderr: '' };
+
+function fakeRunner(replies) {
+ const calls = [];
+ const run = (args) => {
+ calls.push(args.join(' '));
+ const reply = replies[args.join(' ')];
+ return reply ?? { status: 1, stdout: '', stderr: `unexpected: ${args.join(' ')}` };
+ };
+ return { run, calls };
+}
+
+test('OpenCode 2.x: reads the /api/model catalogue once and never tries --verbose', () => {
+ const { run, calls } = fakeRunner({
+ '--version': V2,
+ 'api --standalone GET /api/model': { status: 0, stdout: CATALOGUE_V2, stderr: '' },
+ });
+ const caps = loadCapabilities(['opencode-go', 'openrouter'], run);
+
+ expect(seesImages(caps.get('opencode-go').get('glm-5.3-flash'))).toBe(true);
+ expect(seesImages(caps.get('opencode-go').get('glm-5.3'))).toBe(false);
+ expect(caps.get('opencode-go').has('kimi-k3')).toBe(false);
+ // A model id containing a slash stays whole under its provider.
+ expect(seesImages(caps.get('openrouter').get('liquid/lfm-3-1.7b'))).toBe(true);
+ expect(calls).toEqual(['--version', 'api --standalone GET /api/model']);
+});
+
+test('OpenCode 1.x: reads --verbose per provider and never calls the 2.x API', () => {
+ const { run, calls } = fakeRunner({
+ '--version': V1,
+ 'models opencode-go --verbose': { status: 0, stdout: VERBOSE_V1, stderr: '' },
+ });
+ const caps = loadCapabilities(['opencode-go'], run);
+
+ expect(seesImages(caps.get('opencode-go').get('kimi-k3'))).toBe(true);
+ expect(seesImages(caps.get('opencode-go').get('glm-5.3'))).toBe(false);
+ expect(calls).toEqual(['--version', 'models opencode-go --verbose']);
+});
+
+test('an unreadable catalogue leaves models unlisted rather than throwing', () => {
+ const { run } = fakeRunner({
+ '--version': V2,
+ 'api --standalone GET /api/model': { status: 1, stdout: '', stderr: 'boom' },
+ });
+ expect(loadCapabilities(['opencode-go'], run).get('opencode-go').size).toBe(0);
+});
+
+test('a missing opencode leaves models unlisted rather than throwing', () => {
+ const { run } = fakeRunner({});
+ expect(loadCapabilities(['opencode-go'], run).get('opencode-go').size).toBe(0);
+});
+
+test('seesImages reads both capability shapes', () => {
+ expect(seesImages({ input: ['text', 'image'] })).toBe(true);
+ expect(seesImages({ input: ['text', 'pdf'] })).toBe(false);
+ expect(seesImages({ attachment: true })).toBe(true);
+ expect(seesImages({ attachment: false })).toBe(false);
+ expect(seesImages({ attachment: true, input: { image: false, pdf: true } })).toBe(false);
+ expect(seesImages(undefined)).toBe(false);
+});
+
+test('the CLI still prints its usage without arguments', () => {
+ const r = spawnSync(process.execPath, [script], { encoding: 'utf8' });
+ expect(r.status).toBe(2);
+ expect(r.stderr).toContain('usage: check-vision.mjs');
+});