https://arxiv.org/abs/2606.28344. The end of web parsing. The beginning of scalable pixel-native search. link: https://pixelrag.ai/
feat(agent): add Codex chat backend and deployment support (#166)
Zhifei Lida69a6507053
diff --git a/deploy/CODEX.md b/deploy/CODEX.mdnew file mode 100644index 0000000..472d3dd--- /dev/null+++ b/deploy/CODEX.md@@ -0,0 +1,61 @@+# Codex chat backend++The standalone agent server supports `CHAT_BACKEND=claude` (the existing+default) and `CHAT_BACKEND=codex`. Both use the same search and screenshot+handlers and expose the existing `/chat` SSE contract to the Next.js proxy.+The Codex path uses the logged-in Codex CLI via `codex exec --json`, with a+per-conversation stdio MCP adapter for `pixelrag_search` and `pixelrag_tile`.+It supports text history and an uploaded image, search/gallery events,+disconnect cancellation, tool-call limits, and a wall-time timeout.++Prerequisites: Node 22+, the existing web dependencies, Codex CLI 0.159.1+(the tested version), and `codex login` under the service's OS account.+No OpenAI API key is required when using that account's ChatGPT login.++```sh+CHAT_BACKEND=codex CODEX_BIN=/home/zlab-ssd1/.local/bin/codex \+ PIXELRAG_SEARCH_URL=http://localhost:30001 AGENT_PORT=30011 \+ /home/yichuan/.nix-profile/bin/node web/agent-server.mjs+```++`CHAT_CODEX_MODEL` optionally selects a model; otherwise Codex chooses its+default. `CHAT_CODEX_TIMEOUT_MS` defaults to 180000. `CHAT_MAX_TURNS` bounds+MCP tool calls for Codex. Existing request/concurrency limits apply to both+providers. `CHAT_MAX_BUDGET_USD` and Claude thinking-token limits apply only+to Claude; Codex subscription use is bounded by request, tool, and time limits.++The model runs in an empty temporary directory with a read-only sandbox,+shell, web search, browser, external apps, plugins, and image-file reading+disabled. Only the two PixelRAG MCP tools are preapproved. A random local+bearer token isolates each conversation's tool endpoint. Tool arguments+are validated with the same Zod schemas as Claude's tools.++The production drop-in `pixelrag-agent-codex.conf` runs the original+`/home/yichuan/visrag` checkout under `zlab-ssd1`, the account with the+working Codex login. It preserves the existing port, rate limits, CORS,+and search-backend configuration. Adjust its paths and account before+using it on another host. Do not copy authentication tokens between users.++An administrator can apply it with:++```sh+sudo install -m 0644 deploy/pixelrag-agent-codex.conf \+ /etc/systemd/system/pixelrag-agent.service.d/codex.conf+sudo systemctl daemon-reload+sudo systemctl restart pixelrag-agent.service+curl --fail http://localhost:30010/health+curl --fail http://localhost:30010/ready+```++The health response includes `backend: "codex"`. Removing the drop-in and+restarting the service restores the original Claude deployment.++Validation:++```sh+node --test web/lib/*.test.mjs+node --check web/agent-server.mjs+```++Official documentation: [Non-interactive execution](https://learn.chatgpt.com/docs/non-interactive-mode)+and [MCP tool configuration](https://learn.chatgpt.com/docs/config-file/config-reference).diff --git a/deploy/deploy.sh b/deploy/deploy.shindex 0894429..4fcbdcb 100755--- a/deploy/deploy.sh+++ b/deploy/deploy.sh@@ -45,8 +45,8 @@ if changed '^uv\.lock$'; thenfi# 2. Agent backend — cheap restart, safe to automate.-if changed '^web/agent-server\.mjs$'; then- say "agent-server.mjs changed -> restart pixelrag-agent"+if changed '^web/(agent-server\.mjs|lib/(agent-tools|codex-backend|codex-mcp|retrieval-health)\.mjs)$'; then+ say "agent backend changed -> restart pixelrag-agent"sudo systemctl restart pixelrag-agent.servicesleep 2say "pixelrag-agent: $(systemctl is-active pixelrag-agent.service)"diff --git a/deploy/pixelrag-agent-codex.conf b/deploy/pixelrag-agent-codex.confnew file mode 100644index 0000000..7f47f94--- /dev/null+++ b/deploy/pixelrag-agent-codex.conf@@ -0,0 +1,14 @@+# Install as /etc/systemd/system/pixelrag-agent.service.d/codex.conf.+# This machine's working Codex login belongs to zlab-ssd1. The service must+# run under that account to use it; do not copy auth tokens between users.+[Service]+User=zlab-ssd1+Group=zlab-ssd1+WorkingDirectory=/home/yichuan/visrag/web+Environment=HOME=/home/zlab-ssd1+Environment=CODEX_HOME=/home/zlab-ssd1/.codex+Environment=CHAT_BACKEND=codex+Environment=CODEX_BIN=/home/zlab-ssd1/.local/bin/codex+Environment=CHAT_CODEX_TIMEOUT_MS=180000+ExecStart=+ExecStart=/home/yichuan/.nix-profile/bin/node /home/yichuan/visrag/web/agent-server.mjsdiff --git a/web/agent-server.mjs b/web/agent-server.mjsindex 9127e93..dbfd2ae 100644--- a/web/agent-server.mjs+++ b/web/agent-server.mjs@@ -2,16 +2,21 @@/*** PixelRAG Agent backend — standalone SSE server.*- * Runs the Claude Agent SDK with subscription auth (uses the logged-in+ * Select CHAT_BACKEND=codex for the logged-in Codex CLI, or claude for the+ * Claude Agent SDK with subscription auth (uses the logged-in* `claude` CLI on this machine — no ANTHROPIC_API_KEY needed). Exposes the* same agent loop + pixelrag tools as the Next.js /api/chat route, so the* deployed Vercel frontend can proxy to it instead of running the SDK in* serverless (where the native CLI binary and credentials don't exist).*- * Run on a machine where `claude` is logged in:- * node deploy/agent-server.mjs+ * Run on a machine where the selected CLI is logged in:+ * CHAT_BACKEND=codex node web/agent-server.mjs** Env:+ * CHAT_BACKEND claude (default) or codex+ * CODEX_BIN Codex CLI executable (default codex on PATH)+ * CHAT_CODEX_MODEL optional model override; otherwise CLI default+ * CHAT_CODEX_TIMEOUT_MS per-request wall time (default 180000)* AGENT_PORT listen port (default 30010)* PIXELRAG_SEARCH_URL search API base (default http://localhost:30001)* CHAT_MAX_BUDGET_USD per-conversation budget cap (default 0.50)@@ -19,21 +24,15 @@*/import http from "node:http"-import { query, tool, createSdkMcpServer } from "@anthropic-ai/claude-agent-sdk"-import { z } from "zod"+import { createTools } from "./lib/agent-tools.mjs"+import { runCodex, codexTool } from "./lib/codex-backend.mjs"-import {- RETRIEVAL_OK,- RETRIEVAL_BACKEND_DOWN,- classifyStatus,- classifyThrow,- createRetrievalHealth,- recordRetrieval,- isUngrounded,- backendDownInstruction,- UNGROUNDED_CLIENT_MESSAGE,-} from "./lib/retrieval-health.mjs"+import { createRetrievalHealth, isUngrounded, UNGROUNDED_CLIENT_MESSAGE } from "./lib/retrieval-health.mjs"+const BACKEND = process.env.CHAT_BACKEND || "claude"+if (!["claude", "codex"].includes(BACKEND)) throw new Error("CHAT_BACKEND must be claude or codex")+const { query, tool, createSdkMcpServer } = BACKEND === "claude"+ ? await import("@anthropic-ai/claude-agent-sdk") : {}const PORT = parseInt(process.env.AGENT_PORT || "30010", 10)const SEARCH_URL = process.env.PIXELRAG_SEARCH_URL || "https://api.pixelrag.ai"const MAX_BUDGET = parseFloat(process.env.CHAT_MAX_BUDGET_USD || "2.00")@@ -90,122 +89,6 @@ function log(...args) {console.log(new Date().toISOString(), ...args)}-// An unreachable index is not information for the model to work around: it-// ends the turn. Recorded on the turn's health so the stream can close as-// degraded instead of done.-function retrievalDown(health, onEvent, label, detail) {- recordRetrieval(health, RETRIEVAL_BACKEND_DOWN, detail)- onEvent("search_unavailable", { query: label, reason: detail })- return { content: [{ type: "text", text: backendDownInstruction(detail) }], isError: true }-}--function safeDecodeURIComponent(str) {- try {- return decodeURIComponent(str)- } catch {- return str- }-}--function createTools(onEvent, uploadedImage, health) {- const searchTool = tool(- "pixelrag_search",- "Search the visual Wikipedia index by text, by the user's uploaded image, or BOTH combined. When the user uploaded an image, you MUST set use_uploaded_image=true AND provide a text query to get joint image+text retrieval — this gives the best results. Returns ranked results with article URLs, tile positions, and `pages` — the article's valid tile:chunk ranges (e.g. '0:0-7,1:0-4' = tile 0 has chunks 0-7, tile 1 has chunks 0-4). Use this first, then pixelrag_tile to view tiles.",- {- query: z.string().optional().describe("Natural language search query. Omit only when searching purely by an uploaded image."),- use_uploaded_image: z.boolean().optional().describe("Set true to include the user's uploaded image in the search (visual similarity). ALWAYS combine with a text query for best results — set this AND provide a query string in the same call."),- n_results: z.number().int().min(1).max(20).optional().describe("Number of results (default 5)"),- },- async (args) => {- if (args.use_uploaded_image && !uploadedImage) {- return { content: [{ type: "text", text: "No image was uploaded in this conversation — use a text query instead." }] }- }- const searchByImage = Boolean(args.use_uploaded_image && uploadedImage)- if (!searchByImage && !args.query) {- return { content: [{ type: "text", text: "Provide a text query, or set use_uploaded_image:true when the user uploaded an image." }] }- }- // Text and image can be combined in one query for joint image+text retrieval.- const queryObj = {}- if (searchByImage) queryObj.image = uploadedImage- if (args.query) queryObj.text = args.query- const label = searchByImage && args.query ? `${args.query} + uploaded image` : args.query || "uploaded image"- onEvent("searching", { query: label })- let resp- try {- resp = await fetch(`${SEARCH_URL}/search`, {- method: "POST",- headers: { "Content-Type": "application/json", "X-Source": "chat" },- body: JSON.stringify({ queries: [queryObj], n_docs: args.n_results ?? 5, articles_only: true }),- signal: AbortSignal.timeout(30000),- })- } catch (err) {- return retrievalDown(health, onEvent, label, String(err))- }- const outcome = classifyStatus(resp.status)- if (outcome === RETRIEVAL_BACKEND_DOWN) {- return retrievalDown(health, onEvent, label, `HTTP ${resp.status}`)- }- if (outcome !== RETRIEVAL_OK) {- recordRetrieval(health, outcome, `HTTP ${resp.status}`)- return { content: [{ type: "text", text: `Search API error: ${resp.status}` }] }- }- recordRetrieval(health, RETRIEVAL_OK)- const data = await resp.json()- const hits = data.results?.[0]?.hits ?? []- const results = hits.map((h) => {- const slug = h.url.includes("/wiki/") ? h.url.split("/wiki/").pop() : h.url- return {- title: safeDecodeURIComponent(slug || "").replace(/_/g, " "),- url: h.url.startsWith("http") ? h.url : `https://en.wikipedia.org/wiki/${slug}`,- score: Math.round(h.score * 1000) / 1000,- article_id: h.article_id,- tile_index: h.tile_index,- chunk_index: h.chunk_index,- pages: h.article_pages,- }- })- onEvent("search_results", { query: label, hits })- return {- content: [{ type: "text", text: JSON.stringify({ query: label, results, count: results.length }, null, 2) }],- }- }- )-- const tileTool = tool(- "pixelrag_tile",- "View a Wikipedia screenshot tile by its coordinates. Returns the tile as an image so you can read the visual content. Only request coordinates within the article's `pages` ranges from search results (e.g. pages '0:0-7,1:0-4' means tile 1 ends at chunk 4) — coordinates beyond them do not exist.",- {- article_id: z.number().int().describe("Article ID from search results"),- tile_index: z.number().int().describe("Tile index from search results"),- chunk_index: z.number().int().describe("Chunk index from search results"),- },- async (args) => {- const tileUrl = `${SEARCH_URL}/tile/${args.article_id}/${args.tile_index}/${args.chunk_index}`- try {- const resp = await fetch(tileUrl, { signal: AbortSignal.timeout(30000) })- // The agent pages through articles by guessing chunk coordinates, so- // 404s are normal exploration — only surface tiles that actually load,- // otherwise the chat gallery renders broken images.- if (!resp.ok) {- if (classifyStatus(resp.status) === RETRIEVAL_BACKEND_DOWN) {- recordRetrieval(health, RETRIEVAL_BACKEND_DOWN, `tile HTTP ${resp.status}`)- }- return { content: [{ type: "text", text: `Tile not found: ${resp.status}` }] }- }- onEvent("viewing_tile", { article_id: args.article_id, tile_index: args.tile_index, chunk_index: args.chunk_index })- const buffer = await resp.arrayBuffer()- const base64 = Buffer.from(buffer).toString("base64")- const mimeType = resp.headers.get("content-type") || "image/png"- return { content: [{ type: "image", data: base64, mimeType }] }- } catch (err) {- recordRetrieval(health, classifyThrow(err), `tile ${err}`)- return { content: [{ type: "text", text: `Failed to fetch tile: ${err}` }] }- }- }- )-- return [searchTool, tileTool]-}function sse(event, data) {return `event: ${event}\ndata: ${JSON.stringify(data)}\n\n`@@ -220,7 +103,7 @@ const server = http.createServer(async (req, res) => {if (req.method === "GET" && req.url === "/health") {res.writeHead(200, { "Content-Type": "application/json" })- res.end(JSON.stringify({ status: "ok" }))+ res.end(JSON.stringify({ status: "ok", backend: BACKEND }))return}@@ -316,40 +199,47 @@ const server = http.createServer(async (req, res) => {const send = (event, data) => res.write(sse(event, data))const uploadedImage = last?.image && typeof last.image === "string" ? last.image : nullconst health = createRetrievalHealth()- const tools = createTools(send, uploadedImage, health)- const mcpServer = createSdkMcpServer({ name: "pixelrag", version: "1.0.0", tools })+ const tools = createTools(send, uploadedImage, health, BACKEND === "codex" ? codexTool : tool, SEARCH_URL)+ const controller = new AbortController()+ res.on("close", () => controller.abort())inFlight++let sentText = falsetry {- for await (const message of query({- prompt,- options: {- systemPrompt: SYSTEM_PROMPT,- mcpServers: { pixelrag: mcpServer },- allowedTools: ["mcp__pixelrag__pixelrag_search", "mcp__pixelrag__pixelrag_tile"],- maxTurns: MAX_TURNS,- maxBudgetUsd: MAX_BUDGET,- maxThinkingTokens: THINKING_TOKENS,- includePartialMessages: true,- model: "sonnet",- },- })) {- // Stream extended-thinking deltas (Claude Code-style reasoning trace)- if (message.type === "stream_event") {- const ev = message.event- if (ev?.type === "content_block_delta" && ev.delta?.type === "thinking_delta") {- send("thinking", { text: ev.delta.thinking })+ if (BACKEND === "codex") {+ await runCodex({ textPrompt, uploadedImage, systemPrompt: SYSTEM_PROMPT, tools, send, signal: controller.signal, maxToolCalls: MAX_TURNS })+ sentText = true+ } else {+ const mcpServer = createSdkMcpServer({ name: "pixelrag", version: "1.0.0", tools })+ for await (const message of query({+ prompt,+ options: {+ systemPrompt: SYSTEM_PROMPT,+ mcpServers: { pixelrag: mcpServer },+ allowedTools: ["mcp__pixelrag__pixelrag_search", "mcp__pixelrag__pixelrag_tile"],+ maxTurns: MAX_TURNS,+ maxBudgetUsd: MAX_BUDGET,+ maxThinkingTokens: THINKING_TOKENS,+ includePartialMessages: true,+ model: "sonnet",+ },+ })) {+ // Stream extended-thinking deltas (Claude Code-style reasoning trace)+ if (message.type === "stream_event") {+ const ev = message.event+ if (ev?.type === "content_block_delta" && ev.delta?.type === "thinking_delta") {+ send("thinking", { text: ev.delta.thinking })+ }+ continue}- continue- }- if (message.type === "assistant" && message.message) {- for (const block of message.message.content) {- if (block.type === "text" && block.text) { send("text", { text: block.text }); sentText = true }+ if (message.type === "assistant" && message.message) {+ for (const block of message.message.content) {+ if (block.type === "text" && block.text) { send("text", { text: block.text }); sentText = true }+ }+ }+ if (message.type === "result" && message.subtype === "success" && !sentText) {+ send("text", { text: message.result })}- }- if (message.type === "result" && message.subtype === "success" && !sentText) {- send("text", { text: message.result })}}const elapsed = ((Date.now() - t0) / 1000).toFixed(1)@@ -387,5 +277,5 @@ const server = http.createServer(async (req, res) => {})server.listen(PORT, () => {- log(`PixelRAG agent server on :${PORT} → search ${SEARCH_URL}, budget $${MAX_BUDGET}/conv`)+ log(`PixelRAG ${BACKEND} agent server on :${PORT} → search ${SEARCH_URL}, ${BACKEND === "codex" ? "Codex timeout " + (process.env.CHAT_CODEX_TIMEOUT_MS || 180000) + "ms" : "budget $" + MAX_BUDGET + "/conv"}`)})diff --git a/web/lib/agent-tools.mjs b/web/lib/agent-tools.mjsnew file mode 100644index 0000000..d947223--- /dev/null+++ b/web/lib/agent-tools.mjs@@ -0,0 +1,119 @@+import { z } from "zod"+import { RETRIEVAL_OK, RETRIEVAL_BACKEND_DOWN, classifyStatus, classifyThrow, recordRetrieval, backendDownInstruction } from "./retrieval-health.mjs"++// An unreachable index is not information for the model to work around: it+// ends the turn. Recorded on the turn's health so the stream can close as+// degraded instead of done.+function retrievalDown(health, onEvent, label, detail) {+ recordRetrieval(health, RETRIEVAL_BACKEND_DOWN, detail)+ onEvent("search_unavailable", { query: label, reason: detail })+ return { content: [{ type: "text", text: backendDownInstruction(detail) }], isError: true }+}++function safeDecodeURIComponent(str) {+ try {+ return decodeURIComponent(str)+ } catch {+ return str+ }+}++export function createTools(onEvent, uploadedImage, health, tool, SEARCH_URL) {+ const searchTool = tool(+ "pixelrag_search",+ "Search the visual Wikipedia index by text, by the user's uploaded image, or BOTH combined. When the user uploaded an image, you MUST set use_uploaded_image=true AND provide a text query to get joint image+text retrieval — this gives the best results. Returns ranked results with article URLs, tile positions, and `pages` — the article's valid tile:chunk ranges (e.g. '0:0-7,1:0-4' = tile 0 has chunks 0-7, tile 1 has chunks 0-4). Use this first, then pixelrag_tile to view tiles.",+ {+ query: z.string().optional().describe("Natural language search query. Omit only when searching purely by an uploaded image."),+ use_uploaded_image: z.boolean().optional().describe("Set true to include the user's uploaded image in the search (visual similarity). ALWAYS combine with a text query for best results — set this AND provide a query string in the same call."),+ n_results: z.number().int().min(1).max(20).optional().describe("Number of results (default 5)"),+ },+ async (args) => {+ if (args.use_uploaded_image && !uploadedImage) {+ return { content: [{ type: "text", text: "No image was uploaded in this conversation — use a text query instead." }] }+ }+ const searchByImage = Boolean(args.use_uploaded_image && uploadedImage)+ if (!searchByImage && !args.query) {+ return { content: [{ type: "text", text: "Provide a text query, or set use_uploaded_image:true when the user uploaded an image." }] }+ }+ // Text and image can be combined in one query for joint image+text retrieval.+ const queryObj = {}+ if (searchByImage) queryObj.image = uploadedImage+ if (args.query) queryObj.text = args.query+ const label = searchByImage && args.query ? `${args.query} + uploaded image` : args.query || "uploaded image"+ onEvent("searching", { query: label })+ let resp+ try {+ resp = await fetch(`${SEARCH_URL}/search`, {+ method: "POST",+ headers: { "Content-Type": "application/json", "X-Source": "chat" },+ body: JSON.stringify({ queries: [queryObj], n_docs: args.n_results ?? 5, articles_only: true }),+ signal: AbortSignal.timeout(30000),+ })+ } catch (err) {+ return retrievalDown(health, onEvent, label, String(err))+ }+ const outcome = classifyStatus(resp.status)+ if (outcome === RETRIEVAL_BACKEND_DOWN) {+ return retrievalDown(health, onEvent, label, `HTTP ${resp.status}`)+ }+ if (outcome !== RETRIEVAL_OK) {+ recordRetrieval(health, outcome, `HTTP ${resp.status}`)+ return { content: [{ type: "text", text: `Search API error: ${resp.status}` }] }+ }+ recordRetrieval(health, RETRIEVAL_OK)+ const data = await resp.json()+ const hits = data.results?.[0]?.hits ?? []+ const results = hits.map((h) => {+ const slug = h.url.includes("/wiki/") ? h.url.split("/wiki/").pop() : h.url+ return {+ title: safeDecodeURIComponent(slug || "").replace(/_/g, " "),+ url: h.url.startsWith("http") ? h.url : `https://en.wikipedia.org/wiki/${slug}`,+ score: Math.round(h.score * 1000) / 1000,+ article_id: h.article_id,+ tile_index: h.tile_index,+ chunk_index: h.chunk_index,+ pages: h.article_pages,+ }+ })+ onEvent("search_results", { query: label, hits })+ return {+ content: [{ type: "text", text: JSON.stringify({ query: label, results, count: results.length }, null, 2) }],+ }+ }+ )++ const tileTool = tool(+ "pixelrag_tile",+ "View a Wikipedia screenshot tile by its coordinates. Returns the tile as an image so you can read the visual content. Only request coordinates within the article's `pages` ranges from search results (e.g. pages '0:0-7,1:0-4' means tile 1 ends at chunk 4) — coordinates beyond them do not exist.",+ {+ article_id: z.number().int().describe("Article ID from search results"),+ tile_index: z.number().int().describe("Tile index from search results"),+ chunk_index: z.number().int().describe("Chunk index from search results"),+ },+ async (args) => {+ const tileUrl = `${SEARCH_URL}/tile/${args.article_id}/${args.tile_index}/${args.chunk_index}`+ try {+ const resp = await fetch(tileUrl, { signal: AbortSignal.timeout(30000) })+ // The agent pages through articles by guessing chunk coordinates, so+ // 404s are normal exploration — only surface tiles that actually load,+ // otherwise the chat gallery renders broken images.+ if (!resp.ok) {+ if (classifyStatus(resp.status) === RETRIEVAL_BACKEND_DOWN) {+ recordRetrieval(health, RETRIEVAL_BACKEND_DOWN, `tile HTTP ${resp.status}`)+ }+ return { content: [{ type: "text", text: `Tile not found: ${resp.status}` }] }+ }+ onEvent("viewing_tile", { article_id: args.article_id, tile_index: args.tile_index, chunk_index: args.chunk_index })+ const buffer = await resp.arrayBuffer()+ const base64 = Buffer.from(buffer).toString("base64")+ const mimeType = resp.headers.get("content-type") || "image/png"+ return { content: [{ type: "image", data: base64, mimeType }] }+ } catch (err) {+ recordRetrieval(health, classifyThrow(err), `tile ${err}`)+ return { content: [{ type: "text", text: `Failed to fetch tile: ${err}` }] }+ }+ }+ )++ return [searchTool, tileTool]+}diff --git a/web/lib/codex-backend.mjs b/web/lib/codex-backend.mjsnew file mode 100644index 0000000..bb879f3--- /dev/null+++ b/web/lib/codex-backend.mjs@@ -0,0 +1,118 @@+import http from "node:http"+import { spawn } from "node:child_process"+import { randomBytes } from "node:crypto"+import { mkdtemp, writeFile, rm } from "node:fs/promises"+import { tmpdir } from "node:os"+import { join } from "node:path"+import { fileURLToPath } from "node:url"+import readline from "node:readline"+import { z } from "zod"++export function codexTool(name, description, shape, handler) {+ const schema = z.object(shape)+ return { name, description, inputSchema: z.toJSONSchema(schema), annotations: { readOnlyHint: true, destructiveHint: false, openWorldHint: false }, call: (args) => handler(schema.parse(args)) }+}++export async function runCodex({ textPrompt, uploadedImage, systemPrompt, tools, send, signal, maxToolCalls = 12, onDiagnostic = () => {} }) {+ const dir = await mkdtemp(join(tmpdir(), "pixelrag-codex-"))+ const token = randomBytes(32).toString("hex")+ let calls = 0+ let child+ let timedOut = false+ let exited = false+ const bridge = http.createServer(async (req, res) => {+ const respond = (status, data) => { res.writeHead(status, { "Content-Type": "application/json" }); res.end(JSON.stringify(data)) }+ if (req.method !== "POST" || req.headers.authorization !== `Bearer ${token}`) return respond(403, {})+ try {+ let body = ""+ for await (const chunk of req) {+ body += chunk+ if (body.length > 65536) return respond(413, {})+ }+ const request = JSON.parse(body)+ if (request.method === "tools/list") return respond(200, { tools: tools.map(({ name, description, inputSchema, annotations }) => ({ name, description, inputSchema, annotations })) })+ if (request.method !== "tools/call") return respond(400, {})+ if (++calls > maxToolCalls) return respond(200, { isError: true, content: [{ type: "text", text: "Tool limit reached. Finish the answer using the tiles already read." }] })+ const tool = tools.find((t) => t.name === request.params?.name)+ if (!tool) throw new Error("Unknown tool")+ respond(200, await tool.call(request.params.arguments || {}))+ } catch (error) {+ respond(200, { isError: true, content: [{ type: "text", text: String(error) }] })+ }+ })+ let timer+ let forceKill+ const cancel = () => {+ if (child && !exited) {+ child.kill("SIGTERM")+ if (!forceKill) forceKill = setTimeout(() => { if (!exited) child.kill("SIGKILL") }, 2000)+ }+ }+ try {+ await new Promise((resolve, reject) => { bridge.once("error", reject); bridge.listen(0, "127.0.0.1", resolve) })+ const endpoint = `http://127.0.0.1:${bridge.address().port}`+ const args = ["exec", "--ignore-user-config", "--ephemeral", "--skip-git-repo-check", "--json", "--sandbox", "read-only", "--cd", dir,+ "--disable", "shell_tool", "--disable", "code_mode", "--enable", "code_mode_host", "--disable", "multi_agent", "--disable", "apps", "--disable", "plugins", "--disable", "view_image", "--disable", "browser_use_external", "--disable", "browser_use_full_cdp_access", "--disable", "browser_use", "--disable", "computer_use", "--disable", "image_generation",+ "-c", "approval_policy=\"never\"", "-c", "web_search=\"disabled\"",+ "-c", `developer_instructions=${JSON.stringify(systemPrompt)}`,+ "-c", `mcp_servers.pixelrag.command=${JSON.stringify(process.execPath)}`,+ "-c", `mcp_servers.pixelrag.args=${JSON.stringify([fileURLToPath(new URL("./codex-mcp.mjs", import.meta.url)), endpoint, token])}`,+ "-c", "mcp_servers.pixelrag.required=true",+ "-c", 'mcp_servers.pixelrag.tools.pixelrag_search.approval_mode="approve"',+ "-c", 'mcp_servers.pixelrag.tools.pixelrag_tile.approval_mode="approve"']+ if (process.env.CHAT_CODEX_MODEL) args.push("--model", process.env.CHAT_CODEX_MODEL)+ if (uploadedImage) {+ const match = uploadedImage.match(/^data:(image\/(?:png|jpeg|webp|gif));base64,(.+)$/is)+ if (!match) throw new Error("Unsupported uploaded image; use a PNG, JPEG, WebP, or GIF data URL")+ const path = join(dir, "uploaded-image")+ await writeFile(path, Buffer.from(match[2], "base64"), { mode: 0o600 })+ args.push("--image", path)+ }+ args.push("-")+ if (signal?.aborted) throw new Error("Chat cancelled")+ child = spawn(process.env.CODEX_BIN || "codex", args, { cwd: dir, stdio: ["pipe", "pipe", "pipe"] })+ let diagnostics = ""+ child.stderr.on("data", (chunk) => {+ const safe = String(chunk).replaceAll(token, "[redacted]")+ diagnostics = (diagnostics + safe).slice(-4000)+ onDiagnostic(safe)+ })+ const completed = new Promise((resolve, reject) => {+ child.once("error", reject)+ child.once("close", (code) => {+ exited = true+ clearTimeout(forceKill)+ code === 0 ? resolve() : reject(new Error(signal?.aborted ? "Chat cancelled" : timedOut ? "Codex request timed out" : `Codex failed (${code}): ${diagnostics}`))+ })+ })+ // Attach a rejection handler immediately, before consuming stdout.+ completed.catch(() => {})+ signal?.addEventListener("abort", cancel, { once: true })+ timer = setTimeout(() => { timedOut = true; cancel() }, Number(process.env.CHAT_CODEX_TIMEOUT_MS || 180000))+ child.stdin.on("error", () => {})+ child.stdin.end(textPrompt)+ let sentText = false+ let failure+ for await (const line of readline.createInterface({ input: child.stdout })) {+ let event+ try { event = JSON.parse(line) } catch { continue }+ if (event.type === "item.completed" && event.item?.type === "agent_message" && event.item.text) {+ send("text", { text: event.item.text }); sentText = true+ } else if (event.type === "item.completed" && event.item?.type === "reasoning" && event.item.text) {+ send("thinking", { text: event.item.text })+ } else if (event.type === "turn.failed" || event.type === "error") {+ failure = event.error?.message || event.message || "Codex turn failed"+ }+ }+ await completed+ if (failure) throw new Error(failure)+ if (!sentText) throw new Error("Codex returned no answer")+ } finally {+ clearTimeout(timer)+ signal?.removeEventListener("abort", cancel)+ cancel()+ bridge.closeAllConnections()+ await new Promise((resolve) => bridge.close(resolve))+ await rm(dir, { recursive: true, force: true })+ }+}diff --git a/web/lib/codex-backend.test.mjs b/web/lib/codex-backend.test.mjsnew file mode 100644index 0000000..cb1df5c--- /dev/null+++ b/web/lib/codex-backend.test.mjs@@ -0,0 +1,101 @@+import test from "node:test"+import assert from "node:assert/strict"+import { mkdtemp, writeFile, rm } from "node:fs/promises"+import { join } from "node:path"+import { tmpdir } from "node:os"+import { z } from "zod"+import { runCodex, codexTool } from "./codex-backend.mjs"++// A fake CLI uses the actual stdio adapter, making this test cover the MCP+// protocol, tool validation, and SSE event conversion without model calls.+const fakeCli = `+import {spawn} from 'node:child_process';+import readline from 'node:readline';+import assert from 'node:assert/strict';+import {readFile} from 'node:fs/promises';+const args=process.argv.slice(2);+assert(args.includes('read-only'));+assert(args.includes('shell_tool'));+if(args.includes('--image')) assert((await readFile(args[args.indexOf('--image')+1])).length>0);+const config=args.find(x=>x.startsWith('mcp_servers.pixelrag.args='));+const bridgeArgs=JSON.parse(config.slice(config.indexOf('=')+1));+const child=spawn(process.execPath,bridgeArgs,{stdio:['pipe','pipe','inherit']});+const lines=readline.createInterface({input:child.stdout});+const iterator=lines[Symbol.asyncIterator]();+let id=0;+async function call(method,params={}) {+ child.stdin.write(JSON.stringify({jsonrpc:'2.0',id:++id,method,params})+'\\n');+ return JSON.parse((await iterator.next()).value).result;+}+try {+ const init=await call('initialize',{protocolVersion:'2024-11-05'});+ assert.equal(init.serverInfo.name,'pixelrag');+ child.stdin.write(JSON.stringify({jsonrpc:'2.0',method:'notifications/initialized'})+'\\n');+ const list=await call('tools/list');+ assert.equal(list.tools[0].name,'pixelrag_search');+ const result=await call('tools/call',{name:'pixelrag_search',arguments:{query:'Taj Mahal'}});+ console.log(JSON.stringify({type:'item.completed',item:{type:'reasoning',text:'Reading tiles'}}));+ console.log(JSON.stringify({type:'item.completed',item:{type:'agent_message',text:JSON.stringify(result)}}));+ if(process.env.PIXELRAG_TEST_FAILURE) console.log(JSON.stringify({type:'turn.failed',error:{message:'test failure'}}));+} finally {child.stdin.end();lines.close();}+`++test("Codex CLI adapter calls MCP tools and emits SSE-compatible events", async () => {+ const dir = await mkdtemp(join(tmpdir(), "codex-test-"))+ const cli = join(dir, "cli.mjs")+ const previous = process.env.CODEX_BIN+ try {+ await writeFile(cli, `#!${process.execPath}\n${fakeCli}`, { mode: 0o700 })+ process.env.CODEX_BIN = cli+ const events = []+ const options = {+ textPrompt: "What is the Taj Mahal?", systemPrompt: "Read tiles.",+ tools: [codexTool("pixelrag_search", "Search", { query: z.string() }, async (args) => {+ events.push(["searching", args])+ return { content: [{ type: "text", text: "Retrieved " + args.query }] }+ })],+ send: (event, data) => events.push([event, data]),+ }+ await runCodex(options)+ assert.equal(events[0][0], "searching")+ assert.equal(events[1][0], "thinking")+ assert.match(events[2][1].text, /Retrieved Taj Mahal/)+ process.env.PIXELRAG_TEST_FAILURE = "1"+ await assert.rejects(runCodex(options), /test failure/)+ delete process.env.PIXELRAG_TEST_FAILURE+ const controller = new AbortController()+ controller.abort()+ await assert.rejects(runCodex({ ...options, signal: controller.signal }), /cancelled/)+ await assert.rejects(runCodex({ ...options, uploadedImage: "invalid image" }), /Unsupported uploaded image/)+ await runCodex({ ...options, uploadedImage: "data:image/png;base64,iVBORw0KGgo=" })+ events.length = 0+ await runCodex({ ...options, maxToolCalls: 0 })+ assert.match(events.at(-1)[1].text, /Tool limit reached/)+ assert.equal(events.filter(([event]) => event === "searching").length, 0)+ // Exercise cancellation of an already-running subprocess and the timeout.+ await writeFile(cli, `#!${process.execPath}\nsetInterval(() => {}, 1000)`, { mode: 0o700 })+ const running = new AbortController()+ const cancelTimer = setTimeout(() => running.abort(), 100)+ await assert.rejects(runCodex({ ...options, signal: running.signal }), /cancelled/)+ clearTimeout(cancelTimer)+ process.env.CHAT_CODEX_TIMEOUT_MS = "100"+ await assert.rejects(runCodex(options), /timed out/)+ delete process.env.CHAT_CODEX_TIMEOUT_MS+ process.env.CODEX_BIN = join(dir, "nonexistent-cli")+ await assert.rejects(runCodex(options), /ENOENT/)+ } finally {+ if (previous === undefined) delete process.env.CODEX_BIN+ else process.env.CODEX_BIN = previous+ delete process.env.PIXELRAG_TEST_FAILURE+ delete process.env.CHAT_CODEX_TIMEOUT_MS+ await rm(dir, { recursive: true, force: true })+ }+})++test("MCP schemas reject invalid tile coordinates before invoking retrieval", async () => {+ let called = false+ const tool = codexTool("pixelrag_tile", "Read tile", { article_id: z.number().int() }, async () => { called = true })+ assert.equal(tool.inputSchema.type, "object")+ assert.throws(() => tool.call({ article_id: "wrong" }))+ assert.equal(called, false)+})diff --git a/web/lib/codex-mcp.mjs b/web/lib/codex-mcp.mjsnew file mode 100644index 0000000..d9dc95d--- /dev/null+++ b/web/lib/codex-mcp.mjs@@ -0,0 +1,29 @@+// A stdio MCP adapter for a single conversation's local tool endpoint.+// No filesystem, shell, or arbitrary URL tools are exposed to the model.+import readline from "node:readline"++const [endpoint, token] = process.argv.slice(2)+const reply = (id, result) => process.stdout.write(JSON.stringify({ jsonrpc: "2.0", id, result }) + "\n")+for await (const line of readline.createInterface({ input: process.stdin })) {+ let request+ try {+ request = JSON.parse(line)+ if (request.id === undefined) continue+ if (request.method === "initialize") {+ reply(request.id, { protocolVersion: request.params.protocolVersion, capabilities: { tools: {} }, serverInfo: { name: "pixelrag", version: "1.0.0" } })+ } else if (request.method === "ping") {+ reply(request.id, {})+ } else if (["tools/list", "tools/call"].includes(request.method)) {+ const response = await fetch(endpoint, {+ method: "POST", headers: { "Content-Type": "application/json", Authorization: `Bearer ${token}` },+ body: JSON.stringify(request), signal: AbortSignal.timeout(65000),+ })+ if (!response.ok) throw new Error(`Tool endpoint HTTP ${response.status}`)+ reply(request.id, await response.json())+ } else {+ process.stdout.write(JSON.stringify({ jsonrpc: "2.0", id: request.id, error: { code: -32601, message: "Method not found" } }) + "\n")+ }+ } catch (error) {+ if (request?.id !== undefined) process.stdout.write(JSON.stringify({ jsonrpc: "2.0", id: request.id, error: { code: -32603, message: String(error) } }) + "\n")+ }+}