Make your computer a home for your AI partner. Avibe is the local-first Agent OS: Vibey, its built-in AI partner, works out of the box, and Claude Code, Codex & OpenCode run unchanged beside it — reach them from your browser, phone, or any chat app.
Python agentagent-osaiai-agentsai-partnerchatopsclaudeclaude-codecodexdevtoolsdiscord-botlark-botllmlocal-firstlong-term-memoryopencodeslack-botvibe-codingvibeywechat
feat(benchmark): evaluate full Avibe runtime with owned remote jobs (#2627)
Call Whiteaed967b61a97
diff --git a/core/agent_core/agent/loop.py b/core/agent_core/agent/loop.pyindex 923b4b9ab..b7a0ab90f 100644--- a/core/agent_core/agent/loop.py+++ b/core/agent_core/agent/loop.py@@ -926,12 +926,16 @@ class Agent:)response, final = await self._commit_model_message(message, emit)empty_reply = final and not has_reply(message)- if empty_reply:+ truncated_reply = final and message.stop_reason == "length"+ if empty_reply or truncated_reply:decided = self._outcome.primary("error")await self._error(emit,- message.stop_reason if message.stop_reason in {"refusal", "safety"} else "empty_response",- message.error_message or f"Model stopped with {message.stop_reason} without a reply.",+ message.stop_reason if message.stop_reason in {"length", "refusal", "safety"} else "empty_response",+ message.error_message or (+ "The model reached its output limit before completing the reply."+ if truncated_reply else f"Model stopped with {message.stop_reason} without a reply."+ ),"source",cause=decided,)@@ -982,7 +986,7 @@ class Agent:return "aborted" if message.stop_reason == "aborted" else "error"if not message.tool_calls:await self._save_state()- if empty_reply:+ if empty_reply or truncated_reply:return "error"if end:return "ended_by_hook"diff --git a/docs/plans/2026-10-14-terminal-bench-reliability.md b/docs/plans/2026-10-14-terminal-bench-reliability.mdnew file mode 100644index 000000000..81f27c667--- /dev/null+++ b/docs/plans/2026-10-14-terminal-bench-reliability.md@@ -0,0 +1,123 @@+# Terminal-Bench reliability and failure diagnosis++## Scope and evidence++Continue the full-runtime benchmark branch; preserve the original smoke and+paired-pilot artifacts. The installed-package pilot ended with a real failed+Run and verifier reward 0. Its summary says `empty_response`, but the retained+artifacts do not contain the final provider stop reason or response shape.+The reference lost its Harbor owner without a terminal trial receipt and left+an orphan task container. Neither observation proves model capability or the+historical root cause. No additional paid trial is authorized by this repair.++The prior owner has explicitly stopped writing. This session is the sole writer;+independent audits are read-only. All execution tests run in the user-authorized,+dedicated remote benchmark environment, never the local VM or another tenant.+Do not modify shared host services, resource caps, provider budgets, official+tasks/verifiers, or old results.++## Repair contract++1. Reproduce final empty, reasoning-only, and output-truncated provider responses+ with scripted, zero-paid input. Preserve the distinction between a provider+ stop with no answer and an explicit output-limit stop. Do not silently increase+ output limits, retry a whole task, replay successful tools, or mark incomplete+ work successful. Any core change needs a failing before/fixed after test and+ must agree with the shared loop contract, not special-case a benchmark/model.+2. Add a bounded diagnostic projection to the wrapper's existing `execution.json`.+ Read through normal product CLI/API, not raw SQLite/transcript files. Export+ only allowlisted error/stop categories, response block counts/lengths, reported+ token usage and completeness status. No raw prompts, reasoning text, provider+ keys, source configs, raw logs or internal database copies. Missing diagnostics+ must be explicit, must not change the real Run outcome, and must not block+ cleanup. Capture before the owned service is stopped. Keep old receipts intact.+3. Add a source-only Linux evaluation-host supervisor around Harbor. A systemd+ task-owned unit owns its process tree independently of SSH; Avibe Harness+ observes the durable receipt rather than re-executing the paid command. Claim+ a fresh run directory and a single-host benchmark lock before launch; validate+ config paths, wheel hash and limits without a provider call; preserve the input+ config and an explicit local-path relocation manifest. No mutation of product+ dependencies or user/live Avibe state.+4. Every supervisor exit writes a separate orchestration receipt. A dead Harbor+ parent is failure even if a stale result says running; success requires a+ complete trial receipt. Preserve Harbor's originals and never invent reward,+ usage, or verifier completion. A remote hard wall cap is independent of SSH+ and bounds setup/execution/verification; the configured Harbor agent budget is+ unchanged and recorded separately. Stop/delete only containers/networks with+ positively established trial labels, after preserving task artifacts/logs.+ Never broad-prune a Docker daemon or pattern-kill processes.+5. Secrets travel via protected files/stdin, never command arguments or logs.+ Reject missing/empty/wrong-mode credential files before Harbor starts. Consume+ transport files promptly; diagnostics and receipts must pass sentinel scans.++## Diagnosis decision++The read-only audit confirmed the parser preserves `length` vs `stop` and+thinking vs text; recovery does not replay a failed task. A visible tool-free+`length` final was incorrectly considered completed by both the loop and the+transcript outcome function. The orchestrator chooses one consistent final+classification: every final `length` is failed/`length`, including no-answer+shapes, with generic output-limit copy. This deliberately corrects the old+no-answer matrix expectation too, rather than retaining its misleading+`empty_response` label. Ordinary empty `stop` remains failed/`empty_response`;+non-final continuations and bounded tool-call retries are unchanged. No automatic+continuation or larger budget is introduced.++Before-code evidence: the extended no-answer matrix failed four cases with+`empty_response != length` on the dedicated remote host; 14 controls passed.+The historical pilot still lacks evidence identifying its final stop reason.++## Validation gates++- Table-driven provider/loop regressions: empty vs reasoning-only vs length,+ valid tools/text, refusal/interruption and retry budget preservation.+- Diagnostic projection: synthetic secret/text sentinels never exported,+ correct final stop/error shape, bounded query, explicit unavailable state.+- Supervisor: normal terminal result, preflight rejection, nonzero/zero exit with+ missing result, hard timeout, exact-owned cleanup with an untouched bystander,+ and two-submit/single-owner refusal.+- Remote no-paid acceptance: drop SSH after submission; work continues, then+ produces a terminal receipt. Kill the child or hit a short deadline; receipt+ and cleanup still happen. Verify unrelated services were not targeted.+- Focused tests and lint remotely, independent review, then normal PR/CI gates.+ No merge, deployment to the live user runtime, official-task rerun, or expanded+ benchmark batch is included.++## Verified repair evidence++- Core red test: four final no-answer `length` cases reproduced the lost cause+ (`empty_response` rather than `length`); 14 controls passed.+- Core/adapter green: 83 focused cases, including persisted error rows and+ localized length causes for empty, thinking-only and visible partial finals.+- Wrapper/host final targeted pass: 33 cases, including split-secret redaction,+ global ownership, malformed evidence, exact trial identity, systemd expansion+ refusal (also behind symlinks), the Python 3.10 TOML-import fallback, and opaque+ recovery archives. An expanded run passed 1120 of 1122 cases; one missing remote+ schema fixture was synchronized and one stale Run-show error-code assertion+ was corrected. Both failures plus neighboring cases then passed (9 tests).+- Real full-runtime source acceptance: two simulated provider cases (empty stop,+ thinking-only length) both completed failure reporting, retained the distinct+ canonical stop/block diagnostics and proved owned cleanup. No model bill.+- Dedicated remote systemd + real Harbor/Docker acceptance after review fixes:+ SSH submit/exit completed normally; a seven-second hard wall, SIGKILL of the+ supervisor, and SIGKILL of the Harbor child each produced their own terminal+ orchestration receipt, retained fixture artifacts, and left no owned container+ or network. A separate bystander container remained running and was removed+ only by the test owner after all assertions. An additional real Docker case+ recovered overlapping directory/file declarations containing an absolute+ symlink as two opaque tar files; the host sentinel remained unchanged and all+ owned containers/networks were removed. No paid task was repeated.++Review findings fixed as a class: parse failures cannot block cleanup; one+host-wide reservation plus permanent destination claim; post-stop-only receipt+publication; exact trial terminal evidence; total finalization budget; ownership+journal before network creation; absolute-path preflight; profile-bound secret+variable. Original Harbor results and historical failures remain untouched.++## Historical limitations to retain in reporting++- The candidate's final raw response was not retained. A reproduction of one+ failure class does not retroactively prove that class caused this pilot.+- The reference has partial trajectory usage, not a completed score or bill.+- The initial synthetic reward 1 is only integration evidence.+- A successful repair simulation is not a claim of improved benchmark scores.diff --git a/docs/plans/agent-core-contracts/loop-control.md b/docs/plans/agent-core-contracts/loop-control.mdindex 71ac633a3..526b3cf27 100644--- a/docs/plans/agent-core-contracts/loop-control.md+++ b/docs/plans/agent-core-contracts/loop-control.md@@ -46,7 +46,16 @@ commit input → loop:```Finality is decided before the response row is inserted, so a final (`result` or `error`) row is always the run's last-response. A final response with `stop_reason` `refusal` or `safety` and no text is still committed as the final+response. A final response with `stop_reason=length` is incomplete, even when it+contains partial text or an image: commit it as `error`, preserve usable content,+and end with an error of kind `length`. A no-answer length stop is not an+`empty_response`; the upstream already supplied the cause. Ordinary empty or+reasoning-only `stop` responses still fail as `empty_response`. This does not+add a model call, increase an output limit, or replay a task. Non-final responses+with pending steer/follow-up keep continuing normally; truncated tool calls keep+the existing bounded reissue path without executing their partial arguments.++A final response with `stop_reason` `refusal` or `safety` and no text is still committed as the finalresponse, typed `error` ([`transcript.md`](transcript.md) §1); its commit-time display text is the localizedexplanation. (That it is written in the same transaction so that re-delivery after a crash shows it is moot: Avibedelivers committed rows through the shared emit path and nothing re-delivers them, [`recovery.md`](recovery.md)diff --git a/docs/plans/full-avibe-runtime-terminal-bench.md b/docs/plans/full-avibe-runtime-terminal-bench.mdnew file mode 100644index 000000000..cb28bdab8--- /dev/null+++ b/docs/plans/full-avibe-runtime-terminal-bench.md@@ -0,0 +1,145 @@+# Full-runtime Terminal-Bench vertical slice++## Scope and ownership++Worktree: `feature/vibey-terminal-bench-full-runtime`, starting base `9a89da41b`.+This change adds source-only evaluation tooling around the installed product.+The earlier standalone-core experiment is out of scope and is not used for+product benchmark results. Deployment, merge, public leaderboard submission,+and remote/GPU provisioning remain separate decisions.++## Source-backed correction to the investigation++A new embedded Controller or a second Agent loop is unnecessary. Existing paths:++- `main.main()` / installed `vibe.service_main.main()` owns service lock, config,+ migrations, Controller construction, signals, readiness and shutdown.+- Empty `platforms.enabled` is already a supported workbench-only installation.+ `MultiIMClient` keeps that runtime alive; `AvibeBot` is the internal transport.+- `AVIBE_WEB_UI_SUPERVISOR` already suppresses the UI watchdog's UI launch. The+ benchmark supervisor intentionally starts no UI; no browser or tunnel is needed.+- `vibe agent run --sync --cwd ...` creates a real durable Run/Session/delivery and+ reaches the real Controller, MessageHandler, AgentService and VibeyAgent.+- `vibe.model_hub_client.ModelHubRemoteService` already creates Sources and routes+ through the controller-owned IPC API. Do not bootstrap raw credentials or+ model routes by writing private storage, and do not bypass Model Hub.+- `AVIBE_HOME` controls product state, including SQLite, native transcript,+ compaction, scheduler, Watches, and job recovery. There is no separate removed+ memory component to resurrect; the shipped transcript/context implementation+ is the measurement subject.++## First implementation contract++Add only source-side tooling under `scripts/terminal_bench/` and focused tests,+not product wheel dependencies, not a Harbor dependency in the product lock.+The wrapper invokes the normal service entrypoint in an owned subprocess.+All product imports and CLI operations+run in child processes with their own environment; the parent environment and+running user service are never modified.++Per trial:++1. Validate an existing task cwd and claim a *new* private trial root outside it.+ Reject overlap in either direction, existing state, and symlink aliases.+2. Prepare an isolated config with no external messaging platforms, remote pairing,+ updates, native backend launches, or UI. Keep scheduler/Watch/Session machinery:+ disabling Harness would not measure the product. Use explicit local catalog+ fixtures and offline managed-runtime settings in the hermetic slice.+3. Start the real service and require controller IPC health, not merely a PID or+ fabricated readiness file. Do not run the service from the task cwd: task+ `config.py`/`core` modules must not shadow product modules.+4. Configure one Source through Model Hub IPC using a key supplied only on a+ private stdin pipe. Never persist the provider key in a command, wrapper config,+ parent artifact, or task shell environment. Runtime engine custody stays owned+ by the product. No Source means a reported `blocked` bootstrap, not a fake pass.+5. Use the real CLI Run path with explicit Agent/model/effort, task cwd, and+ instruction. Collect the durable run outcome and redacted terminal result.+ A result is not a verifier reward. The optional Harbor adapter runs the+ separate verifier; ATIF export and public submission are not pilot gates.+6. Gracefully terminate the exact owned service and collect its descendants while+ alive. Join/terminate/kill by recorded process identity within bounded grace;+ never use a global `vibe stop`, pattern kill, or the user's runtime records.+ Refuse to mark a trial closed if an owned process survives. Export artifacts+ only after shutdown; never copy raw SQLite, runtime logs, jobs or config.++Task environment and runtime state are separate. On a host, the wrapper uses a+throwaway HOME/XDG to avoid user state. In a sandbox, its caller may explicitly+supply the task image environment; preserve toolchain variables and task HOME,+while stripping inherited Avibe caller routing and credential variables. The+wrapper is not a security sandbox: paid runs require an external isolated+container with no host mounts or secret access. Tasks may write scripts,+SQLite databases, logs, and notes in their normal cwd.++## Tests and evidence++Existing tests do not exercise this source-side supervisor boundary. New focused+pytest cases protect fresh-state refusal, environment/cwd separation, no parent+state mutation, actual service/IPC/Run dispatch, task writes (including Unicode+and SQLite), explicit blocked bootstrap, redaction, and exact-owned shutdown.++The integration test may replace only the external Model Hub *engine adapter*+with a scripted test double, while keeping production Controller, Model Hub+service/router/gateway, native providers, AgentService, Vibey and Harness real.+Report that boundary honestly: it proves local product integration, not real CPA+installation, credential custody, provider connectivity, or a benchmark score.++## Implemented evidence++- `scripts/terminal_bench/{profile,control,runner}.py` and `README.md` are the+ single source-only wrapper. Optional `harbor_agent.py`, `launch.py` and+ `credential.py` adapt its lifecycle and private credential transport to+ Harbor; no extracted product packages are uploaded.+- The initial wrapper left product dependency, service, Agent and storage files+ unchanged. The follow-up reliability repair also changes shared final-response+ classification so an explicit output-limit stop cannot be persisted as success;+ dependency files, service ownership and database schema remain unchanged.+- The subprocess test uses normal `main.py`, IPC Source/catalog/route APIs and+ the real CLI Run path. It proves a native Bash tool writes a Unicode note and+ task SQLite, with no framework files in task cwd; task `config.py` cannot+ shadow product imports. Provider keys never enter the service/tool environment.+- A real product Watch creates a delayed note after the initial Run result.+ The wrapper waits for Harness quiescence instead of prematurely tearing down+ background work. Missing credentials are explicitly blocked, not fake passes.+- The deadline case interrupts a continuously writing real tool, closes the+ runtime, then verifies no later writes. A separate process-lifecycle case+ covers a setsid child, a stale unrelated pid identity, and clock-shifted+ identity without signalling the bystander.+- Exports name the Run/Session, root-Run outcome scope, actual installation,+ reported usage (completeness explicitly unknown), cleanup and null verifier+ reward. Raw product state/logs are not copied.++## Installed-runtime acceptance and remaining gates++An independent container check used the official `avibe-os 3.2.0rc5` wheel+(source `98cc70a1c`, SHA256+`5b1b59cd1cf93f36b00f1bef8490612da180366244e2cee163e0d59977fffea0`), its real+managed Model Hub engine, and a local scripted HTTP upstream. It confirmed+ordinary CLI Run dispatch, native file writes/reads, task-owned SQLite, Harness+quiescence and process cleanup. The full Harbor path subsequently passed its+synthetic separate verifier with reward 1 and zero infrastructure exceptions.+These are integration checks, **not model capability scores**. Source-checkout+unit tests use an external-engine fixture and are a separate evidence layer.++The optional adapter requires Harbor 0.24.0, verifies wheel bytes before upload,+and records the actual wrapper source and installation. It declares `atif=False`;+a local verifier score does not depend on ATIF or official leaderboard submission.+Docker Compose can preserve a host UID on uploaded files; credential setup+adopts only a validated single-link regular inode under its private directory,+while normal consumption retains strict ownership and permission checks.++The first remote paired pilot reached a real candidate verifier (reward 0 after+an empty-reply Run failure); its final raw provider stop was not retained. The+reference lost its owner without terminal verification. These do not form a valid+paired capability comparison. Reliability diagnosis and repair are tracked in+[the follow-up contract](2026-10-14-terminal-bench-reliability.md).++Remaining gates: a separately authorized paired run after reliability acceptance,+cost assessment before larger batches, and current-head review/CI before code+delivery. Benchmark results must name the tested wheel and wrapper revisions,+not assume they match the latest product branch. Preserve failed attempts and+resource/protocol differences; do not extrapolate a small pilot to a leaderboard+rank. See `scripts/terminal_bench/HARBOR.md` for supported images and lifecycle+limitations.++Run only focused tests through `~/.local/bin/hd1-test`; retain product dependency+files unchanged. Any unproven lifecycle/provider capability remains a named gate.diff --git a/modules/agents/vibey/agent.py b/modules/agents/vibey/agent.pyindex fe8820161..b4f0969d3 100644--- a/modules/agents/vibey/agent.py+++ b/modules/agents/vibey/agent.py@@ -121,7 +121,7 @@ logger = logging.getLogger(__name__)BACKEND = "vibey"# Final responses whose empty text is explained by the stop itself (loop-control.md section 2).-_EXPLAINED_STOPS = ("refusal", "safety")+_EXPLAINED_STOPS = ("length", "refusal", "safety")_COMPLETED = ("completed", "ended_by_hook")_JOB_PRUNE_INTERVAL_S = 3600.0diff --git a/modules/agents/vibey/errors.py b/modules/agents/vibey/errors.pyindex e13bf9cf1..cb0a51794 100644--- a/modules/agents/vibey/errors.py+++ b/modules/agents/vibey/errors.py@@ -40,7 +40,7 @@ _KIND_KEYS: dict[str, str] = {# A provider failure C-2 could not classify: an unknown code, or an error frame no rule names."unknown": "unclassified","error": "modelError",- # The model kept truncating a tool call at its output limit (C-3 section 2).+ # An explicit output-limit stop with no answer, or repeatedly truncated tools."length": "outputLimit",# Tool calls under a stop reason that cannot carry them."stop": "providerProtocol",diff --git a/scripts/terminal_bench/HARBOR.md b/scripts/terminal_bench/HARBOR.mdnew file mode 100644index 000000000..a2c19e3d5--- /dev/null+++ b/scripts/terminal_bench/HARBOR.md@@ -0,0 +1,224 @@+# Optional Harbor bridge++`harbor_agent.FullAvibeAgent` is a source-only `BaseInstalledAgent` for the full+runtime wrapper in [README.md](README.md). It declares **`atif=False`**. It does+not claim leaderboard submission or ATIF support; a local official verifier+reward is possible without them.++## Host setup++Use a **separate** evaluation-host environment. Harbor's dependencies must not+change the product `pyproject.toml`, `uv.lock`, or wheel:++```sh+# From this checkout, in a dedicated venv:+uv pip install --python /path/to/eval-venv/bin/python \+ -r scripts/terminal_bench/requirements-harbor.txt+export PYTHONPATH="$(pwd)" # Evaluation HOST only; never passed into the task.+```++Example Harbor job file (replace paths, SHA and profile values explicitly):++```json+{+ "job_name": "avibe-pilot-one",+ "jobs_dir": "/path/to/private-job-artifacts",+ "n_concurrent_trials": 1,+ "n_attempts": 1,+ "agents": [{+ "import_path": "scripts.terminal_bench.harbor_agent:FullAvibeAgent",+ "model_name": "deepseek-v4-pro",+ "override_timeout_sec": 1800,+ "kwargs": {+ "wheel_path": "/path/to/avibe_os-<pinned-version>-py3-none-any.whl",+ "wheel_sha256": "<verified-64-character-lowercase-sha256>",+ "profile": {+ "model": "deepseek-v4-pro",+ "base_url": "https://api.deepseek.com/v1",+ "credential_env": "DEEPSEEK_API_KEY",+ "protocol": "openai_chat",+ "reasoning_effort": "high",+ "context_window": 128000,+ "max_output_tokens": 32000,+ "wall_seconds": 1800,+ "setup_seconds": 600,+ "cleanup_seconds": 30,+ "poll_seconds": 1,+ "offline": false+ }+ }+ }],+ "tasks": [{"path": "/path/to/pinned-official-task"}],+ "environment": {"type": "docker", "delete": true}+}+```++Run with the named credential injected into the evaluation HOST by an authorized+secret injector. Do not put its value in job JSON or argv. Harbor 0.24's Docker+`exec(env=...)` itself expands to `docker compose exec -e KEY=value` on the host,+so this adapter deliberately does **not** pass credential values through exec+env. It reads the named host environment variable via Harbor's `_get_env` and+uses the narrow ephemeral-file transport below instead.++```sh+/path/to/eval-venv/bin/harbor run --config pilot.json --max-retries 0+```++The adapter requires Harbor 0.24.0, a trusted local **ordinary Avibe wheel**, its+explicit SHA256, a Profile, and a matching explicit model. It hashes an immutable+copy **before uploading anything**. It installs the wheel into a private venv,+not editable source. Only seven tracked wrapper files are copied beside it;+product packages, tests, oracle solutions and user home are never source-uploaded.+Wheel version/SHA, actual Harbor version, wrapper commit/hash/dirty state and+execution metadata are recorded. Real paired runs should use a clean committed+checkout and an independently verified pinned wheel.++## Ephemeral API-key transport++This is temporary transport to the ordinary Model Hub Source-create API, **not**+a Vault/key-custody replacement or a generic secret-export feature.++- First pilot supports **root task users only**; nonroot images fail explicitly+ before installation rather than changing task ownership silently.+- Claim a unique trial root and its credential subdirectory mode0700. Create one+ host mode0600 `provider.key` under a mode0700 tempfile directory; transfer only+ with Harbor `upload_file`, never a shell literal, echo, heredoc, env value, or+ JSON configuration. Inspect remote regular-file/type/mode/size metadata+ without printing or comparing secret content in logs. Docker Compose may+ preserve a host UID during upload. The root-only setup helper adopts that+ copied inode using `fchown` only after validating the owned 0700 parent,+ non-symlink 0600 regular file, single link and size bound. Later metadata and+ consumption still require runner ownership; they never adopt an arbitrary file.+- The wrapper opens without following links, rejects nonregular/multiply-linked,+ oversized or improperly protected files, reads and unlinks the input **before+ starting the service**, then uses the existing private-stdin bootstrap. Provider+ keys never enter wrapper/service/tool exec env or argv in this Harbor mode.+- A cancelled upload is joined before cleanup, so late copy completion cannot+ recreate a deleted secret. Upload/read/metadata failures, cancellation and+ timeout clear that exact owned remote file and host tempfile. Unknown remote+ cleanup blocks handoff and requires container destruction. No recursive+ deletion or another trial's path is used.+- Direct wrapper invocation may retain its named-environment mode, but the Docker+ exec bridge always supplies `--credential-file` containing only a private path.++## Task and lifetime boundary++- POSIX/Linux task containers with Python >=3.10 (Harbor may bootstrap Python and+ venv) are the first supported target. Windows is not claimed.+- Setup installs the normal wheel/dependencies once. Model Hub/bootstrap work+ inside `run()` counts against Harbor's outer task deadline: equal wall limits+ are **not** a claim of equal model-active time between Agent implementations.+- Resolve the actual task cwd with `pwd -P`; do not assume `/app` or `/root`.+ The original task HOME/toolchains are retained. A private directory containing+ only the installed `vibe` executable is prefixed to PATH, not the Python venv.+- Source/venv/state live in a unique `/installed-agent/avibe-full-*` directory;+ exported reports are under `/logs/agent/full-runtime-*`. Reject overlapping+ task roots rather than silently contaminating them.+- Model requests still use the real product Model Hub. The adapter does not+ override the official task's network/resources or secretly forbid legitimate+ task Internet use. Supply the provider endpoint as an agent-phase allowance+ in the normal Harbor job/network policy where needed; installer egress belongs+ to setup. Do not change the official task's verifier/artifact declaration.+- Killing a `docker exec` client is insufficient. A small in-container launcher+ records the exact wrapper PID/creation identity/nonce under a locked start+ fence. Cancellation first fences a late start, then signals the exact wrapper,+ waits for it to exit and requires its inner owned-process-cleanup receipt.+ The adapter joins this path before downloading reports or letting Harbor grade.+- A missing/mismatched/unconfirmed shutdown receipt is an infrastructure error,+ **not a task failure score**. The outer owner must destroy that sandbox before+ any grading; forced wrapper death alone cannot prove a detached engine/job+ stopped. Use an init/reaper in the container. A cancelled-before-start fence+ does not invent an execution report.++`execution.json` is projected to Harbor AgentContext metadata and token counts+only when the product reports tokens. Cost stays null (unknown provider bill),+accounting completeness stays explicit, and verifier reward stays separate.+No model text is passed to Harbor's regex-based CLI error classifier. Reports,+not the runner's exit status, identify completion/failure/time limit.++## Failure diagnosis and remote ownership++`execution.json.diagnostics` is a bounded, allowlisted projection fetched via+`vibe data query` before the service stops. It records canonical stop reasons,+text/thinking/tool block counts, character counts and reported token counts for+up to eight recent responses in the root Session's last 32 context messages.+`responses_truncated` and `history_complete=false` make its limits explicit.+No prompt, reasoning text, tool arguments, raw service log or database is exported.+If the query fails, diagnostics say `unavailable`; the real outcome and shutdown+still take precedence. `has_error_message` is presence, not an error taxonomy.++For remote execution, do not keep Harbor owned by an SSH connection. On the+**dedicated Linux evaluation instance**, use the source-only systemd supervisor:++```sh+# Config paths must be absolute ON THE REMOTE INSTANCE. Preserve the original+# config and record any explicit path relocation; do not silently change budgets.+/path/to/harbor-venv/bin/python -m scripts.terminal_bench.remote_job preflight \+ --config /private/bench/job.json+/path/to/harbor-venv/bin/python -m scripts.terminal_bench.remote_job submit \+ --config /private/bench/job.json --root /private/bench/runs/unique-run \+ --credential-file /private/staging/credential/provider.key --wall-seconds 1800+/path/to/harbor-venv/bin/python -m scripts.terminal_bench.remote_job status \+ --root /private/bench/runs/unique-run+```++The credential parent must be owned mode0700 and the regular file mode0600;+the supervisor consumes it before starting Harbor. Full-runtime jobs derive the+environment-variable name from the profile, and explicit mismatches are refused.+For a reference agent specify its provider variable with `--credential-env`.+No secret value enters systemd properties or command arguments. Paths containing+systemd `%` specifiers, `$` expansions or line breaks are refused (including+resolved symlink targets) rather than interpreted by the service manager.++The host-wide `/run/lock/avibe-benchmark.active.json` reservation rejects a second+submission regardless of run-root parent. A permanent per-job destination claim+also prevents accidental reuse after completion. An unconfirmed owner remains a+blocker; never delete its reservation merely to launch another paid attempt.+The supervisor requires Harbor 0.24.0, one local single-step task, one attempt,+zero retries and concurrency one. It keeps the frozen config and wheel hash;+preflight never calls a model. The remote hard wall covers the whole job and is+recorded separately from Harbor's agent-execution budget. It does not secretly+extend setup, output, model or verification budgets.++The pinned Harbor driver's awaited START hook records exact agent and verifier+Compose project labels **before** resources are created, including network-only+startup failures. systemd owns the worker and descendants; ExecStopPost runs+only after their process group is fenced. This phase recovers declared task+artifacts as separate, size/time-bounded **opaque tar files**, never extracts+container-controlled symlinks onto the host, removes only proven-owned containers+and networks, and writes `receipt.json`. Inspect recovered archives in an isolated+workspace; do not blindly extract untrusted paths on the host. Missing/malformed Harbor output cannot skip+cleanup. A zero child exit plus a job timestamp is not enough: completion needs+the one exact trial's terminal receipt as well. Harbor results are never rewritten+and orchestration failure never invents a reward or usage.++Observe `status` with one Avibe Harness Watch (exit75 = still active). A dead unit+without a terminal receipt is an explicit failure, not an endless wait. Reconnect+and inspect after an SSH failure; **do not resubmit** the workload. The supervisor+needs systemd and Docker inside an explicitly authorized dedicated environment;+it does not provision hosts, relax Incus security, restart shared services, or+change other projects. Read-only diagnostics and no-paid lifecycle fixtures are+not benchmark capability scores.++## Focused optional tests++Normal product tests skip the Harbor-only file when Harbor is absent. Run it+explicitly in the isolated optional environment after installing product dev+test prerequisites and `requirements-harbor.txt`:++```sh+~/.local/bin/hd1-test /path/to/eval-venv/bin/python -m pytest -q \+ tests/test_terminal_bench_harbor.py tests/test_terminal_bench_launch.py \+ tests/test_terminal_bench_credential.py+```++The tests use Harbor's actual class/model interfaces, a recording Environment+transport, and owned local processes. They verify wheel mismatch refusal before+mutation, restricted source transfer, explicit model/profile binding, unknown+cost, cancellation join before artifact reads, failed cleanup blocking handoff,+late-start fencing and exact-owned TERM/receipt behavior. Credential cases prove+sentinel absence from all exec argv/env/config and logs, consume-before-service,+stdin-only bootstrap, rejected input shapes and cleanup on failed/cancelled+uploads. They do **not** prove+a Docker install or real engine execution. Those remain the parent's one-task+container smoke gate before a paid paired trial.diff --git a/scripts/terminal_bench/README.md b/scripts/terminal_bench/README.mdnew file mode 100644index 000000000..80f0b55fb--- /dev/null+++ b/scripts/terminal_bench/README.md@@ -0,0 +1,165 @@+# Full Avibe Runtime trial wrapper++Optional Harbor entrypoint:+`scripts.terminal_bench.harbor_agent:FullAvibeAgent` — see [HARBOR.md](HARBOR.md).+It installs a SHA-verified normal wheel, uploads only tracked wrapper files,+declares `atif=False`, and joins owned shutdown before grading. Its ephemeral+credential-file transport avoids Harbor Docker's key-bearing exec-env argv;+the file is consumed/unlinked before service startup. Direct invocation below+may retain the named-environment input mode.++This source-only tool drives **normal Avibe**, not an extracted agent core:++1. a fresh `AVIBE_HOME`, ordinary V2 config and ordinary service main;+2. the real controller-owned Model Hub IPC for Source and route setup;+3. `vibe agent update`, followed by `vibe agent run` and stable Run polling;+4. exact-Run cancellation on deadline and bounded owned-process shutdown;+5. `execution.json`, separate from a later Harbor verifier reward.++The native scheduler, Watches, Session/history/data-query machinery and Vibey+context compaction stay intact. No external IM platform, browser, Cloud pairing,+tunnel or Web UI is started. The loopback/internal Model Hub and control IPC are+real product services. No `Controller`, `Agent`, Session row or provider loop is+reimplemented here. This tool is **not** a sandbox.++## Prerequisites++Use a dedicated interpreter with the ordinary, pinned Avibe wheel installed,+including its normal dependencies. Verify the wheel SHA externally and keep it+in the experiment receipt. The interpreter chosen to launch this wrapper is+also used for the service and CLI. Do not run `uv` in the product repository in a+way that replaces its lock with Harbor dependencies.++For example, inside a fresh task container (with an init/reaper such as Docker's+`--init`), after installing the verified wheel into `/opt/avibe`:++```sh+# /opt/eval contains only this source-side scripts package, NOT a dataset,+# oracle solution, the user's home, or another agent's trajectory.+cd /opt/eval+/opt/avibe/bin/python -m scripts.terminal_bench.runner \+ --workdir /app \+ --state /var/lib/avibe-eval/trial-001 \+ --output /logs/agent/full-runtime \+ --profile /opt/eval/profile.json \+ --instruction /opt/eval/instruction.txt \+ --sandbox-environment+```++`state` and `output` must **not exist** yet. They and `workdir` must be disjoint+(no aliases, nesting, or filesystem root). Do not mount user state or the host+Docker socket. Source-code/test helpers, task instructions, raw service logs,+and model keys must not be collected as task outputs. Review the task's declared+artifacts to ensure they cannot copy the private state directory.++Example profile (freeze before trials; these are pilot limits, not the official+Terminal-Bench protocol):++```json+{+ "model": "deepseek-v4-pro",+ "base_url": "https://api.deepseek.com/v1",+ "credential_env": "DEEPSEEK_API_KEY",+ "protocol": "openai_chat",+ "reasoning_effort": "high",+ "context_window": 128000,+ "max_output_tokens": 32000,+ "wall_seconds": 1200,+ "setup_seconds": 600,+ "cleanup_seconds": 30,+ "poll_seconds": 1,+ "offline": false+}+```++Supply the named credential through an authorized secret injector, not argv or a+profile value. The supervisor passes it only to the short-lived bootstrap helper+on stdin, and that helper uses the ordinary Model Hub API. The service and tool+subprocess environment do not inherit the provider key. Real custody is still+Model Hub's responsibility. Tool processes share the container user; this is not+a key-security boundary against a malicious root process.++`--sandbox-environment` is only for an already isolated container: preserve the+image's HOME, XDG, PATH, Python/Rust/Java/conda toolchains and other task variables,+while clearing inherited Avibe routing and credential variables. Without it, the+wrapper uses a private host-test HOME. It never modifies its parent's environment.+This prevents tests from touching the developer's account; the host-test mode is+not a leaderboard-equivalent HOME configuration.++`offline=true` prevents managed binary downloads and is used in local tests. Real+runs with it require already installed/cached dependencies in that fresh home.+`offline=false` allows normal product dependency setup; arrange installer egress+in the sandbox setup phase. This wrapper does not replace Harbor's network policy+or impose a model-provider-only rule on tasks that legitimately need the network.++## Result and lifecycle contract++- A ready service with no supplied credential is `blocked`; no model Run or score+ is invented.+- Configure the isolated Agent's model/effort with the existing update command.+ There are no unsupported model override flags on non-forked `agent run`.+- Creation is asynchronous to capture the Run/Session IDs **before** waiting.+ This avoids mistaking CLI `--wait-timeout` for cancellation. A terminal Run+ does not automatically finish a trial: wait for the fresh home's real Harness+ to quiesce (including Watches, scheduled work and delegated runs). Check twice,+ and refuse incomplete/truncated snapshots. A delayed Watch note is covered by+ the integration test; it must not be killed at the initial final reply.+ `outcome` and `final_text` identify the initial Run, not an invented aggregate+ verdict over all continuations; grading still belongs to the verifier.+- `time_limit` requests cancellation of that exact Run. A terminal Run may still+ have Watch work, so the service and its recorded descendants are stopped before+ the final report is written. TERM reaches normal Controller shutdown; forced+ cleanup checks recorded process identity, including detached descendants.+- A failed cleanup prevents a normal handoff. Keep the surrounding sandbox alive+ until all writers are gone; if the supervisor is killed outright, its container+ owner must stop/remove the entire task container before any separate grading.+- Wrapper failures retain a bounded symbolic `error_code` (for example,+ `cli_helper_timeout` or `harness_quiescence_unconfirmed`) in `execution.json`+ and the CLI stdout/stderr summaries. Only wrapper-owned codes are accepted;+ arbitrary upstream exception text and captured helper output are never echoed.+- `outcome` is the product Run state, `time_limit`, `blocked`, or a wrapper failure;+ `verifier_reward` remains null. Do not treat wrapper/Harbor exit status as a+ success rate.+- `usage` is the product's **reported fresh-home usage, including bootstrap**,+ not a provider invoice. `accounting_complete=false` deliberately avoids claiming+ knowledge of unreported/cancelled provider work. No inferred zero-dollar cost.+- Only the structured report is exported. Raw SQLite, job/scratch state, secrets,+ setup files and runtime logs remain private. Model text in the report is+ redacted against the supplied key; it is not an unredacted token-delta stream.+- The report identifies the imported installation and whether source-checkout+ test mode or installed-package mode was used. Do not conflate their revisions.++## Tests++```sh+uv sync --python 3.12 --group dev --no-install-project --frozen+~/.local/bin/hd1-test .venv/bin/python -m pytest -q tests/test_terminal_bench_full_runtime.py+ruff check scripts/terminal_bench tests/test_terminal_bench_full_runtime.py tests/fixtures/full_runtime_service.py+```++The process test runs the real service, Model Hub aggregate/gateway, product CLI,+Run/Session persistence, Vibey and its tools. It substitutes **only the external+engine adapter** with the repository's existing deterministic fixture. It proves+Unicode note and SQLite task writes, module-shadow isolation, actual deadline+cancellation, no writes after cleanup, secret-safe exports, and no task-workspace+pollution. These are local integration tests, **not** acceptance of the real CPA+engine or a benchmark score. The parent's installed-wheel + real-engine + local+HTTP upstream check is a separate evidence layer.++The subprocess fixture has a separate 120-second setup budget for fresh SQLite+migrations on cold/contended filesystems. Its Agent execution budgets remain+30 seconds (normal) and 2 seconds (deadline cancellation); a service that exits+before readiness is still reported immediately. This is not a retry policy.++The startup updater log may initially show compatibility defaults (60 minutes,+`auto_update=true`). The real `_do_check()` reloads the saved config before its+interval-zero early return; the intervening askill reconcile is explicitly+disabled by this wrapper. That log alone does not prove updater egress. No product+updater behavior is changed by this integration.++Still required: parent-run installed-wheel wrapper smoke in the real sandbox,+Harbor installed-agent wiring (declare `atif=False` until conversion is implemented),+pinned task verification, same-profile paired trials, network-policy/resource+equivalence, review/CI, and cost assessment before any larger paid batch. ATIF+conversion is a later submission feature, not a prerequisite to an honest local+pilot with official verifier rewards and structured/native execution evidence.diff --git a/scripts/terminal_bench/__init__.py b/scripts/terminal_bench/__init__.pynew file mode 100644index 000000000..f5ececbb1--- /dev/null+++ b/scripts/terminal_bench/__init__.py@@ -0,0 +1 @@+"""Source-only Terminal-Bench integration; not part of the Avibe wheel."""diff --git a/scripts/terminal_bench/control.py b/scripts/terminal_bench/control.pynew file mode 100644index 000000000..025548aeb--- /dev/null+++ b/scripts/terminal_bench/control.py@@ -0,0 +1,131 @@+"""Child-only bridge to the normal installed Avibe service, CLI, and Model Hub IPC.++No Controller/Agent replacement, no storage seeding, no provider implementation.+The supervisor supplies a fresh AVIBE_HOME before this module imports Avibe.+"""+from __future__ import annotations++import argparse+import asyncio+import importlib.metadata+import json+import os+from pathlib import Path+import runpy+import sys+++def initialize(workdir: Path) -> dict:+ from config import paths+ from config.v2_config import V2Config++ config_path = paths.get_config_path()+ if config_path.exists():+ raise RuntimeError("refusing to overwrite an existing Avibe configuration")+ config = V2Config.default()+ config.runtime.default_cwd = str(workdir.resolve())+ config.platforms.enabled = []+ config.platforms.primary = config.platform = "avibe"+ for backend in (config.agents.claude, config.agents.codex, config.agents.opencode):+ backend.enabled = backend.auto_update = False+ config.update.auto_update = config.update.notify_admins = False+ config.update.check_interval_minutes = 0+ config.ui.open_browser = False+ config.ui.show_agent_activity = True+ config.remote_access.vibe_cloud.enabled = False+ config.setup_completed = True+ # Empty-home startup must not fetch or start an engine. The normal Hub+ # runtime_start operation below owns that transition, not a hand-written key.+ config.model_hub.enabled = False+ config.save(config_path)+ from vibe.runtime import get_service_main_path+ try:+ version = importlib.metadata.version("avibe-os")+ except importlib.metadata.PackageNotFoundError:+ version = None+ import vibe+ return {"service_main": str(get_service_main_path()), "avibe_version": version,+ "python": sys.version,+ "backend_catalog": str(Path(vibe.__file__).parent / "data" / "backend_models.json")}+++async def configure(profile: dict, key: str) -> dict:+ from vibe.model_hub_client import ModelHubRemoteService++ hub = ModelHubRemoteService()+ await hub.runtime_start()+ created = await hub.create_source({+ "kind": "api_key", "vendor": "custom", "display_name": "Benchmark provider",+ "base_url": profile["base_url"], "protocol": profile["protocol"], "key": key,+ "save_unverified": True,+ "models": [{"id": profile["model"], "origin": "manual",+ "reasoning_efforts": [profile["reasoning_effort"]]}],+ })+ source_id = created["source"]["id"]+ supply = await asyncio.to_thread(hub.get_agent_sources, "vibey")+ baseline = supply.get("catalog_models", [])+ chosen = {+ "id": profile["model"], "origin": "manual",+ "context_window": profile["context_window"],+ "max_output_tokens": profile["max_output_tokens"],+ "input_modalities": ["text"], "output_modalities": ["text"],+ "supports_tools": True, "supports_reasoning": True,+ "reasoning_efforts": [profile["reasoning_effort"]],+ }+ await hub.set_agent_sources("vibey", {"order": [source_id]})+ await hub.set_agent_models("vibey", baseline, [chosen])+ await hub.set_agent_chain("vibey", profile["model"], {+ "hops": [{"source_id": source_id, "model_id": profile["model"]}],+ })+ return {"source_id": source_id, "model": profile["model"], "protocol": profile["protocol"],+ "runtime": await hub.runtime_status()}+++def main(argv=None) -> int:+ parser = argparse.ArgumentParser(description=__doc__)+ parser.add_argument("command", choices=["initialize", "serve", "health", "configure", "usage", "versions"])+ parser.add_argument("--workdir", type=Path)+ parser.add_argument("--profile", type=Path)+ args = parser.parse_args(argv)+ if not os.environ.get("AVIBE_HOME") or not os.environ.get("AVIBE_BENCHMARK_OWNER"):+ parser.error("a supervisor-owned isolated AVIBE_HOME is required")+ try:+ if args.command == "initialize":+ if args.workdir is None:+ parser.error("initialize requires --workdir")+ result = initialize(args.workdir)+ elif args.command == "serve":+ from vibe.runtime import get_service_main_path+ runpy.run_path(str(get_service_main_path()), run_name="__main__")+ return 0+ elif args.command == "health":+ from vibe.internal_client import health_identity_sync+ identity = health_identity_sync()+ result = {"ready": identity is not None, "identity": identity}+ elif args.command == "configure":+ if args.profile is None:+ parser.error("configure requires --profile")+ profile = json.loads(args.profile.read_text(encoding="utf-8"))+ # Only this short-lived helper receives the key, over a private pipe.+ key = sys.stdin.read(65537)+ if not key.strip() or len(key) > 65536:+ raise ValueError("missing or oversized credential")+ result = asyncio.run(configure(profile, key))+ elif args.command == "usage":+ from vibe.model_hub_client import ModelHubRemoteService+ result = asyncio.run(ModelHubRemoteService().usage_summary(window="24h"))+ else:+ result = {"python": sys.version, "packages": {+ dist.metadata["Name"]: dist.version for dist in importlib.metadata.distributions()+ if dist.metadata.get("Name")+ }}+ print(json.dumps(result, ensure_ascii=False))+ return 0+ except Exception as exc:+ # Upstream exception text may echo the key. Never print it on the bridge.+ print(json.dumps({"ok": False, "error_type": type(exc).__name__}))+ return 2+++if __name__ == "__main__":+ raise SystemExit(main())diff --git a/scripts/terminal_bench/credential.py b/scripts/terminal_bench/credential.pynew file mode 100644index 000000000..63c7ab026--- /dev/null+++ b/scripts/terminal_bench/credential.py@@ -0,0 +1,118 @@+"""Ephemeral benchmark API-key file transport, not a key-custody service.++Harbor Docker exec env values appear in host argv. Instead upload one 0600 file+into a fresh 0700 per-trial directory, validate metadata, then read+unlink before+starting Avibe. Secret bytes never occur in this helper's argv or printed output.+"""+from __future__ import annotations++import argparse+from contextlib import contextmanager+import json+import os+from pathlib import Path+import stat++LIMIT = 65536+++@contextmanager+def _directory(path: Path):+ if path.name != "provider.key" or path.parent.name != "credential":+ raise ValueError("unexpected credential input path")+ fd = os.open(path.parent, os.O_RDONLY | os.O_DIRECTORY | os.O_NOFOLLOW)+ try:+ info = os.fstat(fd)+ if stat.S_IMODE(info.st_mode) != 0o700 or info.st_uid != os.geteuid():+ raise ValueError("credential directory must be private and owned by the runner")+ yield fd+ finally:+ os.close(fd)+++@contextmanager+def _file(path: Path, directory: int, *, adopt_uploaded: bool = False):+ fd = os.open(path.name, os.O_RDONLY | os.O_NOFOLLOW | os.O_NONBLOCK, dir_fd=directory)+ try:+ info = os.fstat(fd)+ if (not stat.S_ISREG(info.st_mode) or stat.S_IMODE(info.st_mode) != 0o600+ or info.st_nlink != 1 or not 0 < info.st_size <= LIMIT):+ raise ValueError("credential input must be one private regular bounded file")+ if info.st_uid != os.geteuid():+ # Compose cp can preserve the host UID (e.g. macOS 501). Only the+ # setup path may adopt that copied inode, after its containing+ # directory and all file properties have been checked. Never chown+ # a pathname: that could follow a replaced symlink or hardlink.+ if not adopt_uploaded or os.geteuid() != 0:+ raise ValueError("credential input must be owned by the runner")+ os.fchown(fd, os.geteuid(), -1)+ info = os.fstat(fd)+ if info.st_uid != os.geteuid() or stat.S_IMODE(info.st_mode) != 0o600:+ raise ValueError("credential input ownership could not be established")+ yield fd, info+ finally:+ os.close(fd)+++def prepare_upload(path: Path) -> dict:+ """Adopt only a validated copied inode in our new private staging directory."""+ with _directory(path) as directory:+ with _file(path, directory, adopt_uploaded=True) as (_fd, info):+ return {"regular": True, "mode": "0600", "directory_mode": "0700", "size": info.st_size}+++def metadata(path: Path) -> dict:+ with _directory(path) as directory:+ with _file(path, directory) as (_fd, info):+ return {"regular": True, "mode": "0600", "directory_mode": "0700", "size": info.st_size}+++def consume(path: Path) -> str:+ with _directory(path) as directory:+ try:+ with _file(path, directory) as (fd, _info):+ with os.fdopen(os.dup(fd), "rb") as reader:+ data = reader.read(LIMIT + 1)+ if len(data) > LIMIT:+ raise ValueError("credential input grew beyond its bound")+ value = data.decode("utf-8").strip()+ if not value:+ raise ValueError("empty credential input")+ return value+ finally:+ # A rejected symlink is removed as a link, never followed. This+ # fixed name sits in the freshly claimed per-trial private directory.+ try:+ os.unlink(path.name, dir_fd=directory)+ except FileNotFoundError:+ pass+++def clean(path: Path) -> dict:+ if not path.parent.exists() and not path.parent.is_symlink():+ return {"removed": True}+ with _directory(path) as directory:+ try:+ os.unlink(path.name, dir_fd=directory)+ except FileNotFoundError:+ pass+ path.parent.rmdir() # Refuse unknown files rather than delete recursively.+ return {"removed": True}+++def main(argv=None):+ parser = argparse.ArgumentParser(description=__doc__)+ parser.add_argument("operation", choices=["prepare", "metadata", "clean"])+ parser.add_argument("path", type=Path)+ args = parser.parse_args(argv)+ try:+ operation = {"prepare": prepare_upload, "metadata": metadata, "clean": clean}[args.operation]+ print(json.dumps(operation(args.path)))+ return 0+ except Exception as exc:+ print(json.dumps({"ok": False, "error_type": type(exc).__name__}))+ return 2+++if __name__ == "__main__":+ raise SystemExit(main())diff --git a/scripts/terminal_bench/diagnostics.py b/scripts/terminal_bench/diagnostics.pynew file mode 100644index 000000000..de01c647d--- /dev/null+++ b/scripts/terminal_bench/diagnostics.py@@ -0,0 +1,82 @@+"""Bounded, text-free diagnosis through the normal product data-query CLI.++The SQL projects counts and protocol categories inside the product, rather than+exporting a transcript/SQLite file and trying to redact secrets afterwards.+"""+from __future__ import annotations++import re++_STOPS = ("stop", "length", "tool_use", "error", "aborted", "refusal", "safety", "pause")+_COUNTS = (+ "text_blocks", "text_chars", "thinking_blocks", "thinking_chars",+ "tool_calls", "hosted_blocks", "input_tokens", "output_tokens",+ "cache_read_tokens", "reasoning_tokens", "has_error_message",+)+++def response_query(session_id: str) -> str:+ # The CLI deliberately accepts SQL, not bound parameters. Only product IDs+ # are interpolated, after a strict ASCII alphabet and length check.+ if not re.fullmatch(r"[A-Za-z0-9_-]{1,128}", session_id):+ raise ValueError("invalid diagnostic session id")+ allowed = ",".join(repr(s) for s in _STOPS)+ fields = [+ "m.context_seq",+ "CASE WHEN m.type IN ('assistant','result','error') THEN m.type ELSE 'other' END AS row_type",+ f"CASE WHEN json_extract(m.content_json,'$.model.message.stop_reason') IN ({allowed}) "+ "THEN json_extract(m.content_json,'$.model.message.stop_reason') ELSE 'unknown' END AS stop_reason",+ "CASE WHEN json_extract(m.content_json,'$.model.message.error_message') IS NOT NULL "+ "THEN 1 ELSE 0 END AS has_error_message",+ ]+ for block, prefix in (("text", "text"), ("thinking", "thinking")):+ for expr, suffix in (("count(*)", "blocks"), ("coalesce(sum(length(json_extract(b.value,'$.text'))),0)", "chars")):+ fields.append(+ f"(SELECT {expr} FROM json_each(m.content_json,'$.model.message.content') b "+ f"WHERE json_extract(b.value,'$.type')='{block}') AS {prefix}_{suffix}"+ )+ for block, name in (("tool_call", "tool_calls"), ("hosted_tool", "hosted_blocks")):+ fields.append(+ "(SELECT count(*) FROM json_each(m.content_json,'$.model.message.content') b "+ f"WHERE json_extract(b.value,'$.type')='{block}') AS {name}"+ )+ for name in ("input_tokens", "output_tokens", "cache_read_tokens", "reasoning_tokens"):+ path = f"$.model.message.usage.{name}"+ fields.append(+ f"CASE WHEN json_type(m.content_json,'{path}')='integer' "+ f"THEN json_extract(m.content_json,'{path}') ELSE NULL END AS {name}"+ )+ return (+ "WITH recent AS (SELECT context_seq,type,content_json FROM messages "+ f"WHERE session_id='{session_id}' AND context_seq IS NOT NULL "+ "ORDER BY context_seq DESC LIMIT 32) SELECT " + ", ".join(fields) ++ " FROM recent m WHERE json_extract(m.content_json,'$.model.message.role')='assistant' "+ "ORDER BY m.context_seq DESC LIMIT 9"+ )+++def project_responses(payload: dict) -> dict:+ """Second allowlist fence: even a malformed/new CLI shape cannot leak text."""+ rows = payload.get("rows")+ if payload.get("ok") is not True or not isinstance(rows, list):+ raise ValueError("diagnostic query unavailable")+ output = []+ for row in rows[:8]:+ if not isinstance(row, dict):+ raise ValueError("invalid diagnostic row")+ item = {"stop_reason": row.get("stop_reason") if row.get("stop_reason") in _STOPS else "unknown"}+ item["row_type"] = row.get("row_type") if row.get("row_type") in {"assistant", "result", "error"} else "other"+ for name in ("context_seq", *_COUNTS):+ value = row.get(name)+ item[name] = value if type(value) is int and 0 <= value <= 2**63 - 1 else None+ output.append(item)+ return {+ "schema_version": 1,+ "status": "available" if output else "no_response_recorded",+ "source": "product_data_query",+ "scope": "last_8_responses_within_last_32_context_messages_of_root_session",+ "raw_text_exported": False,+ "responses_truncated": len(rows) > 8,+ "history_complete": False,+ "responses": output,+ }diff --git a/scripts/terminal_bench/harbor_agent.py b/scripts/terminal_bench/harbor_agent.pynew file mode 100644index 000000000..f47e3dc5c--- /dev/null+++ b/scripts/terminal_bench/harbor_agent.py@@ -0,0 +1,277 @@+"""Optional Harbor 0.24 installed-agent bridge to the full Avibe product.++No extracted product code, no alternate loop, no dependency on Harbor in the+product wheel. Install requirements-harbor.txt in the evaluation host only.+"""+from __future__ import annotations++import asyncio+from email.parser import Parser+import hashlib+from importlib.metadata import version+import json+from pathlib import Path, PurePosixPath+import re+import shlex+import shutil+import subprocess+import tempfile+import uuid+import zipfile++from harbor.agents.capabilities import AgentCapabilities+from harbor.agents.installed.base import BaseInstalledAgent, with_prompt_template++from .profile import Profile+++_WRAPPER_FILES = ("__init__.py", "profile.py", "control.py", "runner.py", "launch.py", "credential.py", "diagnostics.py")+++async def _join_transfer(awaitable):+ """A cancelled upload must settle before remote credential cleanup."""+ task = asyncio.create_task(awaitable)+ cancelled = False+ while True:+ try:+ result = await asyncio.shield(task)+ break+ except asyncio.CancelledError:+ cancelled = True+ if task.done():+ result = await task+ break+ if cancelled:+ raise asyncio.CancelledError+ return result+++class FullAvibeAgent(BaseInstalledAgent):+ capabilities = AgentCapabilities(atif=False, resume=False)++ @staticmethod+ def name():+ return "avibe-full-runtime"++ def __init__(self, *args, wheel_path: str, wheel_sha256: str, profile: dict, **kwargs):+ if version("harbor") != "0.24.0":+ raise ValueError("this integration requires Harbor 0.24.0")+ self.profile = Profile.parse(profile)+ if kwargs.get("model_name") != self.profile.model:+ raise ValueError("explicit Harbor model_name must equal profile.model")+ self.wheel = Path(wheel_path).expanduser().resolve()+ if self.wheel.suffix != ".whl" or not self.wheel.is_file():+ raise ValueError("a trusted local Avibe wheel file is required")+ if not re.fullmatch(r"[0-9a-f]{64}", wheel_sha256):+ raise ValueError("wheel_sha256 must be an explicit lowercase SHA256")+ self.wheel_sha256 = wheel_sha256+ self.nonce = uuid.uuid4().hex+ self.root = PurePosixPath("/installed-agent") / ("avibe-full-" + self.nonce)+ self.python = self.root / "venv/bin/python"+ self.provenance = {}+ self._credential_claimed = False+ self.credential_path = self.root / "credential/provider.key"+ super().__init__(*args, **kwargs)+ if self.profile.credential_env in self.extra_env:+ raise ValueError("supply the named credential in the host environment, not Harbor agent-env/config")+ self.execution_dir = self.environment_logs_dir / ("full-runtime-" + self.nonce)++ def _snapshot(self, temporary: Path):+ # Hash the immutable COPY that is uploaded, not an earlier observation+ # of a file somebody might replace during container setup.+ wheel = temporary / self.wheel.name+ shutil.copyfile(self.wheel, wheel)+ with wheel.open("rb") as handle:+ if hashlib.file_digest(handle, "sha256").hexdigest() != self.wheel_sha256:+ raise ValueError("Avibe wheel SHA256 mismatch")+ with zipfile.ZipFile(wheel) as archive:+ metadata_names = [name for name in archive.namelist() if name.endswith(".dist-info/METADATA")]+ if len(metadata_names) != 1:+ raise ValueError("invalid wheel metadata")+ metadata = Parser().parsestr(archive.read(metadata_names[0]).decode("utf-8"))+ if metadata.get("Name", "").lower().replace("_", "-") != "avibe-os":+ raise ValueError("the pinned wheel must be avibe-os")+ source = Path(__file__).resolve().parent+ repo = source.parents[1]+ staged = temporary / "source/scripts/terminal_bench"+ staged.mkdir(parents=True)+ digest = hashlib.sha256()+ for name in _WRAPPER_FILES:+ relative = (source / name).relative_to(repo).as_posix()+ check = subprocess.run(["git", "ls-files", "--error-unmatch", "--", relative],+ cwd=repo, capture_output=True, timeout=10)+ if check.returncode:+ raise ValueError("wrapper upload requires tracked source: " + relative)+ content = (source / name).read_bytes()+ (staged / name).write_bytes(content)+ digest.update(relative.encode() + b"\0" + content + b"\0")+ (staged.parent / "__init__.py").write_text("", encoding="utf-8")+ commit = subprocess.check_output(["git", "rev-parse", "HEAD"], cwd=repo, text=True, timeout=10).strip()+ dirty = subprocess.check_output(["git", "status", "--porcelain"], cwd=repo, text=True, timeout=10)+ self.provenance = {+ "agent": self.name(), "harbor_version": version("harbor"), "atif": False,+ "wheel_filename": wheel.name, "wheel_sha256": self.wheel_sha256,+ "avibe_version": metadata["Version"], "wrapper_commit": commit,+ "wrapper_source_sha256": digest.hexdigest(), "wrapper_source_dirty": bool(dirty.strip()),+ }+ return wheel, temporary / "source"++ async def install(self, environment):+ with tempfile.TemporaryDirectory(prefix="avibe-full-source-") as temp:+ wheel, source = await asyncio.to_thread(self._snapshot, Path(temp))+ # Wheel/provenance checks occur before any mutation of the task.+ uid = await environment.exec(command="id -u")+ if uid.return_code or (uid.stdout or "").strip() != "0":+ raise RuntimeError("first full-runtime Harbor pilot requires a root task user")+ await self.ensure_system_dependencies(environment, ("python3", "python_venv"))+ await self.exec_as_root(environment, "mkdir -m 700 " + shlex.quote(str(self.root)))+ await environment.upload_file(wheel, str(self.root / self.wheel.name))+ await environment.upload_dir(source, str(self.root / "source"))+ # Installed distributions, not editable source. A .pth installs ONLY+ # this small wrapper for -I imports; task PYTHONPATH remains unmodified.+ pth = "import pathlib,sysconfig; pathlib.Path(sysconfig.get_path('purelib'),'avibe_benchmark.pth').write_text(" + repr(str(self.root / "source") + "\n") + ")"+ command = (+ "set -eu; python3 -c 'import sys; assert sys.version_info >= (3, 10)' && "+ f"python3 -m venv {shlex.quote(str(self.root / 'venv'))} && "+ f"{shlex.quote(str(self.python))} -m pip install --disable-pip-version-check --no-input "+ f"{shlex.quote(str(self.root / self.wheel.name))} && "+ f"{shlex.quote(str(self.python))} -c {shlex.quote(pth)}"+ )+ await self.exec_as_root(environment, command, timeout_sec=int(self.profile.setup_seconds))+ # Add ONLY a private vibe shim to PATH during execution, not the venv's+ # Python binaries: task Python/conda/java resolution must stay unchanged.+ await self.exec_as_root(environment,+ "mkdir -p " + shlex.quote(str(self.root / "bin")) + "; ln -s "+ + shlex.quote(str(self.root / "venv/bin/vibe")) + " " + shlex.quote(str(self.root / "bin/vibe")))+ async def _stage_credential(self, environment, key):+ result = await environment.exec(command="mkdir -m 700 " + shlex.quote(str(self.credential_path.parent)), user="root")+ if result.return_code:+ raise RuntimeError("could not exclusively claim credential directory")+ self._credential_claimed = True+ with tempfile.TemporaryDirectory(prefix="avibe-full-key-") as temp:+ path = Path(temp) / "provider.key"+ # The directory is 0700 already; explicitly make the file 0600+ # before writing and use upload_file rather than any shell literal.+ import os+ fd = os.open(path, os.O_CREAT | os.O_EXCL | os.O_WRONLY, 0o600)+ with os.fdopen(fd, "w", encoding="utf-8") as handle:+ handle.write(key)+ await _join_transfer(environment.upload_file(path, str(self.credential_path)))+ checked = await environment.exec(command=shlex.join([+ str(self.python), "-I", "-m", "scripts.terminal_bench.credential", "prepare", str(self.credential_path),+ ]), user="root")+ if checked.return_code:+ raise RuntimeError("credential metadata refused")+ metadata = json.loads(checked.stdout)+ if metadata.get("regular") is not True or metadata.get("mode") != "0600" or metadata.get("directory_mode") != "0700":+ raise RuntimeError("credential metadata refused")++ async def _clear_credential(self, environment):+ if not self._credential_claimed:+ return+ result = await environment.exec(command=shlex.join([+ str(self.python), "-I", "-m", "scripts.terminal_bench.credential", "clean", str(self.credential_path),+ ]), user="root", timeout_sec=15)+ if result.return_code or json.loads(result.stdout).get("removed") is not True:+ raise RuntimeError("credential cleanup unconfirmed; destroy task container before grading")+ self._credential_claimed = False++ async def _cleanup(self, environment):+ command = shlex.join([+ str(self.python), "-I", "-m", "scripts.terminal_bench.launch", "stop",+ "--root", str(self.root / "launch"), "--nonce", self.nonce,+ "--receipt", str(self.execution_dir / "execution.json"),+ "--timeout", str(self.profile.cleanup_seconds + 60),+ ])+ try:+ result = await environment.exec(command=command, user="root", timeout_sec=int(self.profile.cleanup_seconds + 75))+ finally:+ await self._clear_credential(environment)+ if result.return_code != 0:+ raise RuntimeError("full-runtime cleanup unconfirmed; destroy task container before grading")+ try:+ receipt = json.loads(result.stdout)+ if receipt.get("quiescent") is not True:+ raise ValueError+ except (ValueError, TypeError):+ raise RuntimeError("full-runtime cleanup receipt invalid; destroy task container before grading") from None+ return receipt++ @with_prompt_template+ async def run(self, instruction, environment, context):+ if not self.provenance:+ raise RuntimeError("verified installation is required before execution")+ cwd_result = await environment.exec(command="pwd -P")+ workdir = (cwd_result.stdout or "").strip()+ if cwd_result.return_code or not workdir.startswith("/") or "\n" in workdir:+ raise RuntimeError("task cwd could not be determined")+ for private in (self.root, self.execution_dir):+ task = PurePosixPath(workdir)+ if task == private or task in private.parents or private in task.parents:+ raise RuntimeError("task and framework directories must be disjoint")+ key = self._get_env(self.profile.credential_env)+ if not key:+ raise ValueError("the named benchmark credential is missing")+ with tempfile.TemporaryDirectory(prefix="avibe-full-config-") as temp:+ for name, text in (("profile.json", json.dumps(self.profile.payload())), ("instruction.txt", instruction)):+ path = Path(temp) / name+ path.write_text(text, encoding="utf-8")+ await environment.upload_file(path, str(self.root / name))+ command = shlex.join([+ str(self.python), "-I", "-m", "scripts.terminal_bench.launch", "run",+ "--root", str(self.root / "launch"), "--nonce", self.nonce, "--",+ "--workdir", workdir, "--state", str(self.root / "state"),+ "--output", str(self.execution_dir), "--profile", str(self.root / "profile.json"),+ "--instruction", str(self.root / "instruction.txt"), "--sandbox-environment",+ "--credential-file", str(self.credential_path),+ ])+ command = "export PATH=" + shlex.quote(str(self.root / "bin")) + ":\"$PATH\"; exec " + command+ try:+ await self._stage_credential(environment, key)+ # Model setup is part of actual full-product execution, hence the+ # outer exec allowance includes it. Harbor's task timeout remains+ # authoritative and may cancel this earlier; cleanup is mandatory.+ allowance = int(self.profile.setup_seconds * 4 + self.profile.wall_seconds+ + self.profile.cleanup_seconds * 2 + 90)+ result = await environment.exec(command=command, cwd=str(self.root), timeout_sec=allowance)+ if result.return_code:+ raise RuntimeError("full-runtime wrapper infrastructure failure; inspect its structured report")+ finally:+ # Shield cleanup from the cancellation that interrupted docker exec,+ # but JOIN it: verification must never race an orphan writer.+ cleanup = asyncio.create_task(self._cleanup(environment))+ while True:+ try:+ cleanup_receipt = await asyncio.shield(cleanup)+ break+ except asyncio.CancelledError:+ if cleanup.done():+ cleanup_receipt = await cleanup+ break+ if cleanup_receipt.get("started") is not False:+ await self._collect(environment, context)++ async def _collect(self, environment, context):+ local = self.logs_dir / "full-runtime"+ local.mkdir(parents=True, exist_ok=True)+ report_path = local / "execution.json"+ await environment.download_file(str(self.execution_dir / "execution.json"), report_path)+ report = json.loads(report_path.read_text(encoding="utf-8"))+ if report.get("cleanup", {}).get("owned_processes_stopped") is not True:+ raise RuntimeError("full-runtime report did not confirm shutdown")+ totals = (report.get("usage") or {}).get("totals", {})+ # A missing token report is unknown, not zero-cost inference.+ if totals.get("token_reports", 0) > 0:+ context.n_input_tokens = totals.get("input_tokens")+ context.n_cache_tokens = totals.get("cached_input_tokens")+ context.n_output_tokens = totals.get("output_tokens")+ context.cost_usd = None+ context.metadata = {**self.provenance, "execution_outcome": report.get("outcome"),+ "accounting_complete": report.get("accounting_complete", False),+ "run_id": report.get("run_id"), "session_id": report.get("session_id"),+ "verifier_reward": None}+ (local / "adapter-provenance.json").write_text(json.dumps(context.metadata, indent=2), encoding="utf-8")+ if (report.get("outcome") not in {"succeeded", "failed", "canceled", "cancelled", "time_limit"}+ or not isinstance(report.get("run_id"), str) or not report["run_id"]+ or not isinstance(report.get("session_id"), str) or not report["session_id"]):+ raise RuntimeError("full-runtime execution was not admitted; not an Agent failure score")diff --git a/scripts/terminal_bench/harbor_job.py b/scripts/terminal_bench/harbor_job.pynew file mode 100644index 000000000..38f667f0e--- /dev/null+++ b/scripts/terminal_bench/harbor_job.py@@ -0,0 +1,58 @@+"""Pinned Harbor driver: journal exact Compose ownership before environment start."""+from __future__ import annotations++import argparse+import asyncio+from importlib.metadata import version+from pathlib import Path+from types import SimpleNamespace++from .remote_job import load, save+++async def execute(root):+ if version("harbor") != "0.24.0":+ raise ValueError("this supervisor requires Harbor 0.24.0")+ from harbor.environments.docker.docker import _sanitize_docker_compose_project_name+ from harbor.job import Job+ from harbor.models.job.config import JobConfig+ from harbor.trial.hooks import TrialEvent+ from harbor.trial.trial import Trial++ config = JobConfig.model_validate(load(root / "config.json"))+ config.retry.max_retries = 0+ job = await Job.create(config)++ async def before_environment(event):+ # START is awaited before _prepare(), so this journal also covers the+ # network-only window. Use the pinned implementation's exact sanitizer+ # and verifier truncation/hash, not a prefix match on arbitrary labels.+ verifier = Trial._separate_verifier_session_id(SimpleNamespace(config=event.config), "trial")+ save(root / "ownership.json", {+ "schema_version": 1, "job_id": str(event.config.job_id),+ "trial_name": event.trial_name, "trial_id": str(event.trial_id),+ "projects": [_sanitize_docker_compose_project_name(f"{event.trial_name}__env"),+ _sanitize_docker_compose_project_name(verifier)],+ })++ job.add_hook(TrialEvent.START, before_environment)+ await job.run()+++def main():+ parser = argparse.ArgumentParser(description=__doc__)+ parser.add_argument("--root", type=Path, required=True)+ args = parser.parse_args()+ try:+ from harbor.utils.async_cleanup import with_async_cleanup+ asyncio.run(with_async_cleanup(execute(args.root)))+ return 0+ except Exception as exc:+ # No Rich local-variable dump and no upstream exception text. The host+ # log captures Harbor's own output through the supervisor's redactor.+ print(type(exc).__name__)+ return 1+++if __name__ == "__main__":+ raise SystemExit(main())diff --git a/scripts/terminal_bench/launch.py b/scripts/terminal_bench/launch.pynew file mode 100644index 000000000..16ad34470--- /dev/null+++ b/scripts/terminal_bench/launch.py@@ -0,0 +1,132 @@+"""In-container wrapper process ownership for Harbor exec cancellation.++A Docker exec client's death does not terminate its container process. This+launcher records the exact owned wrapper identity before releasing its start+lock. The stop path fences late starts, signals only that identity, and requires+the wrapper's completed inner cleanup receipt. Otherwise grading is refused and+the outer owner must destroy the sandbox; a guessed PID is never signalled.+"""+from __future__ import annotations++import argparse+from contextlib import contextmanager+import json+import os+from pathlib import Path+import subprocess+import sys+import time++import psutil+++@contextmanager+def locked(root: Path):+ import fcntl # This first Harbor integration supports POSIX task containers.+ root.mkdir(parents=True, exist_ok=True)+ with (root / "launch.lock").open("a") as lock:+ fcntl.flock(lock.fileno(), fcntl.LOCK_EX)+ try:+ yield+ finally:+ fcntl.flock(lock.fileno(), fcntl.LOCK_UN)+++def run(root: Path, nonce: str, args: list[str]) -> int:+ with locked(root):+ if (root / "cancelled").exists():+ return 125+ if (root / "wrapper-process.json").exists():+ raise RuntimeError("wrapper_already_started")+ # Persist the launch attempt before spawn. If identity persistence fails+ # (e.g. disk full), stop must NOT mistake an absent pidfile for no child.+ (root / "starting").touch(exist_ok=False)+ try:+ process = subprocess.Popen(+ [sys.executable, "-I", "-m", "scripts.terminal_bench.runner", *args],+ env={**os.environ, "AVIBE_BENCHMARK_LAUNCH_ID": nonce},+ start_new_session=True,+ )+ except BaseException:+ (root / "starting").unlink()+ raise+ try:+ identity = {"pid": process.pid, "created": psutil.Process(process.pid).create_time(), "nonce": nonce}+ temporary = root / "wrapper-process.tmp"+ temporary.write_text(json.dumps(identity), encoding="utf-8")+ temporary.replace(root / "wrapper-process.json")+ except BaseException:+ # Preserve `starting`: the outer sandbox remains responsible if+ # we cannot prove inner shutdown after losing the identity write.+ if process.poll() is None:+ process.terminate()+ raise+ return process.wait()+++def stop(root: Path, nonce: str, receipt: Path, timeout: float) -> dict:+ with locked(root):+ (root / "cancelled").touch(exist_ok=True)+ path = root / "wrapper-process.json"+ if not path.exists():+ if (root / "starting").exists():+ raise RuntimeError("wrapper_identity_not_recorded_destroy_sandbox")+ return {"quiescent": True, "started": False}+ record = json.loads(path.read_text(encoding="utf-8"))+ if record.get("nonce") != nonce:+ raise RuntimeError("wrapper_identity_mismatch")++ def owned_live():+ try:+ process = psutil.Process(record["pid"])+ if process.status() == psutil.STATUS_ZOMBIE:+ return None+ if (process.create_time() != record["created"]+ and process.environ().get("AVIBE_BENCHMARK_LAUNCH_ID") != nonce):+ raise RuntimeError("wrapper_identity_mismatch")+ return process+ except psutil.NoSuchProcess:+ return None++ process = owned_live()+ if process is not None:+ process.terminate()+ deadline = time.monotonic() + timeout+ while owned_live() is not None and time.monotonic() < deadline:+ time.sleep(.1)+ process = owned_live()+ if process is not None:+ process.kill()+ raise RuntimeError("wrapper_cleanup_unconfirmed_destroy_sandbox")+ try:+ result = json.loads(receipt.read_text(encoding="utf-8"))+ except (OSError, ValueError):+ raise RuntimeError("wrapper_receipt_missing_destroy_sandbox") from None+ if result.get("cleanup", {}).get("owned_processes_stopped") is not True:+ raise RuntimeError("wrapper_cleanup_unconfirmed_destroy_sandbox")+ return {"quiescent": True, "started": True}+++def main(argv=None):+ parser = argparse.ArgumentParser(description=__doc__)+ parser.add_argument("operation", choices=["run", "stop"])+ parser.add_argument("--root", type=Path, required=True)+ parser.add_argument("--nonce", required=True)+ parser.add_argument("--receipt", type=Path)+ parser.add_argument("--timeout", type=float, default=90)+ args, wrapper_args = parser.parse_known_args(argv)+ try:+ if args.operation == "run":+ return run(args.root, args.nonce, wrapper_args[1:] if wrapper_args[:1] == ["--"] else wrapper_args)+ if args.receipt is None or args.timeout <= 0:+ parser.error("stop requires --receipt and positive --timeout")+ print(json.dumps(stop(args.root, args.nonce, args.receipt, args.timeout)))+ return 0+ except Exception as exc:+ # Only our closed vocabulary is printed, never arbitrary process output.+ print(json.dumps({"quiescent": False, "error_type": type(exc).__name__}))+ return 2+++if __name__ == "__main__":+ raise SystemExit(main())diff --git a/scripts/terminal_bench/profile.py b/scripts/terminal_bench/profile.pynew file mode 100644index 000000000..7ae52f44c--- /dev/null+++ b/scripts/terminal_bench/profile.py@@ -0,0 +1,82 @@+"""Explicit, credential-free full-runtime trial configuration."""+from __future__ import annotations++from dataclasses import asdict, dataclass, fields+import os+import math+from pathlib import Path+import re+from urllib.parse import urlsplit+++@dataclass(frozen=True)+class Profile:+ model: str+ base_url: str+ credential_env: str = "DEEPSEEK_API_KEY"+ protocol: str = "openai_chat"+ reasoning_effort: str = "high"+ context_window: int = 128000+ max_output_tokens: int = 32000+ wall_seconds: float = 1200+ setup_seconds: float = 600+ cleanup_seconds: float = 30+ poll_seconds: float = 1+ offline: bool = False++ def __post_init__(self):+ if type(self.offline) is not bool:+ raise ValueError("offline must be boolean")+ url = urlsplit(self.base_url)+ if url.scheme not in {"http", "https"} or not url.hostname:+ raise ValueError("base_url must be an HTTP(S) API root")+ if url.username or url.password or url.query or url.fragment:+ raise ValueError("base_url cannot contain credentials, query, or fragment")+ if url.scheme == "http" and url.hostname not in {"localhost", "127.0.0.1", "::1"}:+ raise ValueError("plaintext HTTP is restricted to loopback fixtures")+ if self.protocol not in {"openai_chat", "openai_responses", "anthropic"}:+ raise ValueError("unsupported protocol")+ if not self.model or not self.reasoning_effort:+ raise ValueError("explicit model and effort required")+ if not re.fullmatch(r"[A-Z][A-Z0-9_]*", self.credential_env):+ raise ValueError("credential_env must be an environment variable name")+ for value in (self.context_window, self.max_output_tokens):+ if type(value) is not int or value <= 0:+ raise ValueError("token limits must be positive integers")+ if self.max_output_tokens > self.context_window // 4:+ raise ValueError("output limit exceeds Vibey's quarter-window policy")+ for value in (self.wall_seconds, self.setup_seconds, self.cleanup_seconds, self.poll_seconds):+ if isinstance(value, bool) or not math.isfinite(value) or value <= 0:+ raise ValueError("time limits must be finite and positive")++ def payload(self):+ return asdict(self)++ @classmethod+ def parse(cls, payload):+ if not isinstance(payload, dict) or set(payload) - {f.name for f in fields(cls)}:+ raise ValueError("unknown profile fields")+ return cls(**payload)+++def validate_layout(workdir: Path, state: Path, output: Path) -> None:+ paths = [p.resolve() for p in (workdir, state, output)]+ if not paths[0].is_dir():+ raise ValueError("task workdir must exist")+ if any(p == Path("/") for p in paths):+ raise ValueError("filesystem root is not a trial directory")+ for i, left in enumerate(paths):+ for right in paths[i + 1:]:+ if left == right or left in right.parents or right in left.parents:+ raise ValueError("task, runtime state, and exported artifacts must be disjoint")+ # Reject production roots even if a caller forgets the container-only gate.+ home = Path.home().resolve()+ for path in paths:+ configured = os.environ.get("AVIBE_HOME")+ forbidden_roots = [home / ".avibe", home / ".vibe_remote"]+ if configured:+ forbidden_roots.append(Path(configured).expanduser())+ for forbidden in forbidden_roots:+ forbidden = forbidden.resolve()+ if path == forbidden or forbidden in path.parents or path in forbidden.parents:+ raise ValueError("refusing the current/default Avibe home or its ancestors")diff --git a/scripts/terminal_bench/remote_job.py b/scripts/terminal_bench/remote_job.pynew file mode 100644index 000000000..b4d5361f1--- /dev/null+++ b/scripts/terminal_bench/remote_job.py@@ -0,0 +1,488 @@+"""Single-job Harbor lifetime on a dedicated Linux evaluation host.++systemd owns execution independently of SSH. Only its post-stop phase finalizes+artifacts and releases ownership; Avibe Harness observes the durable receipt.+"""+from __future__ import annotations++import argparse+import fcntl+import hashlib+import json+import os+from pathlib import Path+import re+import selectors+import signal+import subprocess+import sys+import threading+import time+import uuid++try:+ import tomllib+except ModuleNotFoundError: # Product tests still support Python 3.10.+ import tomli as tomllib++from .credential import consume, metadata++HOST_RESERVATION = Path("/run/lock/avibe-benchmark.active.json")+FINALIZE_SECONDS = 45+++def load(path):+ return json.loads(Path(path).read_text(encoding="utf-8"))+++def evidence(path):+ try:+ value = load(path)+ return value if isinstance(value, dict) else None+ except (OSError, ValueError):+ return None+++def save(path, value):+ path = Path(path)+ temporary = path.with_name(path.name + ".new")+ with temporary.open("w", encoding="utf-8") as file:+ os.fchmod(file.fileno(), 0o600)+ json.dump(value, file, indent=2)+ file.write("\n")+ temporary.replace(path)+++def command(argv, *, timeout=10):+ return subprocess.run(argv, capture_output=True, text=True, timeout=timeout, check=True).stdout+++def absolute(path, *, exists=True):+ p = Path(path)+ if any(char in str(p) for char in "%$\n\r\x00"):+ raise ValueError("systemd specifiers and variable expansions are not supported in paths")+ if not p.is_absolute():+ raise ValueError("absolute execution paths required; relocate explicitly before submission")+ resolved = p.resolve(strict=exists)+ if any(char in str(resolved) for char in "%$\n\r\x00"):+ raise ValueError("resolved path contains a systemd expansion")+ return resolved+++def validate(config_path, credential_env=None):+ """No job creation, Docker mutation, credential consumption or model call."""+ config = load(config_path)+ if config.get("n_attempts", 1) != 1 or config.get("n_concurrent_trials", 1) != 1:+ raise ValueError("one attempt and concurrency one required")+ if config.get("retry", {}).get("max_retries", 0) != 0 or any(config.get(k) for k in ("source_jobs", "datasets", "user_agent")):+ raise ValueError("retry, dataset, regrade and simulated-user jobs unsupported")+ if len(config.get("agents", [])) != 1 or len(config.get("tasks", [])) != 1:+ raise ValueError("exactly one agent and local task required")+ name = config.get("job_name", "")+ if not re.fullmatch(r"[A-Za-z0-9_-]{1,100}", name):+ raise ValueError("explicit safe job name required")+ env = config.get("environment", {})+ if env.get("type", "docker") != "docker" or env.get("delete", True) is not True or env.get("import_path"):+ raise ValueError("isolated Docker with delete=true required")+ task = absolute(config["tasks"][0]["path"])+ task_config = tomllib.loads((task / "task.toml").read_text())+ if task_config.get("steps"):+ raise ValueError("single-step tasks only; verifier ownership is pinned to Harbor 0.24")+ for overlay in env.get("extra_docker_compose", []):+ absolute(overlay)+ agent = config["agents"][0]+ seconds = agent.get("override_timeout_sec")+ if type(seconds) not in (int, float) or not 0 < seconds <= 1800:+ raise ValueError("explicit agent limit no greater than 1800 seconds required")+ kwargs = agent.get("kwargs", {})+ if agent.get("import_path") == "scripts.terminal_bench.harbor_agent:FullAvibeAgent":+ from .profile import Profile+ profile = Profile.parse(kwargs["profile"])+ if profile.model != agent.get("model_name"):+ raise ValueError("profile model mismatch")+ if credential_env is not None and credential_env != profile.credential_env:+ raise ValueError("credential variable must match the full-runtime profile")+ credential_env = profile.credential_env+ wheel = absolute(kwargs["wheel_path"])+ with wheel.open("rb") as file:+ digest = hashlib.sha256()+ for chunk in iter(lambda: file.read(1024 * 1024), b""):+ digest.update(chunk)+ if digest.hexdigest() != kwargs["wheel_sha256"]:+ raise ValueError("wheel hash mismatch")+ credential_env = credential_env or "BENCHMARK_API_KEY"+ if not re.fullmatch(r"[A-Z][A-Z0-9_]{0,100}", credential_env):+ raise ValueError("invalid credential variable")+ jobs = absolute(config["jobs_dir"], exists=False)+ if (jobs / name).exists():+ raise ValueError("fresh job destination required")+ return config, jobs / name, task, credential_env+++def release(root):+ if HOST_RESERVATION.is_file() and HOST_RESERVATION.read_text() == str(Path(root).resolve()):+ HOST_RESERVATION.unlink()+++def prepare(config_path, root, wall_seconds, credential_env=None):+ config, job_dir, task, credential_env = validate(config_path, credential_env)+ root = absolute(root, exists=False)+ source_root = absolute(Path(__file__).resolve().parents[2])+ absolute(sys.executable) # Reject manager expansion in the interpreter path too.+ if not 0 < wall_seconds <= 7200:+ raise ValueError("invalid hard wall cap")+ if any(root == p or root in p.parents or p in root.parents for p in (task, job_dir)):+ raise ValueError("supervisor state must be disjoint from task and job")+ root.parent.mkdir(parents=True, exist_ok=True)+ HOST_RESERVATION.parent.mkdir(parents=True, exist_ok=True)+ with HOST_RESERVATION.open("x", encoding="utf-8") as file:+ os.fchmod(file.fileno(), 0o600)+ file.write(str(root))+ try:+ root.mkdir(mode=0o700)+ job_dir.parent.mkdir(parents=True, exist_ok=True)+ # Keep this per-destination claim permanently, including failed starts.+ with (job_dir.parent / ("." + job_dir.name + ".claimed")).open("x", encoding="utf-8") as file:+ file.write(str(root))+ manifest = {"schema_version": 1, "unit": "avibe-benchmark-" + uuid.uuid4().hex,+ "job_dir": str(job_dir), "task": str(task), "wall_seconds": wall_seconds,+ "credential_env": credential_env, "created_at": time.time(),+ "config_sha256": hashlib.sha256(Path(config_path).read_bytes()).hexdigest(),+ "source_root": str(source_root)}+ save(root / "config.json", config)+ save(root / "manifest.json", manifest)+ save(root / "lifecycle.json", {"state": "prepared"})+ return manifest+ except Exception:+ release(root)+ raise+++def terminal_evidence(manifest, ownership):+ """A finished job alone is insufficient: the one exact trial must be final."""+ job_dir = Path(manifest["job_dir"])+ result = evidence(job_dir / "result.json")+ name = ownership.get("trial_name")+ if not isinstance(name, str) or not re.fullmatch(r"[A-Za-z0-9_-]{1,180}", name):+ return False+ trial = evidence(job_dir / name / "result.json")+ if not result or not trial or not result.get("finished_at") or not trial.get("finished_at"):+ return False+ stats = result.get("stats") or {}+ if not isinstance(stats, dict):+ return False+ return (result.get("n_total_trials") == 1 and stats.get("n_completed_trials") == 1+ and stats.get("n_running_trials") == 0 and stats.get("n_pending_trials") == 0+ and result.get("id") == ownership.get("job_id")+ and trial.get("id") == ownership.get("trial_id") and trial.get("trial_name") == name)+++def docker_state(projects, run=command):+ ids = run(["docker", "ps", "-aq"]).split()+ containers = json.loads(run(["docker", "inspect", *ids])) if ids else []+ ids = run(["docker", "network", "ls", "-q"]).split()+ networks = json.loads(run(["docker", "network", "inspect", *ids])) if ids else []+ owned = [c for c in containers if (c.get("Config", {}).get("Labels") or {}).get("com.docker.compose.project") in projects]+ nets = [n for n in networks if (n.get("Labels") or {}).get("com.docker.compose.project") in projects]+ return owned, nets+++def archive_artifact(container_id, source, destination, timeout):+ """Keep untrusted Docker tar bytes opaque; never extract onto the host.++ Separate exclusive files prevent overlapping artifact paths or symlinks in+ one artifact from redirecting a later host write. Bound both bytes and time.+ """+ deadline = time.monotonic() + timeout+ with destination.open("xb") as file:+ os.fchmod(file.fileno(), 0o600)+ child = subprocess.Popen(["docker", "cp", f"{container_id}:{source}", "-"],+ stdout=subprocess.PIPE, stderr=subprocess.DEVNULL)+ try:+ with selectors.DefaultSelector() as selector:+ selector.register(child.stdout, selectors.EVENT_READ)+ size = 0+ while True:+ left = deadline - time.monotonic()+ if left <= 0 or not selector.select(left):+ raise subprocess.TimeoutExpired("artifact archive", timeout)+ chunk = os.read(child.stdout.fileno(), 65536)+ if not chunk:+ break+ size += len(chunk)+ if size > 32 * 1024 * 1024:+ raise OSError("artifact archive exceeds byte budget")+ file.write(chunk)+ child.wait(timeout=max(.1, deadline - time.monotonic()))+ if child.returncode:+ raise subprocess.CalledProcessError(child.returncode, "artifact archive")+ finally:+ if child.poll() is None:+ child.kill()+ child.wait(timeout=2)+ child.stdout.close()+++def copy_artifacts(root, manifest, containers, deadline):+ records = []+ try:+ declared = tomllib.loads((Path(manifest["task"]) / "task.toml").read_text()).get("artifacts", [])+ except (OSError, ValueError):+ return [{"status": "metadata_unavailable"}]+ for container in containers:+ for index, path in enumerate(declared[:32]):+ if time.monotonic() >= deadline:+ return [*records, {"status": "recovery_budget_exhausted"}]+ if not isinstance(path, str):+ continue+ source = Path(path)+ if not source.is_absolute() or ".." in source.parts or len(source.parts) < 3:+ continue+ if source.parts[1] in {"installed-agent", "proc", "sys", "dev", "etc", "root", "run", "logs"}:+ continue+ destination = root / "recovered-artifacts" / container["Id"][:12] / f"{index:02d}.tar"+ destination.parent.mkdir(parents=True, exist_ok=True)+ try:+ archive_artifact(container["Id"], source, destination, min(3, max(.1, deadline - time.monotonic())))+ status = "archived"+ except (subprocess.SubprocessError, OSError):+ status = "unavailable"+ records.append({"path": path, "status": status, "archive": str(destination.relative_to(root)),+ "extracted_on_host": False})+ return records+++def finalize(root, *, never_started=False):+ # Publishing a cleanup receipt while the unit can still launch work would+ # release the slot early. Only ExecStopPost or a proven absent unit may do so.+ if not never_started and "SERVICE_RESULT" not in os.environ:+ raise ValueError("finalization requires the manager post-stop fence")+ root = Path(root)+ with (root / "finalize.lock").open("a") as lock:+ fcntl.flock(lock, fcntl.LOCK_EX)+ prior = evidence(root / "receipt.json")+ if prior and prior.get("finished_at"):+ return prior+ manifest = load(root / "manifest.json")+ lifecycle = evidence(root / "lifecycle.json") or {}+ ownership = evidence(root / "ownership.json") or {}+ finished = terminal_evidence(manifest, ownership)+ manager = os.environ.get("SERVICE_RESULT", "not-started")+ reason = lifecycle.get("state", "supervisor_lost")+ if never_started:+ reason = "submit_failed"+ elif manager == "timeout":+ reason = "time_limit"+ elif reason not in {"completed", "harbor_error", "time_limit", "interrupted", "launch_failed"}:+ reason = "supervisor_lost"+ if reason == "completed" and not finished:+ reason = "missing_terminal_result"+ receipt = {"schema_version": 1, "state": reason, "harbor_exit_code": lifecycle.get("exit_code"),+ "harbor_finished": finished, "harbor_result_path": str(Path(manifest["job_dir"]) / "result.json"),+ "wall_seconds": manifest["wall_seconds"], "finished_at": None,+ "cleanup": {"confirmed": False, "unit_stopped": True}, "artifacts": []}+ save(root / "receipt.json", receipt) # Even a killed finalizer leaves an honest incomplete receipt.+ deadline = time.monotonic() + FINALIZE_SECONDS+ def bounded(argv):+ left = deadline - time.monotonic()+ if left <= 0:+ raise TimeoutError("cleanup budget exhausted")+ return command(argv, timeout=min(5, left))+ projects = ownership.get("projects", [])+ errors = []+ try:+ # This journal is written by the pre-environment Harbor START hook,+ # covering networks created before any container/log mount exists.+ if not projects and (Path(manifest["job_dir"]).exists() or not never_started):+ # Without START there should be no Docker mutation; fail closed+ # if a corrupted/missing ownership journal cannot establish it.+ if not lifecycle.get("no_trial_started"):+ raise ValueError("ownership unavailable")+ owned, networks = docker_state(projects, bounded)+ if not finished:+ try:+ receipt["artifacts"] = copy_artifacts(root, manifest, owned, min(deadline - 15, time.monotonic() + 8))+ except (OSError, ValueError, TypeError):+ receipt["artifacts"] = [{"status": "recovery_unavailable"}]+ for argv in (["docker", "rm", "-f", c["Id"]] for c in owned):+ try:+ bounded(argv)+ except (subprocess.SubprocessError, OSError):+ errors.append("container_removal_failed")+ for net in networks:+ try:+ bounded(["docker", "network", "rm", net["Id"]])+ except (subprocess.SubprocessError, OSError):+ errors.append("network_removal_failed")+ remaining, nets = docker_state(projects, bounded)+ receipt["cleanup"].update(confirmed=not (remaining or nets or errors),+ removed_container_ids=[c["Id"] for c in owned], owned_projects=projects,+ remaining_ids=[c["Id"] for c in remaining], remaining_network_ids=[n["Id"] for n in nets])+ except (subprocess.SubprocessError, OSError, ValueError, TypeError):+ errors.append("owned_cleanup_unconfirmed")+ finally:+ (root / "credential" / "provider.key").unlink(missing_ok=True)+ if errors:+ receipt["cleanup"]["errors"] = errors+ receipt["finished_at"] = time.time()+ save(root / "receipt.json", receipt)+ if receipt["cleanup"]["confirmed"]:+ release(root)+ return receipt+++def redact_stream(pipe, output, secret):+ """Keep a look-behind so split stdout chunks cannot expose a credential."""+ pending, written = "", 0+ with output.open("w", encoding="utf-8") as file:+ os.chmod(output, 0o600)+ while True:+ chunk = pipe.read(4096)+ pending += chunk+ end = len(pending) if not chunk else max(0, len(pending) - len(secret))+ if secret:+ start = pending.find(secret, max(0, end - len(secret)))+ if start >= 0 and start < end < start + len(secret):+ end = start+ text = pending[:end].replace(secret, "[REDACTED]") if secret else pending[:end]+ if written < 16 * 1024 * 1024:+ file.write(text[:16 * 1024 * 1024 - written])+ file.flush()+ written += len(text)+ pending = pending[end:]+ if not chunk:+ break+++def worker(root):+ root = Path(root)+ manifest = load(root / "manifest.json")+ stopped = threading.Event()+ previous_term = signal.signal(signal.SIGTERM, lambda *_: stopped.set())+ previous_int = signal.signal(signal.SIGINT, lambda *_: stopped.set())+ child = thread = None+ reason, code = "launch_failed", None+ try:+ key = consume(root / "credential" / "provider.key")+ env = {k: v for k, v in os.environ.items() if k in {"PATH", "LANG", "LC_ALL", "SSL_CERT_FILE"}}+ env.update(HOME=str(root / "home"), XDG_CONFIG_HOME=str(root / "home/.config"),+ XDG_CACHE_HOME=str(root / "home/.cache"), PYTHONPATH=manifest["source_root"], HARBOR_TELEMETRY="0")+ env[manifest["credential_env"]] = key+ argv = [sys.executable, "-m", "scripts.terminal_bench.harbor_job", "--root", str(root)]+ deadline = time.monotonic() + max(0, manifest["created_at"] + manifest["wall_seconds"] - time.time())+ child = subprocess.Popen(argv, cwd=manifest["source_root"], env=env, stdin=subprocess.DEVNULL,+ stdout=subprocess.PIPE, stderr=subprocess.STDOUT, text=True,+ encoding="utf-8", errors="replace", start_new_session=True)+ thread = threading.Thread(target=redact_stream, args=(child.stdout, root / "harbor.log", key), daemon=True)+ thread.start()+ save(root / "lifecycle.json", {"state": "running", "pid": child.pid})+ while child.poll() is None and not stopped.is_set() and time.monotonic() < deadline:+ stopped.wait(.2)+ if child.poll() is None:+ reason = "time_limit" if time.monotonic() >= deadline else "interrupted"+ os.killpg(child.pid, signal.SIGTERM)+ try:+ child.wait(timeout=5)+ except subprocess.TimeoutExpired:+ os.killpg(child.pid, signal.SIGKILL)+ child.wait(timeout=5)+ else:+ reason = "completed" if child.returncode == 0 else "harbor_error"+ code = child.returncode+ finally:+ try:+ if child is not None and child.poll() is None:+ os.killpg(child.pid, signal.SIGKILL)+ child.wait(timeout=5)+ if thread:+ thread.join(timeout=2)+ # No receipt/slot release here. systemd kills every remaining unit+ # process before ExecStopPost performs the definitive Docker sweep.+ save(root / "lifecycle.json", {"state": reason, "exit_code": code,+ "no_trial_started": not (root / "ownership.json").exists()})+ finally:+ signal.signal(signal.SIGTERM, previous_term)+ signal.signal(signal.SIGINT, previous_int)+ return 0 if reason == "completed" else 1+++def unit_state(unit):+ data = command(["systemctl", "show", unit, "--property=LoadState,ActiveState"], timeout=5)+ return dict(line.split("=", 1) for line in data.splitlines() if "=" in line)+++def submit(config, root, credential, wall, credential_env=None):+ metadata(Path(credential)) # Validate before claiming ownership or consuming.+ root = absolute(root, exists=False)+ manifest = prepare(config, root, wall, credential_env)+ try:+ secret = consume(Path(credential))+ (root / "credential").mkdir(mode=0o700)+ with (root / "credential" / "provider.key").open("x", encoding="utf-8") as file:+ os.fchmod(file.fileno(), 0o600)+ file.write(secret)+ entry = [sys.executable, "-m", "scripts.terminal_bench.remote_job"]+ stop = " ".join(json.dumps(s, ensure_ascii=False) for s in [*entry, "finalize", "--root", str(root)])+ command(["systemd-run", "--quiet", "--unit=" + manifest["unit"],+ "--property=Type=exec", "--property=KillMode=control-group",+ "--property=WorkingDirectory=" + manifest["source_root"],+ "--property=RuntimeMaxSec=" + str(wall), "--property=TimeoutStopSec=90",+ "--property=ExecStopPost=" + stop,+ "--", *entry, "worker", "--root", str(root)])+ except Exception:+ # Launch acknowledgement may have been lost AFTER the manager accepted+ # the unit. Fence that exact unit; never publish an early clean receipt.+ try:+ state = unit_state(manifest["unit"])+ if state.get("LoadState") == "not-found":+ finalize(root, never_started=True)+ else:+ command(["systemctl", "stop", "--no-block", manifest["unit"]], timeout=5)+ except (subprocess.SubprocessError, OSError, ValueError):+ pass # Reservation deliberately retained; status reports the ambiguity.+ raise+ return {"unit": manifest["unit"], "receipt": str(root / "receipt.json")}+++def status(root):+ receipt = evidence(root / "receipt.json")+ if receipt and receipt.get("finished_at"):+ return receipt, 0+ manifest = load(root / "manifest.json")+ state = unit_state(manifest["unit"])+ if state.get("ActiveState") in {"active", "activating", "deactivating"}:+ return {"state": "waiting", "unit": manifest["unit"]}, 75+ return {"state": "missing_finalization_receipt", "unit": manifest["unit"], "cleanup_confirmed": False}, 1+++def main(argv=None):+ parser = argparse.ArgumentParser(description=__doc__)+ parser.add_argument("action", choices=["preflight", "submit", "worker", "finalize", "status"])+ parser.add_argument("--config", type=Path)+ parser.add_argument("--root", type=Path)+ parser.add_argument("--credential-file", type=Path)+ parser.add_argument("--credential-env")+ parser.add_argument("--wall-seconds", type=int, default=1800)+ args = parser.parse_args(argv)+ try:+ code = 0+ if args.action == "preflight":+ validate(args.config, args.credential_env)+ result = {"preflight": "passed", "model_requests": 0}+ elif args.action == "submit":+ result = submit(args.config, args.root, args.credential_file, args.wall_seconds, args.credential_env)+ elif args.action == "worker":+ return worker(args.root)+ elif args.action == "finalize":+ result = finalize(args.root)+ else:+ result, code = status(args.root)+ print(json.dumps(result))+ return code+ except Exception as exc:+ print(json.dumps({"error_type": type(exc).__name__}), file=sys.stderr)+ return 1+++if __name__ == "__main__":+ raise SystemExit(main())diff --git a/scripts/terminal_bench/requirements-harbor.txt b/scripts/terminal_bench/requirements-harbor.txtnew file mode 100644index 000000000..ab9eec1de--- /dev/null+++ b/scripts/terminal_bench/requirements-harbor.txt@@ -0,0 +1,2 @@+# Optional SOURCE-ONLY benchmark host tools. Not a product dependency.+harbor==0.24.0diff --git a/scripts/terminal_bench/runner.py b/scripts/terminal_bench/runner.pynew file mode 100644index 000000000..65740d6ed--- /dev/null+++ b/scripts/terminal_bench/runner.py@@ -0,0 +1,495 @@+"""Thin full-product trial supervisor (no Harbor or product dependency changes).++Use the interpreter in the pinned Avibe wheel's venv. This is not a sandbox: real+benchmarks must run inside an isolated task container. Local tests default to an+owned HOME; sandbox callers may supply their explicit task image environment.+"""+from __future__ import annotations++import argparse+import json+import os+from pathlib import Path+import re+import secrets+import signal+import subprocess+import sys+import tempfile+import threading+import time+from typing import Mapping++import psutil++from .diagnostics import project_responses, response_query+from .profile import Profile, validate_layout++_CONTROL = Path(__file__).with_name("control.py")+_OWNER = "AVIBE_BENCHMARK_OWNER"+_TERMINAL = {"succeeded", "failed", "canceled", "cancelled"}+++class TrialError(RuntimeError):+ """A wrapper-owned symbolic code, never an upstream diagnostic body."""++ _CODES = {+ "owned_process_cleanup_unconfirmed", "service_exited_before_ready",+ "controller_readiness_timeout", "service_not_ready", "harness_quiescence_unconfirmed",+ "credential_input_refused", "cli_helper_invalid_run_payload",+ }+ _HELPER_CODE = re.compile(+ r"(?:initialize|serve|health|configure|usage|versions|cli)_helper_"+ r"(?:timeout|invalid_json|failed_-?[0-9]+)\Z"+ )++ def __init__(self, code: str):+ self.code = code if code in self._CODES or self._HELPER_CODE.fullmatch(code) else "unclassified_trial_error"+ super().__init__(self.code)+++def failure_details(exc: BaseException) -> dict[str, str]:+ result = {"outcome": "infrastructure_error", "error_type": type(exc).__name__}+ if isinstance(exc, TrialError):+ result["error_code"] = exc.code+ return result+++def scrub(value, key: str | None):+ if isinstance(value, str):+ return value.replace(key, "[REDACTED]") if key else value+ if isinstance(value, dict):+ return {scrub(str(k), key): scrub(v, key) for k, v in value.items()}+ if isinstance(value, (list, tuple)):+ return [scrub(v, key) for v in value]+ return value+++class OwnedProcesses:+ """Observe before shutdown, including children that create their own sessions.++ Pids are never authority on their own. Capture creation identity while the+ process is a descendant, or proves our random per-trial environment marker.+ Never read a global Avibe runtime pidfile or kill by executable name.+ """++ def __init__(self, process: subprocess.Popen, marker: str):+ self.process = process+ self.marker = marker+ self.identities: dict[int, float] = {}+ self._stop = threading.Event()+ self._lock = threading.Lock()+ self._remember(psutil.Process(process.pid))+ self._thread = threading.Thread(target=self._observe, daemon=True)+ self._thread.start()++ def _remember(self, process):+ try:+ created = process.create_time()+ except psutil.NoSuchProcess:+ return+ with self._lock:+ self.identities[process.pid] = created++ def _live(self):+ with self._lock:+ recorded = list(self.identities.items())+ live = []+ for pid, created in recorded:+ try:+ process = psutil.Process(pid)+ current_creation = process.create_time()+ if current_creation != created:+ # On macOS create_time can shift with the system clock.+ # A readable matching marker still proves ownership; an+ # unreadable identity cannot prove safe cleanup.+ if process.environ().get(_OWNER) != self.marker:+ continue+ self._remember(process)+ if process.status() != psutil.STATUS_ZOMBIE:+ live.append(process)+ except psutil.NoSuchProcess:+ continue+ return live++ def refresh(self):+ for process in self._live():+ try:+ for child in process.children(recursive=True):+ self._remember(child)+ except psutil.NoSuchProcess:+ pass++ def discover_detached(self):+ # Detached engines/jobs inherit this random marker; records survive+ # reparenting. No environment values are logged or exported.+ for process in psutil.process_iter():+ try:+ if process.environ().get(_OWNER) == self.marker:+ self._remember(process)+ except (psutil.NoSuchProcess, psutil.AccessDenied):+ continue++ def _observe(self):+ while not self._stop.wait(0.1):+ try:+ self.refresh()+ except psutil.Error:+ # Rechecked synchronously and fail-closed by close().+ pass++ @staticmethod+ def _signal(process, *, kill=False):+ try:+ process.kill() if kill else process.terminate()+ except psutil.NoSuchProcess:+ pass++ def close(self, grace: float) -> dict:+ self.refresh()+ self.discover_detached()+ if self.process.poll() is None:+ self.process.terminate() # main.py's real Controller shutdown owner+ deadline = time.monotonic() + grace+ while time.monotonic() < deadline:+ self.refresh()+ if not self._live():+ break+ time.sleep(0.05)+ self.discover_detached()+ for process in self._live():+ self._signal(process)+ _, alive = psutil.wait_procs(self._live(), timeout=1)+ for process in alive:+ # _live rechecks create_time before every forced-signal batch.+ if process in self._live():+ self._signal(process, kill=True)+ psutil.wait_procs(self._live(), timeout=2)+ self.process.wait(timeout=3)+ self._stop.set()+ self._thread.join(timeout=2)+ self.discover_detached()+ if self._live():+ raise TrialError("owned_process_cleanup_unconfirmed")+ return {"owned_processes_stopped": True, "observed_process_count": len(self.identities)}+++class FullRuntimeTrial:+ """A fresh home, the normal service, IPC bootstrap, and ordinary CLI Run.++ source_root is for source-checkout tests only. Omit it to exercise the pinned+ installed wheel. No source package subsets are copied into the task.+ """++ def __init__(self, workdir: Path, state: Path, output: Path, profile: Profile, *,+ task_environment: Mapping[str, str] | None = None, source_root: Path | None = None):+ validate_layout(workdir, state, output)+ self.workdir, self.state, self.output = (p.resolve() for p in (workdir, state, output))+ self.profile = profile+ self.source_root = source_root.resolve() if source_root else None+ self.state.mkdir(parents=True, exist_ok=False, mode=0o700)+ try:+ self.output.mkdir(parents=True, exist_ok=False, mode=0o700)+ except OSError:+ self.state.rmdir() # only our just-created, empty reservation+ raise+ self.marker = secrets.token_hex(24)+ self.home = self.state / "home"+ self.home.mkdir()+ self.avibe_home = self.state / "avibe"+ self.env = self._environment(task_environment)+ self.profile_path = self.state / "profile.json"+ self.profile_path.write_text(json.dumps(profile.payload()), encoding="utf-8")+ self._service = None+ self._processes = None+ self._log = None+ self._socket_dir = None+ self._key = None+ self._result = {"outcome": "not_started", "verifier_reward": None, "usage": None,+ "accounting_complete": False, "cleanup": None, "run_id": None, "session_id": None}++ def _environment(self, task_environment):+ if task_environment is None:+ env = {"PATH": os.pathsep.join([str(Path(sys.executable).parent), os.defpath]),+ "HOME": str(self.home), "USERPROFILE": str(self.home),+ "XDG_CONFIG_HOME": str(self.home / ".config"),+ "XDG_DATA_HOME": str(self.home / ".local/share"),+ "XDG_CACHE_HOME": str(self.home / ".cache"), "TMPDIR": str(self.state / "tmp")}+ (self.state / "tmp").mkdir()+ else:+ # This mapping is the sandbox image's environment, not os.environ+ # of the user's machine. Retain task toolchains and the real HOME.+ env = dict(task_environment)+ for name in list(env):+ upper = name.upper()+ if (upper.startswith(("AVIBE_", "VIBE_", "ANTHROPIC_", "OPENAI_", "CODEX_", "CLAUDE_"))+ or name == self.profile.credential_env):+ env.pop(name)+ env.update({+ "AVIBE_HOME": str(self.avibe_home), _OWNER: self.marker,+ "AVIBE_WEB_UI_SUPERVISOR": "terminal-bench-no-ui",+ "VIBE_SENTRY_DSN": "", "VIBE_STARTUP_DEPENDENCY_RECONCILE": "0",+ "VIBE_INSTALL_SKIP_ASKILL": "1", "VIBE_ASKILL_AUTO_UPDATE": "0",+ "VIBE_SHOW_RUNTIME_OFFLINE": "1" if self.profile.offline else "0",+ "VIBE_TMUX_OFFLINE": "1" if self.profile.offline else "0",+ "VIBE_GIT_OFFLINE": "1" if self.profile.offline else "0",+ "VIBE_MODEL_HUB_ENGINE_OFFLINE": "1" if self.profile.offline else "0",+ "VIBE_MODEL_HUB_ENABLED": "1", "PYTHONUNBUFFERED": "1",+ })+ # Local metadata avoids catalog/network drift; it does not change model+ # requests, Agent prompts, tools, Harness or context management.+ catalog = self.state / "empty-models-dev.json"+ catalog.write_text("{}", encoding="utf-8")+ env["AVIBE_MODELS_DEV_URL"] = catalog.as_uri()+ # Use the installed product's bundled backend catalog, resolved in child.+ return env++ def _command(self, operation: str, *args: str):+ if operation == "cli":+ code = "import runpy; runpy.run_module('vibe',run_name='__main__')"+ command_args = list(args)+ else:+ code = "import runpy,sys; runpy.run_path(sys.argv.pop(1),run_name='__main__')"+ command_args = [str(_CONTROL), operation, *args]+ if self.source_root:+ code = "import sys; sys.path.insert(0,sys.argv.pop(1)); " + code+ command_args.insert(0, str(self.source_root))+ return [sys.executable, "-I", "-c", code, *command_args]++ def _call(self, operation: str, *args: str, timeout=None, stdin=None):+ try:+ proc = subprocess.run(self._command(operation, *args), cwd=self.state, env=self.env,+ input=stdin, text=True, encoding="utf-8", capture_output=True,+ timeout=timeout or self.profile.setup_seconds)+ except subprocess.TimeoutExpired:+ raise TrialError(f"{operation}_helper_timeout") from None+ if proc.returncode:+ raise TrialError(f"{operation}_helper_failed_{proc.returncode}")+ try:+ return json.loads(proc.stdout)+ except ValueError:+ raise TrialError(f"{operation}_helper_invalid_json") from None++ def _call_run_show(self, run_id: str, *, timeout: float = 15):+ """Accept a valid durable Run payload even when CLI exits nonzero.++ ``vibe runs show`` returns a meaningful failed Run alongside a nonzero+ exit code. That is a product outcome, not helper infrastructure.+ """+ argv = self._command("cli", "runs", "show", run_id)+ try:+ proc = subprocess.run(argv, cwd=self.state, env=self.env,+ text=True, encoding="utf-8", capture_output=True,+ timeout=timeout)+ except subprocess.TimeoutExpired:+ raise TrialError("cli_helper_timeout") from None+ try:+ payload = json.loads(proc.stdout)+ row = payload.get("run") if isinstance(payload, dict) else None+ if not isinstance(row, dict) or row.get("id") not in {None, run_id}:+ raise ValueError+ return payload+ except (TypeError, ValueError, json.JSONDecodeError):+ if proc.returncode:+ raise TrialError(f"cli_helper_failed_{proc.returncode}") from None+ raise TrialError("cli_helper_invalid_run_payload") from None++ def start(self):+ if os.name != "nt":+ # UNIX socket names have a ~104-byte bound on macOS. Follow the+ # test/operator TMPDIR (HD1 here), not a long nested trial path.+ self._socket_dir = tempfile.TemporaryDirectory(prefix="tb-ipc-")+ self.env["VIBE_INTERNAL_DISPATCH_SOCKET"] = str(Path(self._socket_dir.name) / "dispatch.sock")+ self._result["installation"] = self._call("initialize", "--workdir", str(self.workdir))+ self._result["installation"].update(self._call("versions"))+ self.env["AVIBE_BACKEND_MODEL_CATALOG_URL"] = Path(self._result["installation"]["backend_catalog"]).as_uri()+ self._log = (self.state / "service.log").open("w", encoding="utf-8")+ self._service = subprocess.Popen(self._command("serve"), cwd=self.state, env=self.env,+ stdout=self._log, stderr=self._log, start_new_session=True)+ self._processes = OwnedProcesses(self._service, self.marker)+ deadline = time.monotonic() + self.profile.setup_seconds+ while time.monotonic() < deadline:+ if self._service.poll() is not None:+ raise TrialError("service_exited_before_ready")+ health = self._call("health", timeout=min(10, self.profile.setup_seconds))+ if health.get("ready"):+ self._result["controller_health"] = health["identity"]+ self._result["service_pid"] = self._service.pid+ self._result["outcome"] = "ready"+ return+ time.sleep(self.profile.poll_seconds)+ raise TrialError("controller_readiness_timeout")++ def execute(self, instruction: str, *, key: str | None):+ if self._result["outcome"] != "ready":+ raise TrialError("service_not_ready")+ if not key:+ self._result.update(outcome="blocked", blocker="model_source_credential_required")+ return+ self._key = key.strip()+ if not self._key:+ self._result.update(outcome="blocked", blocker="model_source_credential_required")+ return+ self._result["source"] = self._call("configure", "--profile", str(self.profile_path), stdin=self._key)+ started_at = time.monotonic()+ self._call("cli", "agent", "update", "vibey", "--model", self.profile.model,+ "--reasoning-effort", self.profile.reasoning_effort)+ instruction_path = self.state / "instruction.txt"+ instruction_path.write_text(instruction, encoding="utf-8")+ # Explicit asynchronous creation preserves stable ids before waiting.+ # Expiry of a CLI waiter does not cancel an Agent turn.+ accepted = self._call("cli", "agent", "run", "--agent", "vibey", "--cwd", str(self.workdir),+ "--message-file", str(instruction_path), "--no-callback")+ run_id, session_id = accepted["run_id"], accepted["session_id"]+ self._result.update(run_id=run_id, session_id=session_id)+ deadline = time.monotonic() + self.profile.wall_seconds+ terminal_row = None+ quiet_observations = 0+ while time.monotonic() < deadline:+ row = self._call_run_show(run_id)["run"]+ if row["status"] in _TERMINAL:+ terminal_row = row+ # A Run's final reply can hand over to a Watch, scheduled Task+ # or subagent. They are real product work, not noise to discard+ # at the first terminal. This home belongs only to this trial.+ status = self._call("cli", "harness", "status", timeout=15)+ if (not status.get("controller", {}).get("available")+ or any(status.get("truncated", {}).values())):+ raise TrialError("harness_quiescence_unconfirmed")+ busy = bool(status.get("runs") or status.get("watches") or status.get("tasks"))+ busy = busy or any(item.get("state") == "active" for item in status.get("runtime_agents", []))+ quiet_observations = 0 if busy else quiet_observations + 1+ if quiet_observations >= 2:+ self._result.update(outcome=terminal_row["status"], final_text=terminal_row.get("result_text"),+ run_error=terminal_row.get("error"), harness_quiescent=True)+ break+ time.sleep(self.profile.poll_seconds)+ else:+ self._result["outcome"] = "time_limit"+ try:+ cancelled = self._call("cli", "runs", "cancel", run_id, timeout=15)+ self._result["cancel_requested"] = cancelled.get("ok") is True+ except TrialError as exc:+ # A concurrent terminal or unavailable IPC must not skip the+ # final service/tree teardown, nor erase the deadline outcome.+ self._result["cancel_requested"] = False+ self._result["cancel_error"] = str(exc)+ # Cancellation is asynchronous; a receipt alone is not quiescence.+ cancel_deadline = time.monotonic() + self.profile.cleanup_seconds+ self._result["run_settled_after_cancel"] = False+ while time.monotonic() < cancel_deadline:+ try:+ row = self._call_run_show(run_id)["run"]+ except TrialError:+ break+ if row["status"] in _TERMINAL:+ self._result["run_settled_after_cancel"] = True+ break+ time.sleep(self.profile.poll_seconds)+ # Service shutdown and exact-owned-process reaping still follow;+ # even a terminal Run may own Watch-backed work after its turn.+ self._result["elapsed_seconds"] = time.monotonic() - started_at+ self._result["outcome_scope"] = "initial_run_after_fresh_home_harness_quiescence_or_deadline"+ self._result["final_text_scope"] = "initial_run_result"+ # Entire fresh-home accounting, including bootstrap calls, is identified+ # as such. Never claim completeness for interrupted/provider-failed calls.+ try:+ self._result["usage"] = self._call("usage", timeout=15)+ except TrialError:+ self._result["usage_unavailable"] = True++ def close(self):+ # Capture only allowlisted response shapes while the normal CLI can+ # still reach this trial's service. Failure is explicit but never+ # changes the measured outcome or prevents owned-process cleanup.+ session_id = self._result.get("session_id")+ self._result["diagnostics"] = {"schema_version": 1, "status": "no_session", "raw_text_exported": False}+ if session_id and self._service and self._service.poll() is None:+ try:+ payload = self._call("cli", "data", "query", "--sql", response_query(session_id), "--limit", "9", timeout=10)+ self._result["diagnostics"] = project_responses(payload)+ except Exception:+ self._result["diagnostics"] = {"schema_version": 1, "status": "unavailable", "raw_text_exported": False}+ elif session_id:+ self._result["diagnostics"]["status"] = "service_unavailable"+ try:+ if self._processes:+ self._result["cleanup"] = self._processes.close(self.profile.cleanup_seconds)+ finally:+ if self._log:+ self._log.close()+ if self._socket_dir:+ self._socket_dir.cleanup()+ self._result["profile"] = self.profile.payload()+ self._result["execution_path"] = "normal_service_model_hub_ipc_and_cli_run"+ self._result["runtime_state_exported"] = False+ self._result.setdefault("installation", {})["mode"] = "source_checkout" if self.source_root else "installed_package"+ self._result["accounting_scope"] = "fresh_home_reported_usage_including_bootstrap_not_actual_bill"+ result = scrub(self._result, self._key)+ (self.output / "execution.json").write_text(json.dumps(result, ensure_ascii=False, indent=2), encoding="utf-8")+ return result++ def __enter__(self):+ try:+ self.start()+ return self+ except BaseException as exc:+ self._result.update(failure_details(exc))+ self.close()+ raise++ def __exit__(self, exc_type, exc, _tb):+ if exc is not None:+ self._result.update(failure_details(exc))+ self.close()+++def _interrupt_trial(signum, frame):+ # A second TERM must not interrupt cleanup. The outer container supervisor+ # can still use KILL if its independent final grace deadline expires.+ signal.signal(signal.SIGTERM, signal.SIG_IGN)+ raise KeyboardInterrupt+++def main(argv=None):+ parser = argparse.ArgumentParser(description=__doc__)+ for name in ("workdir", "state", "output", "profile", "instruction"):+ parser.add_argument(f"--{name}", type=Path, required=True)+ parser.add_argument("--credential-file", type=Path,+ help="Private per-trial input; consumed and unlinked before service startup.")+ parser.add_argument("--sandbox-environment", action="store_true",+ help="Use only inside an externally isolated task container; preserve its HOME/toolchains.")+ args = parser.parse_args(argv)+ profile = Profile.parse(json.loads(args.profile.read_text(encoding="utf-8")))+ if args.credential_file is not None:+ from .credential import consume+ try:+ key = consume(args.credential_file)+ except Exception:+ summary = json.dumps(failure_details(TrialError("credential_input_refused")))+ print(summary)+ print(summary, file=sys.stderr)+ return 2+ else:+ # Direct invocation may retain its named env mode. The Harbor Docker+ # adapter must use credential-file instead: exec env leaks into argv.+ key = os.environ.get(profile.credential_env)+ trial = FullRuntimeTrial(args.workdir, args.state, args.output, profile,+ task_environment=os.environ if args.sandbox_environment else None)+ previous_term = signal.signal(signal.SIGTERM, _interrupt_trial)+ try:+ with trial:+ trial.execute(args.instruction.read_text(encoding="utf-8"), key=key)+ except (Exception, KeyboardInterrupt) as exc:+ summary = json.dumps(failure_details(exc))+ print(summary)+ print(summary, file=sys.stderr)+ return 2+ finally:+ signal.signal(signal.SIGTERM, previous_term)+ print(json.dumps({"outcome": trial._result["outcome"], "report": str(trial.output / "execution.json")}))+ return 0+++if __name__ == "__main__":+ raise SystemExit(main())diff --git a/storage/agent_transcript.py b/storage/agent_transcript.pyindex ccd05b4df..8c6f7d1d5 100644--- a/storage/agent_transcript.py+++ b/storage/agent_transcript.py@@ -125,12 +125,12 @@ def final_outcome(message: AssistantMessage) -> FinalOutcome:"""The Turn outcome a committed final response determines by itself.The one rule, shared by the row's type at commit and live settlement: a final- reply without text of its own (an unexplained refusal or safety stop, or an- empty answer) failed; any other final completed. It is the loop's own- empty-reply test (loop-control.md section 2). Failures only a live run can- observe after the commit are applied by the live caller.+ reply without usable content, or explicitly stopped at its output limit,+ failed. Partial text must not turn a provider's incomplete final into a+ successful result. Pending-input responses are not finals. Failures only+ a live run can observe after the commit are applied by the live caller."""- return "completed" if has_reply(message) else "failed"+ return "completed" if has_reply(message) and message.stop_reason != "length" else "failed"def _response_type(message: AssistantMessage, *, final: bool, display_text: str) -> str:diff --git a/tests/agent_core/agent/test_loop.py b/tests/agent_core/agent/test_loop.pyindex 14831ae48..c437a31c1 100644--- a/tests/agent_core/agent/test_loop.py+++ b/tests/agent_core/agent/test_loop.py@@ -1263,12 +1263,12 @@ async def test_tool_free_final_reply_is_committed_and_empty_success_is_an_error(["input"] if continuation == "final" else ["input", "pending"])errors = [event for event in events if isinstance(event, AgentError)]- if shape in ("visible", "image") or continuation != "final":+ if continuation != "final" or (shape in ("visible", "image") and stop_reason != "length"):assert errors == []assert events[-1].reason == "completed"else:assert len(errors) == 1- assert errors[0].kind == (stop_reason if stop_reason in {"refusal", "safety"} else "empty_response")+ assert errors[0].kind == (stop_reason if stop_reason in {"length", "refusal", "safety"} else "empty_response")assert errors[0].messageassert committed[0].seq < errors[0].seq < events[-1].seqassert events[-1].reason == "error"diff --git a/tests/fixtures/full_runtime_service.py b/tests/fixtures/full_runtime_service.pynew file mode 100644index 000000000..b4378b78a--- /dev/null+++ b/tests/fixtures/full_runtime_service.py@@ -0,0 +1,76 @@+"""Child test fixture: replace ONLY the external engine, run normal main.py.++The real Controller/Model Hub service/router/gateway/provider, Session/Harness,+Vibey tools and SQLite run unchanged. This is not real CPA acceptance evidence.+"""+from __future__ import annotations++import json+import os+import sys+from pathlib import Path++REPO = Path(__file__).resolve().parents[2]+sys.path.insert(0, str(REPO))++from tests import network_guard+sys.addaudithook(network_guard.audit)++from core.handlers.model_hub.adapter import RawOutcomeKind+from tests.scenario_harness.model_hub import ModelHubScenarioAdapter, ScenarioCallResult+++class ScriptedEngine(ModelHubScenarioAdapter):+ async def invoke(self, source_id, model_id, request, stream, origin, *, on_admitted=None):+ messages = request.get("messages", [])+ has_result = any(item.get("role") == "tool" for item in messages)+ if not has_result:+ program = (+ "import os,sqlite3; from pathlib import Path; "+ "Path('笔记.md').write_text('可查询的任务笔记', encoding='utf-8'); "+ "db=sqlite3.connect('task.sqlite'); "+ "db.execute('CREATE TABLE evidence(value TEXT)'); "+ "db.execute(\"INSERT INTO evidence VALUES ('task-data')\"); db.commit(); db.close(); "+ "Path('task-env.json').write_text(__import__('json').dumps({"+ "'cwd':os.getcwd(),'home':os.environ['HOME'],"+ "'credential_present':'DEEPSEEK_API_KEY' in os.environ,"+ "'session':os.environ.get('AVIBE_SESSION_ID'),"+ "'java_home':os.environ.get('JAVA_HOME')}))"+ )+ if os.environ.get("TB_TEST_SCENARIO") == "slow-tool":+ program = "import time; from pathlib import Path\nwhile True:\n Path('still-writing').write_text(str(time.time_ns())); time.sleep(.02)"+ import shlex+ command = "python3 -c " + shlex.quote(program)+ if os.environ.get("TB_TEST_SCENARIO") == "watch":+ program = "import time; from pathlib import Path; time.sleep(4); Path('watch-note.txt').write_text('continued')"+ command = "vibe watch add --message 'The note is ready; finish.' -- python3 -c " + shlex.quote(program)+ delta = {"tool_calls": [{"index": 0, "id": "fixture-call", "type": "function",+ "function": {"name": "bash", "arguments": json.dumps({+ "command": command,+ "timeout": 20, "watch": False})}}]}+ reason = "tool_calls"+ else:+ scenario = os.environ.get("TB_TEST_SCENARIO")+ if scenario == "empty-response":+ delta, reason = {}, "stop"+ elif scenario == "thinking-length":+ delta, reason = {"reasoning_content": "private-reasoning-秘密"}, "length"+ else:+ delta, reason = {"content": "Finished integration fixture. fixture-private-key-not-real"}, "stop"+ rows = [+ {"id": "fixture-response", "model": model_id,+ "choices": [{"index": 0, "delta": delta, "finish_reason": None}]},+ {"id": "fixture-response", "choices": [{"index": 0, "delta": {}, "finish_reason": reason}],+ "usage": {"prompt_tokens": 100, "completion_tokens": 20}},+ ]+ body = ("".join("data: " + json.dumps(row) + "\n\n" for row in rows) + "data: [DONE]\n\n").encode()+ self.invoke_results.append(ScenarioCallResult(RawOutcomeKind.SUCCESS, status=200, body=body, stream_started=True))+ return await super().invoke(source_id, model_id, request, stream, origin, on_admitted=on_admitted)+++if __name__ == "__main__":+ import vibe.model_hub_runtime+ adapter = ScriptedEngine(discovery_models=["fixture-model"])+ vibe.model_hub_runtime.get_model_hub_engine_adapter = lambda: adapter+ from scripts.terminal_bench.control import main+ raise SystemExit(main(["serve"]))diff --git a/tests/test_terminal_bench_credential.py b/tests/test_terminal_bench_credential.pynew file mode 100644index 000000000..efae1b4e3--- /dev/null+++ b/tests/test_terminal_bench_credential.py@@ -0,0 +1,138 @@+"""Ephemeral file transport: metadata-only inspection and consume-before-start."""+from __future__ import annotations++import json+import os+from pathlib import Path+import subprocess++import pytest++from scripts.terminal_bench.credential import LIMIT, clean, consume, metadata, prepare_upload+from scripts.terminal_bench.profile import Profile+from scripts.terminal_bench import runner++SENTINEL = "fake-private-key-not-for-logs"+++def secret_file(tmp_path):+ private = tmp_path / "credential"+ private.mkdir(mode=0o700)+ path = private / "provider.key"+ descriptor = os.open(path, os.O_WRONLY | os.O_CREAT | os.O_EXCL, 0o600)+ with os.fdopen(descriptor, "w") as handle:+ handle.write(SENTINEL)+ return path++[email protected]("shape", ["normal", "symlink", "hardlink", "oversized", "public", "fifo", "empty"])+def test_input_metadata_and_failure_removal(tmp_path, shape):+ path = secret_file(tmp_path)+ external = tmp_path / "unchanged"+ external.write_text("outside")+ if shape == "symlink":+ path.unlink()+ path.symlink_to(external)+ elif shape == "hardlink":+ path.unlink()+ os.link(external, path)+ path.chmod(0o600)+ elif shape == "oversized":+ path.write_bytes(b"x" * (LIMIT + 1))+ elif shape == "public":+ path.chmod(0o644)+ elif shape == "fifo":+ path.unlink()+ os.mkfifo(path, 0o600)+ elif shape == "empty":+ path.write_text("")+ if shape == "normal":+ prepared = prepare_upload(path)+ info = metadata(path)+ assert prepared == info+ assert info == {"regular": True, "mode": "0600", "directory_mode": "0700", "size": len(SENTINEL)}+ assert SENTINEL not in json.dumps(info)+ assert consume(path) == SENTINEL+ else:+ before = external.stat()+ with pytest.raises((OSError, ValueError)):+ prepare_upload(path)+ assert (external.stat().st_uid, external.stat().st_mode) == (before.st_uid, before.st_mode)+ with pytest.raises((OSError, ValueError)):+ consume(path)+ assert not path.exists() and not path.is_symlink()+ assert external.read_text() == "outside"+ assert clean(path) == {"removed": True}+ assert not path.parent.exists()++[email protected](os.geteuid() != 0, reason="real uploaded UID adoption needs a root Linux sandbox")+def test_prepare_adopts_foreign_upload_uid_but_read_stays_strict(tmp_path):+ """Compose cp preserves UID501; strict consume must work only after setup adopts it."""+ path = secret_file(tmp_path)+ os.chown(path, 501, -1)+ with pytest.raises(ValueError, match="owned by the runner"):+ metadata(path)+ assert prepare_upload(path)["mode"] == "0600"+ assert path.stat().st_uid == os.geteuid()+ assert consume(path) == SENTINEL+ assert not path.exists()+++def test_rejects_symlink_parent_without_touching_other_directory(tmp_path):+ elsewhere = tmp_path / "other-trial"+ elsewhere.mkdir()+ actual = secret_file(elsewhere)+ (tmp_path / "credential").symlink_to(actual.parent, target_is_directory=True)+ with pytest.raises(OSError):+ consume(tmp_path / "credential/provider.key")+ assert actual.read_text() == SENTINEL+ with pytest.raises(OSError):+ clean(tmp_path / "credential/provider.key")+ assert actual.read_text() == SENTINEL+++def test_key_unlinked_before_service_and_delivered_only_to_bootstrap_stdin(tmp_path, monkeypatch, capsys):+ path = secret_file(tmp_path)+ work = tmp_path / "task"+ work.mkdir()+ profile = tmp_path / "profile.json"+ profile.write_text(json.dumps(Profile(model="fixture", base_url="http://127.0.0.1:9/v1").payload()))+ instruction = tmp_path / "instruction.txt"+ instruction.write_text("fixture")+ observed = []++ def start(self):+ assert not path.exists(), "credential file must disappear BEFORE measured service starts"+ assert self.profile.credential_env not in self.env+ self._result["outcome"] = "ready"++ def helper(argv, **kwargs):+ observed.append((argv, kwargs))+ assert SENTINEL not in " ".join(argv)+ assert SENTINEL not in json.dumps(kwargs["env"])+ if "configure" in argv:+ assert kwargs["input"] == SENTINEL+ result = {"source_id": "fixture-source"}+ elif "agent" in argv and "run" in argv:+ result = {"run_id": "fixture-run", "session_id": "fixture-session"}+ elif "runs" in argv:+ result = {"run": {"status": "succeeded", "result_text": SENTINEL}}+ elif "harness" in argv:+ result = {"controller": {"available": True}, "runs": [], "watches": [], "tasks": []}+ else:+ result = {}+ return subprocess.CompletedProcess(argv, 0, json.dumps(result), "")++ monkeypatch.delenv("DEEPSEEK_API_KEY", raising=False)+ monkeypatch.setattr(runner.FullRuntimeTrial, "start", start)+ monkeypatch.setattr(subprocess, "run", helper)+ assert runner.main([+ "--workdir", str(work), "--state", str(tmp_path / "state"), "--output", str(tmp_path / "out"),+ "--profile", str(profile), "--instruction", str(instruction), "--credential-file", str(path),+ ]) == 0+ output = capsys.readouterr()+ assert observed and SENTINEL not in output.out + output.err+ assert SENTINEL not in (tmp_path / "out/execution.json").read_text()+ assert SENTINEL not in (tmp_path / "state/profile.json").read_text()+ assert not path.exists()diff --git a/tests/test_terminal_bench_diagnostics.py b/tests/test_terminal_bench_diagnostics.pynew file mode 100644index 000000000..00570babe--- /dev/null+++ b/tests/test_terminal_bench_diagnostics.py@@ -0,0 +1,65 @@+"""Response diagnosis retains stop/shape/usage, never private model content."""+import json+import sqlite3+from unittest.mock import Mock++import pytest++from scripts.terminal_bench.diagnostics import project_responses, response_query+from scripts.terminal_bench.profile import Profile+from scripts.terminal_bench.runner import FullRuntimeTrial++[email protected]("stop,shape", [("stop", "empty"), ("stop", "thinking"), ("length", "thinking"), ("length", "text"), ("tool_use", "tool_call")])+def test_query_preserves_cause_without_exporting_model_text(stop, shape):+ with sqlite3.connect(":memory:") as db:+ db.row_factory = sqlite3.Row+ db.execute("CREATE TABLE messages(session_id,type,context_seq,content_json)")+ blocks = [] if shape == "empty" else [{"type": shape, "text": "SECRET-隐私", "arguments": {"key": "SECRET-隐私"}}]+ message = {"role": "assistant", "content": blocks, "stop_reason": stop, "error_message": "SECRET-隐私", "usage": {"output_tokens": 32000}}+ for sid in ("ses_test", "ses_unrelated"):+ db.execute("INSERT INTO messages VALUES(?,?,?,?)", (sid, "error", 2, json.dumps({"model": {"message": message}})))+ rows = [dict(r) for r in db.execute(response_query("ses_test"))]+ assert len(rows) == 1+ assert "SECRET" not in json.dumps(rows)+ report = project_responses({"ok": True, "rows": rows})+ last = report["responses"][0]+ assert last["stop_reason"] == stop+ assert last["thinking_blocks"] == int(shape == "thinking")+ assert last["text_blocks"] == int(shape == "text")+ assert last["tool_calls"] == int(shape == "tool_call")+ assert last["output_tokens"] == 32000+ assert report["raw_text_exported"] is False+++def test_projection_is_bounded_and_does_not_trust_cli_extra_fields():+ row = {"stop_reason": "SECRET", "row_type": "SECRET", "text_chars": "SECRET", "context_seq": 3,+ "input_tokens": -3, "output_tokens": True, "content_text": "SECRET"}+ report = project_responses({"ok": True, "rows": [row] * 100})+ assert len(report["responses"]) == 8+ assert "SECRET" not in json.dumps(report)+ assert report["responses"][0]["input_tokens"] is None+ assert report["responses"][0]["output_tokens"] is None+ assert "LIMIT 32" in response_query("ses_test")+ for sid in ("x' OR 1=1 --", "", "x" * 129):+ with pytest.raises(ValueError):+ response_query(sid)+ with pytest.raises(ValueError):+ project_responses({"ok": False, "error": "SECRET"})+++def test_missing_diagnostics_does_not_mask_run_failure_or_skip_cleanup(tmp_path, monkeypatch):+ task = tmp_path / "task"+ task.mkdir()+ trial = FullRuntimeTrial(task, tmp_path / "state", tmp_path / "out", Profile(model="fixture", base_url="http://127.0.0.1:9/v1"))+ trial._result.update(outcome="failed", session_id="ses_test")+ trial._service = Mock()+ trial._service.poll.return_value = None+ trial._processes = Mock()+ trial._processes.close.return_value = {"owned_processes_stopped": True}+ monkeypatch.setattr(trial, "_call", Mock(side_effect=ValueError("SECRET")))+ report = trial.close()+ assert report["outcome"] == "failed"+ assert report["diagnostics"]["status"] == "unavailable"+ trial._processes.close.assert_called_once()+ assert "SECRET" not in json.dumps(report)diff --git a/tests/test_terminal_bench_full_runtime.py b/tests/test_terminal_bench_full_runtime.pynew file mode 100644index 000000000..2d664e73f--- /dev/null+++ b/tests/test_terminal_bench_full_runtime.py@@ -0,0 +1,284 @@+"""Boundary regressions for the source-only full-product benchmark supervisor.++Observable contracts: fresh disjoint state; inherited caller credentials cannot+route into the live service; real CLI Run dispatch writes task files while all+runtime state stays out; deadline cleanup is owned and bounded. Existing service+acceptance covers real CPA; these tests substitute ONLY that external engine.+"""+from __future__ import annotations++from dataclasses import replace+import json+import os+from pathlib import Path+import sqlite3+import subprocess+import sys+import time++import psutil+import pytest++from scripts.terminal_bench.profile import Profile, validate_layout+from scripts.terminal_bench.runner import FullRuntimeTrial, OwnedProcesses, TrialError++ROOT = Path(__file__).resolve().parents[1]+# Fresh SQLite migrations reached Controller construction only after 30s in+# cold HD1 runs. Startup gets its own 120s allowance; this does not relax the+# 30s Agent deadline or the deliberately short 2s cancellation scenario.+PROFILE = Profile(model="fixture-model", base_url="http://127.0.0.1:9/v1", setup_seconds=120,+ wall_seconds=30, cleanup_seconds=5, poll_seconds=0.1, offline=True)+++def test_disjoint_fresh_layout_and_environment(tmp_path, monkeypatch):+ task = tmp_path / "任务"+ task.mkdir()+ monkeypatch.setenv("AVIBE_SESSION_ID", "real-user-session")+ monkeypatch.setenv("ANTHROPIC_AUTH_TOKEN", "real-user-key")+ before = dict(os.environ)+ trial = FullRuntimeTrial(task, tmp_path / "private", tmp_path / "export", PROFILE)+ assert dict(os.environ) == before+ assert "AVIBE_SESSION_ID" not in trial.env+ assert "ANTHROPIC_AUTH_TOKEN" not in trial.env+ assert "DEEPSEEK_API_KEY" not in trial.env+ assert Path(trial.env["AVIBE_HOME"]).is_relative_to(trial.state)+ assert not list(task.iterdir())+ with pytest.raises(FileExistsError):+ FullRuntimeTrial(task, tmp_path / "private", tmp_path / "other-export", PROFILE)+ for state, output in [(task / "state", tmp_path / "out"), (tmp_path, tmp_path / "out"),+ (tmp_path / "private", tmp_path / "private" / "export")]:+ with pytest.raises(ValueError):+ validate_layout(task, state, output)+ alias = tmp_path / "alias"+ alias.symlink_to(task, target_is_directory=True)+ with pytest.raises(ValueError):+ validate_layout(task, alias, tmp_path / "out")+ monkeypatch.setenv("AVIBE_HOME", str(tmp_path / "custom-user-state"))+ with pytest.raises(ValueError):+ validate_layout(task, tmp_path / "custom-user-state" / "bench", tmp_path / "out")+++def test_sandbox_environment_retains_task_home_and_toolchains(tmp_path):+ task = tmp_path / "task"+ task.mkdir()+ task_env = {"HOME": "/root", "JAVA_HOME": "/opt/java", "RUSTUP_HOME": "/opt/rust",+ "VIRTUAL_ENV": "/opt/task-venv", "PYTHONPATH": "/task-modules",+ "AVIBE_SESSION_ID": "host-session", "DEEPSEEK_API_KEY": "private",+ "VIBE_INTERNAL_DISPATCH_SOCKET": "/live/socket", "TASK_DB_PASSWORD": "task-local-value"}+ trial = FullRuntimeTrial(task, tmp_path / "state", tmp_path / "out", PROFILE, task_environment=task_env)+ for name in ["HOME", "JAVA_HOME", "RUSTUP_HOME", "VIRTUAL_ENV", "PYTHONPATH", "TASK_DB_PASSWORD"]:+ assert trial.env[name] == task_env[name]+ for name in ["AVIBE_SESSION_ID", "DEEPSEEK_API_KEY", "VIBE_INTERNAL_DISPATCH_SOCKET"]:+ assert name not in trial.env+ assert task_env["AVIBE_SESSION_ID"] == "host-session" # caller mapping was not mutated++[email protected]_processes[email protected]_foreign_signals("Full-runtime supervisor reaps its subprocess's recorded descendants by identity")[email protected]("scenario", ["success", "watch", "deadline", "missing_credential", "empty-response", "thinking-length"])+def test_real_service_cli_hub_vibey_writes_and_isolates_state(tmp_path, monkeypatch, scenario):+ task = tmp_path / "任务"+ task.mkdir()+ # This must not shadow product imports, nor be deleted/rewritten.+ (task / "config.py").write_text("task_owned = True\n")+ profile = replace(PROFILE, wall_seconds=2) if scenario == "deadline" else PROFILE+ trial = FullRuntimeTrial(task, tmp_path / "state", tmp_path / "export", profile, source_root=ROOT)+ trial.env["JAVA_HOME"] = "/fixture/java-toolchain"+ if scenario == "watch":+ trial.env["TB_TEST_SCENARIO"] = "watch"+ # Source-only test interpreter: the measured installed wheel has this+ # executable already. The shim lives outside the task and prompt.+ shim = trial.state / "bin" / "vibe"+ shim.parent.mkdir()+ shim.write_text(f"#!{sys.executable}\nimport sys\nsys.path.insert(0, {str(ROOT)!r})\nfrom vibe.cli import main\nmain()\n")+ shim.chmod(0o755)+ trial.env["PATH"] = str(shim.parent) + os.pathsep + trial.env["PATH"]+ if scenario == "deadline":+ trial.env["TB_TEST_SCENARIO"] = "slow-tool"+ if scenario in {"empty-response", "thinking-length"}:+ trial.env["TB_TEST_SCENARIO"] = scenario+ original_command = trial._command++ def command(operation, *args):+ if operation == "serve":+ return [sys.executable, "-I", str(ROOT / "tests/fixtures/full_runtime_service.py")]+ return original_command(operation, *args)++ monkeypatch.setattr(trial, "_command", command)+ before = dict(os.environ)+ with trial:+ assert trial._result["controller_health"]["service"] == "vibe-remote-internal"+ agents = trial._call("cli", "agent", "list")+ assert any(agent["backend"] == "vibey" for agent in agents["agents"])+ key = None if scenario == "missing_credential" else "fixture-private-key-not-real"+ trial.execute("Complete the synthetic tool-write exercise.", key=key)+ if scenario == "success":+ assert trial._result["outcome"] == "succeeded", trial._result+ assert trial._result["run_id"] and trial._result["session_id"]+ assert (task / "笔记.md").read_text(encoding="utf-8") == "可查询的任务笔记"+ with sqlite3.connect(task / "task.sqlite") as db:+ assert db.execute("SELECT value FROM evidence").fetchall() == [("task-data",)]+ env = json.loads((task / "task-env.json").read_text())+ assert env["cwd"] == str(task)+ assert env["session"] == trial._result["session_id"] # real Harness identity+ assert env["credential_present"] is False+ assert env["java_home"] == "/fixture/java-toolchain"+ assert Path(env["home"]) == trial.home+ elif scenario == "watch":+ assert trial._result["outcome"] == "succeeded", trial._result+ assert trial._result["harness_quiescent"] is True+ assert (task / "watch-note.txt").read_text() == "continued"+ elif scenario == "deadline":+ assert trial._result["outcome"] == "time_limit", trial._result+ assert trial._result["run_id"]+ assert (task / "still-writing").exists(), "the real tool must run before the deadline"+ elif scenario in {"empty-response", "thinking-length"}:+ assert trial._result["outcome"] == "failed", trial._result+ assert trial._result["harness_quiescent"] is True+ expected_error = "output limit" if scenario == "thinking-length" else "empty reply"+ assert expected_error in trial._result["final_text"]+ else:+ assert trial._result["outcome"] == "blocked"+ assert trial._result["blocker"] == "model_source_credential_required"+ assert trial._result["run_id"] is None+ assert (trial.avibe_home / "state" / "vibe.sqlite").exists()+ assert not (trial.avibe_home / "runtime" / "vibe-ui.pid").exists()+ assert not (trial.avibe_home / "runtime" / "remote-access-cloudflared.pid").exists()+ assert dict(os.environ) == before+ expected = {"config.py", "笔记.md", "task.sqlite", "task-env.json"} if scenario in {"success", "empty-response", "thinking-length"} else {"config.py"}+ if scenario == "watch":+ expected.add("watch-note.txt")+ if scenario == "deadline":+ expected.add("still-writing")+ final = (task / "still-writing").read_bytes()+ time.sleep(.15)+ assert (task / "still-writing").read_bytes() == final, "no late write may race the grader"+ assert {p.name for p in task.iterdir()} == expected+ report_text = (trial.output / "execution.json").read_text(encoding="utf-8")+ report = json.loads(report_text)+ assert "fixture-private-key-not-real" not in report_text+ if scenario == "success":+ assert "[REDACTED]" in report["final_text"]+ if scenario in {"success", "empty-response", "thinking-length"}:+ diagnostic = report["diagnostics"]+ assert diagnostic["status"] == "available", diagnostic+ last = diagnostic["responses"][0]+ assert last["stop_reason"] == ("length" if scenario == "thinking-length" else "stop")+ assert last["thinking_blocks"] == int(scenario == "thinking-length")+ assert last["text_blocks"] == int(scenario == "success")+ assert last["tool_calls"] == 0+ assert "private-reasoning" not in json.dumps(diagnostic)+ assert report["cleanup"]["owned_processes_stopped"] is True+ assert report["verifier_reward"] is None+ assert report["accounting_complete"] is False+ assert {p.name for p in trial.output.iterdir()} == {"execution.json"}+ assert not trial._processes._live()++[email protected]_processes[email protected]_foreign_signals("Supervisor owns the spawned process and its captured setsid child")+def test_owned_cleanup_reaps_detached_child_and_preserves_bystander(tmp_path):+ marker = "owned-cleanup-fixture"+ pidfile = tmp_path / "child-pid"+ child = "import time; time.sleep(60)"+ program = (+ "import subprocess,sys,time; from pathlib import Path; "+ f"p=subprocess.Popen([sys.executable,'-c',{child!r}], start_new_session=True); "+ "Path(sys.argv[1]).write_text(str(p.pid)); time.sleep(60)"+ )+ env = {"PATH": os.defpath, "AVIBE_BENCHMARK_OWNER": marker}+ process = subprocess.Popen([sys.executable, "-c", program, str(pidfile)], env=env, start_new_session=True)+ bystander = subprocess.Popen([sys.executable, "-c", child], start_new_session=True)+ owner = OwnedProcesses(process, marker)+ try:+ deadline = time.monotonic() + 5+ while not pidfile.exists() and time.monotonic() < deadline:+ time.sleep(.02)+ assert pidfile.exists()+ child_pid = int(pidfile.read_text())+ # A clock shift must not make an owned, still-live process disappear+ # from verification. A reused unrelated pid must not become authority.+ with owner._lock:+ owner.identities[process.pid] -= .25+ owner.identities[bystander.pid] = psutil.Process(bystander.pid).create_time() - 1.0+ assert process.pid in {p.pid for p in owner._live()}+ assert bystander.pid not in {p.pid for p in owner._live()}+ result = owner.close(.1)+ assert result["owned_processes_stopped"] is True+ assert not psutil.pid_exists(child_pid) or psutil.Process(child_pid).status() == psutil.STATUS_ZOMBIE+ assert bystander.poll() is None+ finally:+ if process.poll() is None:+ owner.close(.1)+ bystander.kill()+ bystander.wait(timeout=5)++[email protected]("stage, failure, code", [+ ("startup", "timeout", "initialize_helper_timeout"),+ ("poll", "timeout", "cli_helper_timeout"),+ ("poll", "exit", "cli_helper_failed_2"),+ ("poll", "signal", "cli_helper_failed_-9"),+ # Run-show validates the durable Run payload, not just generic helper JSON.+ ("poll", "json", "cli_helper_invalid_run_payload"),+ ("poll", "quiescence", "harness_quiescence_unconfirmed"),+ ("poll", "unsafe_trial_error", "unclassified_trial_error"),+ ("poll", "upstream_exception", None),+])+def test_failure_report_preserves_only_wrapper_codes(tmp_path, monkeypatch, capsys, stage, failure, code):+ """The CLI/report boundary lost safe codes; raw exception bodies can leak keys.++ The existing subprocess happy-path test cannot expose that loss. Inject the+ helper failure, but use real _call conversion, context exit, report and main.+ """+ from scripts.terminal_bench import runner++ private = "private-upstream-error-not-for-export"+ task = tmp_path / "task"+ task.mkdir()+ profile_file = tmp_path / "profile.json"+ profile_file.write_text(json.dumps(PROFILE.payload()))+ instruction = tmp_path / "instruction.txt"+ instruction.write_text("fixture")+ monkeypatch.setenv(PROFILE.credential_env, "fixture-key")+ if stage == "poll":+ monkeypatch.setattr(FullRuntimeTrial, "start", lambda self: self._result.update(outcome="ready"))++ def helper(argv, **kwargs):+ is_poll = "runs" in argv and "show" in argv+ if stage == "startup" or is_poll:+ if failure == "timeout":+ raise subprocess.TimeoutExpired(argv, kwargs["timeout"], output=private, stderr=private)+ if failure in {"exit", "signal"}:+ return subprocess.CompletedProcess(argv, 2 if failure == "exit" else -9, private, private)+ if failure == "json":+ return subprocess.CompletedProcess(argv, 0, private, "")+ if failure == "unsafe_trial_error":+ raise TrialError(private)+ if failure == "upstream_exception":+ raise ValueError(private)+ body = {"run": {"status": "succeeded", "result_text": "fixture"}}+ elif "agent" in argv and "run" in argv:+ body = {"run_id": "fixture-run", "session_id": "fixture-session"}+ elif "harness" in argv:+ body = {"controller": {"available": False}}+ else:+ body = {}+ return subprocess.CompletedProcess(argv, 0, json.dumps(body), "")++ monkeypatch.setattr(subprocess, "run", helper)+ assert runner.main([+ "--workdir", str(task), "--state", str(tmp_path / "private-state"),+ "--output", str(tmp_path / "output"), "--profile", str(profile_file),+ "--instruction", str(instruction),+ ]) == 2+ captured = capsys.readouterr()+ report_text = (tmp_path / "output" / "execution.json").read_text()+ assert private not in report_text + captured.out + captured.err+ for payload in (json.loads(report_text), json.loads(captured.out), json.loads(captured.err)):+ assert payload["outcome"] == "infrastructure_error"+ assert payload["error_type"] == ("ValueError" if failure == "upstream_exception" else "TrialError")+ if code is None:+ assert "error_code" not in payload+ else:+ assert payload["error_code"] == codediff --git a/tests/test_terminal_bench_harbor.py b/tests/test_terminal_bench_harbor.pynew file mode 100644index 000000000..717cc78eb--- /dev/null+++ b/tests/test_terminal_bench_harbor.py@@ -0,0 +1,287 @@+"""Optional real-Harbor boundary tests, no container/provider calls.++Normal product CI skips this file when Harbor is not installed. The explicit+benchmark-host test command installs pinned requirements-harbor.txt separately.+The fake Environment is a transport recorder; real Harbor construction, profile,+wheel verification, source snapshot, cancellation and result projection run.+"""+from __future__ import annotations++import asyncio+from hashlib import sha256+from importlib.metadata import version+import json+from pathlib import Path+from types import SimpleNamespace+import subprocess+import zipfile++import pytest++pytest.importorskip("harbor.agents.installed.base")+if version("harbor") != "0.24.0":+ pytest.skip("optional boundary requires pinned Harbor 0.24.0", allow_module_level=True)++from harbor.models.agent.context import AgentContext+from scripts.terminal_bench import runner+from scripts.terminal_bench.harbor_agent import FullAvibeAgent+from scripts.terminal_bench.runner import FullRuntimeTrial+++# These are fixture coordination bounds, not Agent/product timeouts. Cold HD1+# writes and event-loop scheduling can take longer than the former 2s barrier.+_SETUP_BARRIER_SECONDS = 60+++async def _finish_pending(pending, env):+ """A failed setup assertion must not leave a shielded upload/task behind."""+ env.allow_upload.set()+ if not pending.done():+ pending.cancel()+ await asyncio.wait_for(asyncio.gather(pending, return_exceptions=True), _SETUP_BARRIER_SECONDS)+++def wheel(tmp_path):+ path = tmp_path / "avibe_os-9.9.9-py3-none-any.whl"+ with zipfile.ZipFile(path, "w") as archive:+ archive.writestr("avibe_os-9.9.9.dist-info/METADATA", "Name: avibe-os\nVersion: 9.9.9\n")+ return path, sha256(path.read_bytes()).hexdigest()+++def make_agent(tmp_path, wheel, **changes):+ path, digest = wheel+ options = {"logs_dir": tmp_path / "agent-logs", "model_name": "fixture-model",+ "wheel_path": str(path), "wheel_sha256": digest,+ "profile": {"model": "fixture-model", "base_url": "http://127.0.0.1:9/v1",+ "setup_seconds": 20, "wall_seconds": 5, "cleanup_seconds": 1}}+ options.update(changes)+ return FullAvibeAgent(**options)+++class Environment:+ default_user = None++ def __init__(self, *, fail_cleanup=False, block=False):+ self.commands = []+ self.uploads = {}+ self.files = {}+ self.report = {+ "outcome": "succeeded", "run_id": "fixture-run", "session_id": "fixture-session",+ "cleanup": {"owned_processes_stopped": True}, "accounting_complete": False,+ "usage": {"totals": {"token_reports": 2, "input_tokens": 100,+ "cached_input_tokens": 25, "output_tokens": 30}},+ }+ self.fail_cleanup, self.block = fail_cleanup, block+ self.started = asyncio.Event()+ self.cleanup_complete = False+ self.downloaded = False+ self.key_removed = False+ self.secret_upload_modes = None+ self.upload_error = False+ self.block_upload = False+ self.upload_started = asyncio.Event()+ self.allow_upload = asyncio.Event()++ async def upload_file(self, source, target):+ path = Path(source)+ if target.endswith("credential/provider.key"):+ self.secret_upload_modes = (path.stat().st_mode & 0o777, path.parent.stat().st_mode & 0o777)+ self.upload_started.set()+ if self.block_upload:+ await self.allow_upload.wait()+ self.files[target] = path.read_bytes()+ if self.upload_error:+ raise OSError("synthetic upload failed after write")+ else:+ self.files[target] = path.read_bytes()++ async def upload_dir(self, source, target):+ self.uploads[target] = {p.relative_to(source).as_posix(): p.read_bytes()+ for p in Path(source).rglob("*") if p.is_file()}++ async def exec(self, command, **kwargs):+ self.commands.append((command, kwargs))+ if command == "id -u":+ return SimpleNamespace(return_code=0, stdout="0\n", stderr="")+ if command == "pwd -P":+ return SimpleNamespace(return_code=0, stdout="/app\n", stderr="")+ if "scripts.terminal_bench.credential prepare" in command:+ return SimpleNamespace(return_code=0, stdout=json.dumps({"regular": True, "mode": "0600", "directory_mode": "0700"}), stderr="")+ if "scripts.terminal_bench.credential clean" in command:+ self.key_removed = True+ for name in list(self.files):+ if name.endswith("credential/provider.key"):+ self.files.pop(name)+ return SimpleNamespace(return_code=0, stdout=json.dumps({"removed": True}), stderr="")+ if "scripts.terminal_bench.launch run" in command:+ self.started.set()+ if self.block:+ await asyncio.Event().wait()+ if "scripts.terminal_bench.launch stop" in command:+ await asyncio.sleep(.01)+ self.cleanup_complete = not self.fail_cleanup+ return SimpleNamespace(return_code=2 if self.fail_cleanup else 0,+ stdout=json.dumps({"quiescent": not self.fail_cleanup, "started": self.started.is_set()}), stderr="")+ return SimpleNamespace(return_code=0, stdout="", stderr="")++ async def download_file(self, source, target):+ assert self.cleanup_complete, "grading artifact download cannot precede owned shutdown"+ assert source.endswith("execution.json"), "raw service state may not be exported"+ self.downloaded = True+ Path(target).write_text(json.dumps(self.report))+++async def test_verified_wheel_and_only_wrapper_are_installed(tmp_path, wheel, monkeypatch):+ agent = make_agent(tmp_path, wheel)+ env = Environment()+ dependencies = []++ async def ensure(environment, names):+ assert environment is env+ dependencies.extend(names)++ monkeypatch.setattr(agent, "ensure_system_dependencies", ensure)+ await agent.install(env)+ assert set(dependencies) == {"python3", "python_venv"}+ transferred = next(iter(env.uploads.values()))+ assert set(transferred) == {+ "scripts/__init__.py", "scripts/terminal_bench/__init__.py",+ "scripts/terminal_bench/profile.py", "scripts/terminal_bench/control.py",+ "scripts/terminal_bench/runner.py", "scripts/terminal_bench/launch.py", "scripts/terminal_bench/credential.py",+ "scripts/terminal_bench/diagnostics.py",+ }+ assert agent.provenance["wheel_sha256"] == wheel[1]+ assert agent.provenance["avibe_version"] == "9.9.9"+ assert agent.capabilities.atif is False+ assert any("-m pip install" in command and wheel[0].name in command for command, _ in env.commands)+ assert not any("editable" in command or "PYTHONPATH" in command for command, _ in env.commands)+++async def test_nonroot_task_refused_without_silent_ownership_change(tmp_path, wheel, monkeypatch):+ agent = make_agent(tmp_path, wheel)+ env = Environment()++ async def nonroot(command, **kwargs):+ assert command == "id -u"+ return SimpleNamespace(return_code=0, stdout="1000\n", stderr="")++ monkeypatch.setattr(env, "exec", nonroot)+ with pytest.raises(RuntimeError, match="root task user"):+ await agent.install(env)+ assert env.files == {} and env.uploads == {}+++async def test_hash_mismatch_fails_before_task_mutation(tmp_path, wheel):+ agent = make_agent(tmp_path, wheel, wheel_sha256="0" * 64)+ env = Environment()+ with pytest.raises(ValueError, match="SHA256 mismatch"):+ await agent.install(env)+ assert env.commands == [] and env.files == {} and env.uploads == {}++[email protected]("cancel, fail_cleanup", [(False, False), (True, False), (True, True)])+async def test_run_joins_cancel_before_artifact_handoff(tmp_path, wheel, monkeypatch, cancel, fail_cleanup):+ agent = make_agent(tmp_path, wheel)+ agent.provenance = {"wheel_sha256": wheel[1], "atif": False}+ monkeypatch.setenv("DEEPSEEK_API_KEY", "not-real-credential")+ env = Environment(block=cancel, fail_cleanup=fail_cleanup)+ ctx = AgentContext()+ pending = asyncio.create_task(agent.run("Write the requested note.", env, ctx))+ try:+ if cancel:+ await asyncio.wait_for(env.started.wait(), _SETUP_BARRIER_SECONDS)+ pending.cancel()+ with pytest.raises(RuntimeError if fail_cleanup else asyncio.CancelledError):+ await pending+ else:+ await pending+ finally:+ await _finish_pending(pending, env)+ if fail_cleanup:+ assert not env.downloaded+ else:+ assert env.cleanup_complete and env.downloaded+ assert ctx.n_input_tokens == 100+ assert ctx.n_cache_tokens == 25+ assert ctx.n_output_tokens == 30+ assert ctx.cost_usd is None+ assert ctx.metadata["accounting_complete"] is False+ assert ctx.metadata["verifier_reward"] is None+ start_command, options = next((cmd, opts) for cmd, opts in env.commands+ if "scripts.terminal_bench.launch run" in cmd)+ assert "not-real-credential" not in start_command+ assert options.get("env", {}) == {}+ assert "--credential-file" in start_command+ assert env.secret_upload_modes == (0o600, 0o700)+ assert env.key_removed+ assert all("not-real-credential" not in cmd + json.dumps(opts) for cmd, opts in env.commands)+ assert options["cwd"] == str(agent.root)+ assert "--sandbox-environment" in start_command+ assert "--workdir /app" in start_command+ assert options["timeout_sec"] > agent.profile.wall_seconds + agent.profile.cleanup_seconds+ assert not any("not-real-credential".encode() in data for data in env.files.values())++[email protected]("cancel", [False, True])+async def test_upload_failure_or_cancel_joins_then_removes_staged_key(tmp_path, wheel, monkeypatch, cancel):+ agent = make_agent(tmp_path, wheel)+ agent.provenance = {"wheel_sha256": wheel[1]}+ monkeypatch.setenv("DEEPSEEK_API_KEY", "not-real-credential")+ env = Environment()+ env.upload_error = not cancel+ env.block_upload = cancel+ pending = asyncio.create_task(agent.run("fixture", env, AgentContext()))+ try:+ if cancel:+ await asyncio.wait_for(env.upload_started.wait(), _SETUP_BARRIER_SECONDS)+ pending.cancel()+ await asyncio.sleep(.01)+ assert not env.key_removed, "in-flight transfer must settle before cleanup"+ env.allow_upload.set()+ with pytest.raises(asyncio.CancelledError if cancel else OSError):+ await pending+ finally:+ await _finish_pending(pending, env)+ assert env.key_removed+ assert not env.started.is_set()+ assert not env.downloaded+ assert not any("not-real-credential".encode() in data for data in env.files.values())+ assert all("not-real-credential" not in cmd + json.dumps(opts) for cmd, opts in env.commands)+++++def test_runs_show_nonzero_valid_json_is_a_product_failure_not_helper_failure(tmp_path, monkeypatch):+ trial = object.__new__(FullRuntimeTrial)+ trial.state = tmp_path+ trial.env = {"AVIBE_HOME": str(tmp_path / "home")}+ trial._command = lambda operation, *args: ["vibe", operation, *args]+ sentinel = "provider-secret-text-not-for-export"++ def show(argv, **kwargs):+ return subprocess.CompletedProcess(+ argv, 1,+ json.dumps({"run": {"id": "run-1", "status": "failed",+ "error": f"Automatic recovery has ended: {sentinel}",+ "result_text": "safe failure"}}),+ "",+ )++ monkeypatch.setattr(subprocess, "run", show)+ payload = trial._call_run_show("run-1")+ assert payload["run"]["status"] == "failed"+ assert payload["run"]["error"].endswith(sentinel)+ scrubbed = runner.scrub(payload, sentinel)+++def test_explicit_model_must_match_profile(tmp_path, wheel):+ with pytest.raises(ValueError, match="model_name"):+ make_agent(tmp_path, wheel, model_name="different-model")+ with pytest.raises(ValueError, match="host environment"):+ make_agent(tmp_path, wheel, extra_env={"DEEPSEEK_API_KEY": "sentinel-not-config-or-argv"})diff --git a/tests/test_terminal_bench_launch.py b/tests/test_terminal_bench_launch.pynew file mode 100644index 000000000..df182ba39--- /dev/null+++ b/tests/test_terminal_bench_launch.py@@ -0,0 +1,60 @@+"""Local process acceptance for Harbor's outer cancel fence; no Docker/Harbor."""+from __future__ import annotations++import json+import os+from pathlib import Path+import subprocess+import sys+import time++import pytest++from scripts.terminal_bench.launch import run, stop+++def test_late_start_is_fenced_and_untrusted_identity_not_signalled(tmp_path):+ root = tmp_path / "launch"+ root.mkdir()+ assert stop(root, "nonce", tmp_path / "execution.json", .1) == {"quiescent": True, "started": False}+ assert run(root, "nonce", ["must-not-start"]) == 125+ (root / "starting").touch()+ with pytest.raises(RuntimeError, match="identity_not_recorded"):+ stop(root, "nonce", tmp_path / "execution.json", .1)+ (root / "wrapper-process.json").write_text(json.dumps({"nonce": "other", "pid": os.getpid(), "created": 0}))+ with pytest.raises(RuntimeError, match="identity_mismatch"):+ stop(root, "nonce", tmp_path / "execution.json", .1)++[email protected]_processes+def test_stop_joins_exact_wrapper_and_demands_inner_cleanup_receipt(tmp_path):+ import psutil+ root = tmp_path / "launch"+ root.mkdir()+ receipt = tmp_path / "execution.json"+ ready = tmp_path / "ready"+ code = (+ "import sys,json,signal,time; from pathlib import Path; "+ "signal.signal(signal.SIGTERM, lambda *_: (Path(sys.argv[1]).write_text(json.dumps("+ "{'cleanup':{'owned_processes_stopped':True}})),sys.exit(0))); "+ "Path(sys.argv[2]).touch(); time.sleep(60)"+ )+ process = subprocess.Popen([sys.executable, "-c", code, str(receipt), str(ready)],+ env={"PATH": os.defpath, "AVIBE_BENCHMARK_LAUNCH_ID": "nonce"}, start_new_session=True)+ try:+ deadline = time.monotonic() + 3+ while not ready.exists() and time.monotonic() < deadline:+ time.sleep(.01)+ assert ready.exists()+ (root / "wrapper-process.json").write_text(json.dumps({+ "nonce": "nonce", "pid": process.pid, "created": psutil.Process(process.pid).create_time(),+ }))+ assert stop(root, "nonce", receipt, 2)["quiescent"] is True+ process.wait(timeout=3)+ receipt.unlink()+ with pytest.raises(RuntimeError, match="receipt_missing"):+ stop(root, "nonce", receipt, .1)+ finally:+ if process.poll() is None:+ process.kill()+ process.wait(timeout=3)diff --git a/tests/test_terminal_bench_remote_job.py b/tests/test_terminal_bench_remote_job.pynew file mode 100644index 000000000..611da894f--- /dev/null+++ b/tests/test_terminal_bench_remote_job.py@@ -0,0 +1,338 @@+"""Remote owner fences and cleanup failure classes; no provider calls."""+import io+import json+from pathlib import Path+import signal+import subprocess+import sys+from types import SimpleNamespace++import pytest++pytest.importorskip("fcntl")+from scripts.terminal_bench import remote_job as job++[email protected](autouse=True)+def host_scope(tmp_path, monkeypatch):+ monkeypatch.setattr(job, "HOST_RESERVATION", tmp_path / "host-reservation")+ monkeypatch.setenv("SERVICE_RESULT", "success")+ monkeypatch.setattr(job, "archive_artifact", lambda cid, source, destination, timeout: destination.write_bytes(b"opaque-fixture"))+++def config(tmp_path):+ task = tmp_path / "task"+ task.mkdir()+ (task / "task.toml").write_text('artifacts=["/app/result.txt"]\n')+ data = {"job_name": "one-job", "jobs_dir": str(tmp_path / "jobs"), "n_attempts": 1, "n_concurrent_trials": 1,+ "agents": [{"name": "nop", "override_timeout_sec": 2}], "tasks": [{"path": str(task)}],+ "environment": {"type": "docker", "delete": True}}+ p = tmp_path / "config.json"+ job.save(p, data)+ return p+++def prepared(tmp_path, seconds=5):+ p = config(tmp_path)+ root = tmp_path / "runs" / "one"+ job.prepare(p, root, seconds, "BENCHMARK_API_KEY")+ (root / "credential").mkdir(mode=0o700)+ key = root / "credential/provider.key"+ key.write_text("not-real-secret")+ key.chmod(0o600)+ job.save(root / "ownership.json", {"trial_name": "task__nonce", "trial_id": "trial-1", "job_id": "job-1", "projects": ["task_nonce_env"]})+ return root+++def complete(root):+ m = job.load(root / "manifest.json")+ trial = Path(m["job_dir"]) / "task__nonce"+ trial.mkdir(parents=True, exist_ok=True)+ job.save(trial / "result.json", {"id": "trial-1", "trial_name": "task__nonce", "finished_at": "2026-10-14T00:00:00Z"})+ job.save(trial.parent / "result.json", {"id": "job-1", "finished_at": "2026-10-14T00:00:00Z", "n_total_trials": 1,+ "stats": {"n_completed_trials": 1, "n_running_trials": 0, "n_pending_trials": 0}})+++class Docker:+ def __init__(self):+ self.containers = [{"Id": "a" * 64, "Config": {"Labels": {"com.docker.compose.project": "task_nonce_env"}}},+ {"Id": "b" * 64, "Config": {"Labels": {"com.docker.compose.project": "bystander"}}}]+ self.networks = [{"Id": "owned-net", "Labels": {"com.docker.compose.project": "task_nonce_env"}},+ {"Id": "bystander-net", "Labels": {"com.docker.compose.project": "bystander"}}]+ self.calls = []++ def __call__(self, argv, **kwargs):+ self.calls.append((argv, kwargs))+ if argv[1:3] == ["ps", "-aq"]:+ return "\n".join(c["Id"] for c in self.containers)+ if argv[1] == "inspect":+ return json.dumps(self.containers)+ if argv[1:3] == ["rm", "-f"]:+ self.containers[:] = [c for c in self.containers if c["Id"] != argv[-1]]+ return ""+ if argv[1:3] == ["network", "ls"]:+ return "\n".join(n["Id"] for n in self.networks)+ if argv[1:3] == ["network", "inspect"]:+ return json.dumps(self.networks)+ if argv[1:3] == ["network", "rm"]:+ self.networks[:] = [n for n in self.networks if n["Id"] != argv[-1]]+ return ""+ if argv[1] == "cp":+ return ""+ raise AssertionError(argv)+++def test_preflight_rejects_paths_and_budget_before_any_job(tmp_path):+ p = config(tmp_path)+ d = job.load(p)+ for path in ["task", "/Volumes/HD1/absent-task"]:+ d["tasks"][0]["path"] = path+ job.save(p, d)+ with pytest.raises((ValueError, FileNotFoundError)):+ job.prepare(p, tmp_path / "state", 2)+ assert not job.HOST_RESERVATION.exists()+ d["tasks"][0]["path"] = str(tmp_path / "task")+ d["n_attempts"] = 2+ job.save(p, d)+ with pytest.raises(ValueError):+ job.validate(p)+++def test_profile_credential_name_is_derived_and_mismatch_refused(tmp_path):+ import hashlib+ p = config(tmp_path)+ wheel = tmp_path / "avibe.whl"+ wheel.write_bytes(b"fixture-only-wheel")+ d = job.load(p)+ d["agents"] = [{"import_path": "scripts.terminal_bench.harbor_agent:FullAvibeAgent", "model_name": "fixture",+ "override_timeout_sec": 30, "kwargs": {"wheel_path": str(wheel),+ "wheel_sha256": hashlib.sha256(wheel.read_bytes()).hexdigest(),+ "profile": {"model": "fixture", "base_url": "http://127.0.0.1:9/v1", "credential_env": "ACTUAL_PROVIDER_KEY"}}}]+ job.save(p, d)+ assert job.validate(p)[3] == "ACTUAL_PROVIDER_KEY"+ with pytest.raises(ValueError, match="credential variable"):+ job.validate(p, "WRONG_KEY")+ assert not job.HOST_RESERVATION.exists()+++def test_lock_is_host_wide_and_destination_claim_is_permanent(tmp_path):+ root = prepared(tmp_path)+ other = tmp_path / "totally-different-parent" / "second"+ with pytest.raises(FileExistsError):+ job.prepare(tmp_path / "config.json", other, 2)+ assert not other.exists()+ job.release(root)+ with pytest.raises(FileExistsError):+ job.prepare(tmp_path / "config.json", other, 2)+ assert not job.HOST_RESERVATION.exists()++[email protected]("broken_result", ["{truncated", "[]", "null", '{"finished_at":"not-a-trial"}'])+def test_bad_results_never_skip_exact_owned_cleanup(tmp_path, monkeypatch, broken_result):+ root = prepared(tmp_path)+ manifest = job.load(root / "manifest.json")+ jobdir = Path(manifest["job_dir"])+ jobdir.mkdir(parents=True)+ (jobdir / "result.json").write_text(broken_result)+ job.save(root / "lifecycle.json", {"state": "completed", "exit_code": 0})+ docker = Docker()+ monkeypatch.setattr(job, "command", docker)+ receipt = job.finalize(root)+ assert receipt["state"] == "missing_terminal_result"+ assert receipt["harbor_finished"] is False+ assert receipt["cleanup"]["confirmed"] is True+ assert [c["Id"] for c in docker.containers] == ["b" * 64]+ assert [n["Id"] for n in docker.networks] == ["bystander-net"]+ assert not job.HOST_RESERVATION.exists()+ assert not (root / "credential/provider.key").exists()+++def test_network_only_and_bad_artifact_metadata_are_cleaned(tmp_path, monkeypatch):+ root = prepared(tmp_path)+ (tmp_path / "task/task.toml").write_text("broken = [")+ docker = Docker()+ docker.containers = []+ monkeypatch.setattr(job, "command", docker)+ receipt = job.finalize(root)+ assert receipt["cleanup"]["confirmed"]+ assert docker.networks[0]["Id"] == "bystander-net"+ assert receipt["artifacts"] == [{"status": "metadata_unavailable"}]+++def test_cleanup_failure_retains_reservation_and_receipt(tmp_path, monkeypatch):+ root = prepared(tmp_path)+ monkeypatch.setattr(job, "command", lambda *a, **k: (_ for _ in ()).throw(subprocess.TimeoutExpired("docker", 1)))+ receipt = job.finalize(root)+ assert receipt["cleanup"]["confirmed"] is False+ assert job.HOST_RESERVATION.exists()+ assert not (root / "credential/provider.key").exists()+ assert receipt["finished_at"]+++def test_completion_requires_exact_trial_and_no_pending_work(tmp_path):+ root = prepared(tmp_path)+ m, o = job.load(root / "manifest.json"), job.load(root / "ownership.json")+ complete(root)+ assert job.terminal_evidence(m, o)+ assert not job.terminal_evidence(m, {**o, "trial_id": "some-other-trial"})+ p = Path(m["job_dir"]) / "result.json"+ d = job.load(p); d["stats"]["n_running_trials"] = 1; job.save(p, d)+ assert not job.terminal_evidence(m, o)++[email protected]("chunk_size", [1, 3, 4096])+def test_stream_redaction_handles_split_keys(tmp_path, chunk_size):+ class Chunks(io.StringIO):+ def read(self, n):+ return super().read(min(n, chunk_size))+ p = tmp_path / "log"+ job.redact_stream(Chunks("before-secret-非ASCII-after\n"), p, "secret-非ASCII")+ assert p.read_text() == "before-[REDACTED]-after\n"+ assert p.stat().st_mode & 0o777 == 0o600++[email protected]_foreign_signals("Supervisor owns this test child process group")[email protected]_processes[email protected]("mode,expected", [("ok", "completed"), ("missing", "missing_terminal_result"), ("crash", "harbor_error"), ("slow", "time_limit")])+def test_worker_never_releases_slot_before_post_stop(tmp_path, monkeypatch, mode, expected):+ root = prepared(tmp_path, seconds=1 if mode == "slow" else 10)+ program = tmp_path / "child-python"+ program.write_text(f'''#!{sys.executable}+import sys,time+print('not-real-secret',flush=True)+if {mode!r}=='slow': time.sleep(30)+if {mode!r}=='crash': sys.exit(3)+''')+ program.chmod(0o700)+ if mode == "ok":+ complete(root)+ monkeypatch.setattr(job, "sys", SimpleNamespace(executable=str(program)))+ before = signal.getsignal(signal.SIGTERM)+ code = job.worker(root)+ assert not (root / "receipt.json").exists()+ assert job.HOST_RESERVATION.exists()+ assert signal.getsignal(signal.SIGTERM) == before+ monkeypatch.setattr(job, "command", Docker())+ receipt = job.finalize(root)+ assert receipt["state"] == expected+ assert code == (0 if mode in {"ok", "missing"} else 1)+ assert "not-real-secret" not in (root / "harbor.log").read_text()+ assert not job.HOST_RESERVATION.exists()+++def test_finalize_refuses_live_owner_and_status_does_not_wait_on_dead_unit(tmp_path, monkeypatch):+ root = prepared(tmp_path)+ monkeypatch.delenv("SERVICE_RESULT")+ with pytest.raises(ValueError):+ job.finalize(root)+ monkeypatch.setattr(job, "unit_state", lambda unit: {"ActiveState": "failed"})+ result, code = job.status(root)+ assert code == 1 and result["cleanup_confirmed"] is False+++def test_lost_launch_ack_fences_unit_without_publishing_clean_receipt(tmp_path, monkeypatch):+ p = config(tmp_path)+ private = tmp_path / "credential"; private.mkdir(mode=0o700)+ key = private / "provider.key"; key.write_text("KEY-SENTINEL"); key.chmod(0o600)+ calls = []+ def manager(argv, **kwargs):+ calls.append(argv)+ if argv[0] == "systemd-run":+ raise subprocess.TimeoutExpired("systemd-run", 10)+ if argv[:2] == ["systemctl", "show"]:+ return "LoadState=loaded\nActiveState=active\n"+ return ""+ monkeypatch.setattr(job, "command", manager)+ root = tmp_path / "runs/one"+ with pytest.raises(subprocess.TimeoutExpired):+ job.submit(p, root, key, 5)+ assert not (root / "receipt.json").exists()+ assert job.HOST_RESERVATION.exists()+ assert calls[-1][:3] == ["systemctl", "stop", "--no-block"]+ assert "KEY-SENTINEL" not in json.dumps(calls)+ assert any(a.startswith("--property=ExecStopPost=") for a in calls[0])+++def test_artifact_recovery_stops_at_its_total_budget(tmp_path, monkeypatch):+ root = prepared(tmp_path)+ (tmp_path / "task/task.toml").write_text('artifacts=' + json.dumps(['/app/f' + str(i) for i in range(32)]))+ clock = [0.0]+ monkeypatch.setattr(job.time, "monotonic", lambda: clock[0])+ calls = []+ def copy(cid, source, destination, timeout):+ calls.append(source); clock[0] += timeout+ raise subprocess.TimeoutExpired("docker cp", timeout)+ monkeypatch.setattr(job, "archive_artifact", copy)+ result = job.copy_artifacts(root, job.load(root / "manifest.json"), [{"Id": "a" * 64}], 8)+ assert clock[0] <= 8 and len(calls) <= 3+ assert result[-1]["status"] == "recovery_budget_exhausted"++[email protected]("component", ["%n", "$NAME", "a\nb"])+def test_manager_expansion_paths_fail_before_ownership(tmp_path, component):+ p = config(tmp_path)+ with pytest.raises(ValueError, match="systemd"):+ job.prepare(p, tmp_path / "runs" / component, 5)+ assert not job.HOST_RESERVATION.exists()+++def test_manager_expansion_cannot_hide_behind_a_symlink(tmp_path):+ p = config(tmp_path)+ target = tmp_path / "%n"+ target.mkdir()+ alias = tmp_path / "alias"+ alias.symlink_to(target, target_is_directory=True)+ with pytest.raises(ValueError, match="systemd"):+ job.prepare(p, alias / "job", 5)+ assert not job.HOST_RESERVATION.exists()+++def test_tomli_fallback_keeps_module_collectable_on_python310(monkeypatch):+ import builtins+ import importlib.util+ original = builtins.__import__+ used = []+ def compat(name, *args, **kwargs):+ if name == "tomllib":+ raise ModuleNotFoundError(name)+ if name == "tomli":+ used.append(name)+ return job.tomllib+ return original(name, *args, **kwargs)+ monkeypatch.setattr(builtins, "__import__", compat)+ spec = importlib.util.spec_from_file_location("scripts.terminal_bench.compat_fixture", job.__file__)+ module = importlib.util.module_from_spec(spec)+ spec.loader.exec_module(module)+ assert used == ["tomli"]+ assert module.tomllib.loads("x=1") == {"x": 1}+++def test_overlapping_artifact_symlinks_remain_opaque_archives(tmp_path, monkeypatch):+ import tarfile+ root = prepared(tmp_path)+ outside = tmp_path / "outside"+ outside.mkdir()+ sentinel = outside / "payload"+ sentinel.write_text("must-stay-unchanged")+ (tmp_path / "task/task.toml").write_text('artifacts=["/app/out","/app/out/link/payload"]')+ destinations = []+ def archive(cid, source, destination, timeout):+ destinations.append(destination)+ with tarfile.open(destination, "w") as tar:+ if str(source) == "/app/out":+ link = tarfile.TarInfo("out/link")+ link.type = tarfile.SYMTYPE+ link.linkname = str(outside)+ tar.addfile(link)+ else:+ member = tarfile.TarInfo("payload")+ data = b"overwrite-sentinel"+ member.size = len(data)+ tar.addfile(member, io.BytesIO(data))+ monkeypatch.setattr(job, "archive_artifact", archive)+ result = job.copy_artifacts(root, job.load(root / "manifest.json"), [{"Id": "a" * 64}], job.time.monotonic() + 5)+ assert len(destinations) == 2 and destinations[0] != destinations[1]+ assert all(p.is_relative_to(root) and p.suffix == ".tar" for p in destinations)+ assert all(r["status"] == "archived" and r["extracted_on_host"] is False for r in result)+ assert sentinel.read_text() == "must-stay-unchanged"+ assert not list((root / "recovered-artifacts").rglob("link"))diff --git a/tests/test_vibey_agent.py b/tests/test_vibey_agent.pyindex 0c788a341..6e1c1244f 100644--- a/tests/test_vibey_agent.py+++ b/tests/test_vibey_agent.py@@ -1211,6 +1211,25 @@ async def test_a_failed_run_shows_localized_copy_and_a_refusal_shows_its_explanaassert len(reported) == 2[email protected]("language", ["en", "zh"])[email protected]("content", [(), (ThinkingBlock("private reasoning"),), (text("unfinished answer"),)])+async def test_output_limited_final_is_a_failed_row_with_length_cause(+ engine, session, tmp_path, published, caplog, language, content+) -> None:+ message = AssistantMessage(content, ORIGIN, "length")+ harness = _Harness(engine, tmp_path, "telegram", [[Done(message)]], language=language)+ with caplog.at_level(logging.WARNING, logger="modules.agents.vibey.agent"):+ await harness.agent.handle_message(harness.request("finish the task"))+ [row] = harness.rows("error")+ assert json.loads(row["content_json"])["model"]["message"]["stop_reason"] == "length"+ assert not harness.rows("result")+ assert harness.controller.terminals[-1]["is_error"] is True+ assert len(harness.provider.requests) == 1 # No whole-task replay or hidden budget increase.+ expected = "unfinished answer" if any(isinstance(b, TextBlock) for b in content) else i18n_t("vibeyAgent.error.outputLimit", language)+ assert row["content_text"] == expected+ assert "kind=length" in caplog.text++@pytest.mark.parametrize("language", ["en", "zh"])async def test_a_provider_stall_is_not_shown_as_a_connection_failure(engine, session, tmp_path, published, languagediff --git a/vibe/i18n/en.json b/vibe/i18n/en.jsonindex 173ab7e08..506a61c62 100644--- a/vibe/i18n/en.json+++ b/vibe/i18n/en.json@@ -1724,7 +1724,7 @@"toolsUnsupported": "The selected model does not support tools, which {backend} needs. Choose another model.","providerProtocol": "The model provider sent a reply {backend} cannot use.","unclassified": "The request to the model failed with an error {backend} could not classify. Try again, or choose another model.",- "outputLimit": "The model kept running out of output space while writing a tool call. Ask for smaller steps, or choose a model with a larger output limit.",+ "outputLimit": "The model reached its output limit before producing a usable answer or tool call. Ask for smaller steps, or choose a model with a larger output limit.","internal": "{backend} stopped because of an internal error. Try again; if it happens again, the service log names the cause.","generic": "{backend}'s run failed."}diff --git a/vibe/i18n/zh.json b/vibe/i18n/zh.jsonindex 5648425d0..3b1948333 100644--- a/vibe/i18n/zh.json+++ b/vibe/i18n/zh.json@@ -1724,7 +1724,7 @@"toolsUnsupported": "所选模型不支持工具,而 {backend} 需要工具。请换一个模型。","providerProtocol": "模型服务商返回的回复 {backend} 无法使用。","unclassified": "请求模型时出现了 {backend} 无法归类的错误。可以重试,或换一个模型。",- "outputLimit": "模型在写工具调用时多次超出输出上限。请让它分成更小的步骤,或换用输出上限更大的模型。",+ "outputLimit": "模型在生成可用的答案或工具调用前达到了输出上限。请让它分成更小的步骤,或换用输出上限更大的模型。","internal": "{backend} 因内部错误而停止。可以重试;如果再次出现,服务日志里记录了原因。","generic": "{backend} 运行失败。"}