Skip to content

armgit / avibe

Public Imported from GitHub
0

Make your computer a home for your AI partner. Avibe is the local-first Agent OS: Vibey, its built-in AI partner, works out of the box, and Claude Code, Codex & OpenCode run unchanged beside it — reach them from your browser, phone, or any chat app.

feat(benchmark): evaluate full Avibe runtime with owned remote jobs (#2627)

Call Whiteaed967b61a97
diff --git a/core/agent_core/agent/loop.py b/core/agent_core/agent/loop.py
index 923b4b9ab..b7a0ab90f 100644
--- a/core/agent_core/agent/loop.py
+++ b/core/agent_core/agent/loop.py
@@ -926,12 +926,16 @@ class Agent:
)
response, final = await self._commit_model_message(message, emit)
empty_reply = final and not has_reply(message)
- if empty_reply:
+ truncated_reply = final and message.stop_reason == "length"
+ if empty_reply or truncated_reply:
decided = self._outcome.primary("error")
await self._error(
emit,
- message.stop_reason if message.stop_reason in {"refusal", "safety"} else "empty_response",
- message.error_message or f"Model stopped with {message.stop_reason} without a reply.",
+ message.stop_reason if message.stop_reason in {"length", "refusal", "safety"} else "empty_response",
+ message.error_message or (
+ "The model reached its output limit before completing the reply."
+ if truncated_reply else f"Model stopped with {message.stop_reason} without a reply."
+ ),
"source",
cause=decided,
)
@@ -982,7 +986,7 @@ class Agent:
return "aborted" if message.stop_reason == "aborted" else "error"
if not message.tool_calls:
await self._save_state()
- if empty_reply:
+ if empty_reply or truncated_reply:
return "error"
if end:
return "ended_by_hook"
diff --git a/docs/plans/2026-10-14-terminal-bench-reliability.md b/docs/plans/2026-10-14-terminal-bench-reliability.md
new file mode 100644
index 000000000..81f27c667
--- /dev/null
+++ b/docs/plans/2026-10-14-terminal-bench-reliability.md
@@ -0,0 +1,123 @@
+# Terminal-Bench reliability and failure diagnosis
+
+## Scope and evidence
+
+Continue the full-runtime benchmark branch; preserve the original smoke and
+paired-pilot artifacts. The installed-package pilot ended with a real failed
+Run and verifier reward 0. Its summary says `empty_response`, but the retained
+artifacts do not contain the final provider stop reason or response shape.
+The reference lost its Harbor owner without a terminal trial receipt and left
+an orphan task container. Neither observation proves model capability or the
+historical root cause. No additional paid trial is authorized by this repair.
+
+The prior owner has explicitly stopped writing. This session is the sole writer;
+independent audits are read-only. All execution tests run in the user-authorized,
+dedicated remote benchmark environment, never the local VM or another tenant.
+Do not modify shared host services, resource caps, provider budgets, official
+tasks/verifiers, or old results.
+
+## Repair contract
+
+1. Reproduce final empty, reasoning-only, and output-truncated provider responses
+ with scripted, zero-paid input. Preserve the distinction between a provider
+ stop with no answer and an explicit output-limit stop. Do not silently increase
+ output limits, retry a whole task, replay successful tools, or mark incomplete
+ work successful. Any core change needs a failing before/fixed after test and
+ must agree with the shared loop contract, not special-case a benchmark/model.
+2. Add a bounded diagnostic projection to the wrapper's existing `execution.json`.
+ Read through normal product CLI/API, not raw SQLite/transcript files. Export
+ only allowlisted error/stop categories, response block counts/lengths, reported
+ token usage and completeness status. No raw prompts, reasoning text, provider
+ keys, source configs, raw logs or internal database copies. Missing diagnostics
+ must be explicit, must not change the real Run outcome, and must not block
+ cleanup. Capture before the owned service is stopped. Keep old receipts intact.
+3. Add a source-only Linux evaluation-host supervisor around Harbor. A systemd
+ task-owned unit owns its process tree independently of SSH; Avibe Harness
+ observes the durable receipt rather than re-executing the paid command. Claim
+ a fresh run directory and a single-host benchmark lock before launch; validate
+ config paths, wheel hash and limits without a provider call; preserve the input
+ config and an explicit local-path relocation manifest. No mutation of product
+ dependencies or user/live Avibe state.
+4. Every supervisor exit writes a separate orchestration receipt. A dead Harbor
+ parent is failure even if a stale result says running; success requires a
+ complete trial receipt. Preserve Harbor's originals and never invent reward,
+ usage, or verifier completion. A remote hard wall cap is independent of SSH
+ and bounds setup/execution/verification; the configured Harbor agent budget is
+ unchanged and recorded separately. Stop/delete only containers/networks with
+ positively established trial labels, after preserving task artifacts/logs.
+ Never broad-prune a Docker daemon or pattern-kill processes.
+5. Secrets travel via protected files/stdin, never command arguments or logs.
+ Reject missing/empty/wrong-mode credential files before Harbor starts. Consume
+ transport files promptly; diagnostics and receipts must pass sentinel scans.
+
+## Diagnosis decision
+
+The read-only audit confirmed the parser preserves `length` vs `stop` and
+thinking vs text; recovery does not replay a failed task. A visible tool-free
+`length` final was incorrectly considered completed by both the loop and the
+transcript outcome function. The orchestrator chooses one consistent final
+classification: every final `length` is failed/`length`, including no-answer
+shapes, with generic output-limit copy. This deliberately corrects the old
+no-answer matrix expectation too, rather than retaining its misleading
+`empty_response` label. Ordinary empty `stop` remains failed/`empty_response`;
+non-final continuations and bounded tool-call retries are unchanged. No automatic
+continuation or larger budget is introduced.
+
+Before-code evidence: the extended no-answer matrix failed four cases with
+`empty_response != length` on the dedicated remote host; 14 controls passed.
+The historical pilot still lacks evidence identifying its final stop reason.
+
+## Validation gates
+
+- Table-driven provider/loop regressions: empty vs reasoning-only vs length,
+ valid tools/text, refusal/interruption and retry budget preservation.
+- Diagnostic projection: synthetic secret/text sentinels never exported,
+ correct final stop/error shape, bounded query, explicit unavailable state.
+- Supervisor: normal terminal result, preflight rejection, nonzero/zero exit with
+ missing result, hard timeout, exact-owned cleanup with an untouched bystander,
+ and two-submit/single-owner refusal.
+- Remote no-paid acceptance: drop SSH after submission; work continues, then
+ produces a terminal receipt. Kill the child or hit a short deadline; receipt
+ and cleanup still happen. Verify unrelated services were not targeted.
+- Focused tests and lint remotely, independent review, then normal PR/CI gates.
+ No merge, deployment to the live user runtime, official-task rerun, or expanded
+ benchmark batch is included.
+
+## Verified repair evidence
+
+- Core red test: four final no-answer `length` cases reproduced the lost cause
+ (`empty_response` rather than `length`); 14 controls passed.
+- Core/adapter green: 83 focused cases, including persisted error rows and
+ localized length causes for empty, thinking-only and visible partial finals.
+- Wrapper/host final targeted pass: 33 cases, including split-secret redaction,
+ global ownership, malformed evidence, exact trial identity, systemd expansion
+ refusal (also behind symlinks), the Python 3.10 TOML-import fallback, and opaque
+ recovery archives. An expanded run passed 1120 of 1122 cases; one missing remote
+ schema fixture was synchronized and one stale Run-show error-code assertion
+ was corrected. Both failures plus neighboring cases then passed (9 tests).
+- Real full-runtime source acceptance: two simulated provider cases (empty stop,
+ thinking-only length) both completed failure reporting, retained the distinct
+ canonical stop/block diagnostics and proved owned cleanup. No model bill.
+- Dedicated remote systemd + real Harbor/Docker acceptance after review fixes:
+ SSH submit/exit completed normally; a seven-second hard wall, SIGKILL of the
+ supervisor, and SIGKILL of the Harbor child each produced their own terminal
+ orchestration receipt, retained fixture artifacts, and left no owned container
+ or network. A separate bystander container remained running and was removed
+ only by the test owner after all assertions. An additional real Docker case
+ recovered overlapping directory/file declarations containing an absolute
+ symlink as two opaque tar files; the host sentinel remained unchanged and all
+ owned containers/networks were removed. No paid task was repeated.
+
+Review findings fixed as a class: parse failures cannot block cleanup; one
+host-wide reservation plus permanent destination claim; post-stop-only receipt
+publication; exact trial terminal evidence; total finalization budget; ownership
+journal before network creation; absolute-path preflight; profile-bound secret
+variable. Original Harbor results and historical failures remain untouched.
+
+## Historical limitations to retain in reporting
+
+- The candidate's final raw response was not retained. A reproduction of one
+ failure class does not retroactively prove that class caused this pilot.
+- The reference has partial trajectory usage, not a completed score or bill.
+- The initial synthetic reward 1 is only integration evidence.
+- A successful repair simulation is not a claim of improved benchmark scores.
diff --git a/docs/plans/agent-core-contracts/loop-control.md b/docs/plans/agent-core-contracts/loop-control.md
index 71ac633a3..526b3cf27 100644
--- a/docs/plans/agent-core-contracts/loop-control.md
+++ b/docs/plans/agent-core-contracts/loop-control.md
@@ -46,7 +46,16 @@ commit input → loop:
```
Finality is decided before the response row is inserted, so a final (`result` or `error`) row is always the run's last
-response. A final response with `stop_reason` `refusal` or `safety` and no text is still committed as the final
+response. A final response with `stop_reason=length` is incomplete, even when it
+contains partial text or an image: commit it as `error`, preserve usable content,
+and end with an error of kind `length`. A no-answer length stop is not an
+`empty_response`; the upstream already supplied the cause. Ordinary empty or
+reasoning-only `stop` responses still fail as `empty_response`. This does not
+add a model call, increase an output limit, or replay a task. Non-final responses
+with pending steer/follow-up keep continuing normally; truncated tool calls keep
+the existing bounded reissue path without executing their partial arguments.
+
+A final response with `stop_reason` `refusal` or `safety` and no text is still committed as the final
response, typed `error` ([`transcript.md`](transcript.md) §1); its commit-time display text is the localized
explanation. (That it is written in the same transaction so that re-delivery after a crash shows it is moot: Avibe
delivers committed rows through the shared emit path and nothing re-delivers them, [`recovery.md`](recovery.md)
diff --git a/docs/plans/full-avibe-runtime-terminal-bench.md b/docs/plans/full-avibe-runtime-terminal-bench.md
new file mode 100644
index 000000000..cb28bdab8
--- /dev/null
+++ b/docs/plans/full-avibe-runtime-terminal-bench.md
@@ -0,0 +1,145 @@
+# Full-runtime Terminal-Bench vertical slice
+
+## Scope and ownership
+
+Worktree: `feature/vibey-terminal-bench-full-runtime`, starting base `9a89da41b`.
+This change adds source-only evaluation tooling around the installed product.
+The earlier standalone-core experiment is out of scope and is not used for
+product benchmark results. Deployment, merge, public leaderboard submission,
+and remote/GPU provisioning remain separate decisions.
+
+## Source-backed correction to the investigation
+
+A new embedded Controller or a second Agent loop is unnecessary. Existing paths:
+
+- `main.main()` / installed `vibe.service_main.main()` owns service lock, config,
+ migrations, Controller construction, signals, readiness and shutdown.
+- Empty `platforms.enabled` is already a supported workbench-only installation.
+ `MultiIMClient` keeps that runtime alive; `AvibeBot` is the internal transport.
+- `AVIBE_WEB_UI_SUPERVISOR` already suppresses the UI watchdog's UI launch. The
+ benchmark supervisor intentionally starts no UI; no browser or tunnel is needed.
+- `vibe agent run --sync --cwd ...` creates a real durable Run/Session/delivery and
+ reaches the real Controller, MessageHandler, AgentService and VibeyAgent.
+- `vibe.model_hub_client.ModelHubRemoteService` already creates Sources and routes
+ through the controller-owned IPC API. Do not bootstrap raw credentials or
+ model routes by writing private storage, and do not bypass Model Hub.
+- `AVIBE_HOME` controls product state, including SQLite, native transcript,
+ compaction, scheduler, Watches, and job recovery. There is no separate removed
+ memory component to resurrect; the shipped transcript/context implementation
+ is the measurement subject.
+
+## First implementation contract
+
+Add only source-side tooling under `scripts/terminal_bench/` and focused tests,
+not product wheel dependencies, not a Harbor dependency in the product lock.
+The wrapper invokes the normal service entrypoint in an owned subprocess.
+All product imports and CLI operations
+run in child processes with their own environment; the parent environment and
+running user service are never modified.
+
+Per trial:
+
+1. Validate an existing task cwd and claim a *new* private trial root outside it.
+ Reject overlap in either direction, existing state, and symlink aliases.
+2. Prepare an isolated config with no external messaging platforms, remote pairing,
+ updates, native backend launches, or UI. Keep scheduler/Watch/Session machinery:
+ disabling Harness would not measure the product. Use explicit local catalog
+ fixtures and offline managed-runtime settings in the hermetic slice.
+3. Start the real service and require controller IPC health, not merely a PID or
+ fabricated readiness file. Do not run the service from the task cwd: task
+ `config.py`/`core` modules must not shadow product modules.
+4. Configure one Source through Model Hub IPC using a key supplied only on a
+ private stdin pipe. Never persist the provider key in a command, wrapper config,
+ parent artifact, or task shell environment. Runtime engine custody stays owned
+ by the product. No Source means a reported `blocked` bootstrap, not a fake pass.
+5. Use the real CLI Run path with explicit Agent/model/effort, task cwd, and
+ instruction. Collect the durable run outcome and redacted terminal result.
+ A result is not a verifier reward. The optional Harbor adapter runs the
+ separate verifier; ATIF export and public submission are not pilot gates.
+6. Gracefully terminate the exact owned service and collect its descendants while
+ alive. Join/terminate/kill by recorded process identity within bounded grace;
+ never use a global `vibe stop`, pattern kill, or the user's runtime records.
+ Refuse to mark a trial closed if an owned process survives. Export artifacts
+ only after shutdown; never copy raw SQLite, runtime logs, jobs or config.
+
+Task environment and runtime state are separate. On a host, the wrapper uses a
+throwaway HOME/XDG to avoid user state. In a sandbox, its caller may explicitly
+supply the task image environment; preserve toolchain variables and task HOME,
+while stripping inherited Avibe caller routing and credential variables. The
+wrapper is not a security sandbox: paid runs require an external isolated
+container with no host mounts or secret access. Tasks may write scripts,
+SQLite databases, logs, and notes in their normal cwd.
+
+## Tests and evidence
+
+Existing tests do not exercise this source-side supervisor boundary. New focused
+pytest cases protect fresh-state refusal, environment/cwd separation, no parent
+state mutation, actual service/IPC/Run dispatch, task writes (including Unicode
+and SQLite), explicit blocked bootstrap, redaction, and exact-owned shutdown.
+
+The integration test may replace only the external Model Hub *engine adapter*
+with a scripted test double, while keeping production Controller, Model Hub
+service/router/gateway, native providers, AgentService, Vibey and Harness real.
+Report that boundary honestly: it proves local product integration, not real CPA
+installation, credential custody, provider connectivity, or a benchmark score.
+
+## Implemented evidence
+
+- `scripts/terminal_bench/{profile,control,runner}.py` and `README.md` are the
+ single source-only wrapper. Optional `harbor_agent.py`, `launch.py` and
+ `credential.py` adapt its lifecycle and private credential transport to
+ Harbor; no extracted product packages are uploaded.
+- The initial wrapper left product dependency, service, Agent and storage files
+ unchanged. The follow-up reliability repair also changes shared final-response
+ classification so an explicit output-limit stop cannot be persisted as success;
+ dependency files, service ownership and database schema remain unchanged.
+- The subprocess test uses normal `main.py`, IPC Source/catalog/route APIs and
+ the real CLI Run path. It proves a native Bash tool writes a Unicode note and
+ task SQLite, with no framework files in task cwd; task `config.py` cannot
+ shadow product imports. Provider keys never enter the service/tool environment.
+- A real product Watch creates a delayed note after the initial Run result.
+ The wrapper waits for Harness quiescence instead of prematurely tearing down
+ background work. Missing credentials are explicitly blocked, not fake passes.
+- The deadline case interrupts a continuously writing real tool, closes the
+ runtime, then verifies no later writes. A separate process-lifecycle case
+ covers a setsid child, a stale unrelated pid identity, and clock-shifted
+ identity without signalling the bystander.
+- Exports name the Run/Session, root-Run outcome scope, actual installation,
+ reported usage (completeness explicitly unknown), cleanup and null verifier
+ reward. Raw product state/logs are not copied.
+
+## Installed-runtime acceptance and remaining gates
+
+An independent container check used the official `avibe-os 3.2.0rc5` wheel
+(source `98cc70a1c`, SHA256
+`5b1b59cd1cf93f36b00f1bef8490612da180366244e2cee163e0d59977fffea0`), its real
+managed Model Hub engine, and a local scripted HTTP upstream. It confirmed
+ordinary CLI Run dispatch, native file writes/reads, task-owned SQLite, Harness
+quiescence and process cleanup. The full Harbor path subsequently passed its
+synthetic separate verifier with reward 1 and zero infrastructure exceptions.
+These are integration checks, **not model capability scores**. Source-checkout
+unit tests use an external-engine fixture and are a separate evidence layer.
+
+The optional adapter requires Harbor 0.24.0, verifies wheel bytes before upload,
+and records the actual wrapper source and installation. It declares `atif=False`;
+a local verifier score does not depend on ATIF or official leaderboard submission.
+Docker Compose can preserve a host UID on uploaded files; credential setup
+adopts only a validated single-link regular inode under its private directory,
+while normal consumption retains strict ownership and permission checks.
+
+The first remote paired pilot reached a real candidate verifier (reward 0 after
+an empty-reply Run failure); its final raw provider stop was not retained. The
+reference lost its owner without terminal verification. These do not form a valid
+paired capability comparison. Reliability diagnosis and repair are tracked in
+[the follow-up contract](2026-10-14-terminal-bench-reliability.md).
+
+Remaining gates: a separately authorized paired run after reliability acceptance,
+cost assessment before larger batches, and current-head review/CI before code
+delivery. Benchmark results must name the tested wheel and wrapper revisions,
+not assume they match the latest product branch. Preserve failed attempts and
+resource/protocol differences; do not extrapolate a small pilot to a leaderboard
+rank. See `scripts/terminal_bench/HARBOR.md` for supported images and lifecycle
+limitations.
+
+Run only focused tests through `~/.local/bin/hd1-test`; retain product dependency
+files unchanged. Any unproven lifecycle/provider capability remains a named gate.
diff --git a/modules/agents/vibey/agent.py b/modules/agents/vibey/agent.py
index fe8820161..b4f0969d3 100644
--- a/modules/agents/vibey/agent.py
+++ b/modules/agents/vibey/agent.py
@@ -121,7 +121,7 @@ logger = logging.getLogger(__name__)
BACKEND = "vibey"
# Final responses whose empty text is explained by the stop itself (loop-control.md section 2).
-_EXPLAINED_STOPS = ("refusal", "safety")
+_EXPLAINED_STOPS = ("length", "refusal", "safety")
_COMPLETED = ("completed", "ended_by_hook")
_JOB_PRUNE_INTERVAL_S = 3600.0
diff --git a/modules/agents/vibey/errors.py b/modules/agents/vibey/errors.py
index e13bf9cf1..cb0a51794 100644
--- a/modules/agents/vibey/errors.py
+++ b/modules/agents/vibey/errors.py
@@ -40,7 +40,7 @@ _KIND_KEYS: dict[str, str] = {
# A provider failure C-2 could not classify: an unknown code, or an error frame no rule names.
"unknown": "unclassified",
"error": "modelError",
- # The model kept truncating a tool call at its output limit (C-3 section 2).
+ # An explicit output-limit stop with no answer, or repeatedly truncated tools.
"length": "outputLimit",
# Tool calls under a stop reason that cannot carry them.
"stop": "providerProtocol",
diff --git a/scripts/terminal_bench/HARBOR.md b/scripts/terminal_bench/HARBOR.md
new file mode 100644
index 000000000..a2c19e3d5
--- /dev/null
+++ b/scripts/terminal_bench/HARBOR.md
@@ -0,0 +1,224 @@
+# Optional Harbor bridge
+
+`harbor_agent.FullAvibeAgent` is a source-only `BaseInstalledAgent` for the full
+runtime wrapper in [README.md](README.md). It declares **`atif=False`**. It does
+not claim leaderboard submission or ATIF support; a local official verifier
+reward is possible without them.
+
+## Host setup
+
+Use a **separate** evaluation-host environment. Harbor's dependencies must not
+change the product `pyproject.toml`, `uv.lock`, or wheel:
+
+```sh
+# From this checkout, in a dedicated venv:
+uv pip install --python /path/to/eval-venv/bin/python \
+ -r scripts/terminal_bench/requirements-harbor.txt
+export PYTHONPATH="$(pwd)" # Evaluation HOST only; never passed into the task.
+```
+
+Example Harbor job file (replace paths, SHA and profile values explicitly):
+
+```json
+{
+ "job_name": "avibe-pilot-one",
+ "jobs_dir": "/path/to/private-job-artifacts",
+ "n_concurrent_trials": 1,
+ "n_attempts": 1,
+ "agents": [{
+ "import_path": "scripts.terminal_bench.harbor_agent:FullAvibeAgent",
+ "model_name": "deepseek-v4-pro",
+ "override_timeout_sec": 1800,
+ "kwargs": {
+ "wheel_path": "/path/to/avibe_os-<pinned-version>-py3-none-any.whl",
+ "wheel_sha256": "<verified-64-character-lowercase-sha256>",
+ "profile": {
+ "model": "deepseek-v4-pro",
+ "base_url": "https://api.deepseek.com/v1",
+ "credential_env": "DEEPSEEK_API_KEY",
+ "protocol": "openai_chat",
+ "reasoning_effort": "high",
+ "context_window": 128000,
+ "max_output_tokens": 32000,
+ "wall_seconds": 1800,
+ "setup_seconds": 600,
+ "cleanup_seconds": 30,
+ "poll_seconds": 1,
+ "offline": false
+ }
+ }
+ }],
+ "tasks": [{"path": "/path/to/pinned-official-task"}],
+ "environment": {"type": "docker", "delete": true}
+}
+```
+
+Run with the named credential injected into the evaluation HOST by an authorized
+secret injector. Do not put its value in job JSON or argv. Harbor 0.24's Docker
+`exec(env=...)` itself expands to `docker compose exec -e KEY=value` on the host,
+so this adapter deliberately does **not** pass credential values through exec
+env. It reads the named host environment variable via Harbor's `_get_env` and
+uses the narrow ephemeral-file transport below instead.
+
+```sh
+/path/to/eval-venv/bin/harbor run --config pilot.json --max-retries 0
+```
+
+The adapter requires Harbor 0.24.0, a trusted local **ordinary Avibe wheel**, its
+explicit SHA256, a Profile, and a matching explicit model. It hashes an immutable
+copy **before uploading anything**. It installs the wheel into a private venv,
+not editable source. Only seven tracked wrapper files are copied beside it;
+product packages, tests, oracle solutions and user home are never source-uploaded.
+Wheel version/SHA, actual Harbor version, wrapper commit/hash/dirty state and
+execution metadata are recorded. Real paired runs should use a clean committed
+checkout and an independently verified pinned wheel.
+
+## Ephemeral API-key transport
+
+This is temporary transport to the ordinary Model Hub Source-create API, **not**
+a Vault/key-custody replacement or a generic secret-export feature.
+
+- First pilot supports **root task users only**; nonroot images fail explicitly
+ before installation rather than changing task ownership silently.
+- Claim a unique trial root and its credential subdirectory mode0700. Create one
+ host mode0600 `provider.key` under a mode0700 tempfile directory; transfer only
+ with Harbor `upload_file`, never a shell literal, echo, heredoc, env value, or
+ JSON configuration. Inspect remote regular-file/type/mode/size metadata
+ without printing or comparing secret content in logs. Docker Compose may
+ preserve a host UID during upload. The root-only setup helper adopts that
+ copied inode using `fchown` only after validating the owned 0700 parent,
+ non-symlink 0600 regular file, single link and size bound. Later metadata and
+ consumption still require runner ownership; they never adopt an arbitrary file.
+- The wrapper opens without following links, rejects nonregular/multiply-linked,
+ oversized or improperly protected files, reads and unlinks the input **before
+ starting the service**, then uses the existing private-stdin bootstrap. Provider
+ keys never enter wrapper/service/tool exec env or argv in this Harbor mode.
+- A cancelled upload is joined before cleanup, so late copy completion cannot
+ recreate a deleted secret. Upload/read/metadata failures, cancellation and
+ timeout clear that exact owned remote file and host tempfile. Unknown remote
+ cleanup blocks handoff and requires container destruction. No recursive
+ deletion or another trial's path is used.
+- Direct wrapper invocation may retain its named-environment mode, but the Docker
+ exec bridge always supplies `--credential-file` containing only a private path.
+
+## Task and lifetime boundary
+
+- POSIX/Linux task containers with Python >=3.10 (Harbor may bootstrap Python and
+ venv) are the first supported target. Windows is not claimed.
+- Setup installs the normal wheel/dependencies once. Model Hub/bootstrap work
+ inside `run()` counts against Harbor's outer task deadline: equal wall limits
+ are **not** a claim of equal model-active time between Agent implementations.
+- Resolve the actual task cwd with `pwd -P`; do not assume `/app` or `/root`.
+ The original task HOME/toolchains are retained. A private directory containing
+ only the installed `vibe` executable is prefixed to PATH, not the Python venv.
+- Source/venv/state live in a unique `/installed-agent/avibe-full-*` directory;
+ exported reports are under `/logs/agent/full-runtime-*`. Reject overlapping
+ task roots rather than silently contaminating them.
+- Model requests still use the real product Model Hub. The adapter does not
+ override the official task's network/resources or secretly forbid legitimate
+ task Internet use. Supply the provider endpoint as an agent-phase allowance
+ in the normal Harbor job/network policy where needed; installer egress belongs
+ to setup. Do not change the official task's verifier/artifact declaration.
+- Killing a `docker exec` client is insufficient. A small in-container launcher
+ records the exact wrapper PID/creation identity/nonce under a locked start
+ fence. Cancellation first fences a late start, then signals the exact wrapper,
+ waits for it to exit and requires its inner owned-process-cleanup receipt.
+ The adapter joins this path before downloading reports or letting Harbor grade.
+- A missing/mismatched/unconfirmed shutdown receipt is an infrastructure error,
+ **not a task failure score**. The outer owner must destroy that sandbox before
+ any grading; forced wrapper death alone cannot prove a detached engine/job
+ stopped. Use an init/reaper in the container. A cancelled-before-start fence
+ does not invent an execution report.
+
+`execution.json` is projected to Harbor AgentContext metadata and token counts
+only when the product reports tokens. Cost stays null (unknown provider bill),
+accounting completeness stays explicit, and verifier reward stays separate.
+No model text is passed to Harbor's regex-based CLI error classifier. Reports,
+not the runner's exit status, identify completion/failure/time limit.
+
+## Failure diagnosis and remote ownership
+
+`execution.json.diagnostics` is a bounded, allowlisted projection fetched via
+`vibe data query` before the service stops. It records canonical stop reasons,
+text/thinking/tool block counts, character counts and reported token counts for
+up to eight recent responses in the root Session's last 32 context messages.
+`responses_truncated` and `history_complete=false` make its limits explicit.
+No prompt, reasoning text, tool arguments, raw service log or database is exported.
+If the query fails, diagnostics say `unavailable`; the real outcome and shutdown
+still take precedence. `has_error_message` is presence, not an error taxonomy.
+
+For remote execution, do not keep Harbor owned by an SSH connection. On the
+**dedicated Linux evaluation instance**, use the source-only systemd supervisor:
+
+```sh
+# Config paths must be absolute ON THE REMOTE INSTANCE. Preserve the original
+# config and record any explicit path relocation; do not silently change budgets.
+/path/to/harbor-venv/bin/python -m scripts.terminal_bench.remote_job preflight \
+ --config /private/bench/job.json
+/path/to/harbor-venv/bin/python -m scripts.terminal_bench.remote_job submit \
+ --config /private/bench/job.json --root /private/bench/runs/unique-run \
+ --credential-file /private/staging/credential/provider.key --wall-seconds 1800
+/path/to/harbor-venv/bin/python -m scripts.terminal_bench.remote_job status \
+ --root /private/bench/runs/unique-run
+```
+
+The credential parent must be owned mode0700 and the regular file mode0600;
+the supervisor consumes it before starting Harbor. Full-runtime jobs derive the
+environment-variable name from the profile, and explicit mismatches are refused.
+For a reference agent specify its provider variable with `--credential-env`.
+No secret value enters systemd properties or command arguments. Paths containing
+systemd `%` specifiers, `$` expansions or line breaks are refused (including
+resolved symlink targets) rather than interpreted by the service manager.
+
+The host-wide `/run/lock/avibe-benchmark.active.json` reservation rejects a second
+submission regardless of run-root parent. A permanent per-job destination claim
+also prevents accidental reuse after completion. An unconfirmed owner remains a
+blocker; never delete its reservation merely to launch another paid attempt.
+The supervisor requires Harbor 0.24.0, one local single-step task, one attempt,
+zero retries and concurrency one. It keeps the frozen config and wheel hash;
+preflight never calls a model. The remote hard wall covers the whole job and is
+recorded separately from Harbor's agent-execution budget. It does not secretly
+extend setup, output, model or verification budgets.
+
+The pinned Harbor driver's awaited START hook records exact agent and verifier
+Compose project labels **before** resources are created, including network-only
+startup failures. systemd owns the worker and descendants; ExecStopPost runs
+only after their process group is fenced. This phase recovers declared task
+artifacts as separate, size/time-bounded **opaque tar files**, never extracts
+container-controlled symlinks onto the host, removes only proven-owned containers
+and networks, and writes `receipt.json`. Inspect recovered archives in an isolated
+workspace; do not blindly extract untrusted paths on the host. Missing/malformed Harbor output cannot skip
+cleanup. A zero child exit plus a job timestamp is not enough: completion needs
+the one exact trial's terminal receipt as well. Harbor results are never rewritten
+and orchestration failure never invents a reward or usage.
+
+Observe `status` with one Avibe Harness Watch (exit75 = still active). A dead unit
+without a terminal receipt is an explicit failure, not an endless wait. Reconnect
+and inspect after an SSH failure; **do not resubmit** the workload. The supervisor
+needs systemd and Docker inside an explicitly authorized dedicated environment;
+it does not provision hosts, relax Incus security, restart shared services, or
+change other projects. Read-only diagnostics and no-paid lifecycle fixtures are
+not benchmark capability scores.
+
+## Focused optional tests
+
+Normal product tests skip the Harbor-only file when Harbor is absent. Run it
+explicitly in the isolated optional environment after installing product dev
+test prerequisites and `requirements-harbor.txt`:
+
+```sh
+~/.local/bin/hd1-test /path/to/eval-venv/bin/python -m pytest -q \
+ tests/test_terminal_bench_harbor.py tests/test_terminal_bench_launch.py \
+ tests/test_terminal_bench_credential.py
+```
+
+The tests use Harbor's actual class/model interfaces, a recording Environment
+transport, and owned local processes. They verify wheel mismatch refusal before
+mutation, restricted source transfer, explicit model/profile binding, unknown
+cost, cancellation join before artifact reads, failed cleanup blocking handoff,
+late-start fencing and exact-owned TERM/receipt behavior. Credential cases prove
+sentinel absence from all exec argv/env/config and logs, consume-before-service,
+stdin-only bootstrap, rejected input shapes and cleanup on failed/cancelled
+uploads. They do **not** prove
+a Docker install or real engine execution. Those remain the parent's one-task
+container smoke gate before a paid paired trial.
diff --git a/scripts/terminal_bench/README.md b/scripts/terminal_bench/README.md
new file mode 100644
index 000000000..80f0b55fb
--- /dev/null
+++ b/scripts/terminal_bench/README.md
@@ -0,0 +1,165 @@
+# Full Avibe Runtime trial wrapper
+
+Optional Harbor entrypoint:
+`scripts.terminal_bench.harbor_agent:FullAvibeAgent` — see [HARBOR.md](HARBOR.md).
+It installs a SHA-verified normal wheel, uploads only tracked wrapper files,
+declares `atif=False`, and joins owned shutdown before grading. Its ephemeral
+credential-file transport avoids Harbor Docker's key-bearing exec-env argv;
+the file is consumed/unlinked before service startup. Direct invocation below
+may retain the named-environment input mode.
+
+This source-only tool drives **normal Avibe**, not an extracted agent core:
+
+1. a fresh `AVIBE_HOME`, ordinary V2 config and ordinary service main;
+2. the real controller-owned Model Hub IPC for Source and route setup;
+3. `vibe agent update`, followed by `vibe agent run` and stable Run polling;
+4. exact-Run cancellation on deadline and bounded owned-process shutdown;
+5. `execution.json`, separate from a later Harbor verifier reward.
+
+The native scheduler, Watches, Session/history/data-query machinery and Vibey
+context compaction stay intact. No external IM platform, browser, Cloud pairing,
+tunnel or Web UI is started. The loopback/internal Model Hub and control IPC are
+real product services. No `Controller`, `Agent`, Session row or provider loop is
+reimplemented here. This tool is **not** a sandbox.
+
+## Prerequisites
+
+Use a dedicated interpreter with the ordinary, pinned Avibe wheel installed,
+including its normal dependencies. Verify the wheel SHA externally and keep it
+in the experiment receipt. The interpreter chosen to launch this wrapper is
+also used for the service and CLI. Do not run `uv` in the product repository in a
+way that replaces its lock with Harbor dependencies.
+
+For example, inside a fresh task container (with an init/reaper such as Docker's
+`--init`), after installing the verified wheel into `/opt/avibe`:
+
+```sh
+# /opt/eval contains only this source-side scripts package, NOT a dataset,
+# oracle solution, the user's home, or another agent's trajectory.
+cd /opt/eval
+/opt/avibe/bin/python -m scripts.terminal_bench.runner \
+ --workdir /app \
+ --state /var/lib/avibe-eval/trial-001 \
+ --output /logs/agent/full-runtime \
+ --profile /opt/eval/profile.json \
+ --instruction /opt/eval/instruction.txt \
+ --sandbox-environment
+```
+
+`state` and `output` must **not exist** yet. They and `workdir` must be disjoint
+(no aliases, nesting, or filesystem root). Do not mount user state or the host
+Docker socket. Source-code/test helpers, task instructions, raw service logs,
+and model keys must not be collected as task outputs. Review the task's declared
+artifacts to ensure they cannot copy the private state directory.
+
+Example profile (freeze before trials; these are pilot limits, not the official
+Terminal-Bench protocol):
+
+```json
+{
+ "model": "deepseek-v4-pro",
+ "base_url": "https://api.deepseek.com/v1",
+ "credential_env": "DEEPSEEK_API_KEY",
+ "protocol": "openai_chat",
+ "reasoning_effort": "high",
+ "context_window": 128000,
+ "max_output_tokens": 32000,
+ "wall_seconds": 1200,
+ "setup_seconds": 600,
+ "cleanup_seconds": 30,
+ "poll_seconds": 1,
+ "offline": false
+}
+```
+
+Supply the named credential through an authorized secret injector, not argv or a
+profile value. The supervisor passes it only to the short-lived bootstrap helper
+on stdin, and that helper uses the ordinary Model Hub API. The service and tool
+subprocess environment do not inherit the provider key. Real custody is still
+Model Hub's responsibility. Tool processes share the container user; this is not
+a key-security boundary against a malicious root process.
+
+`--sandbox-environment` is only for an already isolated container: preserve the
+image's HOME, XDG, PATH, Python/Rust/Java/conda toolchains and other task variables,
+while clearing inherited Avibe routing and credential variables. Without it, the
+wrapper uses a private host-test HOME. It never modifies its parent's environment.
+This prevents tests from touching the developer's account; the host-test mode is
+not a leaderboard-equivalent HOME configuration.
+
+`offline=true` prevents managed binary downloads and is used in local tests. Real
+runs with it require already installed/cached dependencies in that fresh home.
+`offline=false` allows normal product dependency setup; arrange installer egress
+in the sandbox setup phase. This wrapper does not replace Harbor's network policy
+or impose a model-provider-only rule on tasks that legitimately need the network.
+
+## Result and lifecycle contract
+
+- A ready service with no supplied credential is `blocked`; no model Run or score
+ is invented.
+- Configure the isolated Agent's model/effort with the existing update command.
+ There are no unsupported model override flags on non-forked `agent run`.
+- Creation is asynchronous to capture the Run/Session IDs **before** waiting.
+ This avoids mistaking CLI `--wait-timeout` for cancellation. A terminal Run
+ does not automatically finish a trial: wait for the fresh home's real Harness
+ to quiesce (including Watches, scheduled work and delegated runs). Check twice,
+ and refuse incomplete/truncated snapshots. A delayed Watch note is covered by
+ the integration test; it must not be killed at the initial final reply.
+ `outcome` and `final_text` identify the initial Run, not an invented aggregate
+ verdict over all continuations; grading still belongs to the verifier.
+- `time_limit` requests cancellation of that exact Run. A terminal Run may still
+ have Watch work, so the service and its recorded descendants are stopped before
+ the final report is written. TERM reaches normal Controller shutdown; forced
+ cleanup checks recorded process identity, including detached descendants.
+- A failed cleanup prevents a normal handoff. Keep the surrounding sandbox alive
+ until all writers are gone; if the supervisor is killed outright, its container
+ owner must stop/remove the entire task container before any separate grading.
+- Wrapper failures retain a bounded symbolic `error_code` (for example,
+ `cli_helper_timeout` or `harness_quiescence_unconfirmed`) in `execution.json`
+ and the CLI stdout/stderr summaries. Only wrapper-owned codes are accepted;
+ arbitrary upstream exception text and captured helper output are never echoed.
+- `outcome` is the product Run state, `time_limit`, `blocked`, or a wrapper failure;
+ `verifier_reward` remains null. Do not treat wrapper/Harbor exit status as a
+ success rate.
+- `usage` is the product's **reported fresh-home usage, including bootstrap**,
+ not a provider invoice. `accounting_complete=false` deliberately avoids claiming
+ knowledge of unreported/cancelled provider work. No inferred zero-dollar cost.
+- Only the structured report is exported. Raw SQLite, job/scratch state, secrets,
+ setup files and runtime logs remain private. Model text in the report is
+ redacted against the supplied key; it is not an unredacted token-delta stream.
+- The report identifies the imported installation and whether source-checkout
+ test mode or installed-package mode was used. Do not conflate their revisions.
+
+## Tests
+
+```sh
+uv sync --python 3.12 --group dev --no-install-project --frozen
+~/.local/bin/hd1-test .venv/bin/python -m pytest -q tests/test_terminal_bench_full_runtime.py
+ruff check scripts/terminal_bench tests/test_terminal_bench_full_runtime.py tests/fixtures/full_runtime_service.py
+```
+
+The process test runs the real service, Model Hub aggregate/gateway, product CLI,
+Run/Session persistence, Vibey and its tools. It substitutes **only the external
+engine adapter** with the repository's existing deterministic fixture. It proves
+Unicode note and SQLite task writes, module-shadow isolation, actual deadline
+cancellation, no writes after cleanup, secret-safe exports, and no task-workspace
+pollution. These are local integration tests, **not** acceptance of the real CPA
+engine or a benchmark score. The parent's installed-wheel + real-engine + local
+HTTP upstream check is a separate evidence layer.
+
+The subprocess fixture has a separate 120-second setup budget for fresh SQLite
+migrations on cold/contended filesystems. Its Agent execution budgets remain
+30 seconds (normal) and 2 seconds (deadline cancellation); a service that exits
+before readiness is still reported immediately. This is not a retry policy.
+
+The startup updater log may initially show compatibility defaults (60 minutes,
+`auto_update=true`). The real `_do_check()` reloads the saved config before its
+interval-zero early return; the intervening askill reconcile is explicitly
+disabled by this wrapper. That log alone does not prove updater egress. No product
+updater behavior is changed by this integration.
+
+Still required: parent-run installed-wheel wrapper smoke in the real sandbox,
+Harbor installed-agent wiring (declare `atif=False` until conversion is implemented),
+pinned task verification, same-profile paired trials, network-policy/resource
+equivalence, review/CI, and cost assessment before any larger paid batch. ATIF
+conversion is a later submission feature, not a prerequisite to an honest local
+pilot with official verifier rewards and structured/native execution evidence.
diff --git a/scripts/terminal_bench/__init__.py b/scripts/terminal_bench/__init__.py
new file mode 100644
index 000000000..f5ececbb1
--- /dev/null
+++ b/scripts/terminal_bench/__init__.py
@@ -0,0 +1 @@
+"""Source-only Terminal-Bench integration; not part of the Avibe wheel."""
diff --git a/scripts/terminal_bench/control.py b/scripts/terminal_bench/control.py
new file mode 100644
index 000000000..025548aeb
--- /dev/null
+++ b/scripts/terminal_bench/control.py
@@ -0,0 +1,131 @@
+"""Child-only bridge to the normal installed Avibe service, CLI, and Model Hub IPC.
+
+No Controller/Agent replacement, no storage seeding, no provider implementation.
+The supervisor supplies a fresh AVIBE_HOME before this module imports Avibe.
+"""
+from __future__ import annotations
+
+import argparse
+import asyncio
+import importlib.metadata
+import json
+import os
+from pathlib import Path
+import runpy
+import sys
+
+
+def initialize(workdir: Path) -> dict:
+ from config import paths
+ from config.v2_config import V2Config
+
+ config_path = paths.get_config_path()
+ if config_path.exists():
+ raise RuntimeError("refusing to overwrite an existing Avibe configuration")
+ config = V2Config.default()
+ config.runtime.default_cwd = str(workdir.resolve())
+ config.platforms.enabled = []
+ config.platforms.primary = config.platform = "avibe"
+ for backend in (config.agents.claude, config.agents.codex, config.agents.opencode):
+ backend.enabled = backend.auto_update = False
+ config.update.auto_update = config.update.notify_admins = False
+ config.update.check_interval_minutes = 0
+ config.ui.open_browser = False
+ config.ui.show_agent_activity = True
+ config.remote_access.vibe_cloud.enabled = False
+ config.setup_completed = True
+ # Empty-home startup must not fetch or start an engine. The normal Hub
+ # runtime_start operation below owns that transition, not a hand-written key.
+ config.model_hub.enabled = False
+ config.save(config_path)
+ from vibe.runtime import get_service_main_path
+ try:
+ version = importlib.metadata.version("avibe-os")
+ except importlib.metadata.PackageNotFoundError:
+ version = None
+ import vibe
+ return {"service_main": str(get_service_main_path()), "avibe_version": version,
+ "python": sys.version,
+ "backend_catalog": str(Path(vibe.__file__).parent / "data" / "backend_models.json")}
+
+
+async def configure(profile: dict, key: str) -> dict:
+ from vibe.model_hub_client import ModelHubRemoteService
+
+ hub = ModelHubRemoteService()
+ await hub.runtime_start()
+ created = await hub.create_source({
+ "kind": "api_key", "vendor": "custom", "display_name": "Benchmark provider",
+ "base_url": profile["base_url"], "protocol": profile["protocol"], "key": key,
+ "save_unverified": True,
+ "models": [{"id": profile["model"], "origin": "manual",
+ "reasoning_efforts": [profile["reasoning_effort"]]}],
+ })
+ source_id = created["source"]["id"]
+ supply = await asyncio.to_thread(hub.get_agent_sources, "vibey")
+ baseline = supply.get("catalog_models", [])
+ chosen = {
+ "id": profile["model"], "origin": "manual",
+ "context_window": profile["context_window"],
+ "max_output_tokens": profile["max_output_tokens"],
+ "input_modalities": ["text"], "output_modalities": ["text"],
+ "supports_tools": True, "supports_reasoning": True,
+ "reasoning_efforts": [profile["reasoning_effort"]],
+ }
+ await hub.set_agent_sources("vibey", {"order": [source_id]})
+ await hub.set_agent_models("vibey", baseline, [chosen])
+ await hub.set_agent_chain("vibey", profile["model"], {
+ "hops": [{"source_id": source_id, "model_id": profile["model"]}],
+ })
+ return {"source_id": source_id, "model": profile["model"], "protocol": profile["protocol"],
+ "runtime": await hub.runtime_status()}
+
+
+def main(argv=None) -> int:
+ parser = argparse.ArgumentParser(description=__doc__)
+ parser.add_argument("command", choices=["initialize", "serve", "health", "configure", "usage", "versions"])
+ parser.add_argument("--workdir", type=Path)
+ parser.add_argument("--profile", type=Path)
+ args = parser.parse_args(argv)
+ if not os.environ.get("AVIBE_HOME") or not os.environ.get("AVIBE_BENCHMARK_OWNER"):
+ parser.error("a supervisor-owned isolated AVIBE_HOME is required")
+ try:
+ if args.command == "initialize":
+ if args.workdir is None:
+ parser.error("initialize requires --workdir")
+ result = initialize(args.workdir)
+ elif args.command == "serve":
+ from vibe.runtime import get_service_main_path
+ runpy.run_path(str(get_service_main_path()), run_name="__main__")
+ return 0
+ elif args.command == "health":
+ from vibe.internal_client import health_identity_sync
+ identity = health_identity_sync()
+ result = {"ready": identity is not None, "identity": identity}
+ elif args.command == "configure":
+ if args.profile is None:
+ parser.error("configure requires --profile")
+ profile = json.loads(args.profile.read_text(encoding="utf-8"))
+ # Only this short-lived helper receives the key, over a private pipe.
+ key = sys.stdin.read(65537)
+ if not key.strip() or len(key) > 65536:
+ raise ValueError("missing or oversized credential")
+ result = asyncio.run(configure(profile, key))
+ elif args.command == "usage":
+ from vibe.model_hub_client import ModelHubRemoteService
+ result = asyncio.run(ModelHubRemoteService().usage_summary(window="24h"))
+ else:
+ result = {"python": sys.version, "packages": {
+ dist.metadata["Name"]: dist.version for dist in importlib.metadata.distributions()
+ if dist.metadata.get("Name")
+ }}
+ print(json.dumps(result, ensure_ascii=False))
+ return 0
+ except Exception as exc:
+ # Upstream exception text may echo the key. Never print it on the bridge.
+ print(json.dumps({"ok": False, "error_type": type(exc).__name__}))
+ return 2
+
+
+if __name__ == "__main__":
+ raise SystemExit(main())
diff --git a/scripts/terminal_bench/credential.py b/scripts/terminal_bench/credential.py
new file mode 100644
index 000000000..63c7ab026
--- /dev/null
+++ b/scripts/terminal_bench/credential.py
@@ -0,0 +1,118 @@
+"""Ephemeral benchmark API-key file transport, not a key-custody service.
+
+Harbor Docker exec env values appear in host argv. Instead upload one 0600 file
+into a fresh 0700 per-trial directory, validate metadata, then read+unlink before
+starting Avibe. Secret bytes never occur in this helper's argv or printed output.
+"""
+from __future__ import annotations
+
+import argparse
+from contextlib import contextmanager
+import json
+import os
+from pathlib import Path
+import stat
+
+LIMIT = 65536
+
+
+@contextmanager
+def _directory(path: Path):
+ if path.name != "provider.key" or path.parent.name != "credential":
+ raise ValueError("unexpected credential input path")
+ fd = os.open(path.parent, os.O_RDONLY | os.O_DIRECTORY | os.O_NOFOLLOW)
+ try:
+ info = os.fstat(fd)
+ if stat.S_IMODE(info.st_mode) != 0o700 or info.st_uid != os.geteuid():
+ raise ValueError("credential directory must be private and owned by the runner")
+ yield fd
+ finally:
+ os.close(fd)
+
+
+@contextmanager
+def _file(path: Path, directory: int, *, adopt_uploaded: bool = False):
+ fd = os.open(path.name, os.O_RDONLY | os.O_NOFOLLOW | os.O_NONBLOCK, dir_fd=directory)
+ try:
+ info = os.fstat(fd)
+ if (not stat.S_ISREG(info.st_mode) or stat.S_IMODE(info.st_mode) != 0o600
+ or info.st_nlink != 1 or not 0 < info.st_size <= LIMIT):
+ raise ValueError("credential input must be one private regular bounded file")
+ if info.st_uid != os.geteuid():
+ # Compose cp can preserve the host UID (e.g. macOS 501). Only the
+ # setup path may adopt that copied inode, after its containing
+ # directory and all file properties have been checked. Never chown
+ # a pathname: that could follow a replaced symlink or hardlink.
+ if not adopt_uploaded or os.geteuid() != 0:
+ raise ValueError("credential input must be owned by the runner")
+ os.fchown(fd, os.geteuid(), -1)
+ info = os.fstat(fd)
+ if info.st_uid != os.geteuid() or stat.S_IMODE(info.st_mode) != 0o600:
+ raise ValueError("credential input ownership could not be established")
+ yield fd, info
+ finally:
+ os.close(fd)
+
+
+def prepare_upload(path: Path) -> dict:
+ """Adopt only a validated copied inode in our new private staging directory."""
+ with _directory(path) as directory:
+ with _file(path, directory, adopt_uploaded=True) as (_fd, info):
+ return {"regular": True, "mode": "0600", "directory_mode": "0700", "size": info.st_size}
+
+
+def metadata(path: Path) -> dict:
+ with _directory(path) as directory:
+ with _file(path, directory) as (_fd, info):
+ return {"regular": True, "mode": "0600", "directory_mode": "0700", "size": info.st_size}
+
+
+def consume(path: Path) -> str:
+ with _directory(path) as directory:
+ try:
+ with _file(path, directory) as (fd, _info):
+ with os.fdopen(os.dup(fd), "rb") as reader:
+ data = reader.read(LIMIT + 1)
+ if len(data) > LIMIT:
+ raise ValueError("credential input grew beyond its bound")
+ value = data.decode("utf-8").strip()
+ if not value:
+ raise ValueError("empty credential input")
+ return value
+ finally:
+ # A rejected symlink is removed as a link, never followed. This
+ # fixed name sits in the freshly claimed per-trial private directory.
+ try:
+ os.unlink(path.name, dir_fd=directory)
+ except FileNotFoundError:
+ pass
+
+
+def clean(path: Path) -> dict:
+ if not path.parent.exists() and not path.parent.is_symlink():
+ return {"removed": True}
+ with _directory(path) as directory:
+ try:
+ os.unlink(path.name, dir_fd=directory)
+ except FileNotFoundError:
+ pass
+ path.parent.rmdir() # Refuse unknown files rather than delete recursively.
+ return {"removed": True}
+
+
+def main(argv=None):
+ parser = argparse.ArgumentParser(description=__doc__)
+ parser.add_argument("operation", choices=["prepare", "metadata", "clean"])
+ parser.add_argument("path", type=Path)
+ args = parser.parse_args(argv)
+ try:
+ operation = {"prepare": prepare_upload, "metadata": metadata, "clean": clean}[args.operation]
+ print(json.dumps(operation(args.path)))
+ return 0
+ except Exception as exc:
+ print(json.dumps({"ok": False, "error_type": type(exc).__name__}))
+ return 2
+
+
+if __name__ == "__main__":
+ raise SystemExit(main())
diff --git a/scripts/terminal_bench/diagnostics.py b/scripts/terminal_bench/diagnostics.py
new file mode 100644
index 000000000..de01c647d
--- /dev/null
+++ b/scripts/terminal_bench/diagnostics.py
@@ -0,0 +1,82 @@
+"""Bounded, text-free diagnosis through the normal product data-query CLI.
+
+The SQL projects counts and protocol categories inside the product, rather than
+exporting a transcript/SQLite file and trying to redact secrets afterwards.
+"""
+from __future__ import annotations
+
+import re
+
+_STOPS = ("stop", "length", "tool_use", "error", "aborted", "refusal", "safety", "pause")
+_COUNTS = (
+ "text_blocks", "text_chars", "thinking_blocks", "thinking_chars",
+ "tool_calls", "hosted_blocks", "input_tokens", "output_tokens",
+ "cache_read_tokens", "reasoning_tokens", "has_error_message",
+)
+
+
+def response_query(session_id: str) -> str:
+ # The CLI deliberately accepts SQL, not bound parameters. Only product IDs
+ # are interpolated, after a strict ASCII alphabet and length check.
+ if not re.fullmatch(r"[A-Za-z0-9_-]{1,128}", session_id):
+ raise ValueError("invalid diagnostic session id")
+ allowed = ",".join(repr(s) for s in _STOPS)
+ fields = [
+ "m.context_seq",
+ "CASE WHEN m.type IN ('assistant','result','error') THEN m.type ELSE 'other' END AS row_type",
+ f"CASE WHEN json_extract(m.content_json,'$.model.message.stop_reason') IN ({allowed}) "
+ "THEN json_extract(m.content_json,'$.model.message.stop_reason') ELSE 'unknown' END AS stop_reason",
+ "CASE WHEN json_extract(m.content_json,'$.model.message.error_message') IS NOT NULL "
+ "THEN 1 ELSE 0 END AS has_error_message",
+ ]
+ for block, prefix in (("text", "text"), ("thinking", "thinking")):
+ for expr, suffix in (("count(*)", "blocks"), ("coalesce(sum(length(json_extract(b.value,'$.text'))),0)", "chars")):
+ fields.append(
+ f"(SELECT {expr} FROM json_each(m.content_json,'$.model.message.content') b "
+ f"WHERE json_extract(b.value,'$.type')='{block}') AS {prefix}_{suffix}"
+ )
+ for block, name in (("tool_call", "tool_calls"), ("hosted_tool", "hosted_blocks")):
+ fields.append(
+ "(SELECT count(*) FROM json_each(m.content_json,'$.model.message.content') b "
+ f"WHERE json_extract(b.value,'$.type')='{block}') AS {name}"
+ )
+ for name in ("input_tokens", "output_tokens", "cache_read_tokens", "reasoning_tokens"):
+ path = f"$.model.message.usage.{name}"
+ fields.append(
+ f"CASE WHEN json_type(m.content_json,'{path}')='integer' "
+ f"THEN json_extract(m.content_json,'{path}') ELSE NULL END AS {name}"
+ )
+ return (
+ "WITH recent AS (SELECT context_seq,type,content_json FROM messages "
+ f"WHERE session_id='{session_id}' AND context_seq IS NOT NULL "
+ "ORDER BY context_seq DESC LIMIT 32) SELECT " + ", ".join(fields) +
+ " FROM recent m WHERE json_extract(m.content_json,'$.model.message.role')='assistant' "
+ "ORDER BY m.context_seq DESC LIMIT 9"
+ )
+
+
+def project_responses(payload: dict) -> dict:
+ """Second allowlist fence: even a malformed/new CLI shape cannot leak text."""
+ rows = payload.get("rows")
+ if payload.get("ok") is not True or not isinstance(rows, list):
+ raise ValueError("diagnostic query unavailable")
+ output = []
+ for row in rows[:8]:
+ if not isinstance(row, dict):
+ raise ValueError("invalid diagnostic row")
+ item = {"stop_reason": row.get("stop_reason") if row.get("stop_reason") in _STOPS else "unknown"}
+ item["row_type"] = row.get("row_type") if row.get("row_type") in {"assistant", "result", "error"} else "other"
+ for name in ("context_seq", *_COUNTS):
+ value = row.get(name)
+ item[name] = value if type(value) is int and 0 <= value <= 2**63 - 1 else None
+ output.append(item)
+ return {
+ "schema_version": 1,
+ "status": "available" if output else "no_response_recorded",
+ "source": "product_data_query",
+ "scope": "last_8_responses_within_last_32_context_messages_of_root_session",
+ "raw_text_exported": False,
+ "responses_truncated": len(rows) > 8,
+ "history_complete": False,
+ "responses": output,
+ }
diff --git a/scripts/terminal_bench/harbor_agent.py b/scripts/terminal_bench/harbor_agent.py
new file mode 100644
index 000000000..f47e3dc5c
--- /dev/null
+++ b/scripts/terminal_bench/harbor_agent.py
@@ -0,0 +1,277 @@
+"""Optional Harbor 0.24 installed-agent bridge to the full Avibe product.
+
+No extracted product code, no alternate loop, no dependency on Harbor in the
+product wheel. Install requirements-harbor.txt in the evaluation host only.
+"""
+from __future__ import annotations
+
+import asyncio
+from email.parser import Parser
+import hashlib
+from importlib.metadata import version
+import json
+from pathlib import Path, PurePosixPath
+import re
+import shlex
+import shutil
+import subprocess
+import tempfile
+import uuid
+import zipfile
+
+from harbor.agents.capabilities import AgentCapabilities
+from harbor.agents.installed.base import BaseInstalledAgent, with_prompt_template
+
+from .profile import Profile
+
+
+_WRAPPER_FILES = ("__init__.py", "profile.py", "control.py", "runner.py", "launch.py", "credential.py", "diagnostics.py")
+
+
+async def _join_transfer(awaitable):
+ """A cancelled upload must settle before remote credential cleanup."""
+ task = asyncio.create_task(awaitable)
+ cancelled = False
+ while True:
+ try:
+ result = await asyncio.shield(task)
+ break
+ except asyncio.CancelledError:
+ cancelled = True
+ if task.done():
+ result = await task
+ break
+ if cancelled:
+ raise asyncio.CancelledError
+ return result
+
+
+class FullAvibeAgent(BaseInstalledAgent):
+ capabilities = AgentCapabilities(atif=False, resume=False)
+
+ @staticmethod
+ def name():
+ return "avibe-full-runtime"
+
+ def __init__(self, *args, wheel_path: str, wheel_sha256: str, profile: dict, **kwargs):
+ if version("harbor") != "0.24.0":
+ raise ValueError("this integration requires Harbor 0.24.0")
+ self.profile = Profile.parse(profile)
+ if kwargs.get("model_name") != self.profile.model:
+ raise ValueError("explicit Harbor model_name must equal profile.model")
+ self.wheel = Path(wheel_path).expanduser().resolve()
+ if self.wheel.suffix != ".whl" or not self.wheel.is_file():
+ raise ValueError("a trusted local Avibe wheel file is required")
+ if not re.fullmatch(r"[0-9a-f]{64}", wheel_sha256):
+ raise ValueError("wheel_sha256 must be an explicit lowercase SHA256")
+ self.wheel_sha256 = wheel_sha256
+ self.nonce = uuid.uuid4().hex
+ self.root = PurePosixPath("/installed-agent") / ("avibe-full-" + self.nonce)
+ self.python = self.root / "venv/bin/python"
+ self.provenance = {}
+ self._credential_claimed = False
+ self.credential_path = self.root / "credential/provider.key"
+ super().__init__(*args, **kwargs)
+ if self.profile.credential_env in self.extra_env:
+ raise ValueError("supply the named credential in the host environment, not Harbor agent-env/config")
+ self.execution_dir = self.environment_logs_dir / ("full-runtime-" + self.nonce)
+
+ def _snapshot(self, temporary: Path):
+ # Hash the immutable COPY that is uploaded, not an earlier observation
+ # of a file somebody might replace during container setup.
+ wheel = temporary / self.wheel.name
+ shutil.copyfile(self.wheel, wheel)
+ with wheel.open("rb") as handle:
+ if hashlib.file_digest(handle, "sha256").hexdigest() != self.wheel_sha256:
+ raise ValueError("Avibe wheel SHA256 mismatch")
+ with zipfile.ZipFile(wheel) as archive:
+ metadata_names = [name for name in archive.namelist() if name.endswith(".dist-info/METADATA")]
+ if len(metadata_names) != 1:
+ raise ValueError("invalid wheel metadata")
+ metadata = Parser().parsestr(archive.read(metadata_names[0]).decode("utf-8"))
+ if metadata.get("Name", "").lower().replace("_", "-") != "avibe-os":
+ raise ValueError("the pinned wheel must be avibe-os")
+ source = Path(__file__).resolve().parent
+ repo = source.parents[1]
+ staged = temporary / "source/scripts/terminal_bench"
+ staged.mkdir(parents=True)
+ digest = hashlib.sha256()
+ for name in _WRAPPER_FILES:
+ relative = (source / name).relative_to(repo).as_posix()
+ check = subprocess.run(["git", "ls-files", "--error-unmatch", "--", relative],
+ cwd=repo, capture_output=True, timeout=10)
+ if check.returncode:
+ raise ValueError("wrapper upload requires tracked source: " + relative)
+ content = (source / name).read_bytes()
+ (staged / name).write_bytes(content)
+ digest.update(relative.encode() + b"\0" + content + b"\0")
+ (staged.parent / "__init__.py").write_text("", encoding="utf-8")
+ commit = subprocess.check_output(["git", "rev-parse", "HEAD"], cwd=repo, text=True, timeout=10).strip()
+ dirty = subprocess.check_output(["git", "status", "--porcelain"], cwd=repo, text=True, timeout=10)
+ self.provenance = {
+ "agent": self.name(), "harbor_version": version("harbor"), "atif": False,
+ "wheel_filename": wheel.name, "wheel_sha256": self.wheel_sha256,
+ "avibe_version": metadata["Version"], "wrapper_commit": commit,
+ "wrapper_source_sha256": digest.hexdigest(), "wrapper_source_dirty": bool(dirty.strip()),
+ }
+ return wheel, temporary / "source"
+
+ async def install(self, environment):
+ with tempfile.TemporaryDirectory(prefix="avibe-full-source-") as temp:
+ wheel, source = await asyncio.to_thread(self._snapshot, Path(temp))
+ # Wheel/provenance checks occur before any mutation of the task.
+ uid = await environment.exec(command="id -u")
+ if uid.return_code or (uid.stdout or "").strip() != "0":
+ raise RuntimeError("first full-runtime Harbor pilot requires a root task user")
+ await self.ensure_system_dependencies(environment, ("python3", "python_venv"))
+ await self.exec_as_root(environment, "mkdir -m 700 " + shlex.quote(str(self.root)))
+ await environment.upload_file(wheel, str(self.root / self.wheel.name))
+ await environment.upload_dir(source, str(self.root / "source"))
+ # Installed distributions, not editable source. A .pth installs ONLY
+ # this small wrapper for -I imports; task PYTHONPATH remains unmodified.
+ pth = "import pathlib,sysconfig; pathlib.Path(sysconfig.get_path('purelib'),'avibe_benchmark.pth').write_text(" + repr(str(self.root / "source") + "\n") + ")"
+ command = (
+ "set -eu; python3 -c 'import sys; assert sys.version_info >= (3, 10)' && "
+ f"python3 -m venv {shlex.quote(str(self.root / 'venv'))} && "
+ f"{shlex.quote(str(self.python))} -m pip install --disable-pip-version-check --no-input "
+ f"{shlex.quote(str(self.root / self.wheel.name))} && "
+ f"{shlex.quote(str(self.python))} -c {shlex.quote(pth)}"
+ )
+ await self.exec_as_root(environment, command, timeout_sec=int(self.profile.setup_seconds))
+ # Add ONLY a private vibe shim to PATH during execution, not the venv's
+ # Python binaries: task Python/conda/java resolution must stay unchanged.
+ await self.exec_as_root(environment,
+ "mkdir -p " + shlex.quote(str(self.root / "bin")) + "; ln -s "
+ + shlex.quote(str(self.root / "venv/bin/vibe")) + " " + shlex.quote(str(self.root / "bin/vibe")))
+ async def _stage_credential(self, environment, key):
+ result = await environment.exec(command="mkdir -m 700 " + shlex.quote(str(self.credential_path.parent)), user="root")
+ if result.return_code:
+ raise RuntimeError("could not exclusively claim credential directory")
+ self._credential_claimed = True
+ with tempfile.TemporaryDirectory(prefix="avibe-full-key-") as temp:
+ path = Path(temp) / "provider.key"
+ # The directory is 0700 already; explicitly make the file 0600
+ # before writing and use upload_file rather than any shell literal.
+ import os
+ fd = os.open(path, os.O_CREAT | os.O_EXCL | os.O_WRONLY, 0o600)
+ with os.fdopen(fd, "w", encoding="utf-8") as handle:
+ handle.write(key)
+ await _join_transfer(environment.upload_file(path, str(self.credential_path)))
+ checked = await environment.exec(command=shlex.join([
+ str(self.python), "-I", "-m", "scripts.terminal_bench.credential", "prepare", str(self.credential_path),
+ ]), user="root")
+ if checked.return_code:
+ raise RuntimeError("credential metadata refused")
+ metadata = json.loads(checked.stdout)
+ if metadata.get("regular") is not True or metadata.get("mode") != "0600" or metadata.get("directory_mode") != "0700":
+ raise RuntimeError("credential metadata refused")
+
+ async def _clear_credential(self, environment):
+ if not self._credential_claimed:
+ return
+ result = await environment.exec(command=shlex.join([
+ str(self.python), "-I", "-m", "scripts.terminal_bench.credential", "clean", str(self.credential_path),
+ ]), user="root", timeout_sec=15)
+ if result.return_code or json.loads(result.stdout).get("removed") is not True:
+ raise RuntimeError("credential cleanup unconfirmed; destroy task container before grading")
+ self._credential_claimed = False
+
+ async def _cleanup(self, environment):
+ command = shlex.join([
+ str(self.python), "-I", "-m", "scripts.terminal_bench.launch", "stop",
+ "--root", str(self.root / "launch"), "--nonce", self.nonce,
+ "--receipt", str(self.execution_dir / "execution.json"),
+ "--timeout", str(self.profile.cleanup_seconds + 60),
+ ])
+ try:
+ result = await environment.exec(command=command, user="root", timeout_sec=int(self.profile.cleanup_seconds + 75))
+ finally:
+ await self._clear_credential(environment)
+ if result.return_code != 0:
+ raise RuntimeError("full-runtime cleanup unconfirmed; destroy task container before grading")
+ try:
+ receipt = json.loads(result.stdout)
+ if receipt.get("quiescent") is not True:
+ raise ValueError
+ except (ValueError, TypeError):
+ raise RuntimeError("full-runtime cleanup receipt invalid; destroy task container before grading") from None
+ return receipt
+
+ @with_prompt_template
+ async def run(self, instruction, environment, context):
+ if not self.provenance:
+ raise RuntimeError("verified installation is required before execution")
+ cwd_result = await environment.exec(command="pwd -P")
+ workdir = (cwd_result.stdout or "").strip()
+ if cwd_result.return_code or not workdir.startswith("/") or "\n" in workdir:
+ raise RuntimeError("task cwd could not be determined")
+ for private in (self.root, self.execution_dir):
+ task = PurePosixPath(workdir)
+ if task == private or task in private.parents or private in task.parents:
+ raise RuntimeError("task and framework directories must be disjoint")
+ key = self._get_env(self.profile.credential_env)
+ if not key:
+ raise ValueError("the named benchmark credential is missing")
+ with tempfile.TemporaryDirectory(prefix="avibe-full-config-") as temp:
+ for name, text in (("profile.json", json.dumps(self.profile.payload())), ("instruction.txt", instruction)):
+ path = Path(temp) / name
+ path.write_text(text, encoding="utf-8")
+ await environment.upload_file(path, str(self.root / name))
+ command = shlex.join([
+ str(self.python), "-I", "-m", "scripts.terminal_bench.launch", "run",
+ "--root", str(self.root / "launch"), "--nonce", self.nonce, "--",
+ "--workdir", workdir, "--state", str(self.root / "state"),
+ "--output", str(self.execution_dir), "--profile", str(self.root / "profile.json"),
+ "--instruction", str(self.root / "instruction.txt"), "--sandbox-environment",
+ "--credential-file", str(self.credential_path),
+ ])
+ command = "export PATH=" + shlex.quote(str(self.root / "bin")) + ":\"$PATH\"; exec " + command
+ try:
+ await self._stage_credential(environment, key)
+ # Model setup is part of actual full-product execution, hence the
+ # outer exec allowance includes it. Harbor's task timeout remains
+ # authoritative and may cancel this earlier; cleanup is mandatory.
+ allowance = int(self.profile.setup_seconds * 4 + self.profile.wall_seconds
+ + self.profile.cleanup_seconds * 2 + 90)
+ result = await environment.exec(command=command, cwd=str(self.root), timeout_sec=allowance)
+ if result.return_code:
+ raise RuntimeError("full-runtime wrapper infrastructure failure; inspect its structured report")
+ finally:
+ # Shield cleanup from the cancellation that interrupted docker exec,
+ # but JOIN it: verification must never race an orphan writer.
+ cleanup = asyncio.create_task(self._cleanup(environment))
+ while True:
+ try:
+ cleanup_receipt = await asyncio.shield(cleanup)
+ break
+ except asyncio.CancelledError:
+ if cleanup.done():
+ cleanup_receipt = await cleanup
+ break
+ if cleanup_receipt.get("started") is not False:
+ await self._collect(environment, context)
+
+ async def _collect(self, environment, context):
+ local = self.logs_dir / "full-runtime"
+ local.mkdir(parents=True, exist_ok=True)
+ report_path = local / "execution.json"
+ await environment.download_file(str(self.execution_dir / "execution.json"), report_path)
+ report = json.loads(report_path.read_text(encoding="utf-8"))
+ if report.get("cleanup", {}).get("owned_processes_stopped") is not True:
+ raise RuntimeError("full-runtime report did not confirm shutdown")
+ totals = (report.get("usage") or {}).get("totals", {})
+ # A missing token report is unknown, not zero-cost inference.
+ if totals.get("token_reports", 0) > 0:
+ context.n_input_tokens = totals.get("input_tokens")
+ context.n_cache_tokens = totals.get("cached_input_tokens")
+ context.n_output_tokens = totals.get("output_tokens")
+ context.cost_usd = None
+ context.metadata = {**self.provenance, "execution_outcome": report.get("outcome"),
+ "accounting_complete": report.get("accounting_complete", False),
+ "run_id": report.get("run_id"), "session_id": report.get("session_id"),
+ "verifier_reward": None}
+ (local / "adapter-provenance.json").write_text(json.dumps(context.metadata, indent=2), encoding="utf-8")
+ if (report.get("outcome") not in {"succeeded", "failed", "canceled", "cancelled", "time_limit"}
+ or not isinstance(report.get("run_id"), str) or not report["run_id"]
+ or not isinstance(report.get("session_id"), str) or not report["session_id"]):
+ raise RuntimeError("full-runtime execution was not admitted; not an Agent failure score")
diff --git a/scripts/terminal_bench/harbor_job.py b/scripts/terminal_bench/harbor_job.py
new file mode 100644
index 000000000..38f667f0e
--- /dev/null
+++ b/scripts/terminal_bench/harbor_job.py
@@ -0,0 +1,58 @@
+"""Pinned Harbor driver: journal exact Compose ownership before environment start."""
+from __future__ import annotations
+
+import argparse
+import asyncio
+from importlib.metadata import version
+from pathlib import Path
+from types import SimpleNamespace
+
+from .remote_job import load, save
+
+
+async def execute(root):
+ if version("harbor") != "0.24.0":
+ raise ValueError("this supervisor requires Harbor 0.24.0")
+ from harbor.environments.docker.docker import _sanitize_docker_compose_project_name
+ from harbor.job import Job
+ from harbor.models.job.config import JobConfig
+ from harbor.trial.hooks import TrialEvent
+ from harbor.trial.trial import Trial
+
+ config = JobConfig.model_validate(load(root / "config.json"))
+ config.retry.max_retries = 0
+ job = await Job.create(config)
+
+ async def before_environment(event):
+ # START is awaited before _prepare(), so this journal also covers the
+ # network-only window. Use the pinned implementation's exact sanitizer
+ # and verifier truncation/hash, not a prefix match on arbitrary labels.
+ verifier = Trial._separate_verifier_session_id(SimpleNamespace(config=event.config), "trial")
+ save(root / "ownership.json", {
+ "schema_version": 1, "job_id": str(event.config.job_id),
+ "trial_name": event.trial_name, "trial_id": str(event.trial_id),
+ "projects": [_sanitize_docker_compose_project_name(f"{event.trial_name}__env"),
+ _sanitize_docker_compose_project_name(verifier)],
+ })
+
+ job.add_hook(TrialEvent.START, before_environment)
+ await job.run()
+
+
+def main():
+ parser = argparse.ArgumentParser(description=__doc__)
+ parser.add_argument("--root", type=Path, required=True)
+ args = parser.parse_args()
+ try:
+ from harbor.utils.async_cleanup import with_async_cleanup
+ asyncio.run(with_async_cleanup(execute(args.root)))
+ return 0
+ except Exception as exc:
+ # No Rich local-variable dump and no upstream exception text. The host
+ # log captures Harbor's own output through the supervisor's redactor.
+ print(type(exc).__name__)
+ return 1
+
+
+if __name__ == "__main__":
+ raise SystemExit(main())
diff --git a/scripts/terminal_bench/launch.py b/scripts/terminal_bench/launch.py
new file mode 100644
index 000000000..16ad34470
--- /dev/null
+++ b/scripts/terminal_bench/launch.py
@@ -0,0 +1,132 @@
+"""In-container wrapper process ownership for Harbor exec cancellation.
+
+A Docker exec client's death does not terminate its container process. This
+launcher records the exact owned wrapper identity before releasing its start
+lock. The stop path fences late starts, signals only that identity, and requires
+the wrapper's completed inner cleanup receipt. Otherwise grading is refused and
+the outer owner must destroy the sandbox; a guessed PID is never signalled.
+"""
+from __future__ import annotations
+
+import argparse
+from contextlib import contextmanager
+import json
+import os
+from pathlib import Path
+import subprocess
+import sys
+import time
+
+import psutil
+
+
+@contextmanager
+def locked(root: Path):
+ import fcntl # This first Harbor integration supports POSIX task containers.
+ root.mkdir(parents=True, exist_ok=True)
+ with (root / "launch.lock").open("a") as lock:
+ fcntl.flock(lock.fileno(), fcntl.LOCK_EX)
+ try:
+ yield
+ finally:
+ fcntl.flock(lock.fileno(), fcntl.LOCK_UN)
+
+
+def run(root: Path, nonce: str, args: list[str]) -> int:
+ with locked(root):
+ if (root / "cancelled").exists():
+ return 125
+ if (root / "wrapper-process.json").exists():
+ raise RuntimeError("wrapper_already_started")
+ # Persist the launch attempt before spawn. If identity persistence fails
+ # (e.g. disk full), stop must NOT mistake an absent pidfile for no child.
+ (root / "starting").touch(exist_ok=False)
+ try:
+ process = subprocess.Popen(
+ [sys.executable, "-I", "-m", "scripts.terminal_bench.runner", *args],
+ env={**os.environ, "AVIBE_BENCHMARK_LAUNCH_ID": nonce},
+ start_new_session=True,
+ )
+ except BaseException:
+ (root / "starting").unlink()
+ raise
+ try:
+ identity = {"pid": process.pid, "created": psutil.Process(process.pid).create_time(), "nonce": nonce}
+ temporary = root / "wrapper-process.tmp"
+ temporary.write_text(json.dumps(identity), encoding="utf-8")
+ temporary.replace(root / "wrapper-process.json")
+ except BaseException:
+ # Preserve `starting`: the outer sandbox remains responsible if
+ # we cannot prove inner shutdown after losing the identity write.
+ if process.poll() is None:
+ process.terminate()
+ raise
+ return process.wait()
+
+
+def stop(root: Path, nonce: str, receipt: Path, timeout: float) -> dict:
+ with locked(root):
+ (root / "cancelled").touch(exist_ok=True)
+ path = root / "wrapper-process.json"
+ if not path.exists():
+ if (root / "starting").exists():
+ raise RuntimeError("wrapper_identity_not_recorded_destroy_sandbox")
+ return {"quiescent": True, "started": False}
+ record = json.loads(path.read_text(encoding="utf-8"))
+ if record.get("nonce") != nonce:
+ raise RuntimeError("wrapper_identity_mismatch")
+
+ def owned_live():
+ try:
+ process = psutil.Process(record["pid"])
+ if process.status() == psutil.STATUS_ZOMBIE:
+ return None
+ if (process.create_time() != record["created"]
+ and process.environ().get("AVIBE_BENCHMARK_LAUNCH_ID") != nonce):
+ raise RuntimeError("wrapper_identity_mismatch")
+ return process
+ except psutil.NoSuchProcess:
+ return None
+
+ process = owned_live()
+ if process is not None:
+ process.terminate()
+ deadline = time.monotonic() + timeout
+ while owned_live() is not None and time.monotonic() < deadline:
+ time.sleep(.1)
+ process = owned_live()
+ if process is not None:
+ process.kill()
+ raise RuntimeError("wrapper_cleanup_unconfirmed_destroy_sandbox")
+ try:
+ result = json.loads(receipt.read_text(encoding="utf-8"))
+ except (OSError, ValueError):
+ raise RuntimeError("wrapper_receipt_missing_destroy_sandbox") from None
+ if result.get("cleanup", {}).get("owned_processes_stopped") is not True:
+ raise RuntimeError("wrapper_cleanup_unconfirmed_destroy_sandbox")
+ return {"quiescent": True, "started": True}
+
+
+def main(argv=None):
+ parser = argparse.ArgumentParser(description=__doc__)
+ parser.add_argument("operation", choices=["run", "stop"])
+ parser.add_argument("--root", type=Path, required=True)
+ parser.add_argument("--nonce", required=True)
+ parser.add_argument("--receipt", type=Path)
+ parser.add_argument("--timeout", type=float, default=90)
+ args, wrapper_args = parser.parse_known_args(argv)
+ try:
+ if args.operation == "run":
+ return run(args.root, args.nonce, wrapper_args[1:] if wrapper_args[:1] == ["--"] else wrapper_args)
+ if args.receipt is None or args.timeout <= 0:
+ parser.error("stop requires --receipt and positive --timeout")
+ print(json.dumps(stop(args.root, args.nonce, args.receipt, args.timeout)))
+ return 0
+ except Exception as exc:
+ # Only our closed vocabulary is printed, never arbitrary process output.
+ print(json.dumps({"quiescent": False, "error_type": type(exc).__name__}))
+ return 2
+
+
+if __name__ == "__main__":
+ raise SystemExit(main())
diff --git a/scripts/terminal_bench/profile.py b/scripts/terminal_bench/profile.py
new file mode 100644
index 000000000..7ae52f44c
--- /dev/null
+++ b/scripts/terminal_bench/profile.py
@@ -0,0 +1,82 @@
+"""Explicit, credential-free full-runtime trial configuration."""
+from __future__ import annotations
+
+from dataclasses import asdict, dataclass, fields
+import os
+import math
+from pathlib import Path
+import re
+from urllib.parse import urlsplit
+
+
+@dataclass(frozen=True)
+class Profile:
+ model: str
+ base_url: str
+ credential_env: str = "DEEPSEEK_API_KEY"
+ protocol: str = "openai_chat"
+ reasoning_effort: str = "high"
+ context_window: int = 128000
+ max_output_tokens: int = 32000
+ wall_seconds: float = 1200
+ setup_seconds: float = 600
+ cleanup_seconds: float = 30
+ poll_seconds: float = 1
+ offline: bool = False
+
+ def __post_init__(self):
+ if type(self.offline) is not bool:
+ raise ValueError("offline must be boolean")
+ url = urlsplit(self.base_url)
+ if url.scheme not in {"http", "https"} or not url.hostname:
+ raise ValueError("base_url must be an HTTP(S) API root")
+ if url.username or url.password or url.query or url.fragment:
+ raise ValueError("base_url cannot contain credentials, query, or fragment")
+ if url.scheme == "http" and url.hostname not in {"localhost", "127.0.0.1", "::1"}:
+ raise ValueError("plaintext HTTP is restricted to loopback fixtures")
+ if self.protocol not in {"openai_chat", "openai_responses", "anthropic"}:
+ raise ValueError("unsupported protocol")
+ if not self.model or not self.reasoning_effort:
+ raise ValueError("explicit model and effort required")
+ if not re.fullmatch(r"[A-Z][A-Z0-9_]*", self.credential_env):
+ raise ValueError("credential_env must be an environment variable name")
+ for value in (self.context_window, self.max_output_tokens):
+ if type(value) is not int or value <= 0:
+ raise ValueError("token limits must be positive integers")
+ if self.max_output_tokens > self.context_window // 4:
+ raise ValueError("output limit exceeds Vibey's quarter-window policy")
+ for value in (self.wall_seconds, self.setup_seconds, self.cleanup_seconds, self.poll_seconds):
+ if isinstance(value, bool) or not math.isfinite(value) or value <= 0:
+ raise ValueError("time limits must be finite and positive")
+
+ def payload(self):
+ return asdict(self)
+
+ @classmethod
+ def parse(cls, payload):
+ if not isinstance(payload, dict) or set(payload) - {f.name for f in fields(cls)}:
+ raise ValueError("unknown profile fields")
+ return cls(**payload)
+
+
+def validate_layout(workdir: Path, state: Path, output: Path) -> None:
+ paths = [p.resolve() for p in (workdir, state, output)]
+ if not paths[0].is_dir():
+ raise ValueError("task workdir must exist")
+ if any(p == Path("/") for p in paths):
+ raise ValueError("filesystem root is not a trial directory")
+ for i, left in enumerate(paths):
+ for right in paths[i + 1:]:
+ if left == right or left in right.parents or right in left.parents:
+ raise ValueError("task, runtime state, and exported artifacts must be disjoint")
+ # Reject production roots even if a caller forgets the container-only gate.
+ home = Path.home().resolve()
+ for path in paths:
+ configured = os.environ.get("AVIBE_HOME")
+ forbidden_roots = [home / ".avibe", home / ".vibe_remote"]
+ if configured:
+ forbidden_roots.append(Path(configured).expanduser())
+ for forbidden in forbidden_roots:
+ forbidden = forbidden.resolve()
+ if path == forbidden or forbidden in path.parents or path in forbidden.parents:
+ raise ValueError("refusing the current/default Avibe home or its ancestors")
diff --git a/scripts/terminal_bench/remote_job.py b/scripts/terminal_bench/remote_job.py
new file mode 100644
index 000000000..b4d5361f1
--- /dev/null
+++ b/scripts/terminal_bench/remote_job.py
@@ -0,0 +1,488 @@
+"""Single-job Harbor lifetime on a dedicated Linux evaluation host.
+
+systemd owns execution independently of SSH. Only its post-stop phase finalizes
+artifacts and releases ownership; Avibe Harness observes the durable receipt.
+"""
+from __future__ import annotations
+
+import argparse
+import fcntl
+import hashlib
+import json
+import os
+from pathlib import Path
+import re
+import selectors
+import signal
+import subprocess
+import sys
+import threading
+import time
+import uuid
+
+try:
+ import tomllib
+except ModuleNotFoundError: # Product tests still support Python 3.10.
+ import tomli as tomllib
+
+from .credential import consume, metadata
+
+HOST_RESERVATION = Path("/run/lock/avibe-benchmark.active.json")
+FINALIZE_SECONDS = 45
+
+
+def load(path):
+ return json.loads(Path(path).read_text(encoding="utf-8"))
+
+
+def evidence(path):
+ try:
+ value = load(path)
+ return value if isinstance(value, dict) else None
+ except (OSError, ValueError):
+ return None
+
+
+def save(path, value):
+ path = Path(path)
+ temporary = path.with_name(path.name + ".new")
+ with temporary.open("w", encoding="utf-8") as file:
+ os.fchmod(file.fileno(), 0o600)
+ json.dump(value, file, indent=2)
+ file.write("\n")
+ temporary.replace(path)
+
+
+def command(argv, *, timeout=10):
+ return subprocess.run(argv, capture_output=True, text=True, timeout=timeout, check=True).stdout
+
+
+def absolute(path, *, exists=True):
+ p = Path(path)
+ if any(char in str(p) for char in "%$\n\r\x00"):
+ raise ValueError("systemd specifiers and variable expansions are not supported in paths")
+ if not p.is_absolute():
+ raise ValueError("absolute execution paths required; relocate explicitly before submission")
+ resolved = p.resolve(strict=exists)
+ if any(char in str(resolved) for char in "%$\n\r\x00"):
+ raise ValueError("resolved path contains a systemd expansion")
+ return resolved
+
+
+def validate(config_path, credential_env=None):
+ """No job creation, Docker mutation, credential consumption or model call."""
+ config = load(config_path)
+ if config.get("n_attempts", 1) != 1 or config.get("n_concurrent_trials", 1) != 1:
+ raise ValueError("one attempt and concurrency one required")
+ if config.get("retry", {}).get("max_retries", 0) != 0 or any(config.get(k) for k in ("source_jobs", "datasets", "user_agent")):
+ raise ValueError("retry, dataset, regrade and simulated-user jobs unsupported")
+ if len(config.get("agents", [])) != 1 or len(config.get("tasks", [])) != 1:
+ raise ValueError("exactly one agent and local task required")
+ name = config.get("job_name", "")
+ if not re.fullmatch(r"[A-Za-z0-9_-]{1,100}", name):
+ raise ValueError("explicit safe job name required")
+ env = config.get("environment", {})
+ if env.get("type", "docker") != "docker" or env.get("delete", True) is not True or env.get("import_path"):
+ raise ValueError("isolated Docker with delete=true required")
+ task = absolute(config["tasks"][0]["path"])
+ task_config = tomllib.loads((task / "task.toml").read_text())
+ if task_config.get("steps"):
+ raise ValueError("single-step tasks only; verifier ownership is pinned to Harbor 0.24")
+ for overlay in env.get("extra_docker_compose", []):
+ absolute(overlay)
+ agent = config["agents"][0]
+ seconds = agent.get("override_timeout_sec")
+ if type(seconds) not in (int, float) or not 0 < seconds <= 1800:
+ raise ValueError("explicit agent limit no greater than 1800 seconds required")
+ kwargs = agent.get("kwargs", {})
+ if agent.get("import_path") == "scripts.terminal_bench.harbor_agent:FullAvibeAgent":
+ from .profile import Profile
+ profile = Profile.parse(kwargs["profile"])
+ if profile.model != agent.get("model_name"):
+ raise ValueError("profile model mismatch")
+ if credential_env is not None and credential_env != profile.credential_env:
+ raise ValueError("credential variable must match the full-runtime profile")
+ credential_env = profile.credential_env
+ wheel = absolute(kwargs["wheel_path"])
+ with wheel.open("rb") as file:
+ digest = hashlib.sha256()
+ for chunk in iter(lambda: file.read(1024 * 1024), b""):
+ digest.update(chunk)
+ if digest.hexdigest() != kwargs["wheel_sha256"]:
+ raise ValueError("wheel hash mismatch")
+ credential_env = credential_env or "BENCHMARK_API_KEY"
+ if not re.fullmatch(r"[A-Z][A-Z0-9_]{0,100}", credential_env):
+ raise ValueError("invalid credential variable")
+ jobs = absolute(config["jobs_dir"], exists=False)
+ if (jobs / name).exists():
+ raise ValueError("fresh job destination required")
+ return config, jobs / name, task, credential_env
+
+
+def release(root):
+ if HOST_RESERVATION.is_file() and HOST_RESERVATION.read_text() == str(Path(root).resolve()):
+ HOST_RESERVATION.unlink()
+
+
+def prepare(config_path, root, wall_seconds, credential_env=None):
+ config, job_dir, task, credential_env = validate(config_path, credential_env)
+ root = absolute(root, exists=False)
+ source_root = absolute(Path(__file__).resolve().parents[2])
+ absolute(sys.executable) # Reject manager expansion in the interpreter path too.
+ if not 0 < wall_seconds <= 7200:
+ raise ValueError("invalid hard wall cap")
+ if any(root == p or root in p.parents or p in root.parents for p in (task, job_dir)):
+ raise ValueError("supervisor state must be disjoint from task and job")
+ root.parent.mkdir(parents=True, exist_ok=True)
+ HOST_RESERVATION.parent.mkdir(parents=True, exist_ok=True)
+ with HOST_RESERVATION.open("x", encoding="utf-8") as file:
+ os.fchmod(file.fileno(), 0o600)
+ file.write(str(root))
+ try:
+ root.mkdir(mode=0o700)
+ job_dir.parent.mkdir(parents=True, exist_ok=True)
+ # Keep this per-destination claim permanently, including failed starts.
+ with (job_dir.parent / ("." + job_dir.name + ".claimed")).open("x", encoding="utf-8") as file:
+ file.write(str(root))
+ manifest = {"schema_version": 1, "unit": "avibe-benchmark-" + uuid.uuid4().hex,
+ "job_dir": str(job_dir), "task": str(task), "wall_seconds": wall_seconds,
+ "credential_env": credential_env, "created_at": time.time(),
+ "config_sha256": hashlib.sha256(Path(config_path).read_bytes()).hexdigest(),
+ "source_root": str(source_root)}
+ save(root / "config.json", config)
+ save(root / "manifest.json", manifest)
+ save(root / "lifecycle.json", {"state": "prepared"})
+ return manifest
+ except Exception:
+ release(root)
+ raise
+
+
+def terminal_evidence(manifest, ownership):
+ """A finished job alone is insufficient: the one exact trial must be final."""
+ job_dir = Path(manifest["job_dir"])
+ result = evidence(job_dir / "result.json")
+ name = ownership.get("trial_name")
+ if not isinstance(name, str) or not re.fullmatch(r"[A-Za-z0-9_-]{1,180}", name):
+ return False
+ trial = evidence(job_dir / name / "result.json")
+ if not result or not trial or not result.get("finished_at") or not trial.get("finished_at"):
+ return False
+ stats = result.get("stats") or {}
+ if not isinstance(stats, dict):
+ return False
+ return (result.get("n_total_trials") == 1 and stats.get("n_completed_trials") == 1
+ and stats.get("n_running_trials") == 0 and stats.get("n_pending_trials") == 0
+ and result.get("id") == ownership.get("job_id")
+ and trial.get("id") == ownership.get("trial_id") and trial.get("trial_name") == name)
+
+
+def docker_state(projects, run=command):
+ ids = run(["docker", "ps", "-aq"]).split()
+ containers = json.loads(run(["docker", "inspect", *ids])) if ids else []
+ ids = run(["docker", "network", "ls", "-q"]).split()
+ networks = json.loads(run(["docker", "network", "inspect", *ids])) if ids else []
+ owned = [c for c in containers if (c.get("Config", {}).get("Labels") or {}).get("com.docker.compose.project") in projects]
+ nets = [n for n in networks if (n.get("Labels") or {}).get("com.docker.compose.project") in projects]
+ return owned, nets
+
+
+def archive_artifact(container_id, source, destination, timeout):
+ """Keep untrusted Docker tar bytes opaque; never extract onto the host.
+
+ Separate exclusive files prevent overlapping artifact paths or symlinks in
+ one artifact from redirecting a later host write. Bound both bytes and time.
+ """
+ deadline = time.monotonic() + timeout
+ with destination.open("xb") as file:
+ os.fchmod(file.fileno(), 0o600)
+ child = subprocess.Popen(["docker", "cp", f"{container_id}:{source}", "-"],
+ stdout=subprocess.PIPE, stderr=subprocess.DEVNULL)
+ try:
+ with selectors.DefaultSelector() as selector:
+ selector.register(child.stdout, selectors.EVENT_READ)
+ size = 0
+ while True:
+ left = deadline - time.monotonic()
+ if left <= 0 or not selector.select(left):
+ raise subprocess.TimeoutExpired("artifact archive", timeout)
+ chunk = os.read(child.stdout.fileno(), 65536)
+ if not chunk:
+ break
+ size += len(chunk)
+ if size > 32 * 1024 * 1024:
+ raise OSError("artifact archive exceeds byte budget")
+ file.write(chunk)
+ child.wait(timeout=max(.1, deadline - time.monotonic()))
+ if child.returncode:
+ raise subprocess.CalledProcessError(child.returncode, "artifact archive")
+ finally:
+ if child.poll() is None:
+ child.kill()
+ child.wait(timeout=2)
+ child.stdout.close()
+
+
+def copy_artifacts(root, manifest, containers, deadline):
+ records = []
+ try:
+ declared = tomllib.loads((Path(manifest["task"]) / "task.toml").read_text()).get("artifacts", [])
+ except (OSError, ValueError):
+ return [{"status": "metadata_unavailable"}]
+ for container in containers:
+ for index, path in enumerate(declared[:32]):
+ if time.monotonic() >= deadline:
+ return [*records, {"status": "recovery_budget_exhausted"}]
+ if not isinstance(path, str):
+ continue
+ source = Path(path)
+ if not source.is_absolute() or ".." in source.parts or len(source.parts) < 3:
+ continue
+ if source.parts[1] in {"installed-agent", "proc", "sys", "dev", "etc", "root", "run", "logs"}:
+ continue
+ destination = root / "recovered-artifacts" / container["Id"][:12] / f"{index:02d}.tar"
+ destination.parent.mkdir(parents=True, exist_ok=True)
+ try:
+ archive_artifact(container["Id"], source, destination, min(3, max(.1, deadline - time.monotonic())))
+ status = "archived"
+ except (subprocess.SubprocessError, OSError):
+ status = "unavailable"
+ records.append({"path": path, "status": status, "archive": str(destination.relative_to(root)),
+ "extracted_on_host": False})
+ return records
+
+
+def finalize(root, *, never_started=False):
+ # Publishing a cleanup receipt while the unit can still launch work would
+ # release the slot early. Only ExecStopPost or a proven absent unit may do so.
+ if not never_started and "SERVICE_RESULT" not in os.environ:
+ raise ValueError("finalization requires the manager post-stop fence")
+ root = Path(root)
+ with (root / "finalize.lock").open("a") as lock:
+ fcntl.flock(lock, fcntl.LOCK_EX)
+ prior = evidence(root / "receipt.json")
+ if prior and prior.get("finished_at"):
+ return prior
+ manifest = load(root / "manifest.json")
+ lifecycle = evidence(root / "lifecycle.json") or {}
+ ownership = evidence(root / "ownership.json") or {}
+ finished = terminal_evidence(manifest, ownership)
+ manager = os.environ.get("SERVICE_RESULT", "not-started")
+ reason = lifecycle.get("state", "supervisor_lost")
+ if never_started:
+ reason = "submit_failed"
+ elif manager == "timeout":
+ reason = "time_limit"
+ elif reason not in {"completed", "harbor_error", "time_limit", "interrupted", "launch_failed"}:
+ reason = "supervisor_lost"
+ if reason == "completed" and not finished:
+ reason = "missing_terminal_result"
+ receipt = {"schema_version": 1, "state": reason, "harbor_exit_code": lifecycle.get("exit_code"),
+ "harbor_finished": finished, "harbor_result_path": str(Path(manifest["job_dir"]) / "result.json"),
+ "wall_seconds": manifest["wall_seconds"], "finished_at": None,
+ "cleanup": {"confirmed": False, "unit_stopped": True}, "artifacts": []}
+ save(root / "receipt.json", receipt) # Even a killed finalizer leaves an honest incomplete receipt.
+ deadline = time.monotonic() + FINALIZE_SECONDS
+ def bounded(argv):
+ left = deadline - time.monotonic()
+ if left <= 0:
+ raise TimeoutError("cleanup budget exhausted")
+ return command(argv, timeout=min(5, left))
+ projects = ownership.get("projects", [])
+ errors = []
+ try:
+ # This journal is written by the pre-environment Harbor START hook,
+ # covering networks created before any container/log mount exists.
+ if not projects and (Path(manifest["job_dir"]).exists() or not never_started):
+ # Without START there should be no Docker mutation; fail closed
+ # if a corrupted/missing ownership journal cannot establish it.
+ if not lifecycle.get("no_trial_started"):
+ raise ValueError("ownership unavailable")
+ owned, networks = docker_state(projects, bounded)
+ if not finished:
+ try:
+ receipt["artifacts"] = copy_artifacts(root, manifest, owned, min(deadline - 15, time.monotonic() + 8))
+ except (OSError, ValueError, TypeError):
+ receipt["artifacts"] = [{"status": "recovery_unavailable"}]
+ for argv in (["docker", "rm", "-f", c["Id"]] for c in owned):
+ try:
+ bounded(argv)
+ except (subprocess.SubprocessError, OSError):
+ errors.append("container_removal_failed")
+ for net in networks:
+ try:
+ bounded(["docker", "network", "rm", net["Id"]])
+ except (subprocess.SubprocessError, OSError):
+ errors.append("network_removal_failed")
+ remaining, nets = docker_state(projects, bounded)
+ receipt["cleanup"].update(confirmed=not (remaining or nets or errors),
+ removed_container_ids=[c["Id"] for c in owned], owned_projects=projects,
+ remaining_ids=[c["Id"] for c in remaining], remaining_network_ids=[n["Id"] for n in nets])
+ except (subprocess.SubprocessError, OSError, ValueError, TypeError):
+ errors.append("owned_cleanup_unconfirmed")
+ finally:
+ (root / "credential" / "provider.key").unlink(missing_ok=True)
+ if errors:
+ receipt["cleanup"]["errors"] = errors
+ receipt["finished_at"] = time.time()
+ save(root / "receipt.json", receipt)
+ if receipt["cleanup"]["confirmed"]:
+ release(root)
+ return receipt
+
+
+def redact_stream(pipe, output, secret):
+ """Keep a look-behind so split stdout chunks cannot expose a credential."""
+ pending, written = "", 0
+ with output.open("w", encoding="utf-8") as file:
+ os.chmod(output, 0o600)
+ while True:
+ chunk = pipe.read(4096)
+ pending += chunk
+ end = len(pending) if not chunk else max(0, len(pending) - len(secret))
+ if secret:
+ start = pending.find(secret, max(0, end - len(secret)))
+ if start >= 0 and start < end < start + len(secret):
+ end = start
+ text = pending[:end].replace(secret, "[REDACTED]") if secret else pending[:end]
+ if written < 16 * 1024 * 1024:
+ file.write(text[:16 * 1024 * 1024 - written])
+ file.flush()
+ written += len(text)
+ pending = pending[end:]
+ if not chunk:
+ break
+
+
+def worker(root):
+ root = Path(root)
+ manifest = load(root / "manifest.json")
+ stopped = threading.Event()
+ previous_term = signal.signal(signal.SIGTERM, lambda *_: stopped.set())
+ previous_int = signal.signal(signal.SIGINT, lambda *_: stopped.set())
+ child = thread = None
+ reason, code = "launch_failed", None
+ try:
+ key = consume(root / "credential" / "provider.key")
+ env = {k: v for k, v in os.environ.items() if k in {"PATH", "LANG", "LC_ALL", "SSL_CERT_FILE"}}
+ env.update(HOME=str(root / "home"), XDG_CONFIG_HOME=str(root / "home/.config"),
+ XDG_CACHE_HOME=str(root / "home/.cache"), PYTHONPATH=manifest["source_root"], HARBOR_TELEMETRY="0")
+ env[manifest["credential_env"]] = key
+ argv = [sys.executable, "-m", "scripts.terminal_bench.harbor_job", "--root", str(root)]
+ deadline = time.monotonic() + max(0, manifest["created_at"] + manifest["wall_seconds"] - time.time())
+ child = subprocess.Popen(argv, cwd=manifest["source_root"], env=env, stdin=subprocess.DEVNULL,
+ stdout=subprocess.PIPE, stderr=subprocess.STDOUT, text=True,
+ encoding="utf-8", errors="replace", start_new_session=True)
+ thread = threading.Thread(target=redact_stream, args=(child.stdout, root / "harbor.log", key), daemon=True)
+ thread.start()
+ save(root / "lifecycle.json", {"state": "running", "pid": child.pid})
+ while child.poll() is None and not stopped.is_set() and time.monotonic() < deadline:
+ stopped.wait(.2)
+ if child.poll() is None:
+ reason = "time_limit" if time.monotonic() >= deadline else "interrupted"
+ os.killpg(child.pid, signal.SIGTERM)
+ try:
+ child.wait(timeout=5)
+ except subprocess.TimeoutExpired:
+ os.killpg(child.pid, signal.SIGKILL)
+ child.wait(timeout=5)
+ else:
+ reason = "completed" if child.returncode == 0 else "harbor_error"
+ code = child.returncode
+ finally:
+ try:
+ if child is not None and child.poll() is None:
+ os.killpg(child.pid, signal.SIGKILL)
+ child.wait(timeout=5)
+ if thread:
+ thread.join(timeout=2)
+ # No receipt/slot release here. systemd kills every remaining unit
+ # process before ExecStopPost performs the definitive Docker sweep.
+ save(root / "lifecycle.json", {"state": reason, "exit_code": code,
+ "no_trial_started": not (root / "ownership.json").exists()})
+ finally:
+ signal.signal(signal.SIGTERM, previous_term)
+ signal.signal(signal.SIGINT, previous_int)
+ return 0 if reason == "completed" else 1
+
+
+def unit_state(unit):
+ data = command(["systemctl", "show", unit, "--property=LoadState,ActiveState"], timeout=5)
+ return dict(line.split("=", 1) for line in data.splitlines() if "=" in line)
+
+
+def submit(config, root, credential, wall, credential_env=None):
+ metadata(Path(credential)) # Validate before claiming ownership or consuming.
+ root = absolute(root, exists=False)
+ manifest = prepare(config, root, wall, credential_env)
+ try:
+ secret = consume(Path(credential))
+ (root / "credential").mkdir(mode=0o700)
+ with (root / "credential" / "provider.key").open("x", encoding="utf-8") as file:
+ os.fchmod(file.fileno(), 0o600)
+ file.write(secret)
+ entry = [sys.executable, "-m", "scripts.terminal_bench.remote_job"]
+ stop = " ".join(json.dumps(s, ensure_ascii=False) for s in [*entry, "finalize", "--root", str(root)])
+ command(["systemd-run", "--quiet", "--unit=" + manifest["unit"],
+ "--property=Type=exec", "--property=KillMode=control-group",
+ "--property=WorkingDirectory=" + manifest["source_root"],
+ "--property=RuntimeMaxSec=" + str(wall), "--property=TimeoutStopSec=90",
+ "--property=ExecStopPost=" + stop,
+ "--", *entry, "worker", "--root", str(root)])
+ except Exception:
+ # Launch acknowledgement may have been lost AFTER the manager accepted
+ # the unit. Fence that exact unit; never publish an early clean receipt.
+ try:
+ state = unit_state(manifest["unit"])
+ if state.get("LoadState") == "not-found":
+ finalize(root, never_started=True)
+ else:
+ command(["systemctl", "stop", "--no-block", manifest["unit"]], timeout=5)
+ except (subprocess.SubprocessError, OSError, ValueError):
+ pass # Reservation deliberately retained; status reports the ambiguity.
+ raise
+ return {"unit": manifest["unit"], "receipt": str(root / "receipt.json")}
+
+
+def status(root):
+ receipt = evidence(root / "receipt.json")
+ if receipt and receipt.get("finished_at"):
+ return receipt, 0
+ manifest = load(root / "manifest.json")
+ state = unit_state(manifest["unit"])
+ if state.get("ActiveState") in {"active", "activating", "deactivating"}:
+ return {"state": "waiting", "unit": manifest["unit"]}, 75
+ return {"state": "missing_finalization_receipt", "unit": manifest["unit"], "cleanup_confirmed": False}, 1
+
+
+def main(argv=None):
+ parser = argparse.ArgumentParser(description=__doc__)
+ parser.add_argument("action", choices=["preflight", "submit", "worker", "finalize", "status"])
+ parser.add_argument("--config", type=Path)
+ parser.add_argument("--root", type=Path)
+ parser.add_argument("--credential-file", type=Path)
+ parser.add_argument("--credential-env")
+ parser.add_argument("--wall-seconds", type=int, default=1800)
+ args = parser.parse_args(argv)
+ try:
+ code = 0
+ if args.action == "preflight":
+ validate(args.config, args.credential_env)
+ result = {"preflight": "passed", "model_requests": 0}
+ elif args.action == "submit":
+ result = submit(args.config, args.root, args.credential_file, args.wall_seconds, args.credential_env)
+ elif args.action == "worker":
+ return worker(args.root)
+ elif args.action == "finalize":
+ result = finalize(args.root)
+ else:
+ result, code = status(args.root)
+ print(json.dumps(result))
+ return code
+ except Exception as exc:
+ print(json.dumps({"error_type": type(exc).__name__}), file=sys.stderr)
+ return 1
+
+
+if __name__ == "__main__":
+ raise SystemExit(main())
diff --git a/scripts/terminal_bench/requirements-harbor.txt b/scripts/terminal_bench/requirements-harbor.txt
new file mode 100644
index 000000000..ab9eec1de
--- /dev/null
+++ b/scripts/terminal_bench/requirements-harbor.txt
@@ -0,0 +1,2 @@
+# Optional SOURCE-ONLY benchmark host tools. Not a product dependency.
+harbor==0.24.0
diff --git a/scripts/terminal_bench/runner.py b/scripts/terminal_bench/runner.py
new file mode 100644
index 000000000..65740d6ed
--- /dev/null
+++ b/scripts/terminal_bench/runner.py
@@ -0,0 +1,495 @@
+"""Thin full-product trial supervisor (no Harbor or product dependency changes).
+
+Use the interpreter in the pinned Avibe wheel's venv. This is not a sandbox: real
+benchmarks must run inside an isolated task container. Local tests default to an
+owned HOME; sandbox callers may supply their explicit task image environment.
+"""
+from __future__ import annotations
+
+import argparse
+import json
+import os
+from pathlib import Path
+import re
+import secrets
+import signal
+import subprocess
+import sys
+import tempfile
+import threading
+import time
+from typing import Mapping
+
+import psutil
+
+from .diagnostics import project_responses, response_query
+from .profile import Profile, validate_layout
+
+_CONTROL = Path(__file__).with_name("control.py")
+_OWNER = "AVIBE_BENCHMARK_OWNER"
+_TERMINAL = {"succeeded", "failed", "canceled", "cancelled"}
+
+
+class TrialError(RuntimeError):
+ """A wrapper-owned symbolic code, never an upstream diagnostic body."""
+
+ _CODES = {
+ "owned_process_cleanup_unconfirmed", "service_exited_before_ready",
+ "controller_readiness_timeout", "service_not_ready", "harness_quiescence_unconfirmed",
+ "credential_input_refused", "cli_helper_invalid_run_payload",
+ }
+ _HELPER_CODE = re.compile(
+ r"(?:initialize|serve|health|configure|usage|versions|cli)_helper_"
+ r"(?:timeout|invalid_json|failed_-?[0-9]+)\Z"
+ )
+
+ def __init__(self, code: str):
+ self.code = code if code in self._CODES or self._HELPER_CODE.fullmatch(code) else "unclassified_trial_error"
+ super().__init__(self.code)
+
+
+def failure_details(exc: BaseException) -> dict[str, str]:
+ result = {"outcome": "infrastructure_error", "error_type": type(exc).__name__}
+ if isinstance(exc, TrialError):
+ result["error_code"] = exc.code
+ return result
+
+
+def scrub(value, key: str | None):
+ if isinstance(value, str):
+ return value.replace(key, "[REDACTED]") if key else value
+ if isinstance(value, dict):
+ return {scrub(str(k), key): scrub(v, key) for k, v in value.items()}
+ if isinstance(value, (list, tuple)):
+ return [scrub(v, key) for v in value]
+ return value
+
+
+class OwnedProcesses:
+ """Observe before shutdown, including children that create their own sessions.
+
+ Pids are never authority on their own. Capture creation identity while the
+ process is a descendant, or proves our random per-trial environment marker.
+ Never read a global Avibe runtime pidfile or kill by executable name.
+ """
+
+ def __init__(self, process: subprocess.Popen, marker: str):
+ self.process = process
+ self.marker = marker
+ self.identities: dict[int, float] = {}
+ self._stop = threading.Event()
+ self._lock = threading.Lock()
+ self._remember(psutil.Process(process.pid))
+ self._thread = threading.Thread(target=self._observe, daemon=True)
+ self._thread.start()
+
+ def _remember(self, process):
+ try:
+ created = process.create_time()
+ except psutil.NoSuchProcess:
+ return
+ with self._lock:
+ self.identities[process.pid] = created
+
+ def _live(self):
+ with self._lock:
+ recorded = list(self.identities.items())
+ live = []
+ for pid, created in recorded:
+ try:
+ process = psutil.Process(pid)
+ current_creation = process.create_time()
+ if current_creation != created:
+ # On macOS create_time can shift with the system clock.
+ # A readable matching marker still proves ownership; an
+ # unreadable identity cannot prove safe cleanup.
+ if process.environ().get(_OWNER) != self.marker:
+ continue
+ self._remember(process)
+ if process.status() != psutil.STATUS_ZOMBIE:
+ live.append(process)
+ except psutil.NoSuchProcess:
+ continue
+ return live
+
+ def refresh(self):
+ for process in self._live():
+ try:
+ for child in process.children(recursive=True):
+ self._remember(child)
+ except psutil.NoSuchProcess:
+ pass
+
+ def discover_detached(self):
+ # Detached engines/jobs inherit this random marker; records survive
+ # reparenting. No environment values are logged or exported.
+ for process in psutil.process_iter():
+ try:
+ if process.environ().get(_OWNER) == self.marker:
+ self._remember(process)
+ except (psutil.NoSuchProcess, psutil.AccessDenied):
+ continue
+
+ def _observe(self):
+ while not self._stop.wait(0.1):
+ try:
+ self.refresh()
+ except psutil.Error:
+ # Rechecked synchronously and fail-closed by close().
+ pass
+
+ @staticmethod
+ def _signal(process, *, kill=False):
+ try:
+ process.kill() if kill else process.terminate()
+ except psutil.NoSuchProcess:
+ pass
+
+ def close(self, grace: float) -> dict:
+ self.refresh()
+ self.discover_detached()
+ if self.process.poll() is None:
+ self.process.terminate() # main.py's real Controller shutdown owner
+ deadline = time.monotonic() + grace
+ while time.monotonic() < deadline:
+ self.refresh()
+ if not self._live():
+ break
+ time.sleep(0.05)
+ self.discover_detached()
+ for process in self._live():
+ self._signal(process)
+ _, alive = psutil.wait_procs(self._live(), timeout=1)
+ for process in alive:
+ # _live rechecks create_time before every forced-signal batch.
+ if process in self._live():
+ self._signal(process, kill=True)
+ psutil.wait_procs(self._live(), timeout=2)
+ self.process.wait(timeout=3)
+ self._stop.set()
+ self._thread.join(timeout=2)
+ self.discover_detached()
+ if self._live():
+ raise TrialError("owned_process_cleanup_unconfirmed")
+ return {"owned_processes_stopped": True, "observed_process_count": len(self.identities)}
+
+
+class FullRuntimeTrial:
+ """A fresh home, the normal service, IPC bootstrap, and ordinary CLI Run.
+
+ source_root is for source-checkout tests only. Omit it to exercise the pinned
+ installed wheel. No source package subsets are copied into the task.
+ """
+
+ def __init__(self, workdir: Path, state: Path, output: Path, profile: Profile, *,
+ task_environment: Mapping[str, str] | None = None, source_root: Path | None = None):
+ validate_layout(workdir, state, output)
+ self.workdir, self.state, self.output = (p.resolve() for p in (workdir, state, output))
+ self.profile = profile
+ self.source_root = source_root.resolve() if source_root else None
+ self.state.mkdir(parents=True, exist_ok=False, mode=0o700)
+ try:
+ self.output.mkdir(parents=True, exist_ok=False, mode=0o700)
+ except OSError:
+ self.state.rmdir() # only our just-created, empty reservation
+ raise
+ self.marker = secrets.token_hex(24)
+ self.home = self.state / "home"
+ self.home.mkdir()
+ self.avibe_home = self.state / "avibe"
+ self.env = self._environment(task_environment)
+ self.profile_path = self.state / "profile.json"
+ self.profile_path.write_text(json.dumps(profile.payload()), encoding="utf-8")
+ self._service = None
+ self._processes = None
+ self._log = None
+ self._socket_dir = None
+ self._key = None
+ self._result = {"outcome": "not_started", "verifier_reward": None, "usage": None,
+ "accounting_complete": False, "cleanup": None, "run_id": None, "session_id": None}
+
+ def _environment(self, task_environment):
+ if task_environment is None:
+ env = {"PATH": os.pathsep.join([str(Path(sys.executable).parent), os.defpath]),
+ "HOME": str(self.home), "USERPROFILE": str(self.home),
+ "XDG_CONFIG_HOME": str(self.home / ".config"),
+ "XDG_DATA_HOME": str(self.home / ".local/share"),
+ "XDG_CACHE_HOME": str(self.home / ".cache"), "TMPDIR": str(self.state / "tmp")}
+ (self.state / "tmp").mkdir()
+ else:
+ # This mapping is the sandbox image's environment, not os.environ
+ # of the user's machine. Retain task toolchains and the real HOME.
+ env = dict(task_environment)
+ for name in list(env):
+ upper = name.upper()
+ if (upper.startswith(("AVIBE_", "VIBE_", "ANTHROPIC_", "OPENAI_", "CODEX_", "CLAUDE_"))
+ or name == self.profile.credential_env):
+ env.pop(name)
+ env.update({
+ "AVIBE_HOME": str(self.avibe_home), _OWNER: self.marker,
+ "AVIBE_WEB_UI_SUPERVISOR": "terminal-bench-no-ui",
+ "VIBE_SENTRY_DSN": "", "VIBE_STARTUP_DEPENDENCY_RECONCILE": "0",
+ "VIBE_INSTALL_SKIP_ASKILL": "1", "VIBE_ASKILL_AUTO_UPDATE": "0",
+ "VIBE_SHOW_RUNTIME_OFFLINE": "1" if self.profile.offline else "0",
+ "VIBE_TMUX_OFFLINE": "1" if self.profile.offline else "0",
+ "VIBE_GIT_OFFLINE": "1" if self.profile.offline else "0",
+ "VIBE_MODEL_HUB_ENGINE_OFFLINE": "1" if self.profile.offline else "0",
+ "VIBE_MODEL_HUB_ENABLED": "1", "PYTHONUNBUFFERED": "1",
+ })
+ # Local metadata avoids catalog/network drift; it does not change model
+ # requests, Agent prompts, tools, Harness or context management.
+ catalog = self.state / "empty-models-dev.json"
+ catalog.write_text("{}", encoding="utf-8")
+ env["AVIBE_MODELS_DEV_URL"] = catalog.as_uri()
+ # Use the installed product's bundled backend catalog, resolved in child.
+ return env
+
+ def _command(self, operation: str, *args: str):
+ if operation == "cli":
+ code = "import runpy; runpy.run_module('vibe',run_name='__main__')"
+ command_args = list(args)
+ else:
+ code = "import runpy,sys; runpy.run_path(sys.argv.pop(1),run_name='__main__')"
+ command_args = [str(_CONTROL), operation, *args]
+ if self.source_root:
+ code = "import sys; sys.path.insert(0,sys.argv.pop(1)); " + code
+ command_args.insert(0, str(self.source_root))
+ return [sys.executable, "-I", "-c", code, *command_args]
+
+ def _call(self, operation: str, *args: str, timeout=None, stdin=None):
+ try:
+ proc = subprocess.run(self._command(operation, *args), cwd=self.state, env=self.env,
+ input=stdin, text=True, encoding="utf-8", capture_output=True,
+ timeout=timeout or self.profile.setup_seconds)
+ except subprocess.TimeoutExpired:
+ raise TrialError(f"{operation}_helper_timeout") from None
+ if proc.returncode:
+ raise TrialError(f"{operation}_helper_failed_{proc.returncode}")
+ try:
+ return json.loads(proc.stdout)
+ except ValueError:
+ raise TrialError(f"{operation}_helper_invalid_json") from None
+
+ def _call_run_show(self, run_id: str, *, timeout: float = 15):
+ """Accept a valid durable Run payload even when CLI exits nonzero.
+
+ ``vibe runs show`` returns a meaningful failed Run alongside a nonzero
+ exit code. That is a product outcome, not helper infrastructure.
+ """
+ argv = self._command("cli", "runs", "show", run_id)
+ try:
+ proc = subprocess.run(argv, cwd=self.state, env=self.env,
+ text=True, encoding="utf-8", capture_output=True,
+ timeout=timeout)
+ except subprocess.TimeoutExpired:
+ raise TrialError("cli_helper_timeout") from None
+ try:
+ payload = json.loads(proc.stdout)
+ row = payload.get("run") if isinstance(payload, dict) else None
+ if not isinstance(row, dict) or row.get("id") not in {None, run_id}:
+ raise ValueError
+ return payload
+ except (TypeError, ValueError, json.JSONDecodeError):
+ if proc.returncode:
+ raise TrialError(f"cli_helper_failed_{proc.returncode}") from None
+ raise TrialError("cli_helper_invalid_run_payload") from None
+
+ def start(self):
+ if os.name != "nt":
+ # UNIX socket names have a ~104-byte bound on macOS. Follow the
+ # test/operator TMPDIR (HD1 here), not a long nested trial path.
+ self._socket_dir = tempfile.TemporaryDirectory(prefix="tb-ipc-")
+ self.env["VIBE_INTERNAL_DISPATCH_SOCKET"] = str(Path(self._socket_dir.name) / "dispatch.sock")
+ self._result["installation"] = self._call("initialize", "--workdir", str(self.workdir))
+ self._result["installation"].update(self._call("versions"))
+ self.env["AVIBE_BACKEND_MODEL_CATALOG_URL"] = Path(self._result["installation"]["backend_catalog"]).as_uri()
+ self._log = (self.state / "service.log").open("w", encoding="utf-8")
+ self._service = subprocess.Popen(self._command("serve"), cwd=self.state, env=self.env,
+ stdout=self._log, stderr=self._log, start_new_session=True)
+ self._processes = OwnedProcesses(self._service, self.marker)
+ deadline = time.monotonic() + self.profile.setup_seconds
+ while time.monotonic() < deadline:
+ if self._service.poll() is not None:
+ raise TrialError("service_exited_before_ready")
+ health = self._call("health", timeout=min(10, self.profile.setup_seconds))
+ if health.get("ready"):
+ self._result["controller_health"] = health["identity"]
+ self._result["service_pid"] = self._service.pid
+ self._result["outcome"] = "ready"
+ return
+ time.sleep(self.profile.poll_seconds)
+ raise TrialError("controller_readiness_timeout")
+
+ def execute(self, instruction: str, *, key: str | None):
+ if self._result["outcome"] != "ready":
+ raise TrialError("service_not_ready")
+ if not key:
+ self._result.update(outcome="blocked", blocker="model_source_credential_required")
+ return
+ self._key = key.strip()
+ if not self._key:
+ self._result.update(outcome="blocked", blocker="model_source_credential_required")
+ return
+ self._result["source"] = self._call("configure", "--profile", str(self.profile_path), stdin=self._key)
+ started_at = time.monotonic()
+ self._call("cli", "agent", "update", "vibey", "--model", self.profile.model,
+ "--reasoning-effort", self.profile.reasoning_effort)
+ instruction_path = self.state / "instruction.txt"
+ instruction_path.write_text(instruction, encoding="utf-8")
+ # Explicit asynchronous creation preserves stable ids before waiting.
+ # Expiry of a CLI waiter does not cancel an Agent turn.
+ accepted = self._call("cli", "agent", "run", "--agent", "vibey", "--cwd", str(self.workdir),
+ "--message-file", str(instruction_path), "--no-callback")
+ run_id, session_id = accepted["run_id"], accepted["session_id"]
+ self._result.update(run_id=run_id, session_id=session_id)
+ deadline = time.monotonic() + self.profile.wall_seconds
+ terminal_row = None
+ quiet_observations = 0
+ while time.monotonic() < deadline:
+ row = self._call_run_show(run_id)["run"]
+ if row["status"] in _TERMINAL:
+ terminal_row = row
+ # A Run's final reply can hand over to a Watch, scheduled Task
+ # or subagent. They are real product work, not noise to discard
+ # at the first terminal. This home belongs only to this trial.
+ status = self._call("cli", "harness", "status", timeout=15)
+ if (not status.get("controller", {}).get("available")
+ or any(status.get("truncated", {}).values())):
+ raise TrialError("harness_quiescence_unconfirmed")
+ busy = bool(status.get("runs") or status.get("watches") or status.get("tasks"))
+ busy = busy or any(item.get("state") == "active" for item in status.get("runtime_agents", []))
+ quiet_observations = 0 if busy else quiet_observations + 1
+ if quiet_observations >= 2:
+ self._result.update(outcome=terminal_row["status"], final_text=terminal_row.get("result_text"),
+ run_error=terminal_row.get("error"), harness_quiescent=True)
+ break
+ time.sleep(self.profile.poll_seconds)
+ else:
+ self._result["outcome"] = "time_limit"
+ try:
+ cancelled = self._call("cli", "runs", "cancel", run_id, timeout=15)
+ self._result["cancel_requested"] = cancelled.get("ok") is True
+ except TrialError as exc:
+ # A concurrent terminal or unavailable IPC must not skip the
+ # final service/tree teardown, nor erase the deadline outcome.
+ self._result["cancel_requested"] = False
+ self._result["cancel_error"] = str(exc)
+ # Cancellation is asynchronous; a receipt alone is not quiescence.
+ cancel_deadline = time.monotonic() + self.profile.cleanup_seconds
+ self._result["run_settled_after_cancel"] = False
+ while time.monotonic() < cancel_deadline:
+ try:
+ row = self._call_run_show(run_id)["run"]
+ except TrialError:
+ break
+ if row["status"] in _TERMINAL:
+ self._result["run_settled_after_cancel"] = True
+ break
+ time.sleep(self.profile.poll_seconds)
+ # Service shutdown and exact-owned-process reaping still follow;
+ # even a terminal Run may own Watch-backed work after its turn.
+ self._result["elapsed_seconds"] = time.monotonic() - started_at
+ self._result["outcome_scope"] = "initial_run_after_fresh_home_harness_quiescence_or_deadline"
+ self._result["final_text_scope"] = "initial_run_result"
+ # Entire fresh-home accounting, including bootstrap calls, is identified
+ # as such. Never claim completeness for interrupted/provider-failed calls.
+ try:
+ self._result["usage"] = self._call("usage", timeout=15)
+ except TrialError:
+ self._result["usage_unavailable"] = True
+
+ def close(self):
+ # Capture only allowlisted response shapes while the normal CLI can
+ # still reach this trial's service. Failure is explicit but never
+ # changes the measured outcome or prevents owned-process cleanup.
+ session_id = self._result.get("session_id")
+ self._result["diagnostics"] = {"schema_version": 1, "status": "no_session", "raw_text_exported": False}
+ if session_id and self._service and self._service.poll() is None:
+ try:
+ payload = self._call("cli", "data", "query", "--sql", response_query(session_id), "--limit", "9", timeout=10)
+ self._result["diagnostics"] = project_responses(payload)
+ except Exception:
+ self._result["diagnostics"] = {"schema_version": 1, "status": "unavailable", "raw_text_exported": False}
+ elif session_id:
+ self._result["diagnostics"]["status"] = "service_unavailable"
+ try:
+ if self._processes:
+ self._result["cleanup"] = self._processes.close(self.profile.cleanup_seconds)
+ finally:
+ if self._log:
+ self._log.close()
+ if self._socket_dir:
+ self._socket_dir.cleanup()
+ self._result["profile"] = self.profile.payload()
+ self._result["execution_path"] = "normal_service_model_hub_ipc_and_cli_run"
+ self._result["runtime_state_exported"] = False
+ self._result.setdefault("installation", {})["mode"] = "source_checkout" if self.source_root else "installed_package"
+ self._result["accounting_scope"] = "fresh_home_reported_usage_including_bootstrap_not_actual_bill"
+ result = scrub(self._result, self._key)
+ (self.output / "execution.json").write_text(json.dumps(result, ensure_ascii=False, indent=2), encoding="utf-8")
+ return result
+
+ def __enter__(self):
+ try:
+ self.start()
+ return self
+ except BaseException as exc:
+ self._result.update(failure_details(exc))
+ self.close()
+ raise
+
+ def __exit__(self, exc_type, exc, _tb):
+ if exc is not None:
+ self._result.update(failure_details(exc))
+ self.close()
+
+
+def _interrupt_trial(signum, frame):
+ # A second TERM must not interrupt cleanup. The outer container supervisor
+ # can still use KILL if its independent final grace deadline expires.
+ signal.signal(signal.SIGTERM, signal.SIG_IGN)
+ raise KeyboardInterrupt
+
+
+def main(argv=None):
+ parser = argparse.ArgumentParser(description=__doc__)
+ for name in ("workdir", "state", "output", "profile", "instruction"):
+ parser.add_argument(f"--{name}", type=Path, required=True)
+ parser.add_argument("--credential-file", type=Path,
+ help="Private per-trial input; consumed and unlinked before service startup.")
+ parser.add_argument("--sandbox-environment", action="store_true",
+ help="Use only inside an externally isolated task container; preserve its HOME/toolchains.")
+ args = parser.parse_args(argv)
+ profile = Profile.parse(json.loads(args.profile.read_text(encoding="utf-8")))
+ if args.credential_file is not None:
+ from .credential import consume
+ try:
+ key = consume(args.credential_file)
+ except Exception:
+ summary = json.dumps(failure_details(TrialError("credential_input_refused")))
+ print(summary)
+ print(summary, file=sys.stderr)
+ return 2
+ else:
+ # Direct invocation may retain its named env mode. The Harbor Docker
+ # adapter must use credential-file instead: exec env leaks into argv.
+ key = os.environ.get(profile.credential_env)
+ trial = FullRuntimeTrial(args.workdir, args.state, args.output, profile,
+ task_environment=os.environ if args.sandbox_environment else None)
+ previous_term = signal.signal(signal.SIGTERM, _interrupt_trial)
+ try:
+ with trial:
+ trial.execute(args.instruction.read_text(encoding="utf-8"), key=key)
+ except (Exception, KeyboardInterrupt) as exc:
+ summary = json.dumps(failure_details(exc))
+ print(summary)
+ print(summary, file=sys.stderr)
+ return 2
+ finally:
+ signal.signal(signal.SIGTERM, previous_term)
+ print(json.dumps({"outcome": trial._result["outcome"], "report": str(trial.output / "execution.json")}))
+ return 0
+
+
+if __name__ == "__main__":
+ raise SystemExit(main())
diff --git a/storage/agent_transcript.py b/storage/agent_transcript.py
index ccd05b4df..8c6f7d1d5 100644
--- a/storage/agent_transcript.py
+++ b/storage/agent_transcript.py
@@ -125,12 +125,12 @@ def final_outcome(message: AssistantMessage) -> FinalOutcome:
"""The Turn outcome a committed final response determines by itself.
The one rule, shared by the row's type at commit and live settlement: a final
- reply without text of its own (an unexplained refusal or safety stop, or an
- empty answer) failed; any other final completed. It is the loop's own
- empty-reply test (loop-control.md section 2). Failures only a live run can
- observe after the commit are applied by the live caller.
+ reply without usable content, or explicitly stopped at its output limit,
+ failed. Partial text must not turn a provider's incomplete final into a
+ successful result. Pending-input responses are not finals. Failures only
+ a live run can observe after the commit are applied by the live caller.
"""
- return "completed" if has_reply(message) else "failed"
+ return "completed" if has_reply(message) and message.stop_reason != "length" else "failed"
def _response_type(message: AssistantMessage, *, final: bool, display_text: str) -> str:
diff --git a/tests/agent_core/agent/test_loop.py b/tests/agent_core/agent/test_loop.py
index 14831ae48..c437a31c1 100644
--- a/tests/agent_core/agent/test_loop.py
+++ b/tests/agent_core/agent/test_loop.py
@@ -1263,12 +1263,12 @@ async def test_tool_free_final_reply_is_committed_and_empty_success_is_an_error(
["input"] if continuation == "final" else ["input", "pending"]
)
errors = [event for event in events if isinstance(event, AgentError)]
- if shape in ("visible", "image") or continuation != "final":
+ if continuation != "final" or (shape in ("visible", "image") and stop_reason != "length"):
assert errors == []
assert events[-1].reason == "completed"
else:
assert len(errors) == 1
- assert errors[0].kind == (stop_reason if stop_reason in {"refusal", "safety"} else "empty_response")
+ assert errors[0].kind == (stop_reason if stop_reason in {"length", "refusal", "safety"} else "empty_response")
assert errors[0].message
assert committed[0].seq < errors[0].seq < events[-1].seq
assert events[-1].reason == "error"
diff --git a/tests/fixtures/full_runtime_service.py b/tests/fixtures/full_runtime_service.py
new file mode 100644
index 000000000..b4378b78a
--- /dev/null
+++ b/tests/fixtures/full_runtime_service.py
@@ -0,0 +1,76 @@
+"""Child test fixture: replace ONLY the external engine, run normal main.py.
+
+The real Controller/Model Hub service/router/gateway/provider, Session/Harness,
+Vibey tools and SQLite run unchanged. This is not real CPA acceptance evidence.
+"""
+from __future__ import annotations
+
+import json
+import os
+import sys
+from pathlib import Path
+
+REPO = Path(__file__).resolve().parents[2]
+sys.path.insert(0, str(REPO))
+
+from tests import network_guard
+sys.addaudithook(network_guard.audit)
+
+from core.handlers.model_hub.adapter import RawOutcomeKind
+from tests.scenario_harness.model_hub import ModelHubScenarioAdapter, ScenarioCallResult
+
+
+class ScriptedEngine(ModelHubScenarioAdapter):
+ async def invoke(self, source_id, model_id, request, stream, origin, *, on_admitted=None):
+ messages = request.get("messages", [])
+ has_result = any(item.get("role") == "tool" for item in messages)
+ if not has_result:
+ program = (
+ "import os,sqlite3; from pathlib import Path; "
+ "Path('笔记.md').write_text('可查询的任务笔记', encoding='utf-8'); "
+ "db=sqlite3.connect('task.sqlite'); "
+ "db.execute('CREATE TABLE evidence(value TEXT)'); "
+ "db.execute(\"INSERT INTO evidence VALUES ('task-data')\"); db.commit(); db.close(); "
+ "Path('task-env.json').write_text(__import__('json').dumps({"
+ "'cwd':os.getcwd(),'home':os.environ['HOME'],"
+ "'credential_present':'DEEPSEEK_API_KEY' in os.environ,"
+ "'session':os.environ.get('AVIBE_SESSION_ID'),"
+ "'java_home':os.environ.get('JAVA_HOME')}))"
+ )
+ if os.environ.get("TB_TEST_SCENARIO") == "slow-tool":
+ program = "import time; from pathlib import Path\nwhile True:\n Path('still-writing').write_text(str(time.time_ns())); time.sleep(.02)"
+ import shlex
+ command = "python3 -c " + shlex.quote(program)
+ if os.environ.get("TB_TEST_SCENARIO") == "watch":
+ program = "import time; from pathlib import Path; time.sleep(4); Path('watch-note.txt').write_text('continued')"
+ command = "vibe watch add --message 'The note is ready; finish.' -- python3 -c " + shlex.quote(program)
+ delta = {"tool_calls": [{"index": 0, "id": "fixture-call", "type": "function",
+ "function": {"name": "bash", "arguments": json.dumps({
+ "command": command,
+ "timeout": 20, "watch": False})}}]}
+ reason = "tool_calls"
+ else:
+ scenario = os.environ.get("TB_TEST_SCENARIO")
+ if scenario == "empty-response":
+ delta, reason = {}, "stop"
+ elif scenario == "thinking-length":
+ delta, reason = {"reasoning_content": "private-reasoning-秘密"}, "length"
+ else:
+ delta, reason = {"content": "Finished integration fixture. fixture-private-key-not-real"}, "stop"
+ rows = [
+ {"id": "fixture-response", "model": model_id,
+ "choices": [{"index": 0, "delta": delta, "finish_reason": None}]},
+ {"id": "fixture-response", "choices": [{"index": 0, "delta": {}, "finish_reason": reason}],
+ "usage": {"prompt_tokens": 100, "completion_tokens": 20}},
+ ]
+ body = ("".join("data: " + json.dumps(row) + "\n\n" for row in rows) + "data: [DONE]\n\n").encode()
+ self.invoke_results.append(ScenarioCallResult(RawOutcomeKind.SUCCESS, status=200, body=body, stream_started=True))
+ return await super().invoke(source_id, model_id, request, stream, origin, on_admitted=on_admitted)
+
+
+if __name__ == "__main__":
+ import vibe.model_hub_runtime
+ adapter = ScriptedEngine(discovery_models=["fixture-model"])
+ vibe.model_hub_runtime.get_model_hub_engine_adapter = lambda: adapter
+ from scripts.terminal_bench.control import main
+ raise SystemExit(main(["serve"]))
diff --git a/tests/test_terminal_bench_credential.py b/tests/test_terminal_bench_credential.py
new file mode 100644
index 000000000..efae1b4e3
--- /dev/null
+++ b/tests/test_terminal_bench_credential.py
@@ -0,0 +1,138 @@
+"""Ephemeral file transport: metadata-only inspection and consume-before-start."""
+from __future__ import annotations
+
+import json
+import os
+from pathlib import Path
+import subprocess
+
+import pytest
+
+from scripts.terminal_bench.credential import LIMIT, clean, consume, metadata, prepare_upload
+from scripts.terminal_bench.profile import Profile
+from scripts.terminal_bench import runner
+
+SENTINEL = "fake-private-key-not-for-logs"
+
+
+def secret_file(tmp_path):
+ private = tmp_path / "credential"
+ private.mkdir(mode=0o700)
+ path = private / "provider.key"
+ descriptor = os.open(path, os.O_WRONLY | os.O_CREAT | os.O_EXCL, 0o600)
+ with os.fdopen(descriptor, "w") as handle:
+ handle.write(SENTINEL)
+ return path
+
+
[email protected]("shape", ["normal", "symlink", "hardlink", "oversized", "public", "fifo", "empty"])
+def test_input_metadata_and_failure_removal(tmp_path, shape):
+ path = secret_file(tmp_path)
+ external = tmp_path / "unchanged"
+ external.write_text("outside")
+ if shape == "symlink":
+ path.unlink()
+ path.symlink_to(external)
+ elif shape == "hardlink":
+ path.unlink()
+ os.link(external, path)
+ path.chmod(0o600)
+ elif shape == "oversized":
+ path.write_bytes(b"x" * (LIMIT + 1))
+ elif shape == "public":
+ path.chmod(0o644)
+ elif shape == "fifo":
+ path.unlink()
+ os.mkfifo(path, 0o600)
+ elif shape == "empty":
+ path.write_text("")
+ if shape == "normal":
+ prepared = prepare_upload(path)
+ info = metadata(path)
+ assert prepared == info
+ assert info == {"regular": True, "mode": "0600", "directory_mode": "0700", "size": len(SENTINEL)}
+ assert SENTINEL not in json.dumps(info)
+ assert consume(path) == SENTINEL
+ else:
+ before = external.stat()
+ with pytest.raises((OSError, ValueError)):
+ prepare_upload(path)
+ assert (external.stat().st_uid, external.stat().st_mode) == (before.st_uid, before.st_mode)
+ with pytest.raises((OSError, ValueError)):
+ consume(path)
+ assert not path.exists() and not path.is_symlink()
+ assert external.read_text() == "outside"
+ assert clean(path) == {"removed": True}
+ assert not path.parent.exists()
+
+
[email protected](os.geteuid() != 0, reason="real uploaded UID adoption needs a root Linux sandbox")
+def test_prepare_adopts_foreign_upload_uid_but_read_stays_strict(tmp_path):
+ """Compose cp preserves UID501; strict consume must work only after setup adopts it."""
+ path = secret_file(tmp_path)
+ os.chown(path, 501, -1)
+ with pytest.raises(ValueError, match="owned by the runner"):
+ metadata(path)
+ assert prepare_upload(path)["mode"] == "0600"
+ assert path.stat().st_uid == os.geteuid()
+ assert consume(path) == SENTINEL
+ assert not path.exists()
+
+
+def test_rejects_symlink_parent_without_touching_other_directory(tmp_path):
+ elsewhere = tmp_path / "other-trial"
+ elsewhere.mkdir()
+ actual = secret_file(elsewhere)
+ (tmp_path / "credential").symlink_to(actual.parent, target_is_directory=True)
+ with pytest.raises(OSError):
+ consume(tmp_path / "credential/provider.key")
+ assert actual.read_text() == SENTINEL
+ with pytest.raises(OSError):
+ clean(tmp_path / "credential/provider.key")
+ assert actual.read_text() == SENTINEL
+
+
+def test_key_unlinked_before_service_and_delivered_only_to_bootstrap_stdin(tmp_path, monkeypatch, capsys):
+ path = secret_file(tmp_path)
+ work = tmp_path / "task"
+ work.mkdir()
+ profile = tmp_path / "profile.json"
+ profile.write_text(json.dumps(Profile(model="fixture", base_url="http://127.0.0.1:9/v1").payload()))
+ instruction = tmp_path / "instruction.txt"
+ instruction.write_text("fixture")
+ observed = []
+
+ def start(self):
+ assert not path.exists(), "credential file must disappear BEFORE measured service starts"
+ assert self.profile.credential_env not in self.env
+ self._result["outcome"] = "ready"
+
+ def helper(argv, **kwargs):
+ observed.append((argv, kwargs))
+ assert SENTINEL not in " ".join(argv)
+ assert SENTINEL not in json.dumps(kwargs["env"])
+ if "configure" in argv:
+ assert kwargs["input"] == SENTINEL
+ result = {"source_id": "fixture-source"}
+ elif "agent" in argv and "run" in argv:
+ result = {"run_id": "fixture-run", "session_id": "fixture-session"}
+ elif "runs" in argv:
+ result = {"run": {"status": "succeeded", "result_text": SENTINEL}}
+ elif "harness" in argv:
+ result = {"controller": {"available": True}, "runs": [], "watches": [], "tasks": []}
+ else:
+ result = {}
+ return subprocess.CompletedProcess(argv, 0, json.dumps(result), "")
+
+ monkeypatch.delenv("DEEPSEEK_API_KEY", raising=False)
+ monkeypatch.setattr(runner.FullRuntimeTrial, "start", start)
+ monkeypatch.setattr(subprocess, "run", helper)
+ assert runner.main([
+ "--workdir", str(work), "--state", str(tmp_path / "state"), "--output", str(tmp_path / "out"),
+ "--profile", str(profile), "--instruction", str(instruction), "--credential-file", str(path),
+ ]) == 0
+ output = capsys.readouterr()
+ assert observed and SENTINEL not in output.out + output.err
+ assert SENTINEL not in (tmp_path / "out/execution.json").read_text()
+ assert SENTINEL not in (tmp_path / "state/profile.json").read_text()
+ assert not path.exists()
diff --git a/tests/test_terminal_bench_diagnostics.py b/tests/test_terminal_bench_diagnostics.py
new file mode 100644
index 000000000..00570babe
--- /dev/null
+++ b/tests/test_terminal_bench_diagnostics.py
@@ -0,0 +1,65 @@
+"""Response diagnosis retains stop/shape/usage, never private model content."""
+import json
+import sqlite3
+from unittest.mock import Mock
+
+import pytest
+
+from scripts.terminal_bench.diagnostics import project_responses, response_query
+from scripts.terminal_bench.profile import Profile
+from scripts.terminal_bench.runner import FullRuntimeTrial
+
+
[email protected]("stop,shape", [("stop", "empty"), ("stop", "thinking"), ("length", "thinking"), ("length", "text"), ("tool_use", "tool_call")])
+def test_query_preserves_cause_without_exporting_model_text(stop, shape):
+ with sqlite3.connect(":memory:") as db:
+ db.row_factory = sqlite3.Row
+ db.execute("CREATE TABLE messages(session_id,type,context_seq,content_json)")
+ blocks = [] if shape == "empty" else [{"type": shape, "text": "SECRET-隐私", "arguments": {"key": "SECRET-隐私"}}]
+ message = {"role": "assistant", "content": blocks, "stop_reason": stop, "error_message": "SECRET-隐私", "usage": {"output_tokens": 32000}}
+ for sid in ("ses_test", "ses_unrelated"):
+ db.execute("INSERT INTO messages VALUES(?,?,?,?)", (sid, "error", 2, json.dumps({"model": {"message": message}})))
+ rows = [dict(r) for r in db.execute(response_query("ses_test"))]
+ assert len(rows) == 1
+ assert "SECRET" not in json.dumps(rows)
+ report = project_responses({"ok": True, "rows": rows})
+ last = report["responses"][0]
+ assert last["stop_reason"] == stop
+ assert last["thinking_blocks"] == int(shape == "thinking")
+ assert last["text_blocks"] == int(shape == "text")
+ assert last["tool_calls"] == int(shape == "tool_call")
+ assert last["output_tokens"] == 32000
+ assert report["raw_text_exported"] is False
+
+
+def test_projection_is_bounded_and_does_not_trust_cli_extra_fields():
+ row = {"stop_reason": "SECRET", "row_type": "SECRET", "text_chars": "SECRET", "context_seq": 3,
+ "input_tokens": -3, "output_tokens": True, "content_text": "SECRET"}
+ report = project_responses({"ok": True, "rows": [row] * 100})
+ assert len(report["responses"]) == 8
+ assert "SECRET" not in json.dumps(report)
+ assert report["responses"][0]["input_tokens"] is None
+ assert report["responses"][0]["output_tokens"] is None
+ assert "LIMIT 32" in response_query("ses_test")
+ for sid in ("x' OR 1=1 --", "", "x" * 129):
+ with pytest.raises(ValueError):
+ response_query(sid)
+ with pytest.raises(ValueError):
+ project_responses({"ok": False, "error": "SECRET"})
+
+
+def test_missing_diagnostics_does_not_mask_run_failure_or_skip_cleanup(tmp_path, monkeypatch):
+ task = tmp_path / "task"
+ task.mkdir()
+ trial = FullRuntimeTrial(task, tmp_path / "state", tmp_path / "out", Profile(model="fixture", base_url="http://127.0.0.1:9/v1"))
+ trial._result.update(outcome="failed", session_id="ses_test")
+ trial._service = Mock()
+ trial._service.poll.return_value = None
+ trial._processes = Mock()
+ trial._processes.close.return_value = {"owned_processes_stopped": True}
+ monkeypatch.setattr(trial, "_call", Mock(side_effect=ValueError("SECRET")))
+ report = trial.close()
+ assert report["outcome"] == "failed"
+ assert report["diagnostics"]["status"] == "unavailable"
+ trial._processes.close.assert_called_once()
+ assert "SECRET" not in json.dumps(report)
diff --git a/tests/test_terminal_bench_full_runtime.py b/tests/test_terminal_bench_full_runtime.py
new file mode 100644
index 000000000..2d664e73f
--- /dev/null
+++ b/tests/test_terminal_bench_full_runtime.py
@@ -0,0 +1,284 @@
+"""Boundary regressions for the source-only full-product benchmark supervisor.
+
+Observable contracts: fresh disjoint state; inherited caller credentials cannot
+route into the live service; real CLI Run dispatch writes task files while all
+runtime state stays out; deadline cleanup is owned and bounded. Existing service
+acceptance covers real CPA; these tests substitute ONLY that external engine.
+"""
+from __future__ import annotations
+
+from dataclasses import replace
+import json
+import os
+from pathlib import Path
+import sqlite3
+import subprocess
+import sys
+import time
+
+import psutil
+import pytest
+
+from scripts.terminal_bench.profile import Profile, validate_layout
+from scripts.terminal_bench.runner import FullRuntimeTrial, OwnedProcesses, TrialError
+
+ROOT = Path(__file__).resolve().parents[1]
+# Fresh SQLite migrations reached Controller construction only after 30s in
+# cold HD1 runs. Startup gets its own 120s allowance; this does not relax the
+# 30s Agent deadline or the deliberately short 2s cancellation scenario.
+PROFILE = Profile(model="fixture-model", base_url="http://127.0.0.1:9/v1", setup_seconds=120,
+ wall_seconds=30, cleanup_seconds=5, poll_seconds=0.1, offline=True)
+
+
+def test_disjoint_fresh_layout_and_environment(tmp_path, monkeypatch):
+ task = tmp_path / "任务"
+ task.mkdir()
+ monkeypatch.setenv("AVIBE_SESSION_ID", "real-user-session")
+ monkeypatch.setenv("ANTHROPIC_AUTH_TOKEN", "real-user-key")
+ before = dict(os.environ)
+ trial = FullRuntimeTrial(task, tmp_path / "private", tmp_path / "export", PROFILE)
+ assert dict(os.environ) == before
+ assert "AVIBE_SESSION_ID" not in trial.env
+ assert "ANTHROPIC_AUTH_TOKEN" not in trial.env
+ assert "DEEPSEEK_API_KEY" not in trial.env
+ assert Path(trial.env["AVIBE_HOME"]).is_relative_to(trial.state)
+ assert not list(task.iterdir())
+ with pytest.raises(FileExistsError):
+ FullRuntimeTrial(task, tmp_path / "private", tmp_path / "other-export", PROFILE)
+ for state, output in [(task / "state", tmp_path / "out"), (tmp_path, tmp_path / "out"),
+ (tmp_path / "private", tmp_path / "private" / "export")]:
+ with pytest.raises(ValueError):
+ validate_layout(task, state, output)
+ alias = tmp_path / "alias"
+ alias.symlink_to(task, target_is_directory=True)
+ with pytest.raises(ValueError):
+ validate_layout(task, alias, tmp_path / "out")
+ monkeypatch.setenv("AVIBE_HOME", str(tmp_path / "custom-user-state"))
+ with pytest.raises(ValueError):
+ validate_layout(task, tmp_path / "custom-user-state" / "bench", tmp_path / "out")
+
+
+def test_sandbox_environment_retains_task_home_and_toolchains(tmp_path):
+ task = tmp_path / "task"
+ task.mkdir()
+ task_env = {"HOME": "/root", "JAVA_HOME": "/opt/java", "RUSTUP_HOME": "/opt/rust",
+ "VIRTUAL_ENV": "/opt/task-venv", "PYTHONPATH": "/task-modules",
+ "AVIBE_SESSION_ID": "host-session", "DEEPSEEK_API_KEY": "private",
+ "VIBE_INTERNAL_DISPATCH_SOCKET": "/live/socket", "TASK_DB_PASSWORD": "task-local-value"}
+ trial = FullRuntimeTrial(task, tmp_path / "state", tmp_path / "out", PROFILE, task_environment=task_env)
+ for name in ["HOME", "JAVA_HOME", "RUSTUP_HOME", "VIRTUAL_ENV", "PYTHONPATH", "TASK_DB_PASSWORD"]:
+ assert trial.env[name] == task_env[name]
+ for name in ["AVIBE_SESSION_ID", "DEEPSEEK_API_KEY", "VIBE_INTERNAL_DISPATCH_SOCKET"]:
+ assert name not in trial.env
+ assert task_env["AVIBE_SESSION_ID"] == "host-session" # caller mapping was not mutated
+
+
[email protected]_foreign_signals("Full-runtime supervisor reaps its subprocess's recorded descendants by identity")
[email protected]("scenario", ["success", "watch", "deadline", "missing_credential", "empty-response", "thinking-length"])
+def test_real_service_cli_hub_vibey_writes_and_isolates_state(tmp_path, monkeypatch, scenario):
+ task = tmp_path / "任务"
+ task.mkdir()
+ # This must not shadow product imports, nor be deleted/rewritten.
+ (task / "config.py").write_text("task_owned = True\n")
+ profile = replace(PROFILE, wall_seconds=2) if scenario == "deadline" else PROFILE
+ trial = FullRuntimeTrial(task, tmp_path / "state", tmp_path / "export", profile, source_root=ROOT)
+ trial.env["JAVA_HOME"] = "/fixture/java-toolchain"
+ if scenario == "watch":
+ trial.env["TB_TEST_SCENARIO"] = "watch"
+ # Source-only test interpreter: the measured installed wheel has this
+ # executable already. The shim lives outside the task and prompt.
+ shim = trial.state / "bin" / "vibe"
+ shim.parent.mkdir()
+ shim.write_text(f"#!{sys.executable}\nimport sys\nsys.path.insert(0, {str(ROOT)!r})\nfrom vibe.cli import main\nmain()\n")
+ shim.chmod(0o755)
+ trial.env["PATH"] = str(shim.parent) + os.pathsep + trial.env["PATH"]
+ if scenario == "deadline":
+ trial.env["TB_TEST_SCENARIO"] = "slow-tool"
+ if scenario in {"empty-response", "thinking-length"}:
+ trial.env["TB_TEST_SCENARIO"] = scenario
+ original_command = trial._command
+
+ def command(operation, *args):
+ if operation == "serve":
+ return [sys.executable, "-I", str(ROOT / "tests/fixtures/full_runtime_service.py")]
+ return original_command(operation, *args)
+
+ monkeypatch.setattr(trial, "_command", command)
+ before = dict(os.environ)
+ with trial:
+ assert trial._result["controller_health"]["service"] == "vibe-remote-internal"
+ agents = trial._call("cli", "agent", "list")
+ assert any(agent["backend"] == "vibey" for agent in agents["agents"])
+ key = None if scenario == "missing_credential" else "fixture-private-key-not-real"
+ trial.execute("Complete the synthetic tool-write exercise.", key=key)
+ if scenario == "success":
+ assert trial._result["outcome"] == "succeeded", trial._result
+ assert trial._result["run_id"] and trial._result["session_id"]
+ assert (task / "笔记.md").read_text(encoding="utf-8") == "可查询的任务笔记"
+ with sqlite3.connect(task / "task.sqlite") as db:
+ assert db.execute("SELECT value FROM evidence").fetchall() == [("task-data",)]
+ env = json.loads((task / "task-env.json").read_text())
+ assert env["cwd"] == str(task)
+ assert env["session"] == trial._result["session_id"] # real Harness identity
+ assert env["credential_present"] is False
+ assert env["java_home"] == "/fixture/java-toolchain"
+ assert Path(env["home"]) == trial.home
+ elif scenario == "watch":
+ assert trial._result["outcome"] == "succeeded", trial._result
+ assert trial._result["harness_quiescent"] is True
+ assert (task / "watch-note.txt").read_text() == "continued"
+ elif scenario == "deadline":
+ assert trial._result["outcome"] == "time_limit", trial._result
+ assert trial._result["run_id"]
+ assert (task / "still-writing").exists(), "the real tool must run before the deadline"
+ elif scenario in {"empty-response", "thinking-length"}:
+ assert trial._result["outcome"] == "failed", trial._result
+ assert trial._result["harness_quiescent"] is True
+ expected_error = "output limit" if scenario == "thinking-length" else "empty reply"
+ assert expected_error in trial._result["final_text"]
+ else:
+ assert trial._result["outcome"] == "blocked"
+ assert trial._result["blocker"] == "model_source_credential_required"
+ assert trial._result["run_id"] is None
+ assert (trial.avibe_home / "state" / "vibe.sqlite").exists()
+ assert not (trial.avibe_home / "runtime" / "vibe-ui.pid").exists()
+ assert not (trial.avibe_home / "runtime" / "remote-access-cloudflared.pid").exists()
+ assert dict(os.environ) == before
+ expected = {"config.py", "笔记.md", "task.sqlite", "task-env.json"} if scenario in {"success", "empty-response", "thinking-length"} else {"config.py"}
+ if scenario == "watch":
+ expected.add("watch-note.txt")
+ if scenario == "deadline":
+ expected.add("still-writing")
+ final = (task / "still-writing").read_bytes()
+ time.sleep(.15)
+ assert (task / "still-writing").read_bytes() == final, "no late write may race the grader"
+ assert {p.name for p in task.iterdir()} == expected
+ report_text = (trial.output / "execution.json").read_text(encoding="utf-8")
+ report = json.loads(report_text)
+ assert "fixture-private-key-not-real" not in report_text
+ if scenario == "success":
+ assert "[REDACTED]" in report["final_text"]
+ if scenario in {"success", "empty-response", "thinking-length"}:
+ diagnostic = report["diagnostics"]
+ assert diagnostic["status"] == "available", diagnostic
+ last = diagnostic["responses"][0]
+ assert last["stop_reason"] == ("length" if scenario == "thinking-length" else "stop")
+ assert last["thinking_blocks"] == int(scenario == "thinking-length")
+ assert last["text_blocks"] == int(scenario == "success")
+ assert last["tool_calls"] == 0
+ assert "private-reasoning" not in json.dumps(diagnostic)
+ assert report["cleanup"]["owned_processes_stopped"] is True
+ assert report["verifier_reward"] is None
+ assert report["accounting_complete"] is False
+ assert {p.name for p in trial.output.iterdir()} == {"execution.json"}
+ assert not trial._processes._live()
+
+
[email protected]_foreign_signals("Supervisor owns the spawned process and its captured setsid child")
+def test_owned_cleanup_reaps_detached_child_and_preserves_bystander(tmp_path):
+ marker = "owned-cleanup-fixture"
+ pidfile = tmp_path / "child-pid"
+ child = "import time; time.sleep(60)"
+ program = (
+ "import subprocess,sys,time; from pathlib import Path; "
+ f"p=subprocess.Popen([sys.executable,'-c',{child!r}], start_new_session=True); "
+ "Path(sys.argv[1]).write_text(str(p.pid)); time.sleep(60)"
+ )
+ env = {"PATH": os.defpath, "AVIBE_BENCHMARK_OWNER": marker}
+ process = subprocess.Popen([sys.executable, "-c", program, str(pidfile)], env=env, start_new_session=True)
+ bystander = subprocess.Popen([sys.executable, "-c", child], start_new_session=True)
+ owner = OwnedProcesses(process, marker)
+ try:
+ deadline = time.monotonic() + 5
+ while not pidfile.exists() and time.monotonic() < deadline:
+ time.sleep(.02)
+ assert pidfile.exists()
+ child_pid = int(pidfile.read_text())
+ # A clock shift must not make an owned, still-live process disappear
+ # from verification. A reused unrelated pid must not become authority.
+ with owner._lock:
+ owner.identities[process.pid] -= .25
+ owner.identities[bystander.pid] = psutil.Process(bystander.pid).create_time() - 1.0
+ assert process.pid in {p.pid for p in owner._live()}
+ assert bystander.pid not in {p.pid for p in owner._live()}
+ result = owner.close(.1)
+ assert result["owned_processes_stopped"] is True
+ assert not psutil.pid_exists(child_pid) or psutil.Process(child_pid).status() == psutil.STATUS_ZOMBIE
+ assert bystander.poll() is None
+ finally:
+ if process.poll() is None:
+ owner.close(.1)
+ bystander.kill()
+ bystander.wait(timeout=5)
+
+
[email protected]("stage, failure, code", [
+ ("startup", "timeout", "initialize_helper_timeout"),
+ ("poll", "timeout", "cli_helper_timeout"),
+ ("poll", "exit", "cli_helper_failed_2"),
+ ("poll", "signal", "cli_helper_failed_-9"),
+ # Run-show validates the durable Run payload, not just generic helper JSON.
+ ("poll", "json", "cli_helper_invalid_run_payload"),
+ ("poll", "quiescence", "harness_quiescence_unconfirmed"),
+ ("poll", "unsafe_trial_error", "unclassified_trial_error"),
+ ("poll", "upstream_exception", None),
+])
+def test_failure_report_preserves_only_wrapper_codes(tmp_path, monkeypatch, capsys, stage, failure, code):
+ """The CLI/report boundary lost safe codes; raw exception bodies can leak keys.
+
+ The existing subprocess happy-path test cannot expose that loss. Inject the
+ helper failure, but use real _call conversion, context exit, report and main.
+ """
+ from scripts.terminal_bench import runner
+
+ private = "private-upstream-error-not-for-export"
+ task = tmp_path / "task"
+ task.mkdir()
+ profile_file = tmp_path / "profile.json"
+ profile_file.write_text(json.dumps(PROFILE.payload()))
+ instruction = tmp_path / "instruction.txt"
+ instruction.write_text("fixture")
+ monkeypatch.setenv(PROFILE.credential_env, "fixture-key")
+ if stage == "poll":
+ monkeypatch.setattr(FullRuntimeTrial, "start", lambda self: self._result.update(outcome="ready"))
+
+ def helper(argv, **kwargs):
+ is_poll = "runs" in argv and "show" in argv
+ if stage == "startup" or is_poll:
+ if failure == "timeout":
+ raise subprocess.TimeoutExpired(argv, kwargs["timeout"], output=private, stderr=private)
+ if failure in {"exit", "signal"}:
+ return subprocess.CompletedProcess(argv, 2 if failure == "exit" else -9, private, private)
+ if failure == "json":
+ return subprocess.CompletedProcess(argv, 0, private, "")
+ if failure == "unsafe_trial_error":
+ raise TrialError(private)
+ if failure == "upstream_exception":
+ raise ValueError(private)
+ body = {"run": {"status": "succeeded", "result_text": "fixture"}}
+ elif "agent" in argv and "run" in argv:
+ body = {"run_id": "fixture-run", "session_id": "fixture-session"}
+ elif "harness" in argv:
+ body = {"controller": {"available": False}}
+ else:
+ body = {}
+ return subprocess.CompletedProcess(argv, 0, json.dumps(body), "")
+
+ monkeypatch.setattr(subprocess, "run", helper)
+ assert runner.main([
+ "--workdir", str(task), "--state", str(tmp_path / "private-state"),
+ "--output", str(tmp_path / "output"), "--profile", str(profile_file),
+ "--instruction", str(instruction),
+ ]) == 2
+ captured = capsys.readouterr()
+ report_text = (tmp_path / "output" / "execution.json").read_text()
+ assert private not in report_text + captured.out + captured.err
+ for payload in (json.loads(report_text), json.loads(captured.out), json.loads(captured.err)):
+ assert payload["outcome"] == "infrastructure_error"
+ assert payload["error_type"] == ("ValueError" if failure == "upstream_exception" else "TrialError")
+ if code is None:
+ assert "error_code" not in payload
+ else:
+ assert payload["error_code"] == code
diff --git a/tests/test_terminal_bench_harbor.py b/tests/test_terminal_bench_harbor.py
new file mode 100644
index 000000000..717cc78eb
--- /dev/null
+++ b/tests/test_terminal_bench_harbor.py
@@ -0,0 +1,287 @@
+"""Optional real-Harbor boundary tests, no container/provider calls.
+
+Normal product CI skips this file when Harbor is not installed. The explicit
+benchmark-host test command installs pinned requirements-harbor.txt separately.
+The fake Environment is a transport recorder; real Harbor construction, profile,
+wheel verification, source snapshot, cancellation and result projection run.
+"""
+from __future__ import annotations
+
+import asyncio
+from hashlib import sha256
+from importlib.metadata import version
+import json
+from pathlib import Path
+from types import SimpleNamespace
+import subprocess
+import zipfile
+
+import pytest
+
+pytest.importorskip("harbor.agents.installed.base")
+if version("harbor") != "0.24.0":
+ pytest.skip("optional boundary requires pinned Harbor 0.24.0", allow_module_level=True)
+
+from harbor.models.agent.context import AgentContext
+from scripts.terminal_bench import runner
+from scripts.terminal_bench.harbor_agent import FullAvibeAgent
+from scripts.terminal_bench.runner import FullRuntimeTrial
+
+
+# These are fixture coordination bounds, not Agent/product timeouts. Cold HD1
+# writes and event-loop scheduling can take longer than the former 2s barrier.
+_SETUP_BARRIER_SECONDS = 60
+
+
+async def _finish_pending(pending, env):
+ """A failed setup assertion must not leave a shielded upload/task behind."""
+ env.allow_upload.set()
+ if not pending.done():
+ pending.cancel()
+ await asyncio.wait_for(asyncio.gather(pending, return_exceptions=True), _SETUP_BARRIER_SECONDS)
+
+
+def wheel(tmp_path):
+ path = tmp_path / "avibe_os-9.9.9-py3-none-any.whl"
+ with zipfile.ZipFile(path, "w") as archive:
+ archive.writestr("avibe_os-9.9.9.dist-info/METADATA", "Name: avibe-os\nVersion: 9.9.9\n")
+ return path, sha256(path.read_bytes()).hexdigest()
+
+
+def make_agent(tmp_path, wheel, **changes):
+ path, digest = wheel
+ options = {"logs_dir": tmp_path / "agent-logs", "model_name": "fixture-model",
+ "wheel_path": str(path), "wheel_sha256": digest,
+ "profile": {"model": "fixture-model", "base_url": "http://127.0.0.1:9/v1",
+ "setup_seconds": 20, "wall_seconds": 5, "cleanup_seconds": 1}}
+ options.update(changes)
+ return FullAvibeAgent(**options)
+
+
+class Environment:
+ default_user = None
+
+ def __init__(self, *, fail_cleanup=False, block=False):
+ self.commands = []
+ self.uploads = {}
+ self.files = {}
+ self.report = {
+ "outcome": "succeeded", "run_id": "fixture-run", "session_id": "fixture-session",
+ "cleanup": {"owned_processes_stopped": True}, "accounting_complete": False,
+ "usage": {"totals": {"token_reports": 2, "input_tokens": 100,
+ "cached_input_tokens": 25, "output_tokens": 30}},
+ }
+ self.fail_cleanup, self.block = fail_cleanup, block
+ self.started = asyncio.Event()
+ self.cleanup_complete = False
+ self.downloaded = False
+ self.key_removed = False
+ self.secret_upload_modes = None
+ self.upload_error = False
+ self.block_upload = False
+ self.upload_started = asyncio.Event()
+ self.allow_upload = asyncio.Event()
+
+ async def upload_file(self, source, target):
+ path = Path(source)
+ if target.endswith("credential/provider.key"):
+ self.secret_upload_modes = (path.stat().st_mode & 0o777, path.parent.stat().st_mode & 0o777)
+ self.upload_started.set()
+ if self.block_upload:
+ await self.allow_upload.wait()
+ self.files[target] = path.read_bytes()
+ if self.upload_error:
+ raise OSError("synthetic upload failed after write")
+ else:
+ self.files[target] = path.read_bytes()
+
+ async def upload_dir(self, source, target):
+ self.uploads[target] = {p.relative_to(source).as_posix(): p.read_bytes()
+ for p in Path(source).rglob("*") if p.is_file()}
+
+ async def exec(self, command, **kwargs):
+ self.commands.append((command, kwargs))
+ if command == "id -u":
+ return SimpleNamespace(return_code=0, stdout="0\n", stderr="")
+ if command == "pwd -P":
+ return SimpleNamespace(return_code=0, stdout="/app\n", stderr="")
+ if "scripts.terminal_bench.credential prepare" in command:
+ return SimpleNamespace(return_code=0, stdout=json.dumps({"regular": True, "mode": "0600", "directory_mode": "0700"}), stderr="")
+ if "scripts.terminal_bench.credential clean" in command:
+ self.key_removed = True
+ for name in list(self.files):
+ if name.endswith("credential/provider.key"):
+ self.files.pop(name)
+ return SimpleNamespace(return_code=0, stdout=json.dumps({"removed": True}), stderr="")
+ if "scripts.terminal_bench.launch run" in command:
+ self.started.set()
+ if self.block:
+ await asyncio.Event().wait()
+ if "scripts.terminal_bench.launch stop" in command:
+ await asyncio.sleep(.01)
+ self.cleanup_complete = not self.fail_cleanup
+ return SimpleNamespace(return_code=2 if self.fail_cleanup else 0,
+ stdout=json.dumps({"quiescent": not self.fail_cleanup, "started": self.started.is_set()}), stderr="")
+ return SimpleNamespace(return_code=0, stdout="", stderr="")
+
+ async def download_file(self, source, target):
+ assert self.cleanup_complete, "grading artifact download cannot precede owned shutdown"
+ assert source.endswith("execution.json"), "raw service state may not be exported"
+ self.downloaded = True
+ Path(target).write_text(json.dumps(self.report))
+
+
+async def test_verified_wheel_and_only_wrapper_are_installed(tmp_path, wheel, monkeypatch):
+ agent = make_agent(tmp_path, wheel)
+ env = Environment()
+ dependencies = []
+
+ async def ensure(environment, names):
+ assert environment is env
+ dependencies.extend(names)
+
+ monkeypatch.setattr(agent, "ensure_system_dependencies", ensure)
+ await agent.install(env)
+ assert set(dependencies) == {"python3", "python_venv"}
+ transferred = next(iter(env.uploads.values()))
+ assert set(transferred) == {
+ "scripts/__init__.py", "scripts/terminal_bench/__init__.py",
+ "scripts/terminal_bench/profile.py", "scripts/terminal_bench/control.py",
+ "scripts/terminal_bench/runner.py", "scripts/terminal_bench/launch.py", "scripts/terminal_bench/credential.py",
+ "scripts/terminal_bench/diagnostics.py",
+ }
+ assert agent.provenance["wheel_sha256"] == wheel[1]
+ assert agent.provenance["avibe_version"] == "9.9.9"
+ assert agent.capabilities.atif is False
+ assert any("-m pip install" in command and wheel[0].name in command for command, _ in env.commands)
+ assert not any("editable" in command or "PYTHONPATH" in command for command, _ in env.commands)
+
+
+async def test_nonroot_task_refused_without_silent_ownership_change(tmp_path, wheel, monkeypatch):
+ agent = make_agent(tmp_path, wheel)
+ env = Environment()
+
+ async def nonroot(command, **kwargs):
+ assert command == "id -u"
+ return SimpleNamespace(return_code=0, stdout="1000\n", stderr="")
+
+ monkeypatch.setattr(env, "exec", nonroot)
+ with pytest.raises(RuntimeError, match="root task user"):
+ await agent.install(env)
+ assert env.files == {} and env.uploads == {}
+
+
+async def test_hash_mismatch_fails_before_task_mutation(tmp_path, wheel):
+ agent = make_agent(tmp_path, wheel, wheel_sha256="0" * 64)
+ env = Environment()
+ with pytest.raises(ValueError, match="SHA256 mismatch"):
+ await agent.install(env)
+ assert env.commands == [] and env.files == {} and env.uploads == {}
+
+
[email protected]("cancel, fail_cleanup", [(False, False), (True, False), (True, True)])
+async def test_run_joins_cancel_before_artifact_handoff(tmp_path, wheel, monkeypatch, cancel, fail_cleanup):
+ agent = make_agent(tmp_path, wheel)
+ agent.provenance = {"wheel_sha256": wheel[1], "atif": False}
+ monkeypatch.setenv("DEEPSEEK_API_KEY", "not-real-credential")
+ env = Environment(block=cancel, fail_cleanup=fail_cleanup)
+ ctx = AgentContext()
+ pending = asyncio.create_task(agent.run("Write the requested note.", env, ctx))
+ try:
+ if cancel:
+ await asyncio.wait_for(env.started.wait(), _SETUP_BARRIER_SECONDS)
+ pending.cancel()
+ with pytest.raises(RuntimeError if fail_cleanup else asyncio.CancelledError):
+ await pending
+ else:
+ await pending
+ finally:
+ await _finish_pending(pending, env)
+ if fail_cleanup:
+ assert not env.downloaded
+ else:
+ assert env.cleanup_complete and env.downloaded
+ assert ctx.n_input_tokens == 100
+ assert ctx.n_cache_tokens == 25
+ assert ctx.n_output_tokens == 30
+ assert ctx.cost_usd is None
+ assert ctx.metadata["accounting_complete"] is False
+ assert ctx.metadata["verifier_reward"] is None
+ start_command, options = next((cmd, opts) for cmd, opts in env.commands
+ if "scripts.terminal_bench.launch run" in cmd)
+ assert "not-real-credential" not in start_command
+ assert options.get("env", {}) == {}
+ assert "--credential-file" in start_command
+ assert env.secret_upload_modes == (0o600, 0o700)
+ assert env.key_removed
+ assert all("not-real-credential" not in cmd + json.dumps(opts) for cmd, opts in env.commands)
+ assert options["cwd"] == str(agent.root)
+ assert "--sandbox-environment" in start_command
+ assert "--workdir /app" in start_command
+ assert options["timeout_sec"] > agent.profile.wall_seconds + agent.profile.cleanup_seconds
+ assert not any("not-real-credential".encode() in data for data in env.files.values())
+
+
[email protected]("cancel", [False, True])
+async def test_upload_failure_or_cancel_joins_then_removes_staged_key(tmp_path, wheel, monkeypatch, cancel):
+ agent = make_agent(tmp_path, wheel)
+ agent.provenance = {"wheel_sha256": wheel[1]}
+ monkeypatch.setenv("DEEPSEEK_API_KEY", "not-real-credential")
+ env = Environment()
+ env.upload_error = not cancel
+ env.block_upload = cancel
+ pending = asyncio.create_task(agent.run("fixture", env, AgentContext()))
+ try:
+ if cancel:
+ await asyncio.wait_for(env.upload_started.wait(), _SETUP_BARRIER_SECONDS)
+ pending.cancel()
+ await asyncio.sleep(.01)
+ assert not env.key_removed, "in-flight transfer must settle before cleanup"
+ env.allow_upload.set()
+ with pytest.raises(asyncio.CancelledError if cancel else OSError):
+ await pending
+ finally:
+ await _finish_pending(pending, env)
+ assert env.key_removed
+ assert not env.started.is_set()
+ assert not env.downloaded
+ assert not any("not-real-credential".encode() in data for data in env.files.values())
+ assert all("not-real-credential" not in cmd + json.dumps(opts) for cmd, opts in env.commands)
+
+
+
+
+def test_runs_show_nonzero_valid_json_is_a_product_failure_not_helper_failure(tmp_path, monkeypatch):
+ trial = object.__new__(FullRuntimeTrial)
+ trial.state = tmp_path
+ trial.env = {"AVIBE_HOME": str(tmp_path / "home")}
+ trial._command = lambda operation, *args: ["vibe", operation, *args]
+ sentinel = "provider-secret-text-not-for-export"
+
+ def show(argv, **kwargs):
+ return subprocess.CompletedProcess(
+ argv, 1,
+ json.dumps({"run": {"id": "run-1", "status": "failed",
+ "error": f"Automatic recovery has ended: {sentinel}",
+ "result_text": "safe failure"}}),
+ "",
+ )
+
+ monkeypatch.setattr(subprocess, "run", show)
+ payload = trial._call_run_show("run-1")
+ assert payload["run"]["status"] == "failed"
+ assert payload["run"]["error"].endswith(sentinel)
+ scrubbed = runner.scrub(payload, sentinel)
+
+
+def test_explicit_model_must_match_profile(tmp_path, wheel):
+ with pytest.raises(ValueError, match="model_name"):
+ make_agent(tmp_path, wheel, model_name="different-model")
+ with pytest.raises(ValueError, match="host environment"):
+ make_agent(tmp_path, wheel, extra_env={"DEEPSEEK_API_KEY": "sentinel-not-config-or-argv"})
diff --git a/tests/test_terminal_bench_launch.py b/tests/test_terminal_bench_launch.py
new file mode 100644
index 000000000..df182ba39
--- /dev/null
+++ b/tests/test_terminal_bench_launch.py
@@ -0,0 +1,60 @@
+"""Local process acceptance for Harbor's outer cancel fence; no Docker/Harbor."""
+from __future__ import annotations
+
+import json
+import os
+from pathlib import Path
+import subprocess
+import sys
+import time
+
+import pytest
+
+from scripts.terminal_bench.launch import run, stop
+
+
+def test_late_start_is_fenced_and_untrusted_identity_not_signalled(tmp_path):
+ root = tmp_path / "launch"
+ root.mkdir()
+ assert stop(root, "nonce", tmp_path / "execution.json", .1) == {"quiescent": True, "started": False}
+ assert run(root, "nonce", ["must-not-start"]) == 125
+ (root / "starting").touch()
+ with pytest.raises(RuntimeError, match="identity_not_recorded"):
+ stop(root, "nonce", tmp_path / "execution.json", .1)
+ (root / "wrapper-process.json").write_text(json.dumps({"nonce": "other", "pid": os.getpid(), "created": 0}))
+ with pytest.raises(RuntimeError, match="identity_mismatch"):
+ stop(root, "nonce", tmp_path / "execution.json", .1)
+
+
+def test_stop_joins_exact_wrapper_and_demands_inner_cleanup_receipt(tmp_path):
+ import psutil
+ root = tmp_path / "launch"
+ root.mkdir()
+ receipt = tmp_path / "execution.json"
+ ready = tmp_path / "ready"
+ code = (
+ "import sys,json,signal,time; from pathlib import Path; "
+ "signal.signal(signal.SIGTERM, lambda *_: (Path(sys.argv[1]).write_text(json.dumps("
+ "{'cleanup':{'owned_processes_stopped':True}})),sys.exit(0))); "
+ "Path(sys.argv[2]).touch(); time.sleep(60)"
+ )
+ process = subprocess.Popen([sys.executable, "-c", code, str(receipt), str(ready)],
+ env={"PATH": os.defpath, "AVIBE_BENCHMARK_LAUNCH_ID": "nonce"}, start_new_session=True)
+ try:
+ deadline = time.monotonic() + 3
+ while not ready.exists() and time.monotonic() < deadline:
+ time.sleep(.01)
+ assert ready.exists()
+ (root / "wrapper-process.json").write_text(json.dumps({
+ "nonce": "nonce", "pid": process.pid, "created": psutil.Process(process.pid).create_time(),
+ }))
+ assert stop(root, "nonce", receipt, 2)["quiescent"] is True
+ process.wait(timeout=3)
+ receipt.unlink()
+ with pytest.raises(RuntimeError, match="receipt_missing"):
+ stop(root, "nonce", receipt, .1)
+ finally:
+ if process.poll() is None:
+ process.kill()
+ process.wait(timeout=3)
diff --git a/tests/test_terminal_bench_remote_job.py b/tests/test_terminal_bench_remote_job.py
new file mode 100644
index 000000000..611da894f
--- /dev/null
+++ b/tests/test_terminal_bench_remote_job.py
@@ -0,0 +1,338 @@
+"""Remote owner fences and cleanup failure classes; no provider calls."""
+import io
+import json
+from pathlib import Path
+import signal
+import subprocess
+import sys
+from types import SimpleNamespace
+
+import pytest
+
+pytest.importorskip("fcntl")
+from scripts.terminal_bench import remote_job as job
+
+
[email protected](autouse=True)
+def host_scope(tmp_path, monkeypatch):
+ monkeypatch.setattr(job, "HOST_RESERVATION", tmp_path / "host-reservation")
+ monkeypatch.setenv("SERVICE_RESULT", "success")
+ monkeypatch.setattr(job, "archive_artifact", lambda cid, source, destination, timeout: destination.write_bytes(b"opaque-fixture"))
+
+
+def config(tmp_path):
+ task = tmp_path / "task"
+ task.mkdir()
+ (task / "task.toml").write_text('artifacts=["/app/result.txt"]\n')
+ data = {"job_name": "one-job", "jobs_dir": str(tmp_path / "jobs"), "n_attempts": 1, "n_concurrent_trials": 1,
+ "agents": [{"name": "nop", "override_timeout_sec": 2}], "tasks": [{"path": str(task)}],
+ "environment": {"type": "docker", "delete": True}}
+ p = tmp_path / "config.json"
+ job.save(p, data)
+ return p
+
+
+def prepared(tmp_path, seconds=5):
+ p = config(tmp_path)
+ root = tmp_path / "runs" / "one"
+ job.prepare(p, root, seconds, "BENCHMARK_API_KEY")
+ (root / "credential").mkdir(mode=0o700)
+ key = root / "credential/provider.key"
+ key.write_text("not-real-secret")
+ key.chmod(0o600)
+ job.save(root / "ownership.json", {"trial_name": "task__nonce", "trial_id": "trial-1", "job_id": "job-1", "projects": ["task_nonce_env"]})
+ return root
+
+
+def complete(root):
+ m = job.load(root / "manifest.json")
+ trial = Path(m["job_dir"]) / "task__nonce"
+ trial.mkdir(parents=True, exist_ok=True)
+ job.save(trial / "result.json", {"id": "trial-1", "trial_name": "task__nonce", "finished_at": "2026-10-14T00:00:00Z"})
+ job.save(trial.parent / "result.json", {"id": "job-1", "finished_at": "2026-10-14T00:00:00Z", "n_total_trials": 1,
+ "stats": {"n_completed_trials": 1, "n_running_trials": 0, "n_pending_trials": 0}})
+
+
+class Docker:
+ def __init__(self):
+ self.containers = [{"Id": "a" * 64, "Config": {"Labels": {"com.docker.compose.project": "task_nonce_env"}}},
+ {"Id": "b" * 64, "Config": {"Labels": {"com.docker.compose.project": "bystander"}}}]
+ self.networks = [{"Id": "owned-net", "Labels": {"com.docker.compose.project": "task_nonce_env"}},
+ {"Id": "bystander-net", "Labels": {"com.docker.compose.project": "bystander"}}]
+ self.calls = []
+
+ def __call__(self, argv, **kwargs):
+ self.calls.append((argv, kwargs))
+ if argv[1:3] == ["ps", "-aq"]:
+ return "\n".join(c["Id"] for c in self.containers)
+ if argv[1] == "inspect":
+ return json.dumps(self.containers)
+ if argv[1:3] == ["rm", "-f"]:
+ self.containers[:] = [c for c in self.containers if c["Id"] != argv[-1]]
+ return ""
+ if argv[1:3] == ["network", "ls"]:
+ return "\n".join(n["Id"] for n in self.networks)
+ if argv[1:3] == ["network", "inspect"]:
+ return json.dumps(self.networks)
+ if argv[1:3] == ["network", "rm"]:
+ self.networks[:] = [n for n in self.networks if n["Id"] != argv[-1]]
+ return ""
+ if argv[1] == "cp":
+ return ""
+ raise AssertionError(argv)
+
+
+def test_preflight_rejects_paths_and_budget_before_any_job(tmp_path):
+ p = config(tmp_path)
+ d = job.load(p)
+ for path in ["task", "/Volumes/HD1/absent-task"]:
+ d["tasks"][0]["path"] = path
+ job.save(p, d)
+ with pytest.raises((ValueError, FileNotFoundError)):
+ job.prepare(p, tmp_path / "state", 2)
+ assert not job.HOST_RESERVATION.exists()
+ d["tasks"][0]["path"] = str(tmp_path / "task")
+ d["n_attempts"] = 2
+ job.save(p, d)
+ with pytest.raises(ValueError):
+ job.validate(p)
+
+
+def test_profile_credential_name_is_derived_and_mismatch_refused(tmp_path):
+ import hashlib
+ p = config(tmp_path)
+ wheel = tmp_path / "avibe.whl"
+ wheel.write_bytes(b"fixture-only-wheel")
+ d = job.load(p)
+ d["agents"] = [{"import_path": "scripts.terminal_bench.harbor_agent:FullAvibeAgent", "model_name": "fixture",
+ "override_timeout_sec": 30, "kwargs": {"wheel_path": str(wheel),
+ "wheel_sha256": hashlib.sha256(wheel.read_bytes()).hexdigest(),
+ "profile": {"model": "fixture", "base_url": "http://127.0.0.1:9/v1", "credential_env": "ACTUAL_PROVIDER_KEY"}}}]
+ job.save(p, d)
+ assert job.validate(p)[3] == "ACTUAL_PROVIDER_KEY"
+ with pytest.raises(ValueError, match="credential variable"):
+ job.validate(p, "WRONG_KEY")
+ assert not job.HOST_RESERVATION.exists()
+
+
+def test_lock_is_host_wide_and_destination_claim_is_permanent(tmp_path):
+ root = prepared(tmp_path)
+ other = tmp_path / "totally-different-parent" / "second"
+ with pytest.raises(FileExistsError):
+ job.prepare(tmp_path / "config.json", other, 2)
+ assert not other.exists()
+ job.release(root)
+ with pytest.raises(FileExistsError):
+ job.prepare(tmp_path / "config.json", other, 2)
+ assert not job.HOST_RESERVATION.exists()
+
+
[email protected]("broken_result", ["{truncated", "[]", "null", '{"finished_at":"not-a-trial"}'])
+def test_bad_results_never_skip_exact_owned_cleanup(tmp_path, monkeypatch, broken_result):
+ root = prepared(tmp_path)
+ manifest = job.load(root / "manifest.json")
+ jobdir = Path(manifest["job_dir"])
+ jobdir.mkdir(parents=True)
+ (jobdir / "result.json").write_text(broken_result)
+ job.save(root / "lifecycle.json", {"state": "completed", "exit_code": 0})
+ docker = Docker()
+ monkeypatch.setattr(job, "command", docker)
+ receipt = job.finalize(root)
+ assert receipt["state"] == "missing_terminal_result"
+ assert receipt["harbor_finished"] is False
+ assert receipt["cleanup"]["confirmed"] is True
+ assert [c["Id"] for c in docker.containers] == ["b" * 64]
+ assert [n["Id"] for n in docker.networks] == ["bystander-net"]
+ assert not job.HOST_RESERVATION.exists()
+ assert not (root / "credential/provider.key").exists()
+
+
+def test_network_only_and_bad_artifact_metadata_are_cleaned(tmp_path, monkeypatch):
+ root = prepared(tmp_path)
+ (tmp_path / "task/task.toml").write_text("broken = [")
+ docker = Docker()
+ docker.containers = []
+ monkeypatch.setattr(job, "command", docker)
+ receipt = job.finalize(root)
+ assert receipt["cleanup"]["confirmed"]
+ assert docker.networks[0]["Id"] == "bystander-net"
+ assert receipt["artifacts"] == [{"status": "metadata_unavailable"}]
+
+
+def test_cleanup_failure_retains_reservation_and_receipt(tmp_path, monkeypatch):
+ root = prepared(tmp_path)
+ monkeypatch.setattr(job, "command", lambda *a, **k: (_ for _ in ()).throw(subprocess.TimeoutExpired("docker", 1)))
+ receipt = job.finalize(root)
+ assert receipt["cleanup"]["confirmed"] is False
+ assert job.HOST_RESERVATION.exists()
+ assert not (root / "credential/provider.key").exists()
+ assert receipt["finished_at"]
+
+
+def test_completion_requires_exact_trial_and_no_pending_work(tmp_path):
+ root = prepared(tmp_path)
+ m, o = job.load(root / "manifest.json"), job.load(root / "ownership.json")
+ complete(root)
+ assert job.terminal_evidence(m, o)
+ assert not job.terminal_evidence(m, {**o, "trial_id": "some-other-trial"})
+ p = Path(m["job_dir"]) / "result.json"
+ d = job.load(p); d["stats"]["n_running_trials"] = 1; job.save(p, d)
+ assert not job.terminal_evidence(m, o)
+
+
[email protected]("chunk_size", [1, 3, 4096])
+def test_stream_redaction_handles_split_keys(tmp_path, chunk_size):
+ class Chunks(io.StringIO):
+ def read(self, n):
+ return super().read(min(n, chunk_size))
+ p = tmp_path / "log"
+ job.redact_stream(Chunks("before-secret-非ASCII-after\n"), p, "secret-非ASCII")
+ assert p.read_text() == "before-[REDACTED]-after\n"
+ assert p.stat().st_mode & 0o777 == 0o600
+
+
[email protected]_foreign_signals("Supervisor owns this test child process group")
[email protected]("mode,expected", [("ok", "completed"), ("missing", "missing_terminal_result"), ("crash", "harbor_error"), ("slow", "time_limit")])
+def test_worker_never_releases_slot_before_post_stop(tmp_path, monkeypatch, mode, expected):
+ root = prepared(tmp_path, seconds=1 if mode == "slow" else 10)
+ program = tmp_path / "child-python"
+ program.write_text(f'''#!{sys.executable}
+import sys,time
+print('not-real-secret',flush=True)
+if {mode!r}=='slow': time.sleep(30)
+if {mode!r}=='crash': sys.exit(3)
+''')
+ program.chmod(0o700)
+ if mode == "ok":
+ complete(root)
+ monkeypatch.setattr(job, "sys", SimpleNamespace(executable=str(program)))
+ before = signal.getsignal(signal.SIGTERM)
+ code = job.worker(root)
+ assert not (root / "receipt.json").exists()
+ assert job.HOST_RESERVATION.exists()
+ assert signal.getsignal(signal.SIGTERM) == before
+ monkeypatch.setattr(job, "command", Docker())
+ receipt = job.finalize(root)
+ assert receipt["state"] == expected
+ assert code == (0 if mode in {"ok", "missing"} else 1)
+ assert "not-real-secret" not in (root / "harbor.log").read_text()
+ assert not job.HOST_RESERVATION.exists()
+
+
+def test_finalize_refuses_live_owner_and_status_does_not_wait_on_dead_unit(tmp_path, monkeypatch):
+ root = prepared(tmp_path)
+ monkeypatch.delenv("SERVICE_RESULT")
+ with pytest.raises(ValueError):
+ job.finalize(root)
+ monkeypatch.setattr(job, "unit_state", lambda unit: {"ActiveState": "failed"})
+ result, code = job.status(root)
+ assert code == 1 and result["cleanup_confirmed"] is False
+
+
+def test_lost_launch_ack_fences_unit_without_publishing_clean_receipt(tmp_path, monkeypatch):
+ p = config(tmp_path)
+ private = tmp_path / "credential"; private.mkdir(mode=0o700)
+ key = private / "provider.key"; key.write_text("KEY-SENTINEL"); key.chmod(0o600)
+ calls = []
+ def manager(argv, **kwargs):
+ calls.append(argv)
+ if argv[0] == "systemd-run":
+ raise subprocess.TimeoutExpired("systemd-run", 10)
+ if argv[:2] == ["systemctl", "show"]:
+ return "LoadState=loaded\nActiveState=active\n"
+ return ""
+ monkeypatch.setattr(job, "command", manager)
+ root = tmp_path / "runs/one"
+ with pytest.raises(subprocess.TimeoutExpired):
+ job.submit(p, root, key, 5)
+ assert not (root / "receipt.json").exists()
+ assert job.HOST_RESERVATION.exists()
+ assert calls[-1][:3] == ["systemctl", "stop", "--no-block"]
+ assert "KEY-SENTINEL" not in json.dumps(calls)
+ assert any(a.startswith("--property=ExecStopPost=") for a in calls[0])
+
+
+def test_artifact_recovery_stops_at_its_total_budget(tmp_path, monkeypatch):
+ root = prepared(tmp_path)
+ (tmp_path / "task/task.toml").write_text('artifacts=' + json.dumps(['/app/f' + str(i) for i in range(32)]))
+ clock = [0.0]
+ monkeypatch.setattr(job.time, "monotonic", lambda: clock[0])
+ calls = []
+ def copy(cid, source, destination, timeout):
+ calls.append(source); clock[0] += timeout
+ raise subprocess.TimeoutExpired("docker cp", timeout)
+ monkeypatch.setattr(job, "archive_artifact", copy)
+ result = job.copy_artifacts(root, job.load(root / "manifest.json"), [{"Id": "a" * 64}], 8)
+ assert clock[0] <= 8 and len(calls) <= 3
+ assert result[-1]["status"] == "recovery_budget_exhausted"
+
+
[email protected]("component", ["%n", "$NAME", "a\nb"])
+def test_manager_expansion_paths_fail_before_ownership(tmp_path, component):
+ p = config(tmp_path)
+ with pytest.raises(ValueError, match="systemd"):
+ job.prepare(p, tmp_path / "runs" / component, 5)
+ assert not job.HOST_RESERVATION.exists()
+
+
+def test_manager_expansion_cannot_hide_behind_a_symlink(tmp_path):
+ p = config(tmp_path)
+ target = tmp_path / "%n"
+ target.mkdir()
+ alias = tmp_path / "alias"
+ alias.symlink_to(target, target_is_directory=True)
+ with pytest.raises(ValueError, match="systemd"):
+ job.prepare(p, alias / "job", 5)
+ assert not job.HOST_RESERVATION.exists()
+
+
+def test_tomli_fallback_keeps_module_collectable_on_python310(monkeypatch):
+ import builtins
+ import importlib.util
+ original = builtins.__import__
+ used = []
+ def compat(name, *args, **kwargs):
+ if name == "tomllib":
+ raise ModuleNotFoundError(name)
+ if name == "tomli":
+ used.append(name)
+ return job.tomllib
+ return original(name, *args, **kwargs)
+ monkeypatch.setattr(builtins, "__import__", compat)
+ spec = importlib.util.spec_from_file_location("scripts.terminal_bench.compat_fixture", job.__file__)
+ module = importlib.util.module_from_spec(spec)
+ spec.loader.exec_module(module)
+ assert used == ["tomli"]
+ assert module.tomllib.loads("x=1") == {"x": 1}
+
+
+def test_overlapping_artifact_symlinks_remain_opaque_archives(tmp_path, monkeypatch):
+ import tarfile
+ root = prepared(tmp_path)
+ outside = tmp_path / "outside"
+ outside.mkdir()
+ sentinel = outside / "payload"
+ sentinel.write_text("must-stay-unchanged")
+ (tmp_path / "task/task.toml").write_text('artifacts=["/app/out","/app/out/link/payload"]')
+ destinations = []
+ def archive(cid, source, destination, timeout):
+ destinations.append(destination)
+ with tarfile.open(destination, "w") as tar:
+ if str(source) == "/app/out":
+ link = tarfile.TarInfo("out/link")
+ link.type = tarfile.SYMTYPE
+ link.linkname = str(outside)
+ tar.addfile(link)
+ else:
+ member = tarfile.TarInfo("payload")
+ data = b"overwrite-sentinel"
+ member.size = len(data)
+ tar.addfile(member, io.BytesIO(data))
+ monkeypatch.setattr(job, "archive_artifact", archive)
+ result = job.copy_artifacts(root, job.load(root / "manifest.json"), [{"Id": "a" * 64}], job.time.monotonic() + 5)
+ assert len(destinations) == 2 and destinations[0] != destinations[1]
+ assert all(p.is_relative_to(root) and p.suffix == ".tar" for p in destinations)
+ assert all(r["status"] == "archived" and r["extracted_on_host"] is False for r in result)
+ assert sentinel.read_text() == "must-stay-unchanged"
+ assert not list((root / "recovered-artifacts").rglob("link"))
diff --git a/tests/test_vibey_agent.py b/tests/test_vibey_agent.py
index 0c788a341..6e1c1244f 100644
--- a/tests/test_vibey_agent.py
+++ b/tests/test_vibey_agent.py
@@ -1211,6 +1211,25 @@ async def test_a_failed_run_shows_localized_copy_and_a_refusal_shows_its_explana
assert len(reported) == 2
[email protected]("language", ["en", "zh"])
[email protected]("content", [(), (ThinkingBlock("private reasoning"),), (text("unfinished answer"),)])
+async def test_output_limited_final_is_a_failed_row_with_length_cause(
+ engine, session, tmp_path, published, caplog, language, content
+) -> None:
+ message = AssistantMessage(content, ORIGIN, "length")
+ harness = _Harness(engine, tmp_path, "telegram", [[Done(message)]], language=language)
+ with caplog.at_level(logging.WARNING, logger="modules.agents.vibey.agent"):
+ await harness.agent.handle_message(harness.request("finish the task"))
+ [row] = harness.rows("error")
+ assert json.loads(row["content_json"])["model"]["message"]["stop_reason"] == "length"
+ assert not harness.rows("result")
+ assert harness.controller.terminals[-1]["is_error"] is True
+ assert len(harness.provider.requests) == 1 # No whole-task replay or hidden budget increase.
+ expected = "unfinished answer" if any(isinstance(b, TextBlock) for b in content) else i18n_t("vibeyAgent.error.outputLimit", language)
+ assert row["content_text"] == expected
+ assert "kind=length" in caplog.text
+
+
@pytest.mark.parametrize("language", ["en", "zh"])
async def test_a_provider_stall_is_not_shown_as_a_connection_failure(
engine, session, tmp_path, published, language
diff --git a/vibe/i18n/en.json b/vibe/i18n/en.json
index 173ab7e08..506a61c62 100644
--- a/vibe/i18n/en.json
+++ b/vibe/i18n/en.json
@@ -1724,7 +1724,7 @@
"toolsUnsupported": "The selected model does not support tools, which {backend} needs. Choose another model.",
"providerProtocol": "The model provider sent a reply {backend} cannot use.",
"unclassified": "The request to the model failed with an error {backend} could not classify. Try again, or choose another model.",
- "outputLimit": "The model kept running out of output space while writing a tool call. Ask for smaller steps, or choose a model with a larger output limit.",
+ "outputLimit": "The model reached its output limit before producing a usable answer or tool call. Ask for smaller steps, or choose a model with a larger output limit.",
"internal": "{backend} stopped because of an internal error. Try again; if it happens again, the service log names the cause.",
"generic": "{backend}'s run failed."
}
diff --git a/vibe/i18n/zh.json b/vibe/i18n/zh.json
index 5648425d0..3b1948333 100644
--- a/vibe/i18n/zh.json
+++ b/vibe/i18n/zh.json
@@ -1724,7 +1724,7 @@
"toolsUnsupported": "所选模型不支持工具,而 {backend} 需要工具。请换一个模型。",
"providerProtocol": "模型服务商返回的回复 {backend} 无法使用。",
"unclassified": "请求模型时出现了 {backend} 无法归类的错误。可以重试,或换一个模型。",
- "outputLimit": "模型在写工具调用时多次超出输出上限。请让它分成更小的步骤,或换用输出上限更大的模型。",
+ "outputLimit": "模型在生成可用的答案或工具调用前达到了输出上限。请让它分成更小的步骤,或换用输出上限更大的模型。",
"internal": "{backend} 因内部错误而停止。可以重试;如果再次出现,服务日志里记录了原因。",
"generic": "{backend} 运行失败。"
}