Personal AI, On Personal Devices
Python
hybrid: evaluate six local+cloud paradigms (minions, conductor, archon, advisors, skillorchestra, toolorchestra) (#344)
Andrew Parka7b8597856bf
diff --git a/pyproject.toml b/pyproject.tomlindex 2f568fc8..7c0c67b3 100644--- a/pyproject.toml+++ b/pyproject.toml@@ -179,6 +179,9 @@ select = ["E", "F", "I", "W"][tool.ruff.lint.per-file-ignores]"src/openjarvis/evals/datasets/*.py" = ["E501"]"src/openjarvis/evals/scorers/*.py" = ["E501"]+# hybrid/ is research code with long prompt strings and paradigm-specific+# config dicts — same relaxation as evals research code above.+"src/openjarvis/agents/hybrid/*.py" = ["E501"][dependency-groups]dev = [diff --git a/src/openjarvis/agents/__init__.py b/src/openjarvis/agents/__init__.pyindex c9c7b1bf..9fbbc23e 100644--- a/src/openjarvis/agents/__init__.py+++ b/src/openjarvis/agents/__init__.py@@ -79,6 +79,14 @@ try:except ImportError:pass+# Hybrid local+cloud paradigm agents (Minions, Conductor, Archon, Advisors,+# SkillOrchestra, ToolOrchestra). Each module registers under its own name+# via @AgentRegistry.register(). Optional deps may make some unavailable.+try:+ import openjarvis.agents.hybrid # noqa: F401+except ImportError:+ pass+# Registry alias: "react" -> NativeReActAgent (for backward compat)try:from openjarvis.core.registry import AgentRegistrydiff --git a/src/openjarvis/agents/hybrid/README.md b/src/openjarvis/agents/hybrid/README.mdnew file mode 100644index 00000000..f07be5ef--- /dev/null+++ b/src/openjarvis/agents/hybrid/README.md@@ -0,0 +1,100 @@+# Hybrid local+cloud paradigm agents++Six paradigms ported from+[`/matx/u/aspark/hybrid-local-cloud-compute`](../../../../../) — each is+registered as a standard OpenJarvis agent so the rest of the platform+(SDK, CLI, distillation, evals) can use them like any other agent.++| Agent | Plan shape | Trains what? | Workers |+|-------------------|-----------------|----------------------|---------------------------|+| `minions` | reactive loop | nothing | 1 local + 1 cloud |+| `conductor` | static DAG | (paper: 7B planner) | up to 5 frontier+open |+| `archon` | static recipe | nothing (search) | K local + cloud rank/fuse |+| `advisors` | reactive loop | (paper: local model) | 1 local + 1 cloud |+| `skillorchestra` | per-query pick | (paper: profiler) | 1 local + 1 cloud |+| `toolorchestra` | reactive loop | (paper: 8B RL) | local + tools + LLM pool |++Items in parentheses are what the *paper* trains. These OpenJarvis ports+are **inference-only** — none modify weights. The trained variants (advisor+RL, Orchestrator-8B, SkillOrchestra learn-phase) stay TODOs; the prompted+lower-bounds get you 80-90% of the headline accuracy at zero training cost.++## What's where++```+src/openjarvis/agents/hybrid/+├── _base.py LocalCloudAgent ABC + SDK helpers+├── _prices.py cloud-model pricing + temp-strip quirks+├── _prompts.py GAIA / SWE-bench answer-format instructions+├── advisors.py executor ↔ advisor ↔ executor (3-step)+├── conductor.py static DAG planner+├── minions.py HazyResearch Minions wrapper+├── archon.py Archon (generator → ranker → fuser)+├── skillorchestra.py skill-aware router+├── toolorchestra.py prompted multi-turn tool dispatcher+├── runner.py CLI: python -m ...hybrid.runner --cell NAME+├── registry/ <method>.toml — one cell per (bench, local, cloud, N)+└── scripts/+ └── new_experiment.sh scaffold a new cell, run instructions+```++The Modal-backed SWE-bench-Verified scorer is in+`src/openjarvis/evals/scorers/swebench_harness.py` (next to the existing+structural scorer).++## Quickstart++```bash+cd /matx/u/aspark/OpenJarvis+source .env # API keys++# 1. Start vLLM in another shell (see CLAUDE.md for the full recipe)+# CUDA_VISIBLE_DEVICES=0 .venv/bin/python -m vllm.entrypoints.openai.api_server \+# --model Qwen/Qwen3.5-27B-FP8 --port 8001 ...++# 2. (Optional) for Minions: install the upstream library+.venv/bin/uv pip install -e /matx/u/aspark/hybrid-local-cloud-compute/external/minions++# 3. Run a smoke cell+.venv/bin/python -m openjarvis.agents.hybrid.runner \+ --cell minions-gaia-qwen27b-opus-3+```++Outputs land in+`$OPENJARVIS_HYBRID_EXPERIMENTS_DIR/<cell>/{results.jsonl,summary.json,config.json,logs/}`+(defaults to `~/.openjarvis-hybrid/experiments/`). The schema matches the+hybrid harness so the existing rescore / dashboard scripts work+unmodified.++## Adding a cell++```bash+src/openjarvis/agents/hybrid/scripts/new_experiment.sh \+ --method conductor --bench gaia \+ --local qwen3.5-27b --cloud claude-opus-4-7 --n 30+```++That appends a `[cells.<name>]` block to+`registry/conductor.toml` and prints the runner invocation.++## How good is each paradigm?++Numbers from the upstream hybrid harness+(`/matx/u/aspark/hybrid-local-cloud-compute/docs/results.md`) at full N —+GAIA val n=165, SWE-bench-Verified n=500. Local = Qwen-3.5-27B-FP8, cloud+= Opus 4.7. Cloud-only baseline: GAIA 0.570 / $1.09, SWE 0.238 / $0.95.++| paradigm | shape | GAIA acc / $ | SWE acc / $ | verdict |+|--------------------|----------------------|-----------------|-----------------|--------------------------------------------------|+| **minions** | reactive 1L+1C loop | 0.576 / $0.67 | 0.276 / $0.09 | **keep** — matches cloud, ~10× cheaper on SWE |+| **skillorchestra** | per-query router | 0.570 / $0.02 | 0.298 / $0.05 | **keep** — cloud-tier acc at 1/50× cost |+| conductor | static DAG planner | 0.503 / $0.03 | 0.296 / $0.07 | mixed — wins SWE, −7pp on GAIA |+| advisors | exec ↔ advisor loop | 0.497 / $0.02 | 0.302 / $0.07 | mixed — wins SWE, −7pp on GAIA |+| archon | gen → rank → fuse | 0.376 / $0.14 | 0.288 / $0.17 | dominated on GAIA (−19pp); only mid on SWE |+| toolorchestra | RL'd 8B + tool pool | — | — | port lands but untested at full N (heavy infra) |++Cell configs in `registry/` are copies of the hybrid harness's+`experiments/registry/` — same models, same N, same `method_cfg` — so+these OpenJarvis cells should reproduce the harness numbers within+noise. Until that's validated, the harness stays the authoritative+reference.diff --git a/src/openjarvis/agents/hybrid/__init__.py b/src/openjarvis/agents/hybrid/__init__.pynew file mode 100644index 00000000..87e67091--- /dev/null+++ b/src/openjarvis/agents/hybrid/__init__.py@@ -0,0 +1,42 @@+"""Hybrid local+cloud paradigms — ported from `hybrid-local-cloud-compute`.++Each module here registers one agent under ``@AgentRegistry.register("<name>")``:++ advisors — executor (cloud) ↔ advisor (local) ↔ executor (cloud)+ conductor — zero-shot planner emits a DAG of up to 5 worker calls+ minions — supervisor (cloud) ↔ worker (local) reactive loop+ archon — layered (generator → ranker → fuser) inference-time search+ skillorchestra — skill-aware router picks one agent from a pool+ toolorchestra — prompted multi-turn dispatcher over a mixed tool/model pool++All agents share :class:`LocalCloudAgent` as the base. They are bench-agnostic:+the caller formats the prompt (using ``hybrid_prompts.format_prompt(task, bench)``+or the bench's native formatter) and hands it in via ``run(input=...)``. Task+metadata that the paradigm needs (a problem statement vs. a question, hints,+etc.) goes through ``context.metadata``.++The hybrid harness at ``/matx/u/aspark/hybrid-local-cloud-compute`` is the+reference implementation and stays untouched — these ports are the+OpenJarvis-native versions of the same paradigms.+"""++from __future__ import annotations++import logging++logger = logging.getLogger(__name__)++# Import each paradigm to trigger its @AgentRegistry.register() decorator.+for _modname in (+ "advisors",+ "conductor",+ "minions",+ "archon",+ "skillorchestra",+ "toolorchestra",+ "mini_swe_agent",+):+ try:+ __import__(f"openjarvis.agents.hybrid.{_modname}")+ except Exception as exc: # pragma: no cover — optional deps may be missing+ logger.debug("hybrid agent %s skipped: %s", _modname, exc)diff --git a/src/openjarvis/agents/hybrid/_base.py b/src/openjarvis/agents/hybrid/_base.pynew file mode 100644index 00000000..15ffd114--- /dev/null+++ b/src/openjarvis/agents/hybrid/_base.py@@ -0,0 +1,615 @@+"""LocalCloudAgent — shared base for hybrid local+cloud paradigm agents.++The hybrid paradigms (Minions, Conductor, Archon, Advisors, SkillOrchestra,+ToolOrchestra) all coordinate at least two models: a small **local** model+served by vLLM over an OpenAI-compatible endpoint, and a **cloud** model+reached via the Anthropic or OpenAI SDK.++Why not just use OpenJarvis's :class:`InferenceEngine` for both? Two reasons:++1. The reference hybrid adapters (``hybrid-local-cloud-compute/adapters/``) make+ raw SDK calls because some of them (Minions, Archon) construct external+ library objects that themselves create their own SDK clients. We mirror that+ here so the n=500 numbers stay reproducible during the port.+2. Cloud-side quirks (Opus 4.7 temperature stripping, GPT-5 family+ ``max_completion_tokens``) are paradigm-shaped — Minions needs structured+ outputs on the supervisor turn, SkillOrchestra needs them on the router,+ baseline_cloud does not. Keeping the SDK calls in the agent layer lets each+ paradigm decide the schema rather than fighting a shared engine API.++The base class therefore provides only:++- Standard ``run()`` contract returning an :class:`AgentResult` whose+ ``metadata`` carries the hybrid-result fields (``tokens_local``,+ ``tokens_cloud``, ``cost_usd``, ``latency_s``, ``traces``).+- ``_call_anthropic`` / ``_call_openai`` / ``_call_vllm`` helpers that handle+ Opus 4.7 temperature stripping, GPT-5 token-arg naming, vLLM+ ``enable_thinking`` kwargs, and basic token bookkeeping.+- ``_soft_fail_metadata`` for deterministic failure rows (e.g. Qwen JSON+ malformation) so the runner doesn't crash the whole cell.++Agents register themselves with ``@AgentRegistry.register("name")`` and become+discoverable via the existing SDK / CLI flow. The runner constructs them with+the cloud ``(engine, model)`` as the canonical pair, and paradigm-specific+kwargs (``local_model``, ``local_endpoint``, ``cloud_endpoint``, …) follow.+"""++from __future__ import annotations++import json+import threading+import time+from abc import abstractmethod+from pathlib import Path+from typing import Any, Dict, List, Optional, Tuple++from openjarvis.agents._stubs import AgentContext, AgentResult, BaseAgent+from openjarvis.agents.hybrid._prices import (+ NO_TEMP_PREFIXES,+ is_gpt5_family,+ supports_temperature,+)+from openjarvis.agents.hybrid._prices import (+ cost as estimate_cost,+)+from openjarvis.engine._stubs import InferenceEngine++# Anthropic server-side web_search: $10 per 1000 searches.+WEB_SEARCH_COST_PER_CALL = 0.01++ANTHROPIC_WEB_SEARCH_TOOL = {+ "type": "web_search_20250305",+ "name": "web_search",+ "max_uses": 8,+}+++# ---------- Thread-local trace buffer ----------+#+# Every call through ``_call_anthropic`` / ``_call_openai`` / ``_call_vllm``+# appends an event to the active trace if one is open. ``run()`` opens a fresh+# trace per task and writes the digested log to+# ``<log_dir>/<task_id>.json`` when it's done. Thread-local so concurrent+# tasks in the runner's ThreadPoolExecutor don't stomp each other's trace.++_TRACE_STATE = threading.local()+++def _trace_events() -> Optional[List[Dict[str, Any]]]:+ return getattr(_TRACE_STATE, "events", None)+++def _record_event(event: Dict[str, Any]) -> None:+ events = _trace_events()+ if events is not None:+ events.append(event)+++def _open_trace() -> List[Dict[str, Any]]:+ events: List[Dict[str, Any]] = []+ _TRACE_STATE.events = events+ return events+++def _close_trace() -> None:+ if hasattr(_TRACE_STATE, "events"):+ delattr(_TRACE_STATE, "events")+++def _serialize_block(block: Any) -> Dict[str, Any]:+ """Turn an Anthropic content block (text / tool_use / server_tool_use /+ web_search_tool_result / thinking) into a JSON-safe dict.++ Each block type carries different fields; we extract everything we can+ so the per-task log file is a complete record of what the model emitted+ (including every tool call request and tool result body).+ """+ out: Dict[str, Any] = {"type": getattr(block, "type", type(block).__name__)}+ for attr in (+ "id", "name", "input", "text", "thinking", "signature",+ "tool_use_id", "content",+ ):+ if hasattr(block, attr):+ val = getattr(block, attr)+ # Nested content (e.g. web_search_tool_result.content is a list of+ # citation/result blocks). Recurse for completeness.+ if attr == "content" and isinstance(val, list):+ out[attr] = [_serialize_block(b) for b in val]+ else:+ out[attr] = _jsonable(val)+ return out+++def _serialize_openai_tool_calls(tool_calls: Any) -> List[Dict[str, Any]]:+ """vLLM / OpenAI returns ChatCompletionMessageToolCall objects. Pull out+ id, type, name, and the (JSON-string) arguments so they're round-trippable."""+ out: List[Dict[str, Any]] = []+ if not tool_calls:+ return out+ for tc in tool_calls:+ fn = getattr(tc, "function", None)+ out.append({+ "id": getattr(tc, "id", None),+ "type": getattr(tc, "type", "function"),+ "function": {+ "name": getattr(fn, "name", None) if fn else None,+ "arguments": getattr(fn, "arguments", None) if fn else None,+ },+ })+ return out+++def _jsonable(v: Any) -> Any:+ """Best-effort JSON-friendly conversion. Pydantic models → .model_dump(),+ dataclasses untouched (json.dumps handles them via default=str)."""+ if hasattr(v, "model_dump"):+ try:+ return v.model_dump()+ except Exception:+ pass+ if isinstance(v, (list, tuple)):+ return [_jsonable(x) for x in v]+ if isinstance(v, dict):+ return {k: _jsonable(x) for k, x in v.items()}+ return v+++class LocalCloudAgent(BaseAgent):+ """Base for paradigm agents that coordinate a local + cloud model pair.++ Subclasses implement :meth:`_run_paradigm` rather than ``run`` so the+ base can wrap timing, metadata shaping, and soft-fail handling+ uniformly.++ The :meth:`run` contract takes the formatted task prompt as ``input``+ and reads paradigm-shaped data from ``context.metadata``:++ - ``context.metadata["task"]``: optional dict (the bench's raw task+ row, used by paradigms that look at hints / problem_statement / etc.).+ - ``context.metadata["task_id"]``: optional string identifier.++ Construction args:+ - ``engine``, ``model``: the cloud engine + model id (satisfies+ :class:`BaseAgent`'s contract; only used incidentally — we make raw+ SDK calls).+ - ``local_model``, ``local_endpoint``: vLLM-served local model and its+ OpenAI-compatible endpoint, e.g. ``"http://localhost:8001/v1"``.+ - ``cloud_endpoint``: ``"anthropic"`` or ``"openai"`` — picks the+ cloud SDK.+ - ``cfg``: paradigm-specific knobs (max_tokens, schemas, mode, …).+ """++ accepts_tools: bool = False++ def __init__(+ self,+ engine: InferenceEngine,+ model: str,+ *,+ local_model: Optional[str] = None,+ local_endpoint: Optional[str] = None,+ cloud_endpoint: str = "anthropic",+ cfg: Optional[Dict[str, Any]] = None,+ bus: Optional[Any] = None,+ temperature: Optional[float] = None,+ max_tokens: Optional[int] = None,+ ) -> None:+ super().__init__(+ engine,+ model,+ bus=bus,+ temperature=temperature,+ max_tokens=max_tokens,+ )+ self._cloud_model = model+ self._cloud_endpoint = (cloud_endpoint or "anthropic").lower()+ self._local_model = local_model+ self._local_endpoint = local_endpoint+ self._cfg: Dict[str, Any] = dict(cfg or {})++ # ------------------------------------------------------------------+ # SDK call helpers — raw clients, paradigm-shaped quirks applied+ # ------------------------------------------------------------------++ @staticmethod+ def _call_anthropic(+ model: str,+ *,+ user: str,+ system: Optional[str] = None,+ max_tokens: int = 4096,+ temperature: float = 0.0,+ tools: Optional[list] = None,+ tool_choice: Optional[dict] = None,+ output_config: Optional[dict] = None,+ timeout: float = 600.0,+ max_retries: int = 5,+ trace_role: str = "cloud",+ ) -> Tuple[str, int, int, int]:+ """Single Anthropic call. Returns (text, p_tok, c_tok, n_web_searches).++ Strips ``temperature`` for Opus 4.7+ (rejected by the API). Captures+ the call into the active per-task trace if one is open.+ """+ import anthropic++ client = anthropic.Anthropic(timeout=timeout, max_retries=max_retries)+ kwargs: Dict[str, Any] = {+ "model": model,+ "max_tokens": max_tokens,+ "messages": [{"role": "user", "content": user}],+ }+ if system:+ kwargs["system"] = system+ if supports_temperature(model):+ kwargs["temperature"] = temperature+ if tools:+ kwargs["tools"] = tools+ if tool_choice:+ kwargs["tool_choice"] = tool_choice+ if output_config:+ kwargs["output_config"] = output_config+ t0 = time.time()+ msg = client.messages.create(**kwargs)+ latency = time.time() - t0+ text = "".join(b.text for b in msg.content if hasattr(b, "text"))+ srv = getattr(msg.usage, "server_tool_use", None)+ n_searches = getattr(srv, "web_search_requests", 0) if srv else 0+ content_blocks = [_serialize_block(b) for b in msg.content]+ tool_use_blocks = [b for b in content_blocks if b.get("type") in (+ "tool_use", "server_tool_use",+ )]+ tool_result_blocks = [b for b in content_blocks if b.get("type") in (+ "web_search_tool_result", "tool_result",+ )]+ _record_event({+ "kind": "anthropic",+ "role": trace_role,+ "model": model,+ "system": system,+ "user": user,+ "response": text,+ "content_blocks": content_blocks,+ "tool_calls": tool_use_blocks,+ "tool_results": tool_result_blocks,+ "tokens_in": msg.usage.input_tokens,+ "tokens_out": msg.usage.output_tokens,+ "n_web_searches": n_searches,+ "tools_declared": tools,+ "tool_choice": tool_choice,+ "output_config": output_config,+ "stop_reason": getattr(msg, "stop_reason", None),+ "latency_s": latency,+ "ts": time.time(),+ })+ return text, msg.usage.input_tokens, msg.usage.output_tokens, n_searches++ @staticmethod+ def _call_openai(+ model: str,+ *,+ user: str,+ system: Optional[str] = None,+ max_tokens: int = 4096,+ temperature: float = 0.0,+ response_format: Optional[dict] = None,+ tools: Optional[list] = None,+ tool_choice: Optional[Any] = None,+ timeout: float = 600.0,+ trace_role: str = "cloud",+ ) -> Tuple[str, int, int]:+ """Single OpenAI call. Returns (text, p_tok, c_tok). Trace-captured;+ also records any tool_calls the model emits."""+ from openai import OpenAI++ client = OpenAI(timeout=timeout)+ messages: list = []+ if system:+ messages.append({"role": "system", "content": system})+ messages.append({"role": "user", "content": user})+ kwargs: Dict[str, Any] = {"model": model, "messages": messages}+ if is_gpt5_family(model):+ kwargs["max_completion_tokens"] = max_tokens+ else:+ kwargs["max_tokens"] = max_tokens+ kwargs["temperature"] = temperature+ if response_format is not None:+ kwargs["response_format"] = response_format+ if tools:+ kwargs["tools"] = tools+ if tool_choice is not None:+ kwargs["tool_choice"] = tool_choice+ t0 = time.time()+ resp = client.chat.completions.create(**kwargs)+ latency = time.time() - t0+ choice = resp.choices[0]+ message = choice.message+ text = message.content or ""+ tool_calls = _serialize_openai_tool_calls(getattr(message, "tool_calls", None))+ reasoning = getattr(message, "reasoning_content", None) or getattr(+ message, "reasoning", None+ )+ u = resp.usage+ p = getattr(u, "prompt_tokens", 0) if u else 0+ c = getattr(u, "completion_tokens", 0) if u else 0+ _record_event({+ "kind": "openai",+ "role": trace_role,+ "model": model,+ "system": system,+ "user": user,+ "response": text,+ "tool_calls": tool_calls,+ "reasoning_content": reasoning,+ "tokens_in": p,+ "tokens_out": c,+ "response_format": response_format,+ "tools_declared": tools,+ "tool_choice": tool_choice,+ "finish_reason": getattr(choice, "finish_reason", None),+ "latency_s": latency,+ "ts": time.time(),+ })+ return text, p, c++ @staticmethod+ def _call_vllm(+ model: str,+ endpoint: str,+ *,+ user: str,+ system: Optional[str] = None,+ max_tokens: int = 4096,+ temperature: float = 0.0,+ enable_thinking: bool = False,+ tools: Optional[list] = None,+ tool_choice: Optional[Any] = None,+ timeout: float = 600.0,+ trace_role: str = "local",+ ) -> Tuple[str, int, int]:+ """Local vLLM (OpenAI-compatible) call. Returns (text, p_tok, c_tok).+ Captures the full response into the trace — including any tool_calls+ the local model emits (vLLM exposes them in+ ``resp.choices[0].message.tool_calls`` when ``--enable-auto-tool-choice``+ is on)."""+ from openai import OpenAI++ client = OpenAI(base_url=endpoint, api_key="EMPTY", timeout=timeout)+ messages: list = []+ if system:+ messages.append({"role": "system", "content": system})+ messages.append({"role": "user", "content": user})+ kwargs: Dict[str, Any] = dict(+ model=model,+ messages=messages,+ temperature=temperature,+ max_tokens=max_tokens,+ extra_body={"chat_template_kwargs": {"enable_thinking": enable_thinking}},+ )+ if tools:+ kwargs["tools"] = tools+ if tool_choice is not None:+ kwargs["tool_choice"] = tool_choice+ t0 = time.time()+ resp = client.chat.completions.create(**kwargs)+ latency = time.time() - t0+ choice = resp.choices[0]+ message = choice.message+ text = message.content or ""+ tool_calls = _serialize_openai_tool_calls(getattr(message, "tool_calls", None))+ reasoning = getattr(message, "reasoning_content", None) or getattr(+ message, "reasoning", None+ )+ u = resp.usage+ p = getattr(u, "prompt_tokens", 0) if u else 0+ c = getattr(u, "completion_tokens", 0) if u else 0+ _record_event({+ "kind": "vllm",+ "role": trace_role,+ "model": model,+ "endpoint": endpoint,+ "system": system,+ "user": user,+ "response": text,+ "tool_calls": tool_calls,+ "reasoning_content": reasoning,+ "tokens_in": p,+ "tokens_out": c,+ "temperature": temperature,+ "max_tokens": max_tokens,+ "enable_thinking": enable_thinking,+ "tools_declared": tools,+ "tool_choice": tool_choice,+ "finish_reason": getattr(choice, "finish_reason", None),+ "latency_s": latency,+ "ts": time.time(),+ })+ return text, p, c++ def _call_cloud(+ self,+ *,+ user: str,+ system: Optional[str] = None,+ max_tokens: int = 4096,+ temperature: float = 0.0,+ **kwargs: Any,+ ) -> Tuple[str, int, int]:+ """Dispatch a single cloud call by ``self._cloud_endpoint``.++ Returns (text, p_tok, c_tok). For Anthropic, the web_search count+ is discarded — paradigms that care should call ``_call_anthropic``+ directly.+ """+ if self._cloud_endpoint == "anthropic":+ text, p, c, _ = self._call_anthropic(+ self._cloud_model,+ user=user,+ system=system,+ max_tokens=max_tokens,+ temperature=temperature,+ **kwargs,+ )+ return text, p, c+ if self._cloud_endpoint == "openai":+ return self._call_openai(+ self._cloud_model,+ user=user,+ system=system,+ max_tokens=max_tokens,+ temperature=temperature,+ **kwargs,+ )+ raise ValueError(f"unsupported cloud endpoint: {self._cloud_endpoint!r}")++ # ------------------------------------------------------------------+ # Result shaping+ # ------------------------------------------------------------------++ @staticmethod+ def _soft_fail_metadata(reason: str) -> Dict[str, Any]:+ """Metadata for a soft-fail row (Qwen JSON broke, Anthropic 400, etc.).++ The agent still returns an :class:`AgentResult` with empty content;+ the runner records it as score=0 without crashing the cell.+ """+ return {+ "tokens_local": 0,+ "tokens_cloud": 0,+ "cost_usd": 0.0,+ "latency_s": 0.0,+ "soft_error": reason,+ "traces": {"soft_error": reason},+ }++ @staticmethod+ def cost_usd(model: str, prompt_tokens: int, completion_tokens: int) -> float:+ return estimate_cost(model, prompt_tokens, completion_tokens)++ # ------------------------------------------------------------------+ # Run contract+ # ------------------------------------------------------------------++ def run(+ self,+ input: str,+ context: Optional[AgentContext] = None,+ **kwargs: Any,+ ) -> AgentResult:+ self._emit_turn_start(input)+ t0 = time.time()+ events = _open_trace()+ meta: Dict[str, Any]+ answer: str = ""+ soft_reason: Optional[str] = None+ exc_obj: Optional[BaseException] = None+ try:+ try:+ answer, meta = self._run_paradigm(input, context, **kwargs)+ except Exception as exc:+ soft = self._is_soft_failure(exc)+ if soft is None:+ exc_obj = exc+ raise+ soft_reason = soft+ meta = self._soft_fail_metadata(soft)+ finally:+ # Persist the trace before the trace state is closed (and even on+ # hard failure, so we get a record of what we did before it broke).+ self._write_trace_log(+ context, input, answer, meta if "meta" in locals() else {},+ events, soft_reason, exc_obj,+ )+ _close_trace()+ meta.setdefault("latency_s", time.time() - t0)+ if soft_reason is not None:+ self._emit_turn_end(soft_error=soft_reason)+ return AgentResult(content="", metadata=meta, turns=0)+ self._emit_turn_end(**{k: v for k, v in meta.items() if k != "traces"})+ return AgentResult(+ content=answer,+ metadata=meta,+ turns=int(meta.get("turns", 0) or 0),+ )++ @staticmethod+ def record_trace_event(event: Dict[str, Any]) -> None:+ """Public hook for paradigm code that bypasses the SDK helpers+ (Minions's protocol loop, Archon's layer pipeline, …) to drop a+ custom event into the current task's trace.+ """+ _record_event({**event, "ts": event.get("ts", time.time())})++ def _write_trace_log(+ self,+ context: Optional[AgentContext],+ input: str,+ answer: str,+ meta: Dict[str, Any],+ events: List[Dict[str, Any]],+ soft_reason: Optional[str],+ exc: Optional[BaseException],+ ) -> None:+ log_dir = None+ task_id = "unknown"+ if context is not None:+ log_dir = context.metadata.get("log_dir")+ task_id = context.metadata.get("task_id") or task_id+ if not log_dir:+ return+ try:+ out_dir = Path(log_dir)+ out_dir.mkdir(parents=True, exist_ok=True)+ blob = {+ "task_id": task_id,+ "agent": self.agent_id,+ "cloud_model": self._cloud_model,+ "cloud_endpoint": self._cloud_endpoint,+ "local_model": self._local_model,+ "local_endpoint": self._local_endpoint,+ "cfg": self._cfg,+ "input": input,+ "answer": answer,+ "metadata": meta,+ "events": events,+ "soft_error": soft_reason,+ "error": (+ f"{type(exc).__name__}: {exc}" if exc is not None else None+ ),+ }+ (out_dir / f"{task_id}.json").write_text(+ json.dumps(blob, indent=2, default=str)+ )+ except Exception:+ # Logging must never break a run.+ pass++ @abstractmethod+ def _run_paradigm(+ self,+ input: str,+ context: Optional[AgentContext],+ **kwargs: Any,+ ) -> Tuple[str, Dict[str, Any]]:+ """Run the paradigm. Return ``(final_answer, metadata)``.++ Metadata should include the hybrid-shape fields:+ ``tokens_local``, ``tokens_cloud``, ``cost_usd``, optional+ ``latency_s`` (the base fills it if absent), and a ``traces`` dict.+ """++ # Subclasses override to declare deterministic failure modes they+ # want the base to swallow into a soft-fail row.+ def _is_soft_failure(self, exc: BaseException) -> Optional[str]:+ return None+++__all__ = [+ "ANTHROPIC_WEB_SEARCH_TOOL",+ "LocalCloudAgent",+ "NO_TEMP_PREFIXES",+ "WEB_SEARCH_COST_PER_CALL",+ "estimate_cost",+ "is_gpt5_family",+ "supports_temperature",+]diff --git a/src/openjarvis/agents/hybrid/_prices.py b/src/openjarvis/agents/hybrid/_prices.pynew file mode 100644index 00000000..b0eeba0b--- /dev/null+++ b/src/openjarvis/agents/hybrid/_prices.py@@ -0,0 +1,46 @@+"""Cloud-model pricing + per-family quirks for hybrid paradigm agents.++Ported verbatim from ``hybrid-local-cloud-compute/prices.py``. Kept as a+sibling to the agents rather than merged into ``engine/cloud.py``'s PRICING+on purpose: the hybrid harness is the authoritative cost reference for the+n=500 numbers in ``hybrid-local-cloud-compute/docs/results.md`` and we want+the OpenJarvis ports to charge identically.+"""++from __future__ import annotations++# USD per million tokens, (input, output). Local models = 0.+PRICES: dict[str, tuple[float, float]] = {+ "claude-opus-4-7": (5.00, 25.0),+ "claude-sonnet-4-6": (3.00, 15.0),+ "claude-haiku-4-5": (1.00, 5.00),+ "claude-haiku-4-5-20251001": (1.00, 5.00),+ "gpt-5": (1.25, 10.0),+ "gpt-5-mini": (0.25, 2.00),+ "gpt-5-mini-2025-08-07": (0.25, 2.00),+ "gpt-4o": (0.15, 0.60),+ "gemini-2.5-pro": (1.25, 10.0),+}++# Models whose API rejects an explicit `temperature` param — callers should+# omit it for any model whose name starts with one of these prefixes.+NO_TEMP_PREFIXES: tuple[str, ...] = (+ "claude-opus-4-7",+ "claude-sonnet-4-7",+ "claude-haiku-4-7",+)+++def cost(model: str, prompt_tokens: int, completion_tokens: int) -> float:+ """USD cost for one call. Unknown models price at 0 (e.g. local vLLM)."""+ pi, po = PRICES.get(model, (0.0, 0.0))+ return (prompt_tokens / 1_000_000) * pi + (completion_tokens / 1_000_000) * po+++def supports_temperature(model: str) -> bool:+ return not model.startswith(NO_TEMP_PREFIXES)+++def is_gpt5_family(model: str) -> bool:+ """GPT-5 series requires ``max_completion_tokens`` and forced temp=1."""+ return model.startswith("gpt-5")diff --git a/src/openjarvis/agents/hybrid/_prompts.py b/src/openjarvis/agents/hybrid/_prompts.pynew file mode 100644index 00000000..cf6b4135--- /dev/null+++ b/src/openjarvis/agents/hybrid/_prompts.py@@ -0,0 +1,68 @@+"""Bench-specific answer-format instructions for hybrid paradigm agents.++Ported from ``hybrid-local-cloud-compute/benches/{gaia,swebench_verified}/``.++Adapters in the hybrid harness call ``benches.format_prompt(task)`` to build+the full user prompt — the paradigms here take the formatted prompt directly,+so callers (the experiment runner or eval driver) use these helpers to build+the prompt once and hand the string to the agent's ``run(input=...)``.+"""++from __future__ import annotations++from typing import Any, Dict++GAIA_INSTRUCTION = (+ "GAIA expects a single short, exact answer. Reason as needed, then commit to "+ "one best answer. Your reply MUST end with exactly one line of the form:\n"+ "FINAL ANSWER: <answer>\n"+ "Formatting rules for <answer>:\n"+ " - If a number, write digits only (no commas, no units, no $ or %). "+ "e.g. `34689`, not `34,689 papers`.\n"+ " - If a string, use as few words as possible, no articles, no abbreviations.\n"+ " - If a list, comma-separate it and apply the rules above to each element.\n"+ "Nothing should follow the FINAL ANSWER line."+)++SWEBENCH_INSTRUCTION = (+ "You are fixing a real Python bug from SWE-bench Verified. Your reply MUST "+ "end with a unified diff patch that fixes the bug. The patch MUST:\n"+ " - Start with a `diff --git a/<path> b/<path>` line (one per file changed).\n"+ " - Include the standard `--- a/<path>` and `+++ b/<path>` lines below it.\n"+ " - Use proper hunk headers `@@ -start,len +start,len @@`.\n"+ " - Apply cleanly against the repository at the given base commit.\n"+ "Wrap the patch in a ```diff ... ``` code fence. Do not include any prose "+ "after the fence — only the closing ```."+)+++def format_gaia(task: Dict[str, Any]) -> str:+ return f"{GAIA_INSTRUCTION}\n\nQuestion:\n{task.get('question', '')}"+++def format_swebench(task: Dict[str, Any]) -> str:+ header = SWEBENCH_INSTRUCTION + "\n\n"+ if task.get("repo"):+ header += f"Repository: {task['repo']}\n"+ if task.get("base_commit"):+ header += f"Base commit: {task['base_commit']}\n"+ header += "\nGitHub issue:\n"+ return header + task.get("problem_statement", "")+++def format_prompt(task: Dict[str, Any]) -> str:+ """Dispatch on task shape — same contract as ``benches.format_prompt``."""+ if task.get("question"):+ return format_gaia(task)+ if task.get("problem_statement"):+ return format_swebench(task)+ return ""+++__all__ = [+ "GAIA_INSTRUCTION",+ "SWEBENCH_INSTRUCTION",+ "format_gaia",+ "format_prompt",+ "format_swebench",+]diff --git a/src/openjarvis/agents/hybrid/advisors.py b/src/openjarvis/agents/hybrid/advisors.pynew file mode 100644index 00000000..35797759--- /dev/null+++ b/src/openjarvis/agents/hybrid/advisors.py@@ -0,0 +1,280 @@+"""AdvisorsAgent — inference-only port of advisor-models (Asawa et al., 2026).++Paper: arXiv:2510.02453. A small open-source advisor model writes feedback+that *steers* a black-box cloud executor. The paper trains the advisor with+RL; we don't have a released checkpoint, so this agent is the **inference-+only lower bound**: an untrained Qwen advisor zero-shot prompted with the+paper's structure.++Pipeline (mirrors ``advisor_models/math/env.py``):++1. **Executor (cloud)** answers the question.+2. **Advisor (local)** reads question + initial response and writes+ critique / hint text.+3. **Executor (cloud)** re-answers given question + its own initial+ response + advisor feedback. This final answer is what we score.++Results from the hybrid harness (n=30 GAIA):+``advisors-gaia-qwen9b-opus-30`` = 0.533, $0.02/task — within 3pp of+baseline-cloud at 30× cheaper. The RL-trained variant would land higher.++Ported from ``hybrid-local-cloud-compute/adapters/advisors_adapter.py``.+"""++from __future__ import annotations++import json+import urllib.request+from typing import Any, Dict, Optional, Tuple++from openjarvis.agents._stubs import AgentContext+from openjarvis.agents.hybrid._base import LocalCloudAgent+from openjarvis.agents.hybrid.mini_swe_agent import (+ run_swe_agent_loop,+)+from openjarvis.core.registry import AgentRegistry++# Prompts paraphrased from advisor-models/{math,template}/config.py.++EXECUTOR_INITIAL_SYS = (+ "You are a careful problem-solver. Read the question, reason step by step "+ "as needed, then commit to one best answer following any answer-format "+ "instructions in the question itself."+)++EXECUTOR_FINAL_SYS = (+ "You are a careful problem-solver. You previously gave an initial answer "+ "to a question and an advisor has reviewed it. Incorporate the advisor's "+ "feedback where it improves correctness; ignore it where it is wrong. "+ "Produce your best final answer, following any answer-format instructions "+ "in the question itself."+)++ADVISOR_TEMPLATE = """You are an expert advisor reviewing another model's draft answer to a user question. Your job is NOT to answer the question yourself; it is to give the answering model concrete, actionable feedback so it can improve its next attempt.++The user question was:+{question}++The answering model's initial response was:+{initial_response}++Provide feedback focused on:+1. Specific errors in reasoning, calculation, factual recall, or formatting.+2. Concrete corrections or alternative approaches it should consider.+3. What evidence or sub-step it should verify before committing.++Be concise (a short paragraph or a few bullet points). Do NOT restate the question. Do NOT provide a complete answer — only the feedback the model needs to improve its own next answer."""+++def _resolve_local_model(endpoint: str, registry_model: str) -> str:+ """If the registry-listed model isn't being served by vLLM, fall back to+ whatever the server reports first. Avoids 404s when cell config names+ a model id (e.g. ``Qwen3.5-9B``) that's different from what's loaded.+ """+ try:+ with urllib.request.urlopen(+ endpoint.rstrip("/") + "/models", timeout=5+ ) as r:+ data = json.loads(r.read())+ served = [m["id"] for m in data.get("data", [])]+ except Exception:+ return registry_model+ if registry_model in served:+ return registry_model+ return served[0] if served else registry_model++[email protected]("advisors")+class AdvisorsAgent(LocalCloudAgent):+ """Three-step executor ↔ advisor ↔ executor loop. See module docstring."""++ agent_id = "advisors"++ def _run_paradigm(+ self,+ input: str,+ context: Optional[AgentContext],+ **kwargs: Any,+ ) -> Tuple[str, Dict[str, Any]]:+ question = input+ cfg = self._cfg+ task_meta = (context.metadata.get("task") if context is not None else {}) or {}+ swe_mode = (+ bool(cfg.get("swe_use_agent_loop"))+ and bool(task_meta.get("problem_statement"))+ and bool(task_meta.get("repo"))+ and bool(task_meta.get("base_commit"))+ )+ if swe_mode:+ return self._run_swe(question, task_meta, cfg)++ executor_max_tokens = int(cfg.get("executor_max_tokens", 4096))+ advisor_max_tokens = int(cfg.get("advisor_max_tokens", 1024))+ advisor_temperature = float(cfg.get("advisor_temperature", 0.2))++ # 1. Initial executor pass+ initial_resp, e1_in, e1_out = self._call_cloud(+ user=f"Question:\n{question}",+ system=EXECUTOR_INITIAL_SYS,+ max_tokens=executor_max_tokens,+ temperature=0.0,+ )++ # 2. Advisor pass (local)+ if not self._local_endpoint or not self._local_model:+ raise ValueError(+ "AdvisorsAgent needs local_model + local_endpoint; got "+ f"model={self._local_model!r} endpoint={self._local_endpoint!r}"+ )+ local_model = _resolve_local_model(self._local_endpoint, self._local_model)+ advisor_prompt = ADVISOR_TEMPLATE.format(+ question=question, initial_response=initial_resp,+ )+ advisor_text, adv_in, adv_out = self._call_vllm(+ local_model,+ self._local_endpoint,+ user=advisor_prompt,+ max_tokens=advisor_max_tokens,+ temperature=advisor_temperature,+ enable_thinking=False,+ )++ # 3. Final executor pass with advisor's hints folded in+ final_user = (+ f"Question:\n{question}\n\n"+ f"Your initial response was:\n{initial_resp}\n\n"+ f"Advisor feedback:\n{advisor_text}\n\n"+ f"Produce your best final answer now, respecting the question's "+ f"answer-format rules."+ )+ final_answer, e2_in, e2_out = self._call_cloud(+ user=final_user,+ system=EXECUTOR_FINAL_SYS,+ max_tokens=executor_max_tokens,+ temperature=0.0,+ )++ tokens_local = adv_in + adv_out+ tokens_cloud = e1_in + e1_out + e2_in + e2_out+ cost = self.cost_usd(self._cloud_model, e1_in + e2_in, e1_out + e2_out)++ meta: Dict[str, Any] = {+ "tokens_local": tokens_local,+ "tokens_cloud": tokens_cloud,+ "cost_usd": cost,+ "turns": 3,+ "traces": {+ "initial_response": initial_resp,+ "advisor_feedback": advisor_text,+ "note": "inference-only advisor (untrained); lower bound on the technique.",+ },+ }+ return final_answer, meta++ # ------------------------------------------------------------------+ # SWE-bench variant: each executor pass is a full mini-SWE-agent run.+ # ------------------------------------------------------------------++ def _run_swe(+ self,+ question: str,+ task: Dict[str, Any],+ cfg: Dict[str, Any],+ ) -> Tuple[str, Dict[str, Any]]:+ if not self._local_endpoint or not self._local_model:+ raise ValueError(+ "AdvisorsAgent (swe mode) still needs local_model + local_endpoint "+ "for the advisor critique step."+ )++ # Each executor pass is its own mini-SWE-agent run with a FRESH+ # workdir — the advisor doesn't get a workdir, just the patch text.+ max_turns = int(cfg.get("swe_max_turns", 30))+ bash_timeout = int(cfg.get("swe_bash_timeout_s", 120))+ output_cap = int(cfg.get("swe_output_cap", 10_000))+ turn_max_tokens = int(cfg.get("swe_turn_max_tokens", 4096))++ # 1. Initial executor pass+ initial_out = run_swe_agent_loop(+ task,+ backbone="cloud",+ backbone_model=self._cloud_model,+ cloud_endpoint=self._cloud_endpoint,+ initial_prompt=question,+ max_turns=max_turns,+ bash_timeout=bash_timeout,+ output_cap=output_cap,+ turn_max_tokens=turn_max_tokens,+ trace_prefix="advisors_executor1",+ )++ # 2. Advisor pass — local model critiques the produced patch+ local_model = _resolve_local_model(self._local_endpoint, self._local_model)+ advisor_prompt = ADVISOR_TEMPLATE.format(+ question=question,+ initial_response=(+ f"Summary: {initial_out['final_summary']}\n\n"+ f"Patch produced:\n```diff\n{initial_out['patch']}```"+ ),+ )+ advisor_text, adv_in, adv_out = self._call_vllm(+ local_model,+ self._local_endpoint,+ user=advisor_prompt,+ max_tokens=int(cfg.get("advisor_max_tokens", 2048)),+ temperature=float(cfg.get("advisor_temperature", 0.2)),+ enable_thinking=False,+ )++ # 3. Final executor pass — FRESH workdir, advisor feedback folded+ # into the initial prompt. (Don't reuse initial workdir — that+ # would smuggle the bad-fix into the new attempt; we want the+ # final pass to incorporate ONLY the parts of the advisor's+ # feedback that hold up.)+ final_prompt = (+ f"{question}\n\n"+ f"-----\n"+ f"You previously attempted a fix. Your initial summary was:\n"+ f"{initial_out['final_summary']}\n\n"+ f"An advisor reviewed your attempt and wrote this feedback:\n"+ f"{advisor_text}\n\n"+ f"Incorporate the advisor's feedback where it improves correctness; "+ f"ignore where it is wrong. Produce your best fix now."+ )+ final_out = run_swe_agent_loop(+ task,+ backbone="cloud",+ backbone_model=self._cloud_model,+ cloud_endpoint=self._cloud_endpoint,+ initial_prompt=final_prompt,+ max_turns=max_turns,+ bash_timeout=bash_timeout,+ output_cap=output_cap,+ turn_max_tokens=turn_max_tokens,+ trace_prefix="advisors_executor2",+ )++ tokens_local = adv_in + adv_out+ tokens_cloud = (+ initial_out["tokens_in"] + initial_out["tokens_out"]+ + final_out["tokens_in"] + final_out["tokens_out"]+ )+ cost = initial_out["cost_usd"] + final_out["cost_usd"]+ meta: Dict[str, Any] = {+ "tokens_local": tokens_local,+ "tokens_cloud": tokens_cloud,+ "cost_usd": cost,+ "turns": initial_out["turns"] + 1 + final_out["turns"],+ "traces": {+ "swe_mode": True,+ "initial_summary": initial_out["final_summary"],+ "initial_patch_chars": len(initial_out["patch"]),+ "advisor_feedback": advisor_text,+ "final_summary": final_out["final_summary"],+ "final_patch_chars": len(final_out["patch"]),+ },+ }+ return final_out["answer"], meta+++__all__ = ["AdvisorsAgent"]diff --git a/src/openjarvis/agents/hybrid/archon.py b/src/openjarvis/agents/hybrid/archon.pynew file mode 100644index 00000000..9c0482cf--- /dev/null+++ b/src/openjarvis/agents/hybrid/archon.py@@ -0,0 +1,584 @@+"""ArchonAgent — port of ScalingIntelligence/Archon.++Inference-time architecture search: layered (generator → ranker → fuser)+sampling where a generator proposes K candidates, a ranker scores them,+and a fuser synthesizes a final answer. Paper: arXiv:2409.15254.++How the hybrid harness wires it (and what we mirror here):++- **Local proposers** (generator layer): K samples from vLLM via an+ OpenAI-compatible client at ``local_endpoint``. Injected as a custom+ ``vllm_local`` model_type into Archon's ``GENERATE_MAP`` — that's the+ only way Ranker/Fuser can pick up custom backends (they re-instantiate+ Generator without ``custom_generators``).+- **Cloud ranker + fuser**: Archon's built-in ``OpenAI_API`` /+ ``Anthropic_API``. Patched at import time to strip ``temperature`` for+ Opus 4.7+ and to tally token usage (Archon ignores ``usage`` by default).++``cfg`` knobs:++- ``n_samples`` (int, default 5) — K proposers+- ``architecture`` (str, default ``"ensemble_rank_fuse"``)++ - ``"ensemble_rank_fuse"`` → [K local generators, 1 cloud ranker, 1 cloud fuser]+ - ``"single_local"`` → [1 local generator] (debug)++- ``ranker_model`` / ``fuser_model`` (default: ``cloud_model`` for both)+- ``max_tokens`` (default 2048), ``temperature`` (default 0.7)++Requires the Archon library (cloned at+``hybrid-local-cloud-compute/external/Archon`` — add its ``src`` to+``PYTHONPATH`` or pip-install editable). Import is lazy.++Ported from ``hybrid-local-cloud-compute/adapters/archon_adapter.py``.+"""++from __future__ import annotations++import os+import sys+import threading+import types+from typing import Any, Dict, List, Optional, Tuple++from openjarvis.agents._stubs import AgentContext+from openjarvis.agents.hybrid._base import LocalCloudAgent, _record_event+from openjarvis.agents.hybrid._prices import (+ NO_TEMP_PREFIXES,+)+from openjarvis.agents.hybrid._prices import (+ cost as _cost_cloud,+)+from openjarvis.agents.hybrid.mini_swe_agent import run_swe_agent_loop+from openjarvis.core.registry import AgentRegistry++ARCHON_SWE_RANKER_SYS = (+ "You are ranking K candidate patches for a SWE-bench bug. For each "+ "candidate, you'll see the candidate's summary text and the unified "+ "diff. Pick the index (0-based) of the candidate most likely to "+ "actually fix the issue. Prefer minimal, targeted patches; reject "+ "candidates with no patch or with patches that touch unrelated files. "+ "Respond with a single integer on its own line — the index — "+ "followed by a one-line justification."+)+++# ---------- Stubs for Archon's eager-imported heavy deps we don't need ----------++def _stub_archon_imports() -> None:+ """``utils.py`` imports groq/google/litellm/dotenv at module load. Stub+ the ones we don't use so the import chain doesn't fail when those+ libraries aren't installed in the OpenJarvis venv."""+ for name in ("groq", "google", "google.generativeai", "litellm"):+ if name in sys.modules:+ continue+ sys.modules[name] = types.ModuleType(name)+ sys.modules["groq"].Groq = type("Groq", (), {}) # type: ignore[attr-defined]+ sys.modules["litellm"].completion = lambda *a, **k: None # type: ignore[attr-defined]+ sys.modules["google.generativeai"] = types.ModuleType("google.generativeai")+++def _add_archon_to_path() -> None:+ """Locate Archon's ``src`` dir. The hybrid clone is the default location;+ override with ``ARCHON_SRC`` env var if Archon is installed elsewhere."""+ archon_src = os.environ.get(+ "ARCHON_SRC",+ "/matx/u/aspark/hybrid-local-cloud-compute/external/Archon/src",+ )+ if archon_src not in sys.path and os.path.isdir(archon_src):+ sys.path.insert(0, archon_src)+++# ---------- Anthropic patch for Opus 4.7 ----------++def _patch_anthropic_for_opus() -> None:+ from anthropic.resources.messages import messages as _msgs_mod++ cls = _msgs_mod.Messages+ if getattr(cls.create, "_hybrid_archon_patched", False):+ return+ orig = cls.create++ def patched(self, **kwargs): # type: ignore[no-untyped-def]+ if str(kwargs.get("model", "")).startswith(NO_TEMP_PREFIXES):+ kwargs.pop("temperature", None)+ return orig(self, **kwargs)++ patched._hybrid_archon_patched = True # type: ignore[attr-defined]+ cls.create = patched # type: ignore[assignment]+++# ---------- Per-run token tally + custom generators ----------++# Token tallies live in thread-local storage because the runner reuses one+# ArchonAgent across a ``ThreadPoolExecutor`` (``concurrency > 1``). A plain+# module-global dict would let concurrent ``_run_paradigm`` calls reset and+# read each other's counters, producing wrong ``cost_usd`` and+# ``tokens_local`` / ``tokens_cloud`` in per-task ``meta``.+_TALLY_LOCAL = threading.local()+++def _tally() -> Dict[str, int]:+ """Return the calling thread's token tally, creating it on first touch."""+ counts = getattr(_TALLY_LOCAL, "counts", None)+ if counts is None:+ counts = {+ "cloud_prompt": 0, "cloud_completion": 0,+ "local_prompt": 0, "local_completion": 0,+ }+ _TALLY_LOCAL.counts = counts+ return counts+++def _reset_tally() -> None:+ _TALLY_LOCAL.counts = {+ "cloud_prompt": 0, "cloud_completion": 0,+ "local_prompt": 0, "local_completion": 0,+ }+++def _make_local_generator(local_endpoint: str, local_model: str):+ """Archon custom-generator signature: (model, messages, max_tokens, temperature) -> str."""+ from openai import OpenAI++ client = OpenAI(base_url=local_endpoint, api_key="EMPTY")++ def local_gen(model, messages, max_tokens=2048, temperature=0.7, **_kw): # type: ignore[no-untyped-def]+ import time as _time+ t0 = _time.time()+ try:+ resp = client.chat.completions.create(+ model=local_model,+ messages=messages,+ max_tokens=max_tokens,+ temperature=temperature,+ )+ except Exception as e:+ _record_event({+ "kind": "archon_local_gen_error",+ "model": local_model,+ "messages": messages,+ "error": f"{type(e).__name__}: {e}",+ "ts": _time.time(),+ })+ return f"[local-vllm error: {e!r}]"+ u = resp.usage+ if u:+ _tally()["local_prompt"] += getattr(u, "prompt_tokens", 0) or 0+ _tally()["local_completion"] += getattr(u, "completion_tokens", 0) or 0+ text = (resp.choices[0].message.content or "").strip()+ _record_event({+ "kind": "archon_local_gen",+ "model": local_model,+ "messages": messages,+ "response": text,+ "tokens_in": getattr(u, "prompt_tokens", 0) if u else 0,+ "tokens_out": getattr(u, "completion_tokens", 0) if u else 0,+ "temperature": temperature,+ "max_tokens": max_tokens,+ "latency_s": _time.time() - t0,+ "ts": _time.time(),+ })+ return text++ return local_gen+++def _wrap_archon_cloud_generators() -> None:+ """Replace Archon's GENERATE_MAP entries for OpenAI_API / Anthropic_API+ with token-tallying wrappers. GENERATE_MAP holds function references;+ we update them in place so Ranker/Fuser see the wrapped versions."""+ import anthropic as _anth+ from openai import OpenAI as _OAI++ def gen_openai(model, messages, max_tokens=2048, temperature=0.7, **_kw): # type: ignore[no-untyped-def]+ import time as _time+ client = _OAI()+ kwargs: Dict[str, Any] = dict(+ model=model, messages=messages,+ max_tokens=max_tokens, temperature=temperature,+ )+ # GPT-5/o1/o3 reject non-default temperature and use max_completion_tokens.+ if model.startswith(("gpt-5", "o1", "o3")):+ kwargs.pop("temperature", None)+ kwargs.pop("max_tokens", None)+ kwargs["max_completion_tokens"] = max_tokens+ t0 = _time.time()+ resp = client.chat.completions.create(**kwargs)+ u = resp.usage+ if u:+ _tally()["cloud_prompt"] += getattr(u, "prompt_tokens", 0) or 0+ _tally()["cloud_completion"] += getattr(u, "completion_tokens", 0) or 0+ text = (resp.choices[0].message.content or "").strip()+ _record_event({+ "kind": "archon_cloud_openai",+ "model": model,+ "messages": messages,+ "response": text,+ "tokens_in": getattr(u, "prompt_tokens", 0) if u else 0,+ "tokens_out": getattr(u, "completion_tokens", 0) if u else 0,+ "latency_s": _time.time() - t0,+ "ts": _time.time(),+ })+ return text++ def gen_anthropic(model, messages, max_tokens=2048, temperature=0.7, **_kw): # type: ignore[no-untyped-def]+ import time as _time+ client = _anth.Anthropic(timeout=600.0)+ system = ""+ msgs = []+ for m in messages:+ if m["role"] == "system" and not system:+ system = m["content"]+ else:+ msgs.append(m)+ kwargs: Dict[str, Any] = dict(+ model=model, system=system, messages=msgs, max_tokens=max_tokens,+ )+ if not model.startswith(NO_TEMP_PREFIXES):+ kwargs["temperature"] = temperature+ t0 = _time.time()+ resp = client.messages.create(**kwargs)+ text = "".join(b.text for b in resp.content if hasattr(b, "text"))+ u = resp.usage+ if u:+ _tally()["cloud_prompt"] += getattr(u, "input_tokens", 0) or 0+ _tally()["cloud_completion"] += getattr(u, "output_tokens", 0) or 0+ _record_event({+ "kind": "archon_cloud_anthropic",+ "model": model,+ "system": system,+ "messages": msgs,+ "response": text.strip(),+ "tokens_in": getattr(u, "input_tokens", 0) if u else 0,+ "tokens_out": getattr(u, "output_tokens", 0) if u else 0,+ "latency_s": _time.time() - t0,+ "ts": _time.time(),+ })+ return text.strip()++ from archon.completions.components.Generator import (+ GENERATE_MAP as _GMAP, # type: ignore[import-not-found]+ )+ _GMAP["OpenAI_API"] = gen_openai+ _GMAP["Anthropic_API"] = gen_anthropic+++_FUSER_FORMAT_REMINDER = (+ "\n\nIMPORTANT — output format: your synthesized response MUST honor the "+ "output-format requirements stated in the original user query above. "+ "If the query requires ending with `FINAL ANSWER: <answer>` (GAIA), end "+ "with exactly that one line and nothing after it. If the query requires "+ "a unified diff inside a ```diff … ``` fence (SWE-bench), end with that "+ "fence and nothing after the closing ```. Do not produce free-form "+ "analysis, markdown headers, or commentary that breaks the required "+ "format — the candidate responses may have done so, but your fused "+ "answer must not."+)+++def _patch_archon_prompts() -> None:+ """Make Archon's fuser bench-aware.++ ``Fuser.py`` drops the system message we hand to ``archon.generate()`` and+ builds its own user message via ``make_fuser_prompt``. That template tells+ the fuser to "synthesize into a refined, well-structured response" with+ no reminder of the format the user originally asked for, so on SWE-bench+ Opus drifts to a markdown bug report and scores 0. We append an explicit+ format-honor reminder to the fuser prompt."""+ from archon.completions.components import (+ prompts as _p, # type: ignore[import-not-found]+ )++ if getattr(_p.make_fuser_prompt, "_hybrid_format_patched", False):+ return+ orig = _p.make_fuser_prompt++ def patched(conv, references, critiques=None, length_control=False): # type: ignore[no-untyped-def]+ base = orig(conv, references, critiques=critiques, length_control=length_control)+ return base + _FUSER_FORMAT_REMINDER++ patched._hybrid_format_patched = True # type: ignore[attr-defined]+ _p.make_fuser_prompt = patched+ # Fuser imports the symbol by name at class-body time — rebind there too.+ from archon.completions.components import (+ Fuser as _F, # type: ignore[import-not-found]+ )+ _F.make_fuser_prompt = patched+++_PATCHES_APPLIED = False+++def _apply_patches_once() -> None:+ global _PATCHES_APPLIED+ if _PATCHES_APPLIED:+ return+ _stub_archon_imports()+ _add_archon_to_path()+ _patch_anthropic_for_opus()+ # Trigger Archon imports so GENERATE_MAP exists.+ import archon.completions.components.Generator # type: ignore[import-not-found] # noqa: F401+ _wrap_archon_cloud_generators()+ _patch_archon_prompts()+ _PATCHES_APPLIED = True+++# ---------- Architecture presets ----------++def _presets():+ return {+ "ensemble_rank_fuse": lambda K, local_model, ranker_model, fuser_model, max_tokens, temperature: [+ [{+ "type": "generator",+ "model": local_model,+ "model_type": "vllm_local",+ "top_k": 1,+ "temperature": temperature,+ "max_tokens": max_tokens,+ "samples": K,+ }],+ [{+ "type": "ranker",+ "model": ranker_model,+ "model_type": "Anthropic_API" if ranker_model.startswith("claude") else "OpenAI_API",+ "top_k": min(K, 5),+ "temperature": 0.0,+ "max_tokens": max_tokens,+ }],+ [{+ "type": "fuser",+ "model": fuser_model,+ "model_type": "Anthropic_API" if fuser_model.startswith("claude") else "OpenAI_API",+ "temperature": 0.0,+ "max_tokens": max_tokens,+ "samples": 1,+ }],+ ],+ "single_local": lambda K, local_model, *_a, **_kw: [+ [{+ "type": "generator",+ "model": local_model,+ "model_type": "vllm_local",+ "top_k": 1,+ "temperature": 0.0,+ "max_tokens": 2048,+ "samples": 1,+ }],+ ],+ }++[email protected]("archon")+class ArchonAgent(LocalCloudAgent):+ """Layered (generator → ranker → fuser) inference-time search."""++ agent_id = "archon"++ def _run_paradigm(+ self,+ input: str,+ context: Optional[AgentContext],+ **kwargs: Any,+ ) -> Tuple[str, Dict[str, Any]]:+ if "OPENAI_API_KEY" not in os.environ and "ANTHROPIC_API_KEY" not in os.environ:+ raise RuntimeError("Archon needs OPENAI_API_KEY and/or ANTHROPIC_API_KEY")++ cfg = self._cfg+ task_meta = (context.metadata.get("task") if context is not None else {}) or {}+ swe_mode = (+ bool(cfg.get("swe_use_agent_loop"))+ and bool(task_meta.get("problem_statement"))+ and bool(task_meta.get("repo"))+ and bool(task_meta.get("base_commit"))+ )+ if swe_mode:+ return self._run_swe(input, task_meta, cfg)++ _apply_patches_once()+ from archon.completions import Archon # type: ignore[import-not-found]+ from archon.completions.components.Generator import (+ GENERATE_MAP as _GMAP, # type: ignore[import-not-found]+ )++ arch = cfg.get("architecture", "ensemble_rank_fuse")+ presets = _presets()+ if arch not in presets:+ raise ValueError(+ f"unknown archon architecture {arch!r}, choose from {sorted(presets)}"+ )+ if not self._local_endpoint or not self._local_model:+ raise ValueError(+ "ArchonAgent needs local_model + local_endpoint; got "+ f"model={self._local_model!r} endpoint={self._local_endpoint!r}"+ )++ K = int(cfg.get("n_samples", 5))+ max_tokens = int(cfg.get("max_tokens", 2048))+ temperature = float(cfg.get("temperature", 0.7))+ ranker_model = cfg.get("ranker_model", self._cloud_model)+ fuser_model = cfg.get("fuser_model", self._cloud_model)++ # Register our local-vLLM generator per-run (endpoint can vary per cell).+ _GMAP["vllm_local"] = _make_local_generator(+ self._local_endpoint, self._local_model+ )++ layers = presets[arch](+ K, self._local_model, ranker_model, fuser_model, max_tokens, temperature,+ )+ archon_cfg = {"name": f"hybrid-archon-{arch}", "layers": layers}++ _reset_tally()+ archon = Archon(archon_cfg)++ try:+ answer = archon.generate([+ {"role": "system", "content": "You are a helpful assistant."},+ {"role": "user", "content": input},+ ])+ except Exception as e:+ answer = f"[archon error: {e!r}]"++ if isinstance(answer, list):+ answer = answer[-1] if answer else ""+ answer = str(answer)++ cp = _tally()["cloud_prompt"]+ cc = _tally()["cloud_completion"]+ cost = _cost_cloud(ranker_model, cp, cc)+ if fuser_model != ranker_model:+ # Conservative: charge both at the more expensive of the two.+ cost = max(cost, _cost_cloud(fuser_model, cp, cc))++ meta = {+ "tokens_local": _tally()["local_prompt"] + _tally()["local_completion"],+ "tokens_cloud": cp + cc,+ "cost_usd": cost,+ "turns": (K + 2) if arch == "ensemble_rank_fuse" else 1,+ "traces": {+ "architecture": arch,+ "n_samples": K,+ "ranker_model": ranker_model,+ "fuser_model": fuser_model,+ "local_model": self._local_model,+ "tokens_breakdown": dict(_tally()),+ },+ }+ return answer, meta++ # ------------------------------------------------------------------+ # SWE-bench variant: K diverse mini-SWE-agent runs, cloud ranker picks+ # ------------------------------------------------------------------++ def _run_swe(+ self,+ input: str,+ task: Dict[str, Any],+ cfg: Dict[str, Any],+ ) -> Tuple[str, Dict[str, Any]]:+ if not self._local_endpoint or not self._local_model:+ raise ValueError(+ "ArchonAgent (swe mode) needs local_model + local_endpoint"+ )++ K = int(cfg.get("n_samples", 3))+ max_turns = int(cfg.get("swe_max_turns", 30))+ bash_timeout = int(cfg.get("swe_bash_timeout_s", 120))+ output_cap = int(cfg.get("swe_output_cap", 10_000))+ turn_max_tokens = int(cfg.get("swe_turn_max_tokens", 4096))++ # K independent runs, each on a FRESH workdir → K diverse patches.+ candidates: List[Dict[str, Any]] = []+ total_tokens_local = 0+ total_tokens_cloud = 0+ total_cost = 0.0+ for k in range(K):+ out = run_swe_agent_loop(+ task,+ backbone="local",+ backbone_model=self._local_model,+ local_endpoint=self._local_endpoint,+ initial_prompt=input,+ max_turns=max_turns,+ bash_timeout=bash_timeout,+ output_cap=output_cap,+ turn_max_tokens=turn_max_tokens,+ trace_prefix=f"archon_gen{k}",+ )+ candidates.append({+ "idx": k,+ "summary": out["final_summary"],+ "patch": out["patch"],+ "framed": out["answer"],+ "tokens_in": out["tokens_in"],+ "tokens_out": out["tokens_out"],+ "turns": out["turns"],+ })+ total_tokens_local += out["tokens_in"] + out["tokens_out"]+ self.record_trace_event({+ "kind": "archon_swe_candidate",+ "idx": k,+ "patch_chars": len(out["patch"]),+ "summary": out["final_summary"],+ })++ # Ranker: cloud picks the best candidate.+ ranker_user = (+ f"Issue:\n{task.get('problem_statement','')}\n\n"+ f"K = {K} candidate patches:\n\n"+ + "\n\n".join(+ f"=== Candidate {c['idx']} ===\nSummary: {c['summary']}\n"+ f"Patch (chars={len(c['patch'])}):\n```diff\n{c['patch']}```"+ for c in candidates+ )+ )+ ranker_text, r_in, r_out = self._call_cloud(+ user=ranker_user,+ system=ARCHON_SWE_RANKER_SYS,+ max_tokens=int(cfg.get("ranker_max_tokens", 1024)),+ temperature=0.0,+ )+ total_tokens_cloud += r_in + r_out+ total_cost += self.cost_usd(self._cloud_model, r_in, r_out)++ # Parse — first integer on its own line.+ chosen_idx = 0+ for line in ranker_text.splitlines():+ stripped = line.strip()+ if stripped.isdigit():+ chosen_idx = int(stripped)+ break+ if not (0 <= chosen_idx < K):+ chosen_idx = 0+ chosen = candidates[chosen_idx]++ self.record_trace_event({+ "kind": "archon_swe_rank",+ "chosen_idx": chosen_idx,+ "ranker_raw": ranker_text,+ "tokens_in": r_in,+ "tokens_out": r_out,+ })++ meta = {+ "tokens_local": total_tokens_local,+ "tokens_cloud": total_tokens_cloud,+ "cost_usd": total_cost,+ "turns": sum(c["turns"] for c in candidates) + 1,+ "traces": {+ "swe_mode": True,+ "K": K,+ "candidates": [+ {"idx": c["idx"], "summary": c["summary"],+ "patch_chars": len(c["patch"]), "turns": c["turns"]}+ for c in candidates+ ],+ "chosen_idx": chosen_idx,+ "ranker_text": ranker_text,+ },+ }+ return chosen["framed"], meta+++__all__ = ["ArchonAgent"]diff --git a/src/openjarvis/agents/hybrid/conductor.py b/src/openjarvis/agents/hybrid/conductor.pynew file mode 100644index 00000000..f21e47e9--- /dev/null+++ b/src/openjarvis/agents/hybrid/conductor.py@@ -0,0 +1,514 @@+"""ConductorAgent — static-DAG planner (Sakana AI, arXiv 2512.04388).++Stage-1 inference-only repro. The paper's trained Qwen2.5-7B conductor is+not released; we substitute a strong zero-shot cloud planner (default+Opus) and run the same plan-then-execute machinery.++Pipeline per task:++1. **Plan** — the conductor reads the question + numbered worker pool+ and emits three lists ``(model_id, subtasks, access_list)`` in JSON,+ up to 5 steps.+2. **Execute** — for each step ``i``: build the worker prompt from+ ``subtasks[i]`` + the concatenated prior ``(subtask, output)``+ messages selected by ``access_list[i]``; call worker+ ``model_id[i]``; the final answer is the output of the last step.++On plan parse failure: retry once with a stricter "JSON only" prompt;+on second failure, fall back to a single call to the strongest available+worker (last in the pool by convention).++Workers come from ``cfg["workers"]`` or a sensible default pool+(local Qwen if vLLM is up, plus Opus 4.7 and gpt-5-mini).++Hybrid harness result: ``conductor-swebenchverified-opusplan-30`` = 0.367+acc / $0.22 per task — +10pp vs baseline-cloud at ~15× cheaper.++Ported from ``hybrid-local-cloud-compute/adapters/conductor_adapter.py``.+"""++from __future__ import annotations++import ast+import json+import re+import shutil+import tempfile+import urllib.request+from pathlib import Path+from typing import Any, Dict, List, Optional, Tuple++from openjarvis.agents._stubs import AgentContext+from openjarvis.agents.hybrid._base import LocalCloudAgent+from openjarvis.agents.hybrid._prices import (+ is_gpt5_family,+ supports_temperature,+)+from openjarvis.agents.hybrid.mini_swe_agent import (+ _clone_repo,+ _extract_diff,+ run_swe_agent_loop,+)+from openjarvis.core.registry import AgentRegistry++CONDUCTOR_SYS = """\+Your role as an assistant involves obtaining answers to questions by an iterative \+process of querying powerful language models, each with a different skillset. \+You will be given the user question and a list of available numbered language \+models with their metadata.++Plan up to 5 workflow steps. Output THREE lists of equal length:++ model_id: integers (0..N-1) selecting which numbered model handles each step.+ subtasks: natural-language instructions (one string per step) for that model.+ access_list: for each step, a list of prior step indices whose (subtask, output)+ should be included in that step's prompt; use the string "all" to+ include every prior step, or [] for none. The first step must use [].++Pick the smallest number of steps that will reliably produce a correct final answer. \+The user only sees the output of the LAST step, so make sure the last step both \+solves the task and produces the final user-facing answer in the requested format.++Respond ONLY with a single JSON object (no prose, no markdown fence) with exactly \+these three keys:++ {"model_id": [...], "subtasks": [...], "access_list": [...]}+"""++CONDUCTOR_STRICTER = (+ "Your previous response was not valid JSON or was missing required fields. "+ "Reply with ONLY a single JSON object — no prose, no code fences, no commentary "+ "— containing exactly the three keys model_id (list[int]), subtasks (list[str]), "+ "and access_list (list[list[int] or \"all\"]) of equal length, at most 5 entries, "+ "and access_list[0] must be [] (an empty list)."+)+++# ---------- Plan parsing ----------++def _strip_fences(s: str) -> str:+ s = s.strip()+ if s.startswith("```"):+ first_nl = s.find("\n")+ if first_nl != -1:+ s = s[first_nl + 1:]+ if s.endswith("```"):+ s = s[:-3]+ s = s.strip()+ return s+++def _try_json(s: str):+ try:+ return json.loads(s)+ except Exception:+ return None+++def _try_literal(s: str):+ """Fallback for the paper's literal Python-list output style."""+ out = {}+ for key in ("model_id", "subtasks", "access_list"):+ m = re.search(+ rf"{key}\s*=\s*(\[[^\]]*\](?:\s*\+\s*\[[^\]]*\])*)", s, re.DOTALL+ )+ if not m:+ return None+ try:+ out[key] = ast.literal_eval(m.group(1))+ except Exception:+ return None+ return out+++def _validate_plan(plan: Any, n_workers: int) -> Optional[str]:+ if not isinstance(plan, dict):+ return "plan is not a dict"+ for k in ("model_id", "subtasks", "access_list"):+ if k not in plan:+ return f"missing key {k!r}"+ mi, st, al = plan["model_id"], plan["subtasks"], plan["access_list"]+ if not (isinstance(mi, list) and isinstance(st, list) and isinstance(al, list)):+ return "fields must be lists"+ if not (len(mi) == len(st) == len(al)):+ return f"unequal lengths: {len(mi)}/{len(st)}/{len(al)}"+ if not (1 <= len(mi) <= 5):+ return f"need 1..5 steps, got {len(mi)}"+ for i, m in enumerate(mi):+ if not isinstance(m, int) or not (0 <= m < n_workers):+ return f"model_id[{i}]={m!r} out of range 0..{n_workers - 1}"+ for i, s in enumerate(st):+ if not isinstance(s, str) or not s.strip():+ return f"subtasks[{i}] must be non-empty string"+ for i, a in enumerate(al):+ if a == "all":+ continue+ if not isinstance(a, list):+ return f"access_list[{i}] must be list or \"all\""+ for j in a:+ if not isinstance(j, int) or not (0 <= j < i):+ return f"access_list[{i}] has bad ref {j!r}"+ return None+++def _parse_plan(text: str, n_workers: int):+ s = _strip_fences(text)+ for candidate in (_try_json(s), _try_literal(s)):+ if candidate is None:+ continue+ err = _validate_plan(candidate, n_workers)+ if err is None:+ return candidate, None+ return None, "could not parse a valid plan"+++# ---------- Worker pool ----------++def _vllm_alive(base_url: str) -> bool:+ try:+ with urllib.request.urlopen(+ base_url.rstrip("/") + "/models", timeout=3+ ) as r:+ return r.status == 200+ except Exception:+ return False+++def _default_pool(local_model: Optional[str], local_endpoint: Optional[str]) -> List[Dict[str, Any]]:+ pool: List[Dict[str, Any]] = []+ if local_model and local_endpoint and _vllm_alive(local_endpoint):+ pool.append({+ "id": len(pool),+ "name": "local-qwen",+ "endpoint": "vllm",+ "model": local_model,+ "base_url": local_endpoint,+ "api_key": "EMPTY",+ "description": (+ "Open-weights Qwen3.5 served locally. Cheap and fast. Good at "+ "concise extraction, formatting, arithmetic on given data; "+ "weaker at open-domain factual recall and complex reasoning."+ ),+ })+ pool.append({+ "id": len(pool),+ "name": "frontier-anthropic",+ "endpoint": "anthropic",+ "model": "claude-opus-4-7",+ "description": (+ "Frontier reasoning model. Strongest at multi-step reasoning, "+ "careful instruction following, code, and writing. Expensive; "+ "use sparingly for hard or decisive steps."+ ),+ })+ pool.append({+ "id": len(pool),+ "name": "frontier-openai-mini",+ "endpoint": "openai",+ "model": "gpt-5-mini",+ "description": (+ "Mid-tier OpenAI model. Solid general knowledge and reasoning at "+ "a fraction of frontier cost. Good default for retrieval-style or "+ "broad-knowledge questions."+ ),+ })+ return pool+++def _format_worker_pool(workers: List[Dict[str, Any]]) -> str:+ return "\n".join(+ f"Model {w['id']} ({w['name']}): {w['description']}" for w in workers+ )+++def _build_conductor_prompt(question: str, workers: List[Dict[str, Any]]) -> str:+ return (+ f"Available models:\n{_format_worker_pool(workers)}\n\n"+ f"User question:\n{question}\n"+ )+++def _build_step_prompt(+ question: str,+ subtask: str,+ prior_steps: List[Dict[str, Any]],+ access: "list[int] | str",+) -> str:+ indices = list(range(len(prior_steps))) if access == "all" else list(access)+ pieces = [f"User question:\n{question}\n"]+ if indices:+ pieces.append("Previous routing messages:")+ for j in indices:+ ps = prior_steps[j]+ pieces.append(+ f"[Step {j} subtask]\n{ps['subtask']}\n"+ f"[Step {j} response]\n{ps['output']}"+ )+ pieces.append(f"Your subtask:\n{subtask}")+ return "\n\n".join(pieces)+++# ---------- Worker invocation ----------++def _call_worker(+ worker: Dict[str, Any], prompt: str, cfg: Dict[str, Any]+) -> Tuple[str, int, int, bool]:+ """Returns (text, p_tok, c_tok, is_local)."""+ ep = (worker.get("endpoint") or "openai").lower()+ max_tok = int(cfg.get("worker_max_tokens", 4096))+ temp = float(cfg.get("worker_temperature", 0.2))++ if ep == "vllm":+ text, p, c = LocalCloudAgent._call_vllm(+ worker["model"],+ worker["base_url"],+ user=prompt,+ max_tokens=max_tok,+ temperature=temp,+ enable_thinking=False,+ )+ return text, p, c, True+ if ep == "openai":+ text, p, c = LocalCloudAgent._call_openai(+ worker["model"],+ user=prompt,+ max_tokens=max_tok,+ temperature=(1.0 if is_gpt5_family(worker["model"]) else temp),+ )+ return text, p, c, False+ if ep == "anthropic":+ eff_temp = temp if supports_temperature(worker["model"]) else 0.0+ text, p, c, _ = LocalCloudAgent._call_anthropic(+ worker["model"],+ user=prompt,+ max_tokens=max_tok,+ temperature=eff_temp,+ )+ return text, p, c, False+ raise ValueError(f"unsupported worker endpoint: {ep!r}")+++def _swe_worker_step(+ worker: Dict[str, Any],+ task: Dict[str, Any],+ prompt: str,+ cfg: Dict[str, Any],+ workdir: Path,+ step_idx: int,+) -> Tuple[str, int, int, bool]:+ """Run one Conductor worker step as a mini-SWE-agent subloop on a shared+ workdir. Returns (final_summary_or_diff, tokens_in, tokens_out, is_local)+ in the same shape as ``_call_worker``."""+ ep = (worker.get("endpoint") or "openai").lower()+ if ep == "vllm":+ backbone, model, endpoint, is_local = (+ "local", worker["model"], worker.get("base_url"), True,+ )+ elif ep == "anthropic":+ backbone, model, endpoint, is_local = (+ "cloud", worker["model"], None, False,+ )+ else:+ # OpenAI workers (gpt-5-mini etc.) aren't supported as agent-loop+ # backbones today (the loop's tool-call format is Anthropic- or+ # OpenAI-via-vllm-shaped only). Fall back to one-shot for those —+ # SWE-bench-wise they were already weak; this preserves behavior.+ return _call_worker(worker, prompt, cfg)+ out = run_swe_agent_loop(+ task,+ backbone=backbone,+ backbone_model=model,+ cloud_endpoint="anthropic" if backbone == "cloud" else "anthropic",+ local_endpoint=endpoint,+ initial_prompt=prompt,+ max_turns=int(cfg.get("swe_max_turns", 30)),+ bash_timeout=int(cfg.get("swe_bash_timeout_s", 120)),+ output_cap=int(cfg.get("swe_output_cap", 10_000)),+ turn_max_tokens=int(cfg.get("swe_turn_max_tokens", 4096)),+ trace_prefix=f"conductor_step{step_idx}",+ workdir=workdir,+ )+ return out["final_summary"] or out["answer"], out["tokens_in"], out["tokens_out"], is_local++[email protected]("conductor")+class ConductorAgent(LocalCloudAgent):+ """Plan-then-execute static DAG over a worker pool. See module docstring."""++ agent_id = "conductor"++ def _run_paradigm(+ self,+ input: str,+ context: Optional[AgentContext],+ **kwargs: Any,+ ) -> Tuple[str, Dict[str, Any]]:+ question = input+ cfg = self._cfg+ workers = cfg.get("workers") or _default_pool(+ self._local_model, self._local_endpoint+ )+ if not workers:+ raise RuntimeError("conductor: empty worker pool")++ # 1. Plan+ user = _build_conductor_prompt(question, workers)+ plan_text, p_in, p_out = self._call_cloud(+ user=user,+ system=CONDUCTOR_SYS,+ max_tokens=int(cfg.get("conductor_max_tokens", 2048)),+ temperature=0.0,+ )+ plan, err = _parse_plan(plan_text, len(workers))+ parse_attempts = [{"text": plan_text, "error": err}]+ conductor_p_in, conductor_p_out = p_in, p_out++ if plan is None:+ plan_text2, p_in2, p_out2 = self._call_cloud(+ user=user,+ system=CONDUCTOR_SYS + "\n\n" + CONDUCTOR_STRICTER,+ max_tokens=int(cfg.get("conductor_max_tokens", 2048)),+ temperature=0.0,+ )+ conductor_p_in += p_in2+ conductor_p_out += p_out2+ plan, err2 = _parse_plan(plan_text2, len(workers))+ parse_attempts.append({"text": plan_text2, "error": err2})++ fallback_used = False+ if plan is None:+ fallback_used = True+ plan = {+ "model_id": [len(workers) - 1],+ "subtasks": [question],+ "access_list": [[]],+ }++ self.record_trace_event({+ "kind": "conductor_plan",+ "plan": plan,+ "fallback_used": fallback_used,+ "parse_attempts": parse_attempts,+ "workers": [+ {k: v for k, v in w.items() if k != "api_key"}+ for w in workers+ ],+ })++ # 2. Execute+ # If we're on a SWE-bench task AND cfg["swe_use_agent_loop"] is on,+ # every worker step runs through run_swe_agent_loop on a SHARED+ # workdir so step N+1 builds on step N's edits. The final patch is+ # whatever `git diff` produces after the last step.+ task_meta = (context.metadata.get("task") if context is not None else {}) or {}+ swe_mode = (+ bool(cfg.get("swe_use_agent_loop"))+ and bool(task_meta.get("problem_statement"))+ and bool(task_meta.get("repo"))+ and bool(task_meta.get("base_commit"))+ )+ steps: List[Dict[str, Any]] = []+ tokens_local = 0+ tokens_cloud = 0+ cost = 0.0+ final_answer = ""+ shared_workdir: Optional[Path] = None++ try:+ if swe_mode:+ shared_workdir = Path(tempfile.mkdtemp(+ prefix=f"conductor-swe-{task_meta.get('task_id','x')}-"+ ))+ _clone_repo(task_meta["repo"], task_meta["base_commit"], shared_workdir)+ self.record_trace_event({+ "kind": "conductor_swe_workdir",+ "workdir": str(shared_workdir),+ "repo": task_meta["repo"],+ "base_commit": task_meta["base_commit"],+ })++ for i, (mid, subtask, access) in enumerate(+ zip(plan["model_id"], plan["subtasks"], plan["access_list"])+ ):+ worker = workers[mid]+ prompt = _build_step_prompt(question, subtask, steps, access)+ self.record_trace_event({+ "kind": "conductor_step_dispatch",+ "step_idx": i,+ "worker_id": mid,+ "worker_name": worker["name"],+ "worker_model": worker["model"],+ "subtask": subtask,+ "access": access,+ "prompt": prompt,+ "swe_mode": swe_mode,+ })++ if swe_mode:+ text, w_in, w_out, is_local = _swe_worker_step(+ worker, task_meta, prompt, cfg, shared_workdir, i,+ )+ else:+ text, w_in, w_out, is_local = _call_worker(worker, prompt, cfg)++ if is_local:+ tokens_local += w_in + w_out+ else:+ tokens_cloud += w_in + w_out+ cost += self.cost_usd(worker["model"], w_in, w_out)+ steps.append({+ "step_idx": i,+ "model_id": mid,+ "worker_name": worker["name"],+ "worker_model": worker["model"],+ "subtask": subtask,+ "access": access,+ "output": text,+ "tokens_in": w_in,+ "tokens_out": w_out,+ })+ final_answer = text++ # For SWE mode, the authoritative patch is whatever lives in+ # the working tree at the end — replace whatever the last+ # worker emitted with the full diff (so scoring picks it up).+ if swe_mode and shared_workdir is not None:+ patch = _extract_diff(shared_workdir)+ if patch.strip():+ final_answer = (+ f"{final_answer}\n\n```diff\n{patch}```"+ if final_answer else f"```diff\n{patch}```"+ )+ finally:+ if shared_workdir is not None:+ shutil.rmtree(shared_workdir, ignore_errors=True)++ # Conductor (planner) cost goes into cloud bucket+ cost += self.cost_usd(self._cloud_model, conductor_p_in, conductor_p_out)+ tokens_cloud += conductor_p_in + conductor_p_out++ traces = [+ (s["step_idx"], s["model_id"], s["subtask"], s["output"])+ for s in steps+ ]++ meta = {+ "tokens_local": tokens_local,+ "tokens_cloud": tokens_cloud,+ "cost_usd": cost,+ "turns": len(steps) + 1, # planner + N execution steps+ "traces": {+ "steps": traces,+ "plan": plan,+ "fallback_used": fallback_used,+ "parse_attempts": parse_attempts,+ "workers": [+ {k: v for k, v in w.items() if k != "api_key"}+ for w in workers+ ],+ },+ }+ return final_answer, meta+++__all__ = ["ConductorAgent"]diff --git a/src/openjarvis/agents/hybrid/mini_swe_agent.py b/src/openjarvis/agents/hybrid/mini_swe_agent.pynew file mode 100644index 00000000..7fad36c7--- /dev/null+++ b/src/openjarvis/agents/hybrid/mini_swe_agent.py@@ -0,0 +1,635 @@+"""MiniSWEAgent — vendored, ~330-line port of mini-SWE-agent v2.++Single-LLM agent loop with a ``bash`` tool, run inside a per-task git+clone. The model iterates: read files, grep, run tests, edit, retry —+the environment-interaction loop that turns SWE-bench from "predict the+patch blind" (~0.30) into "actually fix the bug" (~0.77 for frontier+models).++Two ways to use this module:++1. **Standalone agent** — :class:`MiniSWEAgent` registered as+ ``mini_swe_agent``. Use it directly as the agent for a cell.+2. **As a worker subroutine inside another paradigm** — call+ :func:`run_swe_agent_loop(task, ...)`. Returns a dict with the final+ patch, token totals, cost, etc. This is how Minions / Conductor /+ Advisors / SkillOrchestra / ToolOrchestra / Archon swap their+ one-shot worker call for a real agent loop when running SWE-bench.++Differences vs. the upstream+(https://github.com/swe-agent/mini-swe-agent):++- No Docker sandbox. We clone the SWE-bench repo into a tempdir and+ exec bash there. Network is available (pip etc.). Treat outputs as+ untrusted — model can run ``rm -rf`` against its own workdir, but the+ workdir is disposable. Don't run this on a host with secrets in the+ CWD.+- One tool, ``bash``. No separate ``submit`` — the loop ends when the+ model produces a turn with no tool calls. We extract the patch from+ ``git diff`` in the workdir at that point.+- Trace events captured via the LocalCloudAgent thread-local trace+ buffer so every bash invocation + result lands in+ ``experiments/<cell>/logs/<task_id>.json``.+"""++from __future__ import annotations++import json+import shutil+import subprocess+import tempfile+import time+from pathlib import Path+from typing import Any, Dict, List, Optional, Tuple++from openjarvis.agents._stubs import AgentContext+from openjarvis.agents.hybrid._base import LocalCloudAgent, _record_event+from openjarvis.agents.hybrid._prices import (+ cost as estimate_cost,+)+from openjarvis.agents.hybrid._prices import (+ supports_temperature,+)+from openjarvis.core.registry import AgentRegistry++SYSTEM_PROMPT = """\+You are an expert software engineer fixing a bug in a Python repository. \+You have one tool, `bash`, that runs a shell command and returns stdout, \+stderr, and the exit code.++Your task:+1. Read the issue.+2. Use `bash` to explore the repo, read relevant files, and understand the bug.+3. Edit files to fix the bug. You can use `bash` for that too (sed, python -c '...', cat > file <<EOF, etc.).+4. Run the relevant tests with `bash` to confirm your fix.+5. When you are confident the bug is fixed, send one final assistant message \+WITH NO TOOL CALLS containing a brief one-line summary of what you changed. \+That ends the loop; the harness will read your changes via `git diff` against \+the base commit.++Rules:+- Each `bash` call already runs INSIDE the repository's working tree as cwd. \+You do NOT need to `cd` anywhere — just run `ls`, `cat path/to/file`, etc. \+relative to the repo root.+- Each `bash` call is a fresh shell — there's no persistent cwd, env, or \+shell state carried between calls (but cwd is reset to the repo root each \+call, so this is fine for normal exploration).+- Don't run `git commit`, `git stash`, or anything that mutates git state — \+your edits should live in the working tree so `git diff` picks them up.+- Keep individual command outputs under ~10K chars (use `head`, `tail`, \+`grep -n`, `wc`). Long outputs will be truncated.+- Don't ``exit``, ``logout``, or kill the shell.+"""++BASH_TOOL_ANTHROPIC = {+ "name": "bash",+ "description": (+ "Run a bash command in the repository root and return stdout, stderr, "+ "and the exit code. Each call is a fresh shell — no persistent state."+ ),+ "input_schema": {+ "type": "object",+ "properties": {+ "command": {+ "type": "string",+ "description": "The bash command to run.",+ },+ },+ "required": ["command"],+ },+}++BASH_TOOL_OPENAI = {+ "type": "function",+ "function": {+ "name": "bash",+ "description": BASH_TOOL_ANTHROPIC["description"],+ "parameters": BASH_TOOL_ANTHROPIC["input_schema"],+ },+}+++# ---------- Workdir / bash plumbing ----------++def _clone_repo(repo: str, base_commit: str, dest: Path) -> None:+ """Shallow-fetch the SWE-bench repo at the right commit into ``dest``."""+ url = f"https://github.com/{repo}.git"+ subprocess.run(+ ["git", "clone", "--quiet", url, str(dest)],+ check=True, timeout=300, capture_output=True,+ )+ subprocess.run(+ ["git", "checkout", "--quiet", base_commit],+ cwd=str(dest), check=True, timeout=120, capture_output=True,+ )+++def _run_bash(+ command: str, workdir: Path, *, timeout: int = 120, output_cap: int = 10_000+) -> Dict[str, Any]:+ """Run one shell command in ``workdir``. Returns dict with stdout, stderr,+ exit_code, and a ``truncated`` flag if output was clamped."""+ t0 = time.time()+ try:+ proc = subprocess.run(+ ["bash", "-lc", command],+ cwd=str(workdir),+ capture_output=True, text=True, timeout=timeout,+ )+ stdout = proc.stdout+ stderr = proc.stderr+ exit_code = proc.returncode+ timed_out = False+ except subprocess.TimeoutExpired as e:+ stdout = (e.stdout or "") if isinstance(e.stdout, str) else ""+ stderr = (e.stderr or "") if isinstance(e.stderr, str) else ""+ exit_code = -1+ timed_out = True+ truncated = False+ if len(stdout) > output_cap:+ stdout = stdout[:output_cap] + f"\n…[+{len(stdout) - output_cap} chars truncated]"+ truncated = True+ if len(stderr) > output_cap:+ stderr = stderr[:output_cap] + f"\n…[+{len(stderr) - output_cap} chars truncated]"+ truncated = True+ return {+ "stdout": stdout,+ "stderr": stderr,+ "exit_code": exit_code,+ "timed_out": timed_out,+ "truncated": truncated,+ "latency_s": time.time() - t0,+ }+++def _format_observation(result: Dict[str, Any]) -> str:+ parts = [f"exit_code: {result['exit_code']}"]+ if result.get("timed_out"):+ parts.append("[TIMED OUT]")+ if result.get("truncated"):+ parts.append("[output truncated]")+ if result["stdout"]:+ parts.append(f"--- stdout ---\n{result['stdout']}")+ if result["stderr"]:+ parts.append(f"--- stderr ---\n{result['stderr']}")+ return "\n".join(parts)+++def _extract_diff(workdir: Path) -> str:+ """``git diff`` against the base commit — the final SWE-bench patch."""+ proc = subprocess.run(+ ["git", "diff", "--no-color"],+ cwd=str(workdir), capture_output=True, text=True, timeout=60,+ )+ return proc.stdout+++def _anthropic_assistant_block(block: Any) -> Dict[str, Any]:+ """Convert an Anthropic content block back into the dict shape the API+ expects for assistant-role messages."""+ btype = getattr(block, "type", None)+ if btype == "tool_use":+ return {+ "type": "tool_use",+ "id": block.id,+ "name": block.name,+ "input": block.input,+ }+ if hasattr(block, "text"):+ return {"type": "text", "text": block.text}+ return {"type": btype or "unknown"}+++# ---------- Reusable agent-loop entry point ----------++def run_swe_agent_loop(+ task: Dict[str, Any],+ *,+ backbone: str, # "cloud" or "local"+ backbone_model: str,+ cloud_endpoint: str = "anthropic",+ local_endpoint: Optional[str] = None,+ initial_prompt: Optional[str] = None,+ max_turns: int = 30,+ bash_timeout: int = 120,+ output_cap: int = 10_000,+ turn_max_tokens: int = 4096,+ trace_prefix: str = "mini_swe",+ workdir: Optional[Path] = None,+) -> Dict[str, Any]:+ """Run a mini-SWE-agent loop for one SWE-bench task. Returns:++ .. code-block:: python++ {+ "answer": str, # final framed answer with ```diff fence+ "patch": str, # raw unified diff from git diff+ "final_summary": str, # the no-tool-call assistant text (may be empty)+ "tokens_in": int,+ "tokens_out": int,+ "tokens_local": int, # bookkeeping split for paradigms+ "tokens_cloud": int,+ "cost_usd": float,+ "turns": int,+ "max_turns_hit": bool,+ "workdir": str,+ }++ Captures every bash invocation + LLM turn into the active trace buffer+ via :func:`_record_event` from the LocalCloudAgent base, so callers+ don't have to do their own per-call instrumentation.++ Args:+ task: SWE-bench-shaped dict with ``repo`` + ``base_commit`` + ``task_id``+ + (optional) ``problem_statement`` / ``hints_text``.+ backbone: ``"cloud"`` to drive the loop with the cloud model+ (Anthropic only today), ``"local"`` for vLLM.+ backbone_model: model id for the loop's backbone.+ cloud_endpoint / local_endpoint: SDK targets.+ initial_prompt: if set, used as the first user message (paradigms+ embed orchestrator context in here). If None, falls back to the+ task's problem_statement.+ workdir: pre-cloned repo path. If None, this function clones the+ repo into a tempdir and cleans it up at the end. Paradigms that+ want to chain multiple subloops over the same working tree can+ manage their own workdir.+ """+ repo = task.get("repo") or ""+ base_commit = task.get("base_commit") or ""+ if not repo or not base_commit:+ raise ValueError(+ f"run_swe_agent_loop needs task['repo'] + task['base_commit']; "+ f"got repo={repo!r}, base_commit={base_commit!r}"+ )++ own_workdir = workdir is None+ if own_workdir:+ workdir = Path(tempfile.mkdtemp(+ prefix=f"mini-swe-{task.get('task_id','x')}-"+ ))+ try:+ _clone_repo(repo, base_commit, workdir)+ except Exception:+ shutil.rmtree(workdir, ignore_errors=True)+ raise++ _record_event({+ "kind": f"{trace_prefix}_setup",+ "repo": repo,+ "base_commit": base_commit,+ "workdir": str(workdir),+ "owns_workdir": own_workdir,+ "backbone": backbone,+ "backbone_model": backbone_model,+ "ts": time.time(),+ })++ user_prompt = initial_prompt or task.get("problem_statement") or ""++ try:+ if backbone == "cloud":+ result = _loop_cloud(+ user_prompt, workdir,+ model=backbone_model,+ cloud_endpoint=cloud_endpoint,+ max_turns=max_turns,+ bash_timeout=bash_timeout,+ output_cap=output_cap,+ turn_max_tokens=turn_max_tokens,+ trace_prefix=trace_prefix,+ )+ elif backbone == "local":+ if not local_endpoint:+ raise ValueError("run_swe_agent_loop(backbone='local') needs local_endpoint")+ result = _loop_local(+ user_prompt, workdir,+ model=backbone_model,+ endpoint=local_endpoint,+ max_turns=max_turns,+ bash_timeout=bash_timeout,+ output_cap=output_cap,+ turn_max_tokens=turn_max_tokens,+ trace_prefix=trace_prefix,+ )+ else:+ raise ValueError(f"unsupported backbone: {backbone!r}")++ patch = _extract_diff(workdir)+ framed = (result["final_summary"] or "[mini-swe-agent produced no summary text]")+ if patch.strip():+ framed = f"{framed}\n\n```diff\n{patch}```"++ return {+ "answer": framed,+ "patch": patch,+ "final_summary": result["final_summary"],+ "tokens_in": result["tokens_in"],+ "tokens_out": result["tokens_out"],+ "tokens_local": result["tokens_in"] + result["tokens_out"] if backbone == "local" else 0,+ "tokens_cloud": result["tokens_in"] + result["tokens_out"] if backbone == "cloud" else 0,+ "cost_usd": (+ estimate_cost(backbone_model, result["tokens_in"], result["tokens_out"])+ if backbone == "cloud" else 0.0+ ),+ "turns": result["turns"],+ "max_turns_hit": result["max_turns_hit"],+ "workdir": str(workdir),+ }+ finally:+ if own_workdir:+ shutil.rmtree(workdir, ignore_errors=True)+++# ---------- Cloud loop (Anthropic multi-turn with tools) ----------++def _loop_cloud(+ problem: str,+ workdir: Path,+ *,+ model: str,+ cloud_endpoint: str,+ max_turns: int,+ bash_timeout: int,+ output_cap: int,+ turn_max_tokens: int,+ trace_prefix: str,+) -> Dict[str, Any]:+ if cloud_endpoint != "anthropic":+ raise ValueError(+ f"mini-SWE-agent cloud backbone currently supports anthropic only; "+ f"got {cloud_endpoint!r}"+ )+ import anthropic+ client = anthropic.Anthropic(timeout=600.0, max_retries=5)+ messages: List[Dict[str, Any]] = [{"role": "user", "content": problem}]++ tokens_in = 0+ tokens_out = 0+ final_text = ""+ turns = 0+ for turn in range(1, max_turns + 1):+ turns = turn+ kwargs: Dict[str, Any] = {+ "model": model,+ "system": SYSTEM_PROMPT,+ "max_tokens": turn_max_tokens,+ "tools": [BASH_TOOL_ANTHROPIC],+ "messages": messages,+ }+ if supports_temperature(model):+ kwargs["temperature"] = 0.0+ t0 = time.time()+ msg = client.messages.create(**kwargs)+ latency = time.time() - t0+ tokens_in += msg.usage.input_tokens+ tokens_out += msg.usage.output_tokens++ content_blocks: List[Dict[str, Any]] = []+ tool_uses: List[Tuple[str, str, Dict[str, Any]]] = []+ text_parts: List[str] = []+ for block in msg.content:+ btype = getattr(block, "type", None)+ if btype == "tool_use":+ tool_uses.append((block.id, block.name, dict(block.input or {})))+ content_blocks.append({+ "type": "tool_use", "id": block.id, "name": block.name,+ "input": dict(block.input or {}),+ })+ elif hasattr(block, "text"):+ text_parts.append(block.text)+ content_blocks.append({"type": "text", "text": block.text})+ else:+ content_blocks.append({"type": btype or "unknown"})++ _record_event({+ "kind": f"{trace_prefix}_turn",+ "turn": turn,+ "stop_reason": msg.stop_reason,+ "tokens_in": msg.usage.input_tokens,+ "tokens_out": msg.usage.output_tokens,+ "latency_s": latency,+ "content_blocks": content_blocks,+ "ts": time.time(),+ })++ messages.append({"role": "assistant", "content": [+ _anthropic_assistant_block(b) for b in msg.content+ ]})++ if not tool_uses:+ final_text = "\n".join(text_parts).strip()+ break++ tool_result_blocks: List[Dict[str, Any]] = []+ for tu_id, tu_name, tu_input in tool_uses:+ if tu_name != "bash":+ obs = f"unknown tool: {tu_name!r}"+ _record_event({+ "kind": f"{trace_prefix}_unknown_tool",+ "turn": turn, "name": tu_name, "input": tu_input,+ "ts": time.time(),+ })+ else:+ command = str(tu_input.get("command", ""))+ result = _run_bash(+ command, workdir,+ timeout=bash_timeout, output_cap=output_cap,+ )+ _record_event({+ "kind": f"{trace_prefix}_bash",+ "turn": turn, "command": command,+ **result, "ts": time.time(),+ })+ obs = _format_observation(result)+ tool_result_blocks.append({+ "type": "tool_result",+ "tool_use_id": tu_id,+ "content": obs,+ })+ messages.append({"role": "user", "content": tool_result_blocks})++ return {+ "tokens_in": tokens_in,+ "tokens_out": tokens_out,+ "turns": turns,+ "final_summary": final_text,+ "max_turns_hit": turns == max_turns and not final_text,+ }+++# ---------- Local loop (vLLM, OpenAI-compatible multi-turn with tools) ----------++def _loop_local(+ problem: str,+ workdir: Path,+ *,+ model: str,+ endpoint: str,+ max_turns: int,+ bash_timeout: int,+ output_cap: int,+ turn_max_tokens: int,+ trace_prefix: str,+) -> Dict[str, Any]:+ from openai import OpenAI+ client = OpenAI(base_url=endpoint, api_key="EMPTY", timeout=600.0)++ messages: List[Dict[str, Any]] = [+ {"role": "system", "content": SYSTEM_PROMPT},+ {"role": "user", "content": problem},+ ]+ tokens_in = 0+ tokens_out = 0+ final_text = ""+ turns = 0+ for turn in range(1, max_turns + 1):+ turns = turn+ t0 = time.time()+ resp = client.chat.completions.create(+ model=model,+ messages=messages,+ temperature=0.0,+ max_tokens=turn_max_tokens,+ tools=[BASH_TOOL_OPENAI],+ tool_choice="auto",+ extra_body={"chat_template_kwargs": {"enable_thinking": False}},+ )+ latency = time.time() - t0+ u = resp.usage+ tokens_in += getattr(u, "prompt_tokens", 0) if u else 0+ tokens_out += getattr(u, "completion_tokens", 0) if u else 0+ choice = resp.choices[0]+ message = choice.message+ tool_calls = list(getattr(message, "tool_calls", None) or [])+ text = message.content or ""++ _record_event({+ "kind": f"{trace_prefix}_turn",+ "turn": turn,+ "finish_reason": choice.finish_reason,+ "tokens_in": getattr(u, "prompt_tokens", 0) if u else 0,+ "tokens_out": getattr(u, "completion_tokens", 0) if u else 0,+ "latency_s": latency,+ "text": text,+ "tool_calls": [+ {"id": tc.id, "name": tc.function.name, "arguments": tc.function.arguments}+ for tc in tool_calls+ ],+ "ts": time.time(),+ })++ messages.append({+ "role": "assistant",+ "content": text or None,+ "tool_calls": [+ {+ "id": tc.id, "type": "function",+ "function": {+ "name": tc.function.name,+ "arguments": tc.function.arguments,+ },+ }+ for tc in tool_calls+ ] if tool_calls else None,+ })++ if not tool_calls:+ final_text = text.strip()+ break++ for tc in tool_calls:+ try:+ args = json.loads(tc.function.arguments or "{}")+ except json.JSONDecodeError:+ args = {}+ if tc.function.name != "bash":+ obs = f"unknown tool: {tc.function.name!r}"+ else:+ command = str(args.get("command", ""))+ result = _run_bash(+ command, workdir,+ timeout=bash_timeout, output_cap=output_cap,+ )+ _record_event({+ "kind": f"{trace_prefix}_bash",+ "turn": turn, "command": command,+ **result, "ts": time.time(),+ })+ obs = _format_observation(result)+ messages.append({+ "role": "tool",+ "tool_call_id": tc.id,+ "content": obs,+ })++ return {+ "tokens_in": tokens_in,+ "tokens_out": tokens_out,+ "turns": turns,+ "final_summary": final_text,+ "max_turns_hit": turns == max_turns and not final_text,+ }+++# ---------- Standalone agent ----------+[email protected]("mini_swe_agent")+class MiniSWEAgent(LocalCloudAgent):+ """Single-model bash-loop agent for SWE-bench-shaped tasks.++ Configurable knobs via ``cfg``:++ - ``backbone`` (str, default ``"cloud"``): ``"cloud"`` or ``"local"``.+ - ``max_turns`` (int, default 30): hard cap on tool turns.+ - ``bash_timeout_s`` (int, default 120): per-command timeout.+ - ``output_cap`` (int, default 10_000): per-command stdout/stderr cap.+ - ``turn_max_tokens`` (int, default 4096): max_tokens per LLM turn.+ """++ agent_id = "mini_swe_agent"++ def _run_paradigm(+ self,+ input: str,+ context: Optional[AgentContext],+ **kwargs: Any,+ ) -> Tuple[str, Dict[str, Any]]:+ cfg = self._cfg+ task: Dict[str, Any] = {}+ if context is not None:+ task = context.metadata.get("task") or {}++ backbone = cfg.get("backbone", "cloud")+ model = (+ self._cloud_model if backbone == "cloud"+ else (self._local_model or "")+ )++ out = run_swe_agent_loop(+ task,+ backbone=backbone,+ backbone_model=model,+ cloud_endpoint=self._cloud_endpoint,+ local_endpoint=self._local_endpoint,+ initial_prompt=input,+ max_turns=int(cfg.get("max_turns", 30)),+ bash_timeout=int(cfg.get("bash_timeout_s", 120)),+ output_cap=int(cfg.get("output_cap", 10_000)),+ turn_max_tokens=int(cfg.get("turn_max_tokens", 4096)),+ )+ meta = {+ "tokens_local": out["tokens_local"],+ "tokens_cloud": out["tokens_cloud"],+ "cost_usd": out["cost_usd"],+ "turns": out["turns"],+ "traces": {+ "backbone": backbone,+ "max_turns_hit": out["max_turns_hit"],+ "patch_chars": len(out["patch"]),+ "final_summary": out["final_summary"],+ },+ }+ return out["answer"], meta+++__all__ = ["MiniSWEAgent", "run_swe_agent_loop", "SYSTEM_PROMPT"]diff --git a/src/openjarvis/agents/hybrid/minions.py b/src/openjarvis/agents/hybrid/minions.pynew file mode 100644index 00000000..73d064f8--- /dev/null+++ b/src/openjarvis/agents/hybrid/minions.py@@ -0,0 +1,554 @@+"""MinionsAgent — port of HazyResearch Minions protocol.++Cloud supervisor decomposes the task and reads back local-worker output;+local worker(s) do the bulk reading/extraction. Multi-turn loop until the+supervisor commits to a final answer.++Two modes (``cfg["mode"]``):++- ``"minion"`` — single local worker, one cloud supervisor (cheaper).+ Default.+- ``"minions"`` — parallel local workers, cloud aggregator.++Hybrid harness result: ``minions-swebenchverified-qwen27b-opus-500`` =+0.274 acc / $0.09 per task — beats baseline-cloud's 0.236 / $0.95 on+**both** accuracy and cost. GAIA at n=165 ties baseline-cloud at 0.576+acc / $0.67 (vs $1.09).++Requires the ``minions`` library from+https://github.com/HazyResearch/minions installed in the same env (e.g.+``uv pip install -e /matx/u/aspark/hybrid-local-cloud-compute/external/minions``).+Import is lazy — the agent class registers without ``minions`` available,+and the import error only fires on ``run()``.++Compatibility patches applied at first ``run()`` (idempotent):++- Strip ``temperature`` for Opus 4.7+ (rejected with 400).+- Inject server-side ``output_config`` JSON schema on supervisor turns+ so Opus replies in the shape Minions's parser expects (per-turn schema+ picked by sniffing the prompt for ``"decision": "provide_final_answer"``).+- Replace Minions's ``_extract_json`` with a wrapper that short-circuits+ when the response is already valid JSON.+- Inject ``timeout=600``/``max_retries=5`` defaults into+ ``anthropic.Anthropic()`` — Minions builds bare clients which 60s-timeout+ under SWE-bench concurrency=8.++Ported from ``hybrid-local-cloud-compute/adapters/minions_adapter.py``.+"""++from __future__ import annotations++import json as _json+import sys+import types+from typing import Any, Dict, List, Optional, Tuple++from openjarvis.agents._stubs import AgentContext+from openjarvis.agents.hybrid._base import (+ ANTHROPIC_WEB_SEARCH_TOOL,+ WEB_SEARCH_COST_PER_CALL,+ LocalCloudAgent,+)+from openjarvis.agents.hybrid._prices import NO_TEMP_PREFIXES+from openjarvis.agents.hybrid.mini_swe_agent import run_swe_agent_loop+from openjarvis.core.registry import AgentRegistry++MINIONS_SWE_PLANNER_SYS = (+ "You are the cloud supervisor in a Minions setup. The small local model "+ "is about to run an agent loop (with shell access) against a Python "+ "repository to fix a bug. Read the issue and write a concise plan: "+ "what files the local model should look at first, what tests are most "+ "relevant, and 1-3 concrete approaches it should try. Be specific — "+ "this is the local model's only briefing from you. Reply in 8 bullet "+ "points or fewer."+)+++# ---------- Per-turn JSON schemas (server-side enforcement) ----------+#+# Minions's supervisor produces different JSON shapes per turn:+# turn 1 (decompose): {reasoning, message}+# turn 2+ (continue): {decision="request_additional_info", message}+# turn 2+ (final answer): {decision="provide_final_answer", answer}+#+# Anthropic strict mode requires additionalProperties:false + all-props+# required, so we pick the schema PER TURN by sniffing the prompt for+# Minions's turn-2 template marker.++MINIONS_FIRST_TURN_SCHEMA = {+ "format": {+ "type": "json_schema",+ "schema": {+ "type": "object",+ "properties": {+ "reasoning": {"type": "string"},+ "message": {"type": "string"},+ },+ "required": ["reasoning", "message"],+ "additionalProperties": False,+ },+ }+}++MINIONS_CONVERSATION_SCHEMA = {+ "format": {+ "type": "json_schema",+ "schema": {+ "anyOf": [+ {+ "type": "object",+ "properties": {+ "decision": {"const": "request_additional_info"},+ "message": {"type": "string"},+ },+ "required": ["decision", "message"],+ "additionalProperties": False,+ },+ {+ "type": "object",+ "properties": {+ "decision": {"const": "provide_final_answer"},+ "answer": {"type": "string"},+ },+ "required": ["decision", "answer"],+ "additionalProperties": False,+ },+ ],+ },+ }+}++# Markers from Minions's supervisor prompts (prompts/minion.py). Any one+# being present in the call's messages/system is a strong Minions signal.+MINIONS_PROMPT_MARKERS = (+ "small language model that has read", # SUPERVISOR_INITIAL_PROMPT+ "provide_final_answer", # SUPERVISOR_CONVERSATION_PROMPT+ "request_additional_info",+)+++def _looks_like_minions_call(kwargs: Dict[str, Any]) -> bool:+ blob_parts = [str(kwargs.get("system", ""))]+ for msg in kwargs.get("messages", []) or []:+ content = msg.get("content", "") if isinstance(msg, dict) else ""+ blob_parts.append(str(content))+ blob = "\n".join(blob_parts)+ return any(m in blob for m in MINIONS_PROMPT_MARKERS)+++def _minions_turn_schema(kwargs: Dict[str, Any]) -> Dict[str, Any]:+ """Pick the schema by sniffing the prompt — see module docstring."""+ blob_parts = [str(kwargs.get("system", ""))]+ for msg in kwargs.get("messages", []) or []:+ content = msg.get("content", "") if isinstance(msg, dict) else ""+ blob_parts.append(str(content))+ blob = "\n".join(blob_parts)+ if (+ '"decision": "provide_final_answer"' in blob+ or '"decision":"provide_final_answer"' in blob+ ):+ return MINIONS_CONVERSATION_SCHEMA+ return MINIONS_FIRST_TURN_SCHEMA+++# ---------- Compatibility patches (idempotent) ----------++_PATCHES_APPLIED = False+++def _stub_missing_imports() -> None:+ """Minions's clients/__init__.py eager-imports every provider client.++ Two annoyances:+ 1. ``mistralai`` 2.x dropped top-level ``Mistral`` → ImportError.+ 2. The secure-chat path imports ``nv_attestation_sdk`` which writes a log+ file to CWD at import-time. We don't use secure chat.+ """+ try:+ import mistralai+ if not hasattr(mistralai, "Mistral"):+ mistralai.Mistral = type("Mistral", (), {}) # type: ignore[attr-defined]+ except ImportError:+ sys.modules["mistralai"] = types.ModuleType("mistralai")+ sys.modules["mistralai"].Mistral = type("Mistral", (), {}) # type: ignore[attr-defined]+ sys.modules.setdefault("nv_attestation_sdk", None) # type: ignore[arg-type]+++def _patch_anthropic_globally() -> None:+ import anthropic as _anth_mod+ from anthropic.resources.beta.messages import messages as _beta_msgs_mod+ from anthropic.resources.messages import messages as _msgs_mod++ # External Minions builds bare anthropic.Anthropic() clients (no timeout+ # / max_retries). Under concurrency=8 SWE-bench load those default to+ # ~60s and timeout in droves. Inject sane defaults at the constructor.+ if not getattr(_anth_mod.Anthropic.__init__, "_hybrid_patched", False):+ _orig_init = _anth_mod.Anthropic.__init__++ def _patched_init(self, *args, **kwargs): # type: ignore[no-untyped-def]+ kwargs.setdefault("timeout", 600.0)+ kwargs.setdefault("max_retries", 5)+ return _orig_init(self, *args, **kwargs)++ _patched_init._hybrid_patched = True # type: ignore[attr-defined]+ _anth_mod.Anthropic.__init__ = _patched_init # type: ignore[assignment]++ for cls in (_msgs_mod.Messages, _beta_msgs_mod.Messages):+ if getattr(cls.create, "_hybrid_patched", False):+ continue+ orig = cls.create++ def make_patched(orig): # type: ignore[no-untyped-def]+ def patched(self, **kwargs): # type: ignore[no-untyped-def]+ model = kwargs.get("model", "")+ if model.startswith(NO_TEMP_PREFIXES):+ kwargs.pop("temperature", None)+ if (+ "output_config" not in kwargs+ and _looks_like_minions_call(kwargs)+ ):+ kwargs["output_config"] = _minions_turn_schema(kwargs)+ return orig(self, **kwargs)+ patched._hybrid_patched = True # type: ignore[attr-defined]+ return patched++ cls.create = make_patched(orig) # type: ignore[assignment]+++def _patch_minions_extract_json() -> None:+ """Minions's ``_extract_json`` uses a non-greedy regex that grabs the+ first short bracket pair and prefers ```json``` fences. With structured+ outputs the entire response IS valid JSON, so short-circuit on that.+ """+ from minions import minion as _minion_mod # type: ignore[import-not-found]++ if getattr(_minion_mod._extract_json, "_hybrid_patched", False):+ return+ _orig = _minion_mod._extract_json++ def patched(text): # type: ignore[no-untyped-def]+ s = (text or "").strip()+ if s.startswith("{") and s.endswith("}"):+ try:+ return _json.loads(s)+ except _json.JSONDecodeError:+ pass+ return _orig(text)++ patched._hybrid_patched = True # type: ignore[attr-defined]+ _minion_mod._extract_json = patched # type: ignore[assignment]+++def _apply_patches_once() -> None:+ global _PATCHES_APPLIED+ if _PATCHES_APPLIED:+ return+ _stub_missing_imports()+ _patch_anthropic_globally()+ _patch_minions_extract_json()+ _PATCHES_APPLIED = True+++# ---------- Pre-fetch helper (GAIA only) ----------++def _prefetch_context(question: str, cloud_endpoint: str, cloud_model: str) -> Dict[str, Any]:+ """Use Anthropic web_search to fetch real source material the worker can read.++ Minions's premise is "worker reads a doc, asks cloud for help" — but GAIA+ tasks ship with no doc, so we synthesize one by having Opus do an actual+ web search first and dump the results back as the worker's context.++ Returns {text, tokens, cost_usd, n_searches}. On any failure: empty text+ and zeros — the protocol still runs.+ """+ out: Dict[str, Any] = {+ "text": "", "tokens": 0, "cost_usd": 0.0, "n_searches": 0,+ }+ if cloud_endpoint != "anthropic" or not (question or "").strip():+ return out+ try:+ prompt = (+ "Research the following question using web_search. Do NOT answer it. "+ "Instead, gather all relevant facts, numbers, names, dates, sources, "+ "and direct quotes you find, and report them as a dense reference "+ "document with URLs. The downstream reader is a small LLM that "+ "needs raw material to reason over.\n\nQUESTION:\n" + question+ )+ text, p, c, n_searches = LocalCloudAgent._call_anthropic(+ cloud_model,+ user=prompt,+ max_tokens=8192,+ tools=[ANTHROPIC_WEB_SEARCH_TOOL],+ tool_choice={"type": "any"},+ )+ from openjarvis.agents.hybrid._prices import cost as _cost_usd+ out.update(+ text=text,+ tokens=p + c,+ cost_usd=_cost_usd(cloud_model, p, c) + n_searches * WEB_SEARCH_COST_PER_CALL,+ n_searches=n_searches,+ )+ except Exception as e:+ out["error"] = f"{type(e).__name__}: {e}"+ return out+++def _context_for(+ task: Optional[Dict[str, Any]], prefetched: str = ""+) -> List[str]:+ """Minions wants a context list."""+ bits: List[str] = []+ task = task or {}+ if task.get("hints_text"):+ bits.append(task["hints_text"])+ if task.get("problem_statement") and not task.get("question"):+ bits.append(task["problem_statement"])+ if prefetched:+ bits.append(prefetched)+ return bits or [""]+++# ---------- Main agent ----------+[email protected]("minions")+class MinionsAgent(LocalCloudAgent):+ """HazyResearch Minions supervisor/worker protocol. See module docstring."""++ agent_id = "minions"++ def _is_soft_failure(self, exc: BaseException) -> Optional[str]:+ # Known soft-failure modes: Qwen worker JSON malformed, Anthropic+ # 400/529, KeyError on missing schema fields.+ try:+ import anthropic+ if isinstance(exc, anthropic.BadRequestError):+ return f"{type(exc).__name__}: {str(exc)[:120]}"+ except Exception:+ pass+ if isinstance(exc, (_json.JSONDecodeError, ValueError, KeyError)):+ return f"{type(exc).__name__}: {str(exc)[:120]}"+ if "JSONDecodeError" in type(exc).__name__:+ return f"{type(exc).__name__}: {str(exc)[:120]}"+ if "prompt is too long" in str(exc):+ return f"{type(exc).__name__}: {str(exc)[:120]}"+ return None++ def _run_paradigm(+ self,+ input: str,+ context: Optional[AgentContext],+ **kwargs: Any,+ ) -> Tuple[str, Dict[str, Any]]:+ cfg = self._cfg+ task_meta: Dict[str, Any] = {}+ if context is not None:+ task_meta = context.metadata.get("task", {}) or {}++ # SWE-bench branch: the upstream Minions library doesn't fit+ # SWE-bench (it's "small model reads docs, big model summarizes").+ # Instead, mirror Minions's "cloud supervises, local does the+ # work" pattern: cloud writes a high-level fix plan, local Qwen+ # runs mini-SWE-agent with that plan as additional context.+ swe_mode = (+ bool(cfg.get("swe_use_agent_loop"))+ and bool(task_meta.get("problem_statement"))+ and bool(task_meta.get("repo"))+ and bool(task_meta.get("base_commit"))+ )+ if swe_mode:+ return self._run_swe(input, task_meta, cfg)++ _apply_patches_once()+ from minions.clients.anthropic import (+ AnthropicClient, # type: ignore[import-not-found]+ )+ from minions.clients.openai import (+ OpenAIClient, # type: ignore[import-not-found]+ )+ from minions.minion import Minion # type: ignore[import-not-found]+ from minions.minions import Minions # type: ignore[import-not-found]++ mode = cfg.get("mode", "minion")++ if not self._local_endpoint or not self._local_model:+ raise ValueError(+ "MinionsAgent needs local_model + local_endpoint; got "+ f"model={self._local_model!r} endpoint={self._local_endpoint!r}"+ )++ local_client = OpenAIClient(+ model_name=self._local_model,+ base_url=self._local_endpoint,+ api_key="EMPTY",+ temperature=cfg.get("local_temperature", 0.0),+ max_tokens=cfg.get("worker_max_tokens", 4096),+ local=True,+ )+ if self._cloud_endpoint == "openai":+ cloud_client = OpenAIClient(+ model_name=self._cloud_model,+ temperature=0.0,+ max_tokens=4096,+ )+ elif self._cloud_endpoint == "anthropic":+ # Temperature stripping is handled by the global patch above for Opus 4.7+.+ cloud_client = AnthropicClient(+ model_name=self._cloud_model,+ temperature=0.0,+ max_tokens=4096,+ )+ else:+ raise ValueError(f"unsupported cloud endpoint: {self._cloud_endpoint!r}")++ cls = Minions if mode == "minions" else Minion+ log_dir = cfg.get("log_dir") or "/tmp/minions_logs"+ protocol = cls(+ local_client=local_client,+ remote_client=cloud_client,+ max_rounds=cfg.get("max_rounds", 3),+ log_dir=log_dir,+ )++ # GAIA-shape only: prefetch a web_search digest so the worker has+ # something real to read. SWE-bench (problem_statement only) already+ # ships its own doc.+ prefetch: Dict[str, Any] = {+ "text": "", "tokens": 0, "cost_usd": 0.0, "n_searches": 0,+ }+ if task_meta.get("question"):+ prefetch = _prefetch_context(+ task_meta["question"], self._cloud_endpoint, self._cloud_model+ )++ if prefetch.get("text"):+ self.record_trace_event({+ "kind": "minions_prefetch",+ "n_searches": prefetch["n_searches"],+ "tokens": prefetch["tokens"],+ "cost_usd": prefetch["cost_usd"],+ "text": prefetch["text"],+ "error": prefetch.get("error"),+ })++ out = protocol(+ task=input, # full formatted prompt (with bench instruction)+ context=_context_for(task_meta, prefetched=prefetch["text"]),+ doc_metadata=cfg.get("doc_metadata", "task"),+ max_rounds=cfg.get("max_rounds", 3),+ )++ # The Minions library doesn't go through our SDK helpers, so the+ # auto-trace missed every turn. Record the protocol output directly —+ # supervisor_messages + worker_messages contain the full conversation.+ self.record_trace_event({+ "kind": "minions_protocol",+ "mode": mode,+ "supervisor_messages": out.get("supervisor_messages"),+ "worker_messages": out.get("worker_messages"),+ "timing": out.get("timing"),+ "log_file": out.get("log_file"),+ "final_answer": out.get("final_answer", ""),+ })++ local_usage = out.get("local_usage")+ remote_usage = out.get("remote_usage")+ lp = getattr(local_usage, "prompt_tokens", 0)+ lc = getattr(local_usage, "completion_tokens", 0)+ rp = getattr(remote_usage, "prompt_tokens", 0)+ rc = getattr(remote_usage, "completion_tokens", 0)++ meta = {+ "tokens_local": lp + lc,+ "tokens_cloud": (rp + rc) + prefetch["tokens"],+ "cost_usd": self.cost_usd(self._cloud_model, rp, rc) + prefetch["cost_usd"],+ "turns": cfg.get("max_rounds", 3),+ "traces": {+ "mode": mode,+ "supervisor_messages": out.get("supervisor_messages"),+ "worker_messages": out.get("worker_messages"),+ "timing": out.get("timing"),+ "log_file": out.get("log_file"),+ "prefetch": {+ "n_searches": prefetch["n_searches"],+ "tokens": prefetch["tokens"],+ "cost_usd": prefetch["cost_usd"],+ "chars": len(prefetch["text"]),+ "error": prefetch.get("error"),+ },+ },+ }+ return out.get("final_answer", ""), meta+++ # ------------------------------------------------------------------+ # SWE-bench variant+ # ------------------------------------------------------------------++ def _run_swe(+ self,+ input: str,+ task: Dict[str, Any],+ cfg: Dict[str, Any],+ ) -> Tuple[str, Dict[str, Any]]:+ if not self._local_endpoint or not self._local_model:+ raise ValueError(+ "MinionsAgent (swe mode) still needs local_model + local_endpoint"+ )+ # 1. Cloud supervisor writes a high-level plan (no tools).+ plan_text, p_in, p_out = self._call_cloud(+ user=(+ f"Issue:\n{task.get('problem_statement','')}\n\n"+ f"Repo: {task.get('repo','')}\n"+ f"Base commit: {task.get('base_commit','')}\n\n"+ f"{task.get('hints_text','')}"+ ),+ system=MINIONS_SWE_PLANNER_SYS,+ max_tokens=int(cfg.get("supervisor_max_tokens", 1024)),+ temperature=0.0,+ )+ self.record_trace_event({+ "kind": "minions_swe_plan",+ "plan": plan_text,+ "tokens_in": p_in,+ "tokens_out": p_out,+ })+ supervisor_cost = self.cost_usd(self._cloud_model, p_in, p_out)++ # 2. Local worker runs mini-SWE-agent with the plan as context.+ worker_prompt = (+ f"{input}\n\n"+ f"-----\n"+ f"A cloud supervisor reviewed this issue and wrote a fix plan "+ f"for you. Use it as guidance, but verify everything with the "+ f"actual code via your bash tool:\n\n{plan_text}"+ )+ out = run_swe_agent_loop(+ task,+ backbone="local",+ backbone_model=self._local_model,+ local_endpoint=self._local_endpoint,+ initial_prompt=worker_prompt,+ max_turns=int(cfg.get("swe_max_turns", 30)),+ bash_timeout=int(cfg.get("swe_bash_timeout_s", 120)),+ output_cap=int(cfg.get("swe_output_cap", 10_000)),+ turn_max_tokens=int(cfg.get("swe_turn_max_tokens", 4096)),+ trace_prefix="minions_worker",+ )++ meta = {+ "tokens_local": out["tokens_in"] + out["tokens_out"],+ "tokens_cloud": p_in + p_out,+ "cost_usd": supervisor_cost,+ "turns": 1 + out["turns"],+ "traces": {+ "swe_mode": True,+ "supervisor_plan": plan_text,+ "worker_final_summary": out["final_summary"],+ "worker_patch_chars": len(out["patch"]),+ "max_turns_hit": out["max_turns_hit"],+ },+ }+ return out["answer"], meta+++__all__ = ["MinionsAgent"]diff --git a/src/openjarvis/agents/hybrid/registry/advisors.toml b/src/openjarvis/agents/hybrid/registry/advisors.tomlnew file mode 100644index 00000000..4f366360--- /dev/null+++ b/src/openjarvis/agents/hybrid/registry/advisors.toml@@ -0,0 +1,41 @@+# Advisors cells (arXiv:2510.02453).+#+# Inference-only repro of the advisor paradigm: cloud drafts → local advisor+# critiques → cloud rewrites. The paper's RL-trained advisor isn't reproduced+# (no training in this harness) — these are the untrained-advisor lower bound.+# Adapter auto-resolves the local model name against /v1/models.++[cells.advisors-gaia-qwen9b-opus-3]+method = "advisors"+bench = "gaia"+n = 3+local = { model = "Qwen/Qwen3.5-9B", endpoint = "http://localhost:8001/v1" }+cloud = { model = "claude-opus-4-7", endpoint = "anthropic" }+method_cfg = {}++[cells.advisors-swebenchverified-qwen9b-opus-3]+method = "advisors"+bench = "swebench-verified"+n = 3+local = { model = "Qwen/Qwen3.5-27B-FP8", endpoint = "http://localhost:8001/v1" }+cloud = { model = "claude-opus-4-7", endpoint = "anthropic" }+method_cfg = {}+concurrency = 3++[cells.advisors-gaia-qwen9b-opus-30]+method = "advisors"+bench = "gaia"+n = 30+local = { model = "Qwen/Qwen3.5-27B-FP8", endpoint = "http://localhost:8001/v1" }+cloud = { model = "claude-opus-4-7", endpoint = "anthropic" }+method_cfg = {}+concurrency = 4++[cells.advisors-swebenchverified-qwen9b-opus-30]+method = "advisors"+bench = "swebench-verified"+n = 30+local = { model = "Qwen/Qwen3.5-27B-FP8", endpoint = "http://localhost:8001/v1" }+cloud = { model = "claude-opus-4-7", endpoint = "anthropic" }+method_cfg = {}+concurrency = 3diff --git a/src/openjarvis/agents/hybrid/registry/archon.toml b/src/openjarvis/agents/hybrid/registry/archon.tomlnew file mode 100644index 00000000..7a3c3096--- /dev/null+++ b/src/openjarvis/agents/hybrid/registry/archon.toml@@ -0,0 +1,94 @@+# Archon cells. Inference-time architecture search (arXiv:2409.15254).+#+# Hybrid wiring: K local proposers (vLLM on 8001/8004) → 1 cloud ranker →+# 1 cloud fuser. method_cfg.architecture = "ensemble_rank_fuse" is the+# only non-debug preset; "single_local" exists for adapter smoke tests+# (no cloud calls).+#+# Naming: archon-<bench>-<local-short>-<cloud-short>-<arch>-<K>-<N>++# ---- Stage 1 smokes (n=3) ----++[cells.archon-gaia-qwen27b-opus-ensemble-K3-3]+method = "archon"+bench = "gaia"+n = 3+local = { model = "Qwen/Qwen3.5-27B-FP8", endpoint = "http://localhost:8001/v1" }+cloud = { model = "claude-opus-4-7", endpoint = "anthropic" }+method_cfg = { architecture = "ensemble_rank_fuse", n_samples = 3, max_tokens = 1024, temperature = 0.7 }++# Cheap sanity smoke: local-only generator, zero cloud spend. Use to verify+# vLLM + Archon plumbing without paying for Opus.+[cells.archon-gaia-qwen27b-singlelocal-3]+method = "archon"+bench = "gaia"+n = 3+local = { model = "Qwen/Qwen3.5-27B-FP8", endpoint = "http://localhost:8001/v1" }+cloud = { model = "claude-opus-4-7", endpoint = "anthropic" }+method_cfg = { architecture = "single_local", max_tokens = 1024 }++[cells.archon-swebenchverified-qwen27b-opus-ensemble-K3-3]+method = "archon"+bench = "swebench-verified"+n = 3+local = { model = "Qwen/Qwen3.5-27B-FP8", endpoint = "http://localhost:8001/v1" }+cloud = { model = "claude-opus-4-7", endpoint = "anthropic" }+method_cfg = { architecture = "ensemble_rank_fuse", n_samples = 3, max_tokens = 1024, temperature = 0.7 }+concurrency = 3+++# ---- Stage 1.5 mid (n=30) ----++[cells.archon-gaia-qwen27b-opus-ensemble-K5-30]+method = "archon"+bench = "gaia"+n = 30+local = { model = "Qwen/Qwen3.5-27B-FP8", endpoint = "http://localhost:8001/v1" }+cloud = { model = "claude-opus-4-7", endpoint = "anthropic" }+method_cfg = { architecture = "ensemble_rank_fuse", n_samples = 5, max_tokens = 2048, temperature = 0.7 }+concurrency = 4++[cells.archon-gaia-qwen27b-singlelocal-30]+method = "archon"+bench = "gaia"+n = 30+local = { model = "Qwen/Qwen3.5-27B-FP8", endpoint = "http://localhost:8001/v1" }+cloud = { model = "claude-opus-4-7", endpoint = "anthropic" }+method_cfg = { architecture = "single_local", max_tokens = 2048 }+concurrency = 4++[cells.archon-gaia-gemma31b-opus-ensemble-K5-30]+method = "archon"+bench = "gaia"+n = 30+local = { model = "google/gemma-4-31B-it", endpoint = "http://localhost:8004/v1" }+cloud = { model = "claude-opus-4-7", endpoint = "anthropic" }+method_cfg = { architecture = "ensemble_rank_fuse", n_samples = 5, max_tokens = 2048, temperature = 0.7 }++[cells.archon-swebenchverified-qwen27b-opus-ensemble-K5-30]+method = "archon"+bench = "swebench-verified"+n = 30+local = { model = "Qwen/Qwen3.5-27B-FP8", endpoint = "http://localhost:8001/v1" }+cloud = { model = "claude-opus-4-7", endpoint = "anthropic" }+method_cfg = { architecture = "ensemble_rank_fuse", n_samples = 5, max_tokens = 2048, temperature = 0.7 }+concurrency = 3+++# ---- Stage 2 full ----++[cells.archon-gaia-qwen27b-opus-ensemble-K5-165]+method = "archon"+bench = "gaia"+n = 165+local = { model = "Qwen/Qwen3.5-27B-FP8", endpoint = "http://localhost:8001/v1" }+cloud = { model = "claude-opus-4-7", endpoint = "anthropic" }+method_cfg = { architecture = "ensemble_rank_fuse", n_samples = 5, max_tokens = 2048, temperature = 0.7 }++[cells.archon-gaia-gemma31b-opus-ensemble-K5-165]+method = "archon"+bench = "gaia"+n = 165+local = { model = "google/gemma-4-31B-it", endpoint = "http://localhost:8004/v1" }+cloud = { model = "claude-opus-4-7", endpoint = "anthropic" }+method_cfg = { architecture = "ensemble_rank_fuse", n_samples = 5, max_tokens = 2048, temperature = 0.7 }diff --git a/src/openjarvis/agents/hybrid/registry/conductor.toml b/src/openjarvis/agents/hybrid/registry/conductor.tomlnew file mode 100644index 00000000..6e7eda06--- /dev/null+++ b/src/openjarvis/agents/hybrid/registry/conductor.toml@@ -0,0 +1,54 @@+# Conductor cells. Inference-only repro of arXiv 2512.04388 (Sakana, 2025).+#+# Stage-1 substitutes the (untrained) Qwen2.5-7B conductor with a zero-shot+# frontier planner (Opus). Workers default to local Qwen3.5-27B + Opus ++# gpt-5-mini; the adapter auto-drops the local worker if vLLM is down.+# Override `method_cfg.workers` to customize the pool.+#+# Naming: conductor-<bench>-<conductor-short>-<N>++[cells.conductor-gaia-opusplan-5]+method = "conductor"+bench = "gaia"+n = 5+# vLLM is the default local worker; auto-skipped if /v1/models is unreachable.+local = { model = "Qwen/Qwen3.5-27B-FP8", endpoint = "http://localhost:8001/v1" }+cloud = { model = "claude-opus-4-7", endpoint = "anthropic" }+method_cfg = { worker_max_tokens = 4096, worker_temperature = 0.2, conductor_max_tokens = 2048 }++[cells.conductor-gaia-gpt5miniplan-5]+method = "conductor"+bench = "gaia"+n = 5+local = { model = "Qwen/Qwen3.5-27B-FP8", endpoint = "http://localhost:8001/v1" }+cloud = { model = "gpt-5-mini", endpoint = "openai" }+method_cfg = { worker_max_tokens = 4096, worker_temperature = 0.2, conductor_max_tokens = 2048 }++[cells.conductor-swebenchverified-opusplan-5]+method = "conductor"+bench = "swebench-verified"+n = 5+local = { model = "Qwen/Qwen3.5-27B-FP8", endpoint = "http://localhost:8001/v1" }+cloud = { model = "claude-opus-4-7", endpoint = "anthropic" }+method_cfg = { worker_max_tokens = 8192, worker_temperature = 0.2, conductor_max_tokens = 4096 }+++# ---- n=30 (scaled-up cells for paradigms that worked at smoke tier) ----++[cells.conductor-gaia-opusplan-30]+method = "conductor"+bench = "gaia"+n = 30+local = { model = "Qwen/Qwen3.5-27B-FP8", endpoint = "http://localhost:8001/v1" }+cloud = { model = "claude-opus-4-7", endpoint = "anthropic" }+method_cfg = { worker_max_tokens = 4096, worker_temperature = 0.2, conductor_max_tokens = 2048 }+concurrency = 8++[cells.conductor-swebenchverified-opusplan-30]+method = "conductor"+bench = "swebench-verified"+n = 30+local = { model = "Qwen/Qwen3.5-27B-FP8", endpoint = "http://localhost:8001/v1" }+cloud = { model = "claude-opus-4-7", endpoint = "anthropic" }+method_cfg = { worker_max_tokens = 8192, worker_temperature = 0.2, conductor_max_tokens = 4096 }+concurrency = 8diff --git a/src/openjarvis/agents/hybrid/registry/mini_swe_agent.toml b/src/openjarvis/agents/hybrid/registry/mini_swe_agent.tomlnew file mode 100644index 00000000..69bf7bc4--- /dev/null+++ b/src/openjarvis/agents/hybrid/registry/mini_swe_agent.toml@@ -0,0 +1,33 @@+# mini-SWE-agent cells. Single-model bash-tool loop on SWE-bench-Verified.+#+# Vendored ~150-line port of mini-SWE-agent v2. No Docker; each task clones+# the repo into a tempdir and execs bash there. See+# `agents/hybrid/mini_swe_agent.py` docstring for the safety caveats.+#+# Backbone is `cloud` by default (drives the loop with the cloud model).+# Set `backbone = "local"` to drive with the vLLM-served local model+# (requires --enable-auto-tool-choice + a tool-call parser).++[cells.mini-swe-agent-swebenchverified-opus-3]+method = "mini_swe_agent"+bench = "swebench-verified"+n = 3+local = { model = "Qwen/Qwen3.5-27B-FP8", endpoint = "http://localhost:8001/v1" }+cloud = { model = "claude-opus-4-7", endpoint = "anthropic" }+method_cfg = { backbone = "cloud", max_turns = 30, bash_timeout_s = 120, turn_max_tokens = 4096 }++[cells.mini-swe-agent-swebenchverified-opus-30]+method = "mini_swe_agent"+bench = "swebench-verified"+n = 30+local = { model = "Qwen/Qwen3.5-27B-FP8", endpoint = "http://localhost:8001/v1" }+cloud = { model = "claude-opus-4-7", endpoint = "anthropic" }+method_cfg = { backbone = "cloud", max_turns = 50, bash_timeout_s = 120, turn_max_tokens = 4096 }++[cells.mini-swe-agent-swebenchverified-qwen27b-3]+method = "mini_swe_agent"+bench = "swebench-verified"+n = 3+local = { model = "Qwen/Qwen3.5-27B-FP8", endpoint = "http://localhost:8001/v1" }+cloud = { model = "claude-opus-4-7", endpoint = "anthropic" }+method_cfg = { backbone = "local", max_turns = 30, bash_timeout_s = 120, turn_max_tokens = 4096 }diff --git a/src/openjarvis/agents/hybrid/registry/minions.toml b/src/openjarvis/agents/hybrid/registry/minions.tomlnew file mode 100644index 00000000..bf3eeff4--- /dev/null+++ b/src/openjarvis/agents/hybrid/registry/minions.toml@@ -0,0 +1,96 @@+# Minions cells. One [cells.<name>] block per (bench × local × cloud × N).+#+# Naming: minions-<bench>-<local-short>-<cloud-short>-<N>+# HF model IDs verified Feb 2026 / Gemma 4 release:+# Qwen/Qwen3.5-27B (dense, 262K native context)+# google/gemma-4-31B-it (dense instruct, 256K context — 26B is the MoE variant)++# ---- Stage 1 smokes (n=3) ----++[cells.minions-gaia-qwen27b-opus-3]+method = "minions"+bench = "gaia"+n = 3+local = { model = "Qwen/Qwen3.5-27B-FP8", endpoint = "http://localhost:8001/v1" }+cloud = { model = "claude-opus-4-7", endpoint = "anthropic" }+method_cfg = { mode = "minion", max_rounds = 3, worker_max_tokens = 4096 }++[cells.minions-swebenchverified-qwen27b-opus-3]+method = "minions"+bench = "swebench-verified"+n = 3+local = { model = "Qwen/Qwen3.5-27B-FP8", endpoint = "http://localhost:8001/v1" }+cloud = { model = "claude-opus-4-7", endpoint = "anthropic" }+method_cfg = { mode = "minion", max_rounds = 3, worker_max_tokens = 8192 }+++# ---- Stage 1.5 mid (n=30) — catches cells that smoke-pass but full-fail ----++[cells.minions-gaia-qwen27b-opus-30]+method = "minions"+bench = "gaia"+n = 30+local = { model = "Qwen/Qwen3.5-27B-FP8", endpoint = "http://localhost:8001/v1" }+cloud = { model = "claude-opus-4-7", endpoint = "anthropic" }+method_cfg = { mode = "minion", max_rounds = 5, worker_max_tokens = 4096 }++[cells.minions-gaia-gemma31b-opus-30]+method = "minions"+bench = "gaia"+n = 30+local = { model = "google/gemma-4-31B-it", endpoint = "http://localhost:8004/v1" }+cloud = { model = "claude-opus-4-7", endpoint = "anthropic" }+method_cfg = { mode = "minion", max_rounds = 5, worker_max_tokens = 4096 }++[cells.minions-swebenchverified-qwen27b-opus-30]+method = "minions"+bench = "swebench-verified"+n = 30+local = { model = "Qwen/Qwen3.5-27B-FP8", endpoint = "http://localhost:8001/v1" }+cloud = { model = "claude-opus-4-7", endpoint = "anthropic" }+method_cfg = { mode = "minion", max_rounds = 5, worker_max_tokens = 8192 }++[cells.minions-swebenchverified-gemma31b-opus-30]+method = "minions"+bench = "swebench-verified"+n = 30+local = { model = "google/gemma-4-31B-it", endpoint = "http://localhost:8004/v1" }+cloud = { model = "claude-opus-4-7", endpoint = "anthropic" }+method_cfg = { mode = "minion", max_rounds = 5, worker_max_tokens = 8192 }+++# ---- Stage 2 full ----++[cells.minions-gaia-qwen27b-opus-165]+method = "minions"+bench = "gaia"+n = 165+local = { model = "Qwen/Qwen3.5-27B-FP8", endpoint = "http://localhost:8001/v1" }+cloud = { model = "claude-opus-4-7", endpoint = "anthropic" }+method_cfg = { mode = "minion", max_rounds = 5, worker_max_tokens = 4096 }+concurrency = 4++[cells.minions-gaia-gemma31b-opus-165]+method = "minions"+bench = "gaia"+n = 165+local = { model = "google/gemma-4-31B-it", endpoint = "http://localhost:8004/v1" }+cloud = { model = "claude-opus-4-7", endpoint = "anthropic" }+method_cfg = { mode = "minion", max_rounds = 5, worker_max_tokens = 4096 }++[cells.minions-swebenchverified-qwen27b-opus-500]+method = "minions"+bench = "swebench-verified"+n = 500+local = { model = "Qwen/Qwen3.5-27B-FP8", endpoint = "http://localhost:8001/v1" }+cloud = { model = "claude-opus-4-7", endpoint = "anthropic" }+method_cfg = { mode = "minion", max_rounds = 5, worker_max_tokens = 8192 }+concurrency = 4++[cells.minions-swebenchverified-gemma31b-opus-500]+method = "minions"+bench = "swebench-verified"+n = 500+local = { model = "google/gemma-4-31B-it", endpoint = "http://localhost:8004/v1" }+cloud = { model = "claude-opus-4-7", endpoint = "anthropic" }+method_cfg = { mode = "minion", max_rounds = 5, worker_max_tokens = 8192 }diff --git a/src/openjarvis/agents/hybrid/registry/skillorchestra.toml b/src/openjarvis/agents/hybrid/registry/skillorchestra.tomlnew file mode 100644index 00000000..11db4a37--- /dev/null+++ b/src/openjarvis/agents/hybrid/registry/skillorchestra.toml@@ -0,0 +1,47 @@+# SkillOrchestra cells (arXiv:2602.19672).+#+# Deployment-time skill-aware routing only. The paper's offline pipeline+# (explore → learn → select over an SGLang model pool + FAISS wiki index)+# isn't reproduced here — we don't have the SGLang serving stack, the+# retriever index, or a training split to learn a routing policy on. What+# we DO reproduce is the inference-time orchestrator: an Opus router+# analyzes the question into skill weights, picks one of two agents+# (local Qwen-27B vs cloud Opus) under an explicit cost trade-off, then+# the chosen agent answers. Mirrors the eval_orchestrator paradigm in+# external/SkillOrchestra/skillorchestra/prompts/eval_orchestrator.py.+# See adapters/skillorchestra_adapter.py for full notes.++[cells.skillorchestra-gaia-qwen27b-opus-3]+method = "skillorchestra"+bench = "gaia"+n = 3+local = { model = "Qwen/Qwen3.5-27B-FP8", endpoint = "http://localhost:8001/v1" }+cloud = { model = "claude-opus-4-7", endpoint = "anthropic" }+method_cfg = {}++[cells.skillorchestra-swebenchverified-qwen27b-opus-3]+method = "skillorchestra"+bench = "swebench-verified"+n = 3+local = { model = "Qwen/Qwen3.5-27B-FP8", endpoint = "http://localhost:8001/v1" }+cloud = { model = "claude-opus-4-7", endpoint = "anthropic" }+method_cfg = {}+concurrency = 3++[cells.skillorchestra-gaia-qwen27b-opus-30]+method = "skillorchestra"+bench = "gaia"+n = 30+local = { model = "Qwen/Qwen3.5-27B-FP8", endpoint = "http://localhost:8001/v1" }+cloud = { model = "claude-opus-4-7", endpoint = "anthropic" }+method_cfg = {}+concurrency = 4++[cells.skillorchestra-swebenchverified-qwen27b-opus-30]+method = "skillorchestra"+bench = "swebench-verified"+n = 30+local = { model = "Qwen/Qwen3.5-27B-FP8", endpoint = "http://localhost:8001/v1" }+cloud = { model = "claude-opus-4-7", endpoint = "anthropic" }+method_cfg = {}+concurrency = 3diff --git a/src/openjarvis/agents/hybrid/registry/swe_agent_loop.toml b/src/openjarvis/agents/hybrid/registry/swe_agent_loop.tomlnew file mode 100644index 00000000..96a36637--- /dev/null+++ b/src/openjarvis/agents/hybrid/registry/swe_agent_loop.toml@@ -0,0 +1,70 @@+# SWE-bench cells that run each paradigm through mini-SWE-agent.+#+# Same paradigms as the headline cells, same models, but with+# `swe_use_agent_loop = true` so the worker step is a full agent loop+# (shell + repo browse + test runs) instead of one-shot patch prediction.+# Expected: 0.30 → ~0.5-0.77 jump per paradigm (depending on which model+# carries the loop).+#+# These cells are EXPENSIVE — each task is ~30-50 LLM turns with bash, so+# Opus-driven cells run $1-3/task and ~5-10 min/task. Smokes at n=3 first.++# ------------- Minions: cloud planner + local mini-SWE-agent worker -------------++[cells.minions-swe-agent-swebenchverified-qwen27b-opus-3]+method = "minions"+bench = "swebench-verified"+n = 3+local = { model = "Qwen/Qwen3.5-27B-FP8", endpoint = "http://localhost:8001/v1" }+cloud = { model = "claude-opus-4-7", endpoint = "anthropic" }+method_cfg = { swe_use_agent_loop = true, supervisor_max_tokens = 1024, swe_max_turns = 30, swe_bash_timeout_s = 120, swe_turn_max_tokens = 4096 }++# ------------- Conductor: shared workdir across plan steps -------------++[cells.conductor-swe-agent-swebenchverified-opus-3]+method = "conductor"+bench = "swebench-verified"+n = 3+local = { model = "Qwen/Qwen3.5-27B-FP8", endpoint = "http://localhost:8001/v1" }+cloud = { model = "claude-opus-4-7", endpoint = "anthropic" }+method_cfg = { swe_use_agent_loop = true, conductor_max_tokens = 2048, swe_max_turns = 25, swe_bash_timeout_s = 120, swe_turn_max_tokens = 4096 }++# ------------- Advisors: two full agent-loop attempts with local critique between -------------++[cells.advisors-swe-agent-swebenchverified-qwen9b-opus-3]+method = "advisors"+bench = "swebench-verified"+n = 3+local = { model = "Qwen/Qwen3.5-9B", endpoint = "http://localhost:8001/v1" }+cloud = { model = "claude-opus-4-7", endpoint = "anthropic" }+method_cfg = { swe_use_agent_loop = true, swe_max_turns = 25, swe_bash_timeout_s = 120, swe_turn_max_tokens = 4096, advisor_max_tokens = 2048 }++# ------------- SkillOrchestra: router routes to local or cloud, that one runs the loop -------------++[cells.skillorchestra-swe-agent-swebenchverified-qwen27b-opus-3]+method = "skillorchestra"+bench = "swebench-verified"+n = 3+local = { model = "Qwen/Qwen3.5-27B-FP8", endpoint = "http://localhost:8001/v1" }+cloud = { model = "claude-opus-4-7", endpoint = "anthropic" }+method_cfg = { swe_use_agent_loop = true, router_max_tokens = 1024, swe_max_turns = 30, swe_bash_timeout_s = 120, swe_turn_max_tokens = 4096 }++# ------------- ToolOrchestra: orchestrator dispatches solver workers through loop on shared workdir -------------++[cells.toolorchestra-swe-agent-swebenchverified-qwen27b-opus-3]+method = "toolorchestra"+bench = "swebench-verified"+n = 3+local = { model = "Qwen/Qwen3.5-27B-FP8", endpoint = "http://localhost:8001/v1" }+cloud = { model = "claude-opus-4-7", endpoint = "anthropic" }+method_cfg = { swe_use_agent_loop = true, max_turns = 4, orchestrator_max_tokens = 1024, swe_max_turns = 25, swe_bash_timeout_s = 120, swe_turn_max_tokens = 4096 }++# ------------- Archon: K=3 local mini-SWE-agent runs, cloud ranker picks best -------------++[cells.archon-swe-agent-swebenchverified-qwen27b-opus-3]+method = "archon"+bench = "swebench-verified"+n = 3+local = { model = "Qwen/Qwen3.5-27B-FP8", endpoint = "http://localhost:8001/v1" }+cloud = { model = "claude-opus-4-7", endpoint = "anthropic" }+method_cfg = { swe_use_agent_loop = true, n_samples = 3, swe_max_turns = 25, swe_bash_timeout_s = 120, swe_turn_max_tokens = 4096, ranker_max_tokens = 1024 }diff --git a/src/openjarvis/agents/hybrid/registry/toolorchestra.toml b/src/openjarvis/agents/hybrid/registry/toolorchestra.tomlnew file mode 100644index 00000000..c6893725--- /dev/null+++ b/src/openjarvis/agents/hybrid/registry/toolorchestra.toml@@ -0,0 +1,30 @@+# ToolOrchestra cells. Prompted port — uses a cloud model (Opus) as the+# orchestrator, NOT the RL-trained Nemotron-Orchestrator-8B from the paper+# (arXiv:2511.21689). Treat results as preliminary until a real+# Orchestrator-8B deployment is wired up.+#+# Naming: toolorchestra-<bench>-<local-short>-<cloud-short>-<N>++[cells.toolorchestra-gaia-qwen27b-opus-3]+method = "toolorchestra"+bench = "gaia"+n = 3+local = { model = "Qwen/Qwen3.5-27B-FP8", endpoint = "http://localhost:8001/v1" }+cloud = { model = "claude-opus-4-7", endpoint = "anthropic" }+method_cfg = { max_turns = 6, orchestrator_max_tokens = 1024, worker_max_tokens = 4096 }++[cells.toolorchestra-swebenchverified-qwen27b-opus-3]+method = "toolorchestra"+bench = "swebench-verified"+n = 3+local = { model = "Qwen/Qwen3.5-27B-FP8", endpoint = "http://localhost:8001/v1" }+cloud = { model = "claude-opus-4-7", endpoint = "anthropic" }+method_cfg = { max_turns = 6, orchestrator_max_tokens = 1024, worker_max_tokens = 8192 }++[cells.toolorchestra-gaia-qwen27b-opus-30]+method = "toolorchestra"+bench = "gaia"+n = 30+local = { model = "Qwen/Qwen3.5-27B-FP8", endpoint = "http://localhost:8001/v1" }+cloud = { model = "claude-opus-4-7", endpoint = "anthropic" }+method_cfg = { max_turns = 8, orchestrator_max_tokens = 1024, worker_max_tokens = 4096 }diff --git a/src/openjarvis/agents/hybrid/runner.py b/src/openjarvis/agents/hybrid/runner.pynew file mode 100644index 00000000..ad384f03--- /dev/null+++ b/src/openjarvis/agents/hybrid/runner.py@@ -0,0 +1,501 @@+"""CLI runner for hybrid paradigm experiments.++::++ python -m openjarvis.agents.hybrid.runner --cell minions-gaia-qwen27b-opus-3++Reads a cell definition from ``registry/<method>.toml`` (bundled with this+package or pointed at by ``OPENJARVIS_HYBRID_REGISTRY_DIR``), constructs+the registered agent, loads bench tasks via OpenJarvis's existing dataset+providers, runs every task, scores it, and writes+``<EXPERIMENTS_DIR>/<cell>/results.jsonl`` + ``summary.json``.++The output schema matches ``hybrid-local-cloud-compute/runner.py`` so the+existing rescore / dashboard scripts can read OpenJarvis cells without+modification.+"""++from __future__ import annotations++import argparse+import fcntl+import json+import os+import sys+import threading+import time+import traceback+from concurrent.futures import ThreadPoolExecutor, as_completed+from contextlib import contextmanager+from pathlib import Path+from typing import Any, Dict, List, Optional++try:+ import tomllib # py3.11++except ModuleNotFoundError:+ import tomli as tomllib # type: ignore[import-not-found,no-redef]++from openjarvis.agents._stubs import AgentContext, AgentResult+from openjarvis.agents.hybrid._prompts import format_prompt as _format_prompt++PACKAGE_DIR = Path(__file__).parent+DEFAULT_REGISTRY_DIR = PACKAGE_DIR / "registry"+DEFAULT_EXPERIMENTS_DIR = Path(+ os.environ.get(+ "OPENJARVIS_HYBRID_EXPERIMENTS_DIR",+ str(Path.home() / ".openjarvis-hybrid" / "experiments"),+ )+)+++# ---------- Registry ----------++def load_registry(registry_dir: Optional[Path] = None) -> Dict[str, Dict[str, Any]]:+ """Merge every ``<registry_dir>/*.toml``. Cell names must be unique."""+ base = registry_dir or DEFAULT_REGISTRY_DIR+ env_override = os.environ.get("OPENJARVIS_HYBRID_REGISTRY_DIR")+ if env_override:+ base = Path(env_override)+ if not base.is_dir():+ return {}+ cells: Dict[str, Dict[str, Any]] = {}+ for p in sorted(base.glob("*.toml")):+ data = tomllib.loads(p.read_text())+ for name, cell in data.get("cells", {}).items():+ if name in cells:+ raise ValueError(+ f"duplicate cell {name!r} (already defined before {p.name})"+ )+ cells[name] = cell+ return cells+++# ---------- Bench dispatch ----------++def _load_gaia_tasks(n: Optional[int]) -> List[Dict[str, Any]]:+ """GAIA validation. Each task is a dict with `task_id` + `question`."""+ from openjarvis.evals.datasets.gaia import GAIADataset++ ds = GAIADataset()+ ds.load(max_samples=n)+ out: List[Dict[str, Any]] = []+ for rec in ds.iter_records():+ # rec.problem is the formatted question prompt; rec.metadata carries+ # the GAIA-specific fields including any reference answer.+ out.append({+ "task_id": rec.record_id,+ "question": rec.metadata.get("question", rec.problem),+ "reference": rec.reference,+ "metadata": dict(rec.metadata),+ })+ return out+++def _load_swebench_tasks(n: Optional[int]) -> List[Dict[str, Any]]:+ """SWE-bench-Verified test. Each task carries patch-evaluation fields."""+ from openjarvis.evals.datasets.swebench import SWEBenchDataset++ ds = SWEBenchDataset()+ ds.load(max_samples=n)+ out: List[Dict[str, Any]] = []+ for rec in ds.iter_records():+ md = rec.metadata or {}+ out.append({+ "task_id": md.get("instance_id", rec.record_id),+ "repo": md.get("repo", ""),+ "base_commit": md.get("base_commit", ""),+ "problem_statement": md.get("problem_statement", rec.problem),+ "hints_text": md.get("hints_text", ""),+ "test_patch": md.get("test_patch", ""),+ "FAIL_TO_PASS": md.get("FAIL_TO_PASS", []),+ "PASS_TO_PASS": md.get("PASS_TO_PASS", []),+ "version": md.get("version"),+ "reference": rec.reference,+ "metadata": dict(md),+ })+ return out+++def load_tasks(bench: str, n: Optional[int]) -> List[Dict[str, Any]]:+ if bench == "gaia":+ return _load_gaia_tasks(n)+ if bench in ("swebench-verified", "swebench_verified", "swebench"):+ return _load_swebench_tasks(n)+ raise ValueError(f"unknown bench: {bench!r}")+++# ---------- Scoring ----------++def _score_gaia(task: Dict[str, Any], answer: str) -> Dict[str, Any]:+ """Exact-match-with-format-normalization GAIA scorer.++ Lightweight version: extracts the final-answer line and string-compares+ against the reference. Use the OpenJarvis gaia_exact scorer for the+ judge-tiebreaker path.+ """+ import re++ ref = (task.get("reference") or "").strip()+ if not ref:+ return {"success": False, "score": 0.0, "details": {"reason": "no_reference"}}+ m = re.search(+ r"FINAL\s*ANSWER\s*:\s*(.+?)\s*$",+ answer,+ re.IGNORECASE | re.MULTILINE,+ )+ pred = (m.group(1).strip() if m else answer.strip()).rstrip(".").strip()+ success = pred.lower() == ref.lower()+ return {+ "success": success,+ "score": 1.0 if success else 0.0,+ "details": {"prediction": pred, "reference": ref},+ }+++def _score_swebench(task: Dict[str, Any], answer: str) -> Dict[str, Any]:+ """Modal-backed SWE-bench Verified harness scorer."""+ from openjarvis.evals.core.types import EvalRecord+ from openjarvis.evals.scorers.swebench_harness import (+ SWEBenchHarnessScorer,+ extract_patch,+ )++ patch = extract_patch(answer)+ if patch is None:+ return {"success": False, "score": 0.0, "details": {"reason": "no_patch_extracted"}}++ record = EvalRecord(+ record_id=task["task_id"],+ problem=task.get("problem_statement", ""),+ reference="",+ category="agentic",+ metadata={"instance_id": task["task_id"]},+ )+ scorer = SWEBenchHarnessScorer(timeout_s=int(os.environ.get("SWEBENCH_TIMEOUT_S", "1800")))+ is_correct, details = scorer.score(record, answer)+ return {+ "success": bool(is_correct),+ "score": 1.0 if is_correct else 0.0,+ "details": details,+ }+++def score(bench: str, task: Dict[str, Any], answer: str) -> Dict[str, Any]:+ if bench == "gaia":+ return _score_gaia(task, answer)+ if bench in ("swebench-verified", "swebench_verified", "swebench"):+ return _score_swebench(task, answer)+ raise ValueError(f"unknown bench: {bench!r}")+++# ---------- Cell run ----------++def _cell_dir(cell_name: str, root: Path) -> Path:+ d = root / cell_name+ d.mkdir(parents=True, exist_ok=True)+ (d / "logs").mkdir(exist_ok=True)+ return d+++@contextmanager+def _cell_lock(out_dir: Path, cell_name: str):+ """Exclusive flock on ``<cell>/.lock`` to prevent concurrent runner stomps."""+ lock_path = out_dir / ".lock"+ f = lock_path.open("a+")+ try:+ fcntl.flock(f.fileno(), fcntl.LOCK_EX | fcntl.LOCK_NB)+ except BlockingIOError:+ f.seek(0)+ prev = (f.read() or "?").strip() or "?"+ f.close()+ raise SystemExit(+ f"[lock] another runner is already running cell {cell_name!r} "+ f"(holder pid: {prev}). refusing to start a second instance."+ )+ f.seek(0)+ f.truncate()+ f.write(str(os.getpid()))+ f.flush()+ try:+ yield+ finally:+ try:+ fcntl.flock(f.fileno(), fcntl.LOCK_UN)+ except Exception:+ pass+ f.close()+ try:+ lock_path.unlink()+ except FileNotFoundError:+ pass+++def _build_agent(cell: Dict[str, Any]):+ """Construct the registered agent for this cell."""+ import openjarvis.agents # noqa: F401 — populate registry+ from openjarvis.core.registry import AgentRegistry++ method = cell["method"]+ if not AgentRegistry.contains(method):+ raise ValueError(+ f"agent {method!r} not registered. Available: "+ f"{', '.join(sorted(AgentRegistry.keys()))}"+ )+ agent_cls = AgentRegistry.get(method)+ local = cell.get("local") or {}+ cloud = cell.get("cloud") or {}+ method_cfg = dict(cell.get("method_cfg") or {})++ return agent_cls(+ engine=None, # raw SDK calls — engine unused+ model=cloud.get("model", ""),+ local_model=local.get("model"),+ local_endpoint=local.get("endpoint"),+ cloud_endpoint=(cloud.get("endpoint") or "anthropic").lower(),+ cfg=method_cfg,+ )+++def _run_one(agent, bench: str, task: Dict[str, Any], log_dir: str) -> Dict[str, Any]:+ """Run the agent on one task. Returns a hybrid-shape row."""+ prompt = _format_prompt(task)+ ctx = AgentContext(metadata={+ "task": task,+ "task_id": task["task_id"],+ "log_dir": log_dir,+ })+ t0 = time.time()+ try:+ result: AgentResult = agent.run(prompt, ctx)+ meta = dict(result.metadata or {})+ out = {+ "task_id": task["task_id"],+ "answer": result.content or "",+ "tokens_local": int(meta.get("tokens_local", 0)),+ "tokens_cloud": int(meta.get("tokens_cloud", 0)),+ "cost_usd": float(meta.get("cost_usd", 0.0)),+ "latency_s": float(meta.get("latency_s", time.time() - t0)),+ "traces": meta.get("traces", {}),+ }+ if "soft_error" in meta:+ out["soft_error"] = meta["soft_error"]+ return {**out, "error": None}+ except Exception as e:+ return {+ "task_id": task["task_id"],+ "answer": "",+ "tokens_local": 0, "tokens_cloud": 0,+ "cost_usd": 0.0, "latency_s": time.time() - t0,+ "traces": {},+ "error": f"{type(e).__name__}: {e}\n{traceback.format_exc()}",+ }+++def _heartbeat(done: int, total: int, row: Dict[str, Any], t_start: float) -> None:+ elapsed = time.time() - t_start+ eta = (total - done) * (elapsed / max(done, 1))+ ok = "OK" if not row.get("error") else "ERR"+ s = row.get("score") or {}+ sc = s.get("score")+ sc_str = f"{sc:.2f}" if isinstance(sc, (int, float)) else "—"+ print(+ f"[{done}/{total}] {ok} task={row['task_id']} score={sc_str} "+ f"local={row['tokens_local']} cloud={row['tokens_cloud']} "+ f"${row['cost_usd']:.3f} {row['latency_s']:.1f}s eta={eta/60:.1f}m",+ flush=True,+ )+++def _write_summary(+ out_dir: Path,+ cell_name: str,+ cell: Dict[str, Any],+ tasks: List[Dict[str, Any]],+ t_start: float,+) -> None:+ results_path = out_dir / "results.jsonl"+ rows = [+ json.loads(line)+ for line in results_path.read_text().splitlines()+ if line.strip()+ ]+ n_done = len(rows)+ n_err = sum(1 for r in rows if r.get("error"))+ successes = [r for r in rows if r.get("score") and r["score"].get("success")]+ acc = (len(successes) / n_done) if n_done else 0.0+ total_cost = sum(r.get("cost_usd", 0.0) for r in rows)+ total_local = sum(r.get("tokens_local", 0) for r in rows)+ total_cloud = sum(r.get("tokens_cloud", 0) for r in rows)+ elapsed = time.time() - t_start++ summary = {+ "cell": cell_name,+ "method": cell["method"],+ "bench": cell["bench"],+ "n_target": cell["n"],+ "n_done": n_done,+ "n_err": n_err,+ "accuracy": acc,+ "tokens_local_total": total_local,+ "tokens_cloud_total": total_cloud,+ "cost_usd_total": total_cost,+ "wall_time_s": elapsed,+ "task_count": len(tasks),+ }+ (out_dir / "summary.json").write_text(json.dumps(summary, indent=2))+ print(+ f"[summary] {cell_name}: n={n_done}/{cell['n']} err={n_err} "+ f"acc={acc:.3f} cost=${total_cost:.2f} time={elapsed/60:.1f}m",+ flush=True,+ )+++def run_cell(+ cell_name: str,+ cell: Dict[str, Any],+ *,+ do_score: bool = True,+ resume: bool = True,+ root: Optional[Path] = None,+) -> None:+ out_root = root or DEFAULT_EXPERIMENTS_DIR+ out_dir = _cell_dir(cell_name, out_root)+ with _cell_lock(out_dir, cell_name):+ _run_cell_locked(+ cell_name, cell, out_dir,+ do_score=do_score, resume=resume,+ )+++def _run_cell_locked(+ cell_name: str,+ cell: Dict[str, Any],+ out_dir: Path,+ *,+ do_score: bool,+ resume: bool,+) -> None:+ (out_dir / "config.json").write_text(+ json.dumps({"name": cell_name, **cell}, indent=2)+ )++ results_path = out_dir / "results.jsonl"+ done_ids: set = set()+ if resume and results_path.exists():+ # Keep only successful rows; drop errored rows so they retry.+ kept: List[str] = []+ for line in results_path.read_text().splitlines():+ try:+ row = json.loads(line)+ except Exception:+ continue+ if not row.get("error"):+ kept.append(line)+ done_ids.add(row["task_id"])+ results_path.write_text("\n".join(kept) + ("\n" if kept else ""))+ print(+ f"[resume] {len(done_ids)} tasks already done (errored rows dropped)",+ flush=True,+ )++ tasks = load_tasks(cell["bench"], n=cell["n"])+ print(f"[load] {cell['bench']} → {len(tasks)} tasks", flush=True)++ pending = [t for t in tasks if t["task_id"] not in done_ids]+ concurrency = max(1, int(cell.get("concurrency", 1)))+ if concurrency > 1:+ print(f"[concurrency] {concurrency} workers", flush=True)++ agent = _build_agent(cell)++ t_start = time.time()+ write_lock = threading.Lock()+ completed = [0]+ log_dir = str(out_dir / "logs")++ def _process(task: Dict[str, Any]) -> None:+ row = _run_one(agent, cell["bench"], task, log_dir)+ scored: Optional[Dict[str, Any]] = None+ if do_score and row.get("error") is None:+ try:+ scored = score(cell["bench"], task, row["answer"])+ except Exception as e:+ scored = {+ "success": False, "score": 0.0,+ "details": {"score_error": str(e)},+ }+ full_row = {**row, "score": scored}+ with write_lock, results_path.open("a") as f:+ f.write(json.dumps(full_row) + "\n")+ f.flush()+ completed[0] += 1+ _heartbeat(completed[0], len(tasks), full_row, t_start)++ if concurrency == 1:+ for task in pending:+ _process(task)+ else:+ with ThreadPoolExecutor(max_workers=concurrency) as ex:+ futures = [ex.submit(_process, t) for t in pending]+ for fut in as_completed(futures):+ fut.result()++ _write_summary(out_dir, cell_name, cell, tasks, t_start)+++# ---------- CLI ----------++def main(argv: Optional[List[str]] = None) -> int:+ p = argparse.ArgumentParser(+ prog="python -m openjarvis.agents.hybrid.runner",+ description="Run a hybrid paradigm experiment cell.",+ )+ p.add_argument("--cell", required=True, help="Cell name from the registry TOMLs.")+ p.add_argument(+ "--registry-dir",+ default=None,+ help="Override registry dir (defaults to package registry/).",+ )+ p.add_argument(+ "--root",+ default=None,+ help="Override experiments output root.",+ )+ p.add_argument("--no-score", action="store_true", help="Skip scoring.")+ p.add_argument("--no-resume", action="store_true", help="Don't resume from results.jsonl.")+ args = p.parse_args(argv)++ reg_dir = Path(args.registry_dir) if args.registry_dir else None+ cells = load_registry(reg_dir)+ if not cells:+ print(f"[error] no cells found in {reg_dir or DEFAULT_REGISTRY_DIR}", file=sys.stderr)+ return 2+ if args.cell not in cells:+ print(+ f"[error] unknown cell {args.cell!r}. Known: {', '.join(sorted(cells))}",+ file=sys.stderr,+ )+ return 2+ root = Path(args.root) if args.root else None+ run_cell(+ args.cell, cells[args.cell],+ do_score=not args.no_score,+ resume=not args.no_resume,+ root=root,+ )+ return 0+++if __name__ == "__main__":+ raise SystemExit(main())+++__all__ = [+ "DEFAULT_EXPERIMENTS_DIR",+ "DEFAULT_REGISTRY_DIR",+ "load_registry",+ "load_tasks",+ "main",+ "run_cell",+ "score",+]diff --git a/src/openjarvis/agents/hybrid/scripts/new_experiment.sh b/src/openjarvis/agents/hybrid/scripts/new_experiment.shnew file mode 100755index 00000000..4d89ccd6--- /dev/null+++ b/src/openjarvis/agents/hybrid/scripts/new_experiment.sh@@ -0,0 +1,108 @@+#!/usr/bin/env bash+# Scaffold a new hybrid paradigm experiment cell.+#+# Appends a [cells.<name>] block to+# src/openjarvis/agents/hybrid/registry/<method>.toml+# (registry is split by method — minions.toml, conductor.toml, etc.).+#+# Usage:+# src/openjarvis/agents/hybrid/scripts/new_experiment.sh \+# --method minions --bench gaia \+# --local qwen3.5-27b --cloud claude-opus-4-7 --n 50 \+# [--mode minion|minions] [--max-rounds 3]+#+# Then run:+# python -m openjarvis.agents.hybrid.runner --cell <printed name>++set -euo pipefail++SCRIPT_DIR="$(cd "$(dirname "$0")" && pwd)"+REGISTRY_DIR="$(cd "$SCRIPT_DIR/.." && pwd)/registry"++method=""; bench=""; local_model=""; cloud_model=""; n=""+mode="minion"; max_rounds=3++while [[ $# -gt 0 ]]; do+ case "$1" in+ --method) method="$2"; shift 2;;+ --bench) bench="$2"; shift 2;;+ --local) local_model="$2"; shift 2;;+ --cloud) cloud_model="$2"; shift 2;;+ --n) n="$2"; shift 2;;+ --mode) mode="$2"; shift 2;;+ --max-rounds) max_rounds="$2"; shift 2;;+ *) echo "unknown arg: $1" >&2; exit 2;;+ esac+done++for v in method bench local_model cloud_model n; do+ if [[ -z "${!v}" ]]; then+ echo "missing required --${v//_/-}" >&2+ exit 2+ fi+done++# Map shortnames → full HF id + endpoint+case "$local_model" in+ qwen3.5-27b|qwen-27b|qwen27b) local_full="Qwen/Qwen3.5-27B-FP8"; local_ep="http://localhost:8001/v1";;+ qwen3.5-9b|qwen-9b|qwen9b) local_full="Qwen/Qwen3.5-9B"; local_ep="http://localhost:8001/v1";;+ gemma-4-31b|gemma-31b|gemma31b) local_full="google/gemma-4-31B-it"; local_ep="http://localhost:8004/v1";;+ gemma-4-26b|gemma-26b|gemma26b) local_full="google/gemma-4-26B-A4B-it"; local_ep="http://localhost:8004/v1";;+ *) local_full="$local_model"; local_ep="${LOCAL_ENDPOINT:-http://localhost:8001/v1}";;+esac++case "$cloud_model" in+ gpt-5|gpt-5-mini|gpt-4o) cloud_ep="openai";;+ claude-opus-4-7|claude-sonnet-4-6) cloud_ep="anthropic";;+ *) cloud_ep="${CLOUD_ENDPOINT:-anthropic}";;+esac++local_short="$(echo "$local_model" | tr '/' '_')"+cloud_short="$cloud_model"+name="${method}-${bench}-${local_short}-${cloud_short}-${n}"++registry_file="$REGISTRY_DIR/${method}.toml"+if [[ ! -f "$registry_file" ]]; then+ echo "no registry file for method '${method}' — expected $registry_file" >&2+ echo "known: $(ls "$REGISTRY_DIR" | tr '\n' ' ')" >&2+ exit 2+fi++# Method-specific method_cfg defaults — keep simple, override later in TOML.+case "$method" in+ minions)+ cfg_block="method_cfg = { mode = \"${mode}\", max_rounds = ${max_rounds}, worker_max_tokens = 4096 }"+ ;;+ conductor)+ cfg_block="method_cfg = { conductor_max_tokens = 2048, worker_max_tokens = 4096 }"+ ;;+ advisors)+ cfg_block="method_cfg = { executor_max_tokens = 4096, advisor_max_tokens = 1024 }"+ ;;+ archon)+ cfg_block="method_cfg = { architecture = \"ensemble_rank_fuse\", n_samples = 5, max_tokens = 2048 }"+ ;;+ skillorchestra)+ cfg_block="method_cfg = { router_max_tokens = 1024, local_max_tokens = 4096, cloud_max_tokens = 4096 }"+ ;;+ toolorchestra)+ cfg_block="method_cfg = { max_turns = 6, orchestrator_max_tokens = 1024, worker_max_tokens = 4096 }"+ ;;+ *)+ cfg_block="method_cfg = {}"+ ;;+esac++cat >> "$registry_file" <<EOF++[cells.${name}]+method = "${method}"+bench = "${bench}"+n = ${n}+local = { model = "${local_full}", endpoint = "${local_ep}" }+cloud = { model = "${cloud_model}", endpoint = "${cloud_ep}" }+${cfg_block}+EOF++echo "Added cell: ${name} → $(realpath --relative-to="$(pwd)" "$registry_file")"+echo "Run with: python -m openjarvis.agents.hybrid.runner --cell ${name}"diff --git a/src/openjarvis/agents/hybrid/skillorchestra.py b/src/openjarvis/agents/hybrid/skillorchestra.pynew file mode 100644index 00000000..8b179ea7--- /dev/null+++ b/src/openjarvis/agents/hybrid/skillorchestra.py@@ -0,0 +1,363 @@+"""SkillOrchestraAgent — inference-time router (Wang et al., 2026).++Paper: arXiv:2602.19672. The published pipeline is four phases — explore+(run every pool model, collect traces), learn (induce a skill handbook+with per-agent Beta competences and per-skill cost stats), select (Pareto-+optimal handbook subset on a live val set), test. At deployment, the+orchestrator reads the user query, infers skill demands, then picks the+agent that maximizes weighted competence minus λ·cost.++What we reproduce here: the **deployment-time** step only. The full+explore/learn/select pipeline requires multi-model serving + the FRAMES+wiki retriever + a multi-hour LLM-driven learning loop that's out of+scope for the OpenJarvis port (and was out of scope in the hybrid harness).++So this agent uses the orchestrator's *inference logic* with a small+handbook that's synthesized per-task on the fly: cloud (Opus) reads the+question, identifies which skills it needs (from a fixed catalog),+assigns weights, scores each of our two agents (local Qwen-27B vs+cloud Opus) under a cost-discounted weighted-competence rule, then+routes. The chosen agent answers the question.++Hybrid harness result (n=30 GAIA): ``skillorchestra-gaia-qwen27b-opus-30``+= 0.500 acc, $0.02/task — 30× cheaper than baseline-cloud (0.567 / $0.66)+for ~7pp lower accuracy. Best cost-efficient GAIA paradigm by a wide+margin.++Ported from ``hybrid-local-cloud-compute/adapters/skillorchestra_adapter.py``.+"""++from __future__ import annotations++import json+from typing import Any, Dict, List, Optional, Tuple++from openjarvis.agents._stubs import AgentContext+from openjarvis.agents.hybrid._base import LocalCloudAgent+from openjarvis.agents.hybrid._prices import supports_temperature+from openjarvis.agents.hybrid.mini_swe_agent import run_swe_agent_loop+from openjarvis.core.registry import AgentRegistry++# ---------- Skill catalog (compact, GAIA-relevant) ----------+#+# SkillOrchestra learns its taxonomy from execution traces. Without traces+# we use a hand-curated taxonomy covering the kinds of competences GAIA+# actually exercises. Per-agent competences are fixed priors calibrated to+# typical Qwen3.5-27B-FP8 vs Opus 4.7 behavior — what a 1-iteration learn+# would seed before any oracle update.++SKILL_CATALOG: Dict[str, str] = {+ "factual_recall": "Recall named entities, dates, places, well-known facts from training data without external lookup.",+ "multi_step_reasoning": "Chain several inference steps together (e.g. compose dates, traverse relationships, decompose then aggregate).",+ "arithmetic": "Exact numeric computation on values already given in the question.",+ "web_grounding": "Question needs information likely NOT in a small model's parametric memory (rare facts, recent events, niche sources).",+ "long_text_extraction": "Read a long supplied document/context and extract a specific piece.",+ "format_compliance": "Strict output formatting (e.g. GAIA's `FINAL ANSWER: <answer>` rule, comma-separated lists with no units).",+ "code_or_logic": "Write or trace code, or apply logical/symbolic constraints precisely.",+}++DEFAULT_AGENT_COMPETENCE: Dict[str, Dict[str, float]] = {+ "local-qwen-27b": {+ "factual_recall": 0.25,+ "multi_step_reasoning": 0.30,+ "arithmetic": 0.55,+ "web_grounding": 0.10,+ "long_text_extraction": 0.55,+ "format_compliance": 0.65,+ "code_or_logic": 0.45,+ },+ "cloud-opus-4-7": {+ "factual_recall": 0.85,+ "multi_step_reasoning": 0.88,+ "arithmetic": 0.85,+ "web_grounding": 0.70,+ "long_text_extraction": 0.90,+ "format_compliance": 0.92,+ "code_or_logic": 0.90,+ },+}++DEFAULT_AGENT_COST_USD: Dict[str, float] = {+ "local-qwen-27b": 0.0,+ "cloud-opus-4-7": 0.30,+}++ROUTER_SYS_MARKER = "<<SKILLORCHESTRA-ROUTER>>"+++def _format_catalog() -> str:+ return "\n".join(f"- {sid}: {desc}" for sid, desc in SKILL_CATALOG.items())+++def _format_agents(+ competence: Dict[str, Dict[str, float]],+ cost: Dict[str, float],+) -> str:+ lines: List[str] = []+ for aid in competence:+ comp = competence[aid]+ comp_str = ", ".join(f"{k}={v:.2f}" for k, v in comp.items())+ lines.append(f"- **{aid}** (avg cost ${cost.get(aid, 0.0):.2f}/task)")+ lines.append(f" Skill competences: {comp_str}")+ return "\n".join(lines)+++def _build_router_sys(+ competence: Dict[str, Dict[str, float]],+ cost: Dict[str, float],+) -> str:+ return f"""{ROUTER_SYS_MARKER}+You are a skill-aware model router for a compound AI system (the SkillOrchestra deployment-time orchestrator). For each user question you must:++1. Assign weights over the skill catalog (numbers in [0, 1], summing to ~1.0). \+The weights reflect how much each skill matters for *this* question.+2. Score each candidate agent: score(agent) = sum_skill weight_skill * competence(agent, skill) - lambda_cost * avg_cost(agent), with lambda_cost = 0.5.+3. Pick the highest-scoring agent. Tie-break in favor of the cheaper agent.++Skill catalog:+{_format_catalog()}++Agent pool (with learned-prior competences and average costs):+{_format_agents(competence, cost)}++Respond with ONLY a JSON object: {{"chosen_agent": ..., "skill_weights": {{...}}, "reasoning": "..."}}. No prose outside JSON.+"""+++def _build_router_schema(agent_ids: List[str]) -> Dict[str, Any]:+ return {+ "format": {+ "type": "json_schema",+ "schema": {+ "type": "object",+ "properties": {+ "chosen_agent": {"type": "string", "enum": agent_ids},+ "skill_weights": {+ "type": "object",+ "properties": {+ sid: {"type": "number"} for sid in SKILL_CATALOG+ },+ "required": list(SKILL_CATALOG.keys()),+ "additionalProperties": False,+ },+ "reasoning": {"type": "string"},+ },+ "required": ["chosen_agent", "skill_weights", "reasoning"],+ "additionalProperties": False,+ },+ }+ }+++def _parse_router_json(text: str) -> Dict[str, Any]:+ s = (text or "").strip()+ try:+ return json.loads(s)+ except json.JSONDecodeError:+ pass+ start = s.find("{")+ if start == -1:+ raise ValueError(f"router emitted no JSON: {s[:200]!r}")+ depth = 0+ for i in range(start, len(s)):+ c = s[i]+ if c == "{":+ depth += 1+ elif c == "}":+ depth -= 1+ if depth == 0:+ return json.loads(s[start : i + 1])+ raise ValueError(f"router JSON not balanced: {s[:200]!r}")+++def _score_agents(+ skill_weights: Dict[str, float],+ competence: Dict[str, Dict[str, float]],+ cost: Dict[str, float],+) -> Dict[str, Dict[str, float]]:+ lam = 0.5+ scores: Dict[str, Dict[str, float]] = {}+ for aid, comps in competence.items():+ comp = sum(+ skill_weights.get(sid, 0.0) * comps[sid] for sid in SKILL_CATALOG+ )+ cost_pen = lam * cost.get(aid, 0.0)+ scores[aid] = {+ "competence": comp,+ "cost_penalty": cost_pen,+ "final_score": comp - cost_pen,+ }+ return scores++[email protected]("skillorchestra")+class SkillOrchestraAgent(LocalCloudAgent):+ """Inference-time skill-aware router. See module docstring."""++ agent_id = "skillorchestra"++ def _is_soft_failure(self, exc: BaseException) -> Optional[str]:+ # Empty/unbalanced router JSON — treat as soft failure to match the+ # hybrid adapter's behavior (matches `err=1` rows in the n=30 cell).+ if isinstance(exc, (ValueError, json.JSONDecodeError)):+ return f"{type(exc).__name__}: {str(exc)[:120]}"+ return None++ def _run_paradigm(+ self,+ input: str,+ context: Optional[AgentContext],+ **kwargs: Any,+ ) -> Tuple[str, Dict[str, Any]]:+ cfg = self._cfg+ question = input++ competence: Dict[str, Dict[str, float]] = cfg.get(+ "agent_competence", DEFAULT_AGENT_COMPETENCE+ )+ cost: Dict[str, float] = cfg.get("agent_cost_usd", DEFAULT_AGENT_COST_USD)+ agent_ids = list(competence.keys())+ router_sys = _build_router_sys(competence, cost)+ router_schema = _build_router_schema(agent_ids)++ # 1. Route — Anthropic only (output_config schema is Anthropic-specific+ # in the hybrid adapter). If you need OpenAI routing, swap the prompt+ # to JSON-mode and bypass output_config.+ if self._cloud_endpoint != "anthropic":+ raise ValueError(+ "SkillOrchestra router requires cloud_endpoint='anthropic'; "+ f"got {self._cloud_endpoint!r}"+ )+ router_max = int(cfg.get("router_max_tokens", 1024))+ # Strip temperature for Opus 4.7+; Anthropic's output_config does the schema.+ if supports_temperature(self._cloud_model):+ router_text, r_in, r_out, _ = self._call_anthropic(+ self._cloud_model,+ user=f"Question:\n{question}",+ system=router_sys,+ max_tokens=router_max,+ temperature=0.0,+ output_config=router_schema,+ )+ else:+ router_text, r_in, r_out, _ = self._call_anthropic(+ self._cloud_model,+ user=f"Question:\n{question}",+ system=router_sys,+ max_tokens=router_max,+ output_config=router_schema,+ )++ decision = _parse_router_json(router_text)+ skill_weights: Dict[str, float] = decision.get("skill_weights") or {}+ for sid in SKILL_CATALOG:+ skill_weights.setdefault(sid, 0.0)+ chosen = decision.get("chosen_agent") or ""++ scored = _score_agents(skill_weights, competence, cost)+ if chosen not in competence:+ chosen = max(scored, key=lambda a: scored[a]["final_score"])++ self.record_trace_event({+ "kind": "skillorchestra_route",+ "chosen_agent": chosen,+ "skill_weights": skill_weights,+ "agent_scores": scored,+ "reasoning": decision.get("reasoning", ""),+ "router_raw": router_text,+ })++ tokens_local = 0+ tokens_cloud = r_in + r_out+ run_cost = self.cost_usd(self._cloud_model, r_in, r_out)++ # 2. Execute via chosen agent+ task_meta = (context.metadata.get("task") if context is not None else {}) or {}+ swe_mode = (+ bool(cfg.get("swe_use_agent_loop"))+ and bool(task_meta.get("problem_statement"))+ and bool(task_meta.get("repo"))+ and bool(task_meta.get("base_commit"))+ )+ if chosen == "local-qwen-27b":+ if not (self._local_model and self._local_endpoint):+ raise ValueError(+ "SkillOrchestra route hit local agent but local_model/"+ f"local_endpoint missing: {self._local_model!r}/{self._local_endpoint!r}"+ )+ if swe_mode:+ out = run_swe_agent_loop(+ task_meta,+ backbone="local",+ backbone_model=self._local_model,+ local_endpoint=self._local_endpoint,+ initial_prompt=question,+ max_turns=int(cfg.get("swe_max_turns", 30)),+ bash_timeout=int(cfg.get("swe_bash_timeout_s", 120)),+ output_cap=int(cfg.get("swe_output_cap", 10_000)),+ turn_max_tokens=int(cfg.get("swe_turn_max_tokens", 4096)),+ trace_prefix="skillorch_local",+ )+ ans = out["answer"]+ tokens_local += out["tokens_in"] + out["tokens_out"]+ else:+ ans, w_in, w_out = self._call_vllm(+ self._local_model,+ self._local_endpoint,+ user=question,+ max_tokens=int(cfg.get("local_max_tokens", 4096)),+ temperature=float(cfg.get("local_temperature", 0.2)),+ enable_thinking=False,+ )+ tokens_local += w_in + w_out+ worker_model = self._local_model+ else:+ if swe_mode:+ out = run_swe_agent_loop(+ task_meta,+ backbone="cloud",+ backbone_model=self._cloud_model,+ cloud_endpoint=self._cloud_endpoint,+ initial_prompt=question,+ max_turns=int(cfg.get("swe_max_turns", 30)),+ bash_timeout=int(cfg.get("swe_bash_timeout_s", 120)),+ output_cap=int(cfg.get("swe_output_cap", 10_000)),+ turn_max_tokens=int(cfg.get("swe_turn_max_tokens", 4096)),+ trace_prefix="skillorch_cloud",+ )+ ans = out["answer"]+ tokens_cloud += out["tokens_in"] + out["tokens_out"]+ run_cost += out["cost_usd"]+ else:+ ans, w_in, w_out, _ = self._call_anthropic(+ self._cloud_model,+ user=question,+ max_tokens=int(cfg.get("cloud_max_tokens", 4096)),+ temperature=0.0,+ )+ tokens_cloud += w_in + w_out+ run_cost += self.cost_usd(self._cloud_model, w_in, w_out)+ worker_model = self._cloud_model++ meta = {+ "tokens_local": tokens_local,+ "tokens_cloud": tokens_cloud,+ "cost_usd": run_cost,+ "turns": 2, # router + worker+ "traces": {+ "chosen_agent": chosen,+ "worker_model": worker_model,+ "skill_weights": skill_weights,+ "agent_scores": scored,+ "reasoning": decision.get("reasoning", ""),+ },+ }+ return ans, meta+++__all__ = [+ "DEFAULT_AGENT_COMPETENCE",+ "DEFAULT_AGENT_COST_USD",+ "SKILL_CATALOG",+ "SkillOrchestraAgent",+]diff --git a/src/openjarvis/agents/hybrid/toolorchestra.py b/src/openjarvis/agents/hybrid/toolorchestra.pynew file mode 100644index 00000000..b8195e61--- /dev/null+++ b/src/openjarvis/agents/hybrid/toolorchestra.py@@ -0,0 +1,524 @@+"""ToolOrchestraAgent — prompted port of NVlabs ToolOrchestra (arXiv:2511.21689).++The paper RL-trains an 8B Orchestrator (``nvidia/Nemotron-Orchestrator-8B``)+to coordinate basic tools + specialist LLMs + generalist LLMs in+multi-turn agentic loops, ranked #1 on GAIA at release.++The hybrid harness adapter for ToolOrchestra is a documented stub —+running the real thing needs a separate vLLM srun for the Orchestrator-8B+checkpoint, a FAISS wiki retriever, a Tavily API key, and a refactor of+the upstream eval scripts. None of that fits in our cluster allocation.++This port keeps the same scope discipline: **inference-time only,+prompted, no RL**. A cloud model plays the role of the orchestrator,+dispatching to a pool of `(tool | specialist_llm | generalist_llm)`+workers in a reactive loop. The loop is the paradigm; the orchestrator+weights are not.++Why ship this at all if it's not the "real" thing? Because the prompted+upper-bound is useful as a reference point alongside the other paradigms,+and because the OpenJarvis registry needs all six entries for the+distillation pipeline to slot ToolOrchestra in alongside the rest.++Pipeline per task:++1. Orchestrator (cloud) reads question + numbered worker pool.+2. Each turn it emits ``{"action": "call_worker", "worker_id": int,+ "input": str}`` or ``{"action": "final_answer", "answer": str}``.+3. Up to ``max_turns`` (default 6) calls before forcing a final-answer+ prompt; fallback to strongest worker on parse failure.++Workers come from ``cfg["workers"]`` or a sensible default pool (local+Qwen if vLLM up, plus a web-search tool via Anthropic, Opus 4.7,+gpt-5-mini).++Not yet validated end-to-end in the hybrid harness — the hybrid adapter+``raise NotImplementedError``s. Treat results from this paradigm as+preliminary until we have a real ToolOrchestra-8B deployment.+"""++from __future__ import annotations++import json+import re+import shutil+import tempfile+from pathlib import Path+from typing import Any, Dict, List, Optional, Tuple++from openjarvis.agents._stubs import AgentContext+from openjarvis.agents.hybrid._base import (+ ANTHROPIC_WEB_SEARCH_TOOL,+ WEB_SEARCH_COST_PER_CALL,+ LocalCloudAgent,+)+from openjarvis.agents.hybrid._prices import (+ is_gpt5_family,+ supports_temperature,+)+from openjarvis.agents.hybrid.mini_swe_agent import (+ _clone_repo,+ _extract_diff,+ run_swe_agent_loop,+)+from openjarvis.core.registry import AgentRegistry++ORCHESTRATOR_SYS = """\+You are a tool-orchestrating agent. You coordinate a pool of workers to answer the user's question. Each turn you MUST emit exactly one JSON object — no prose, no markdown fences — taking one of two forms:++ {"action": "call_worker", "worker_id": <int>, "input": "<question or instruction for that worker>"}++ {"action": "final_answer", "answer": "<final answer to the user, respecting the question's answer-format rules>"}++Strategy:++- Call cheap / specialized workers first (small local model for extraction or arithmetic on given data; web_search for unknowns; specialist LLMs for code/math).+- Call the frontier worker (Opus / GPT-5) sparingly, for hard reasoning or a final synthesis pass.+- Stop and emit `final_answer` as soon as the previous worker output is sufficient. Do NOT call a worker just to paraphrase.+- The user only sees the `answer` field of `final_answer`, so make sure it follows any answer-format rules in the question.+"""++FORCE_FINAL_PROMPT = (+ "Worker-call budget exhausted. Emit `final_answer` now using everything "+ "you've learned. Respect the question's answer-format rules."+)+++def _build_pool_block(workers: List[Dict[str, Any]]) -> str:+ return "\n".join(+ f"Worker {w['id']} ({w['name']}): {w['description']}" for w in workers+ )+++def _build_user_prompt(+ question: str,+ workers: List[Dict[str, Any]],+ history: List[Dict[str, Any]],+) -> str:+ pieces = [+ f"Worker pool:\n{_build_pool_block(workers)}",+ f"User question:\n{question}",+ ]+ if history:+ pieces.append("Conversation so far (orchestrator turns and worker outputs):")+ for h in history:+ if h["role"] == "orchestrator":+ pieces.append(f"[Orchestrator turn {h['turn']}]\n{h['raw']}")+ else:+ pieces.append(+ f"[Worker {h['worker_id']} ({h['worker_name']}) turn {h['turn']}]\n"+ f"{h['output']}"+ )+ pieces.append(+ "Emit the next JSON action object now — exactly one object, no prose."+ )+ return "\n\n".join(pieces)+++def _strip_fences(s: str) -> str:+ s = s.strip()+ if s.startswith("```"):+ first_nl = s.find("\n")+ if first_nl != -1:+ s = s[first_nl + 1:]+ if s.endswith("```"):+ s = s[:-3]+ s = s.strip()+ return s+++def _parse_action(text: str) -> Optional[Dict[str, Any]]:+ s = _strip_fences(text)+ # First try direct parse, then balanced-brace extraction.+ try:+ obj = json.loads(s)+ if isinstance(obj, dict) and "action" in obj:+ return obj+ except json.JSONDecodeError:+ pass+ start = s.find("{")+ if start == -1:+ return None+ depth = 0+ for i in range(start, len(s)):+ c = s[i]+ if c == "{":+ depth += 1+ elif c == "}":+ depth -= 1+ if depth == 0:+ try:+ obj = json.loads(s[start : i + 1])+ if isinstance(obj, dict) and "action" in obj:+ return obj+ except json.JSONDecodeError:+ return None+ return None+++def _extract_final_answer_text(text: str) -> str:+ """Best-effort: pull the answer string from a malformed action emission.++ Tries `"answer": "..."` regex, then the GAIA-style `FINAL ANSWER:` line.+ """+ m = re.search(r'"answer"\s*:\s*"((?:\\.|[^"\\])*)"', text, re.DOTALL)+ if m:+ return m.group(1).encode("utf-8").decode("unicode_escape")+ m = re.search(r"FINAL\s*ANSWER\s*:\s*(.+?)\s*$", text, re.IGNORECASE | re.MULTILINE)+ if m:+ return m.group(1).strip()+ return text.strip()+++# ---------- Worker pool ----------++def _default_pool(local_model: Optional[str], local_endpoint: Optional[str]) -> List[Dict[str, Any]]:+ pool: List[Dict[str, Any]] = []+ if local_model and local_endpoint:+ pool.append({+ "id": len(pool),+ "name": "local-qwen",+ "type": "vllm",+ "model": local_model,+ "base_url": local_endpoint,+ "description": (+ "Open-weights Qwen3.5 served locally. Cheap and fast. Good at "+ "concise extraction, formatting, arithmetic on given data."+ ),+ })+ pool.append({+ "id": len(pool),+ "name": "web-search",+ "type": "anthropic-web-search",+ "model": "claude-haiku-4-5",+ "description": (+ "Anthropic server-side web_search. Use for facts that need a lookup "+ "(recent events, rare names/dates, niche sources). Returns a digest."+ ),+ })+ pool.append({+ "id": len(pool),+ "name": "frontier-anthropic",+ "type": "anthropic",+ "model": "claude-opus-4-7",+ "description": (+ "Frontier reasoning model. Use for hard multi-step reasoning, "+ "code review, or a final synthesis pass. Expensive — use sparingly."+ ),+ })+ pool.append({+ "id": len(pool),+ "name": "frontier-openai-mini",+ "type": "openai",+ "model": "gpt-5-mini",+ "description": (+ "Mid-tier OpenAI model. Solid general knowledge and reasoning at a "+ "fraction of frontier cost."+ ),+ })+ return pool+++def _call_worker(+ worker: Dict[str, Any], prompt: str, cfg: Dict[str, Any]+) -> Tuple[str, int, int, bool, float, int]:+ """Returns (text, p_tok, c_tok, is_local, extra_cost, n_web_searches)."""+ wtype = worker.get("type", "openai")+ max_tok = int(cfg.get("worker_max_tokens", 4096))+ temp = float(cfg.get("worker_temperature", 0.2))++ if wtype == "vllm":+ text, p, c = LocalCloudAgent._call_vllm(+ worker["model"],+ worker["base_url"],+ user=prompt,+ max_tokens=max_tok,+ temperature=temp,+ enable_thinking=False,+ )+ return text, p, c, True, 0.0, 0+ if wtype == "openai":+ eff_temp = 1.0 if is_gpt5_family(worker["model"]) else temp+ text, p, c = LocalCloudAgent._call_openai(+ worker["model"],+ user=prompt,+ max_tokens=max_tok,+ temperature=eff_temp,+ )+ return text, p, c, False, 0.0, 0+ if wtype == "anthropic":+ eff_temp = temp if supports_temperature(worker["model"]) else 0.0+ text, p, c, _ = LocalCloudAgent._call_anthropic(+ worker["model"],+ user=prompt,+ max_tokens=max_tok,+ temperature=eff_temp,+ )+ return text, p, c, False, 0.0, 0+ if wtype == "anthropic-web-search":+ eff_temp = temp if supports_temperature(worker["model"]) else 0.0+ text, p, c, n_searches = LocalCloudAgent._call_anthropic(+ worker["model"],+ user=prompt,+ max_tokens=max_tok,+ temperature=eff_temp,+ tools=[ANTHROPIC_WEB_SEARCH_TOOL],+ tool_choice={"type": "any"},+ )+ extra = n_searches * WEB_SEARCH_COST_PER_CALL+ return text, p, c, False, extra, n_searches+ raise ValueError(f"unsupported worker type: {wtype!r}")+++def _swe_call_worker(+ worker: Dict[str, Any],+ prompt: str,+ cfg: Dict[str, Any],+ task: Dict[str, Any],+ workdir: Path,+ turn: int,+) -> Tuple[str, int, int, bool, float, int]:+ """SWE-bench worker dispatch: route solver workers through+ run_swe_agent_loop on a shared workdir. Web-search workers fall back+ to the regular one-shot dispatch (search isn't an agent loop)."""+ wtype = worker.get("type", "openai")+ if wtype == "anthropic-web-search":+ # Search workers stay one-shot.+ return _call_worker(worker, prompt, cfg)+ if wtype == "vllm":+ backbone = "local"+ endpoint = worker.get("base_url")+ elif wtype == "anthropic":+ backbone = "cloud"+ endpoint = None+ else:+ # OpenAI workers fall back to one-shot.+ return _call_worker(worker, prompt, cfg)+ out = run_swe_agent_loop(+ task,+ backbone=backbone,+ backbone_model=worker["model"],+ cloud_endpoint="anthropic",+ local_endpoint=endpoint,+ initial_prompt=prompt,+ max_turns=int(cfg.get("swe_max_turns", 30)),+ bash_timeout=int(cfg.get("swe_bash_timeout_s", 120)),+ output_cap=int(cfg.get("swe_output_cap", 10_000)),+ turn_max_tokens=int(cfg.get("swe_turn_max_tokens", 4096)),+ trace_prefix=f"toolorch_turn{turn}",+ workdir=workdir,+ )+ is_local = backbone == "local"+ return (+ out["final_summary"] or out["answer"],+ out["tokens_in"], out["tokens_out"],+ is_local, 0.0, 0,+ )++[email protected]("toolorchestra")+class ToolOrchestraAgent(LocalCloudAgent):+ """Prompted multi-turn dispatcher over a mixed worker pool.++ Inference-only port — does NOT use the RL-trained Nemotron-Orchestrator-8B.+ See module docstring for what's missing relative to the published paper.+ """++ agent_id = "toolorchestra"++ def _run_paradigm(+ self,+ input: str,+ context: Optional[AgentContext],+ **kwargs: Any,+ ) -> Tuple[str, Dict[str, Any]]:+ cfg = self._cfg+ question = input+ workers = cfg.get("workers") or _default_pool(+ self._local_model, self._local_endpoint+ )+ if not workers:+ raise RuntimeError("toolorchestra: empty worker pool")++ max_turns = int(cfg.get("max_turns", 6))+ orch_max_tokens = int(cfg.get("orchestrator_max_tokens", 1024))++ task_meta = (context.metadata.get("task") if context is not None else {}) or {}+ swe_mode = (+ bool(cfg.get("swe_use_agent_loop"))+ and bool(task_meta.get("problem_statement"))+ and bool(task_meta.get("repo"))+ and bool(task_meta.get("base_commit"))+ )+ shared_workdir: Optional[Path] = None+ if swe_mode:+ shared_workdir = Path(tempfile.mkdtemp(+ prefix=f"toolorch-swe-{task_meta.get('task_id','x')}-"+ ))+ try:+ _clone_repo(task_meta["repo"], task_meta["base_commit"], shared_workdir)+ except Exception:+ shutil.rmtree(shared_workdir, ignore_errors=True)+ raise+ self.record_trace_event({+ "kind": "toolorchestra_swe_workdir",+ "workdir": str(shared_workdir),+ "repo": task_meta["repo"],+ "base_commit": task_meta["base_commit"],+ })++ # try/finally guards ``shared_workdir`` against exceptions raised+ # anywhere in the turn loop, the worker calls, the fallback, or+ # the diff-extraction step. Without this, at n=500 SWE-bench an+ # exception leaves hundreds of MB of cloned repos in tempdir.+ try:+ history: List[Dict[str, Any]] = []+ tokens_local = 0+ tokens_cloud = 0+ cost = 0.0+ final_answer: Optional[str] = None+ forced_final = False+ parse_failures = 0++ for turn in range(1, max_turns + 1):+ sys_prompt = ORCHESTRATOR_SYS+ if turn == max_turns and final_answer is None:+ sys_prompt = ORCHESTRATOR_SYS + "\n\n" + FORCE_FINAL_PROMPT+ forced_final = True++ user = _build_user_prompt(question, workers, history)+ text, o_in, o_out = self._call_cloud(+ user=user,+ system=sys_prompt,+ max_tokens=orch_max_tokens,+ temperature=0.0,+ )+ tokens_cloud += o_in + o_out+ cost += self.cost_usd(self._cloud_model, o_in, o_out)++ action = _parse_action(text)+ history.append({+ "role": "orchestrator", "turn": turn, "raw": text, "action": action,+ })+ self.record_trace_event({+ "kind": "toolorchestra_action",+ "turn": turn,+ "action": action,+ "raw": text,+ })++ if action is None:+ parse_failures += 1+ if parse_failures >= 2 or forced_final:+ final_answer = _extract_final_answer_text(text)+ break+ continue++ kind = action.get("action")+ if kind == "final_answer":+ final_answer = str(action.get("answer", "")).strip()+ break+ if kind == "call_worker":+ wid = action.get("worker_id")+ w_input = action.get("input", "")+ if not isinstance(wid, int) or not (0 <= wid < len(workers)):+ parse_failures += 1+ if parse_failures >= 2 or forced_final:+ final_answer = _extract_final_answer_text(text)+ break+ continue+ worker = workers[wid]+ if swe_mode and shared_workdir is not None:+ w_text, w_in, w_out, is_local, extra_cost, n_searches = (+ _swe_call_worker(+ worker, str(w_input), cfg, task_meta,+ shared_workdir, turn,+ )+ )+ else:+ w_text, w_in, w_out, is_local, extra_cost, n_searches = (+ _call_worker(worker, str(w_input), cfg)+ )+ if is_local:+ tokens_local += w_in + w_out+ else:+ tokens_cloud += w_in + w_out+ cost += self.cost_usd(worker["model"], w_in, w_out) + extra_cost+ history.append({+ "role": "worker",+ "turn": turn,+ "worker_id": wid,+ "worker_name": worker["name"],+ "worker_model": worker["model"],+ "output": w_text,+ "tokens_in": w_in,+ "tokens_out": w_out,+ "n_web_searches": n_searches,+ })+ continue+ # Unknown action kind — treat as parse failure.+ parse_failures += 1++ if final_answer is None:+ # Hard fallback: call the strongest worker (last) directly.+ worker = workers[-1]+ if swe_mode and shared_workdir is not None:+ ans, w_in, w_out, is_local, extra_cost, _ = _swe_call_worker(+ worker, question, cfg, task_meta,+ shared_workdir, max_turns + 1,+ )+ else:+ ans, w_in, w_out, is_local, extra_cost, _ = _call_worker(+ worker, question, cfg+ )+ if is_local:+ tokens_local += w_in + w_out+ else:+ tokens_cloud += w_in + w_out+ cost += self.cost_usd(worker["model"], w_in, w_out) + extra_cost+ history.append({+ "role": "worker",+ "turn": max_turns + 1,+ "worker_id": worker["id"],+ "worker_name": worker["name"],+ "worker_model": worker["model"],+ "output": ans,+ "tokens_in": w_in,+ "tokens_out": w_out,+ "fallback": True,+ })+ final_answer = ans++ # In SWE mode, the authoritative output is the working-tree diff —+ # frame it (the runner extracts it via the scorer's ```diff fence).+ if swe_mode and shared_workdir is not None:+ patch = _extract_diff(shared_workdir)+ if patch.strip():+ final_answer = (+ f"{final_answer}\n\n```diff\n{patch}```"+ if final_answer else f"```diff\n{patch}```"+ )++ meta = {+ "tokens_local": tokens_local,+ "tokens_cloud": tokens_cloud,+ "cost_usd": cost,+ "turns": len([h for h in history if h["role"] == "orchestrator"]),+ "traces": {+ "history": history,+ "forced_final": forced_final,+ "parse_failures": parse_failures,+ "workers": workers,+ "note": (+ "inference-only port; the RL-trained Nemotron-Orchestrator-8B "+ "is NOT in the loop. Results are preliminary."+ ),+ },+ }+ return final_answer, meta+ finally:+ if shared_workdir is not None:+ shutil.rmtree(shared_workdir, ignore_errors=True)+++__all__ = ["ToolOrchestraAgent"]diff --git a/src/openjarvis/engine/_stubs.py b/src/openjarvis/engine/_stubs.pyindex 4a940274..c0629502 100644--- a/src/openjarvis/engine/_stubs.py+++ b/src/openjarvis/engine/_stubs.py@@ -21,12 +21,18 @@ class StreamChunk:Used by ``stream_full()`` to yield rich streaming data includingtool_calls fragments and finish_reason, unlike ``stream()`` whichonly yields plain content strings.++ ``content_blocks`` and ``tool_results`` are aggregate fields emitted+ once at end-of-stream so streaming callers reach parity with the+ non-streaming ``generate()`` return shape."""content: Optional[str] = Nonetool_calls: Optional[List[Dict[str, Any]]] = Nonefinish_reason: Optional[str] = Noneusage: Optional[Dict[str, Any]] = None+ content_blocks: Optional[List[Dict[str, Any]]] = None+ tool_results: Optional[List[Dict[str, Any]]] = None@dataclass(slots=True)diff --git a/src/openjarvis/engine/cloud.py b/src/openjarvis/engine/cloud.pyindex 7a6340c0..8a32fc84 100644--- a/src/openjarvis/engine/cloud.py+++ b/src/openjarvis/engine/cloud.py@@ -3,6 +3,7 @@from __future__ import annotationsimport json+import loggingimport osimport timefrom collections.abc import AsyncIterator, Sequence@@ -19,6 +20,8 @@ from openjarvis.engine._base import ()from openjarvis.engine._stubs import StreamChunk+logger = logging.getLogger(__name__)+# Pricing per million tokens (input, output)PRICING: Dict[str, tuple[float, float]] = {"gpt-4o": (2.50, 10.00),@@ -149,6 +152,37 @@ def estimate_cost(model: str, prompt_tokens: int, completion_tokens: int) -> floreturn input_cost + output_cost+def _serialize_anthropic_block(block: Any) -> Dict[str, Any]:+ """Turn an Anthropic content block into a JSON-safe dict for tracing.++ Handles every block type Anthropic returns today (text, tool_use,+ server_tool_use, web_search_tool_result, tool_result, thinking). Nested+ content (e.g. ``web_search_tool_result.content`` is itself a list of+ citation/result blocks) recurses so the trace has the full payload, not+ a truncated summary.+ """+ out: Dict[str, Any] = {+ "type": getattr(block, "type", None) or type(block).__name__,+ }+ for attr in (+ "id", "name", "input", "text", "thinking", "signature",+ "tool_use_id", "content",+ ):+ if not hasattr(block, attr):+ continue+ val = getattr(block, attr)+ if attr == "content" and isinstance(val, list):+ out[attr] = [_serialize_anthropic_block(b) for b in val]+ elif hasattr(val, "model_dump"):+ try:+ out[attr] = val.model_dump()+ except Exception:+ out[attr] = str(val)+ else:+ out[attr] = val+ return out++def _annotate_anthropic_cache(messages: list[dict]) -> list[dict]:"""Add cache_control to system message for Anthropic prompt caching."""result = []@@ -573,20 +607,48 @@ class CloudEngine(InferenceEngine):resp = self._anthropic_client.messages.create(**create_kwargs)elapsed = time.monotonic() - t0- # Extract text and tool_use blocks from response content+ # Walk every block in resp.content. Anthropic returns several kinds:+ # - text (plain assistant text)+ # - tool_use (model wants the caller to run a tool)+ # - server_tool_use (model invoked a server-side tool,+ # e.g. web_search; carries the actual+ # query the model issued)+ # - web_search_tool_result (server-tool result body)+ # - tool_result (caller-side tool result echo)+ # - thinking (Opus reasoning trace)+ # ``content_blocks`` keeps the full serialized list for trace+ # observability. ``tool_calls`` is the narrow caller-executable+ # surface — only ``tool_use`` blocks (server_tool_use lives in+ # content_blocks since Anthropic already ran it server-side).+ # ``tool_results`` aggregates both result kinds.content_parts: list[str] = []tool_calls: list[Dict[str, Any]] = []+ tool_results: list[Dict[str, Any]] = []+ content_blocks: list[Dict[str, Any]] = []for block in resp.content:- if getattr(block, "type", None) == "tool_use":+ btype = getattr(block, "type", None) or type(block).__name__+ serialized = _serialize_anthropic_block(block)+ content_blocks.append(serialized)+ if btype == "tool_use":+ block_id = getattr(block, "id", None)+ if not block_id:+ logger.warning(+ "Anthropic tool_use block without an id; skipping. "+ "Round-trip into the next assistant turn would fail "+ "Anthropic's tool_use_id matching."+ )+ continuetool_calls.append({- "id": block.id,- "name": block.name,- "arguments": json.dumps(block.input)- if isinstance(block.input, dict)- else str(block.input),+ "id": block_id,+ "name": getattr(block, "name", ""),+ "arguments": json.dumps(getattr(block, "input", None))+ if isinstance(getattr(block, "input", None), dict)+ else str(getattr(block, "input", "")),})+ elif btype in ("web_search_tool_result", "tool_result"):+ tool_results.append(serialized)elif hasattr(block, "text"):content_parts.append(block.text)@@ -605,10 +667,13 @@ class CloudEngine(InferenceEngine):"finish_reason": resp.stop_reason or "stop","cost_usd": estimate_cost(model, prompt_tokens, completion_tokens),"ttft": elapsed,+ "content_blocks": content_blocks,}if tool_calls:result["tool_calls"] = tool_calls+ if tool_results:+ result["tool_results"] = tool_resultsreturn result@@ -1281,6 +1346,31 @@ class CloudEngine(InferenceEngine):finish = "tool_calls" if stop_reason == "tool_use" else "stop"yield StreamChunk(finish_reason=finish)+ # End-of-stream parity with ``_generate_anthropic``: emit+ # ``content_blocks`` (every block kind, including server_tool_use+ # and thinking) and ``tool_results`` (web_search_tool_result ++ # tool_result) so streaming traces see what non-streaming traces+ # see.+ try:+ final_msg = stream.get_final_message()+ except Exception as exc: # noqa: BLE001 — SDK shape varies+ logger.debug("Anthropic stream.get_final_message() failed: %s", exc)+ final_msg = None+ if final_msg is not None and getattr(final_msg, "content", None):+ content_blocks: list[Dict[str, Any]] = []+ tool_results: list[Dict[str, Any]] = []+ for block in final_msg.content:+ btype = getattr(block, "type", None) or type(block).__name__+ serialized = _serialize_anthropic_block(block)+ content_blocks.append(serialized)+ if btype in ("web_search_tool_result", "tool_result"):+ tool_results.append(serialized)+ if content_blocks or tool_results:+ yield StreamChunk(+ content_blocks=content_blocks or None,+ tool_results=tool_results or None,+ )+async def stream_full(self,messages: Sequence[Message],diff --git a/src/openjarvis/evals/datasets/gaia.py b/src/openjarvis/evals/datasets/gaia.pyindex 6b519d18..18a89b3b 100644--- a/src/openjarvis/evals/datasets/gaia.py+++ b/src/openjarvis/evals/datasets/gaia.py@@ -159,6 +159,7 @@ class GAIADataset(DatasetProvider):metadata = {"task_id": task_id,+ "question": question,"level": level,"file_name": file_name,"file_path": str(file_path) if file_path else None,diff --git a/src/openjarvis/evals/datasets/swebench.py b/src/openjarvis/evals/datasets/swebench.pyindex 1117649f..d227dab6 100644--- a/src/openjarvis/evals/datasets/swebench.py+++ b/src/openjarvis/evals/datasets/swebench.py@@ -142,6 +142,7 @@ class SWEBenchDataset(DatasetProvider):metadata: dict[str, Any] = {"instance_id": instance_id,+ "problem_statement": problem_statement,"repo": repo,"base_commit": base_commit,"hints_text": hints_text,diff --git a/src/openjarvis/evals/scorers/swebench_harness.py b/src/openjarvis/evals/scorers/swebench_harness.pynew file mode 100644index 00000000..44d27f57--- /dev/null+++ b/src/openjarvis/evals/scorers/swebench_harness.py@@ -0,0 +1,295 @@+"""SWE-bench harness scorer — runs the official `swebench` test harness.++This is the authoritative pass/fail scorer for SWE-bench-Verified.+The lightweight :class:`SWEBenchScorer` in ``swebench_structural.py``+checks only that the model produced *something patch-shaped*; this one+actually applies the patch, runs the targeted tests, and reads the+harness's report JSON.++Backends (selected by ``SWEBENCH_BACKEND`` env var):++- ``modal`` (default) — runs on Modal in the cloud; needs ``swebench[modal]``+ installed and ``modal token new`` configured once.+- ``docker`` — runs locally; needs Docker daemon + user in ``docker`` group.++Ported from ``hybrid-local-cloud-compute/benches/swebench_verified/{runner,parsing}.py``,+with two upstream-`swebench` patches applied at import time:++1. **Modal cgroup-v2 fix**:+ ``swebench/harness/modal_eval/run_evaluation_modal.py:66`` writes to+ ``/sys/fs/cgroup/cpu/cpu.shares`` (cgroup v1). Modal v2 sandboxes use+ cgroup v2 — the path doesn't exist and every sandbox dies on the write.+ Wrap the write in try/except.++2. **Rescore `*_ids` fix**: older harness rescore code read+ ``resolved_instances`` / ``unresolved_instances`` / ``error_instances``+ as lists. Current swebench writes counts there and puts IDs in+ ``*_ids`` fields. Wherever we read these we use ``*_ids``.++Both patches are idempotent and only fire when the harness modules are+imported via this scorer (we don't touch swebench until ``score()`` is+called for the first time).+"""++from __future__ import annotations++import json+import logging+import os+import re+import subprocess+import sys+import tempfile+from pathlib import Path+from typing import Any, Dict, Optional, Tuple++from openjarvis.evals.core.scorer import Scorer+from openjarvis.evals.core.types import EvalRecord++logger = logging.getLogger(__name__)+++# ---------- Patch tracking ----------++_PATCHES_APPLIED = False+++def _patch_modal_cgroup_v2() -> None:+ """Wrap the cgroup-v1 write in run_evaluation_modal.py with try/except.++ The upstream line is ``Path("/sys/fs/cgroup/cpu/cpu.shares").write_text(...)``.+ Modal v2 sandboxes are cgroup-v2 and that path doesn't exist; the write+ raises FileNotFoundError and the sandbox dies. We replace the entire+ ``set_cpu_quota`` function body (if present) with a try/except wrapper.+ """+ try:+ from swebench.harness.modal_eval import (+ run_evaluation_modal as _m, # type: ignore[import-not-found]+ )+ except Exception:+ return+ if getattr(_m, "_hybrid_cgroup_patched", False):+ return+ orig = getattr(_m, "set_cpu_quota", None)+ if orig is None:+ # Upstream API changed: ``set_cpu_quota`` no longer exists. The+ # cgroup-v1 write that used to live inside it is either gone (good)+ # or now lives somewhere we can't patch (bad — every Modal-v2+ # sandbox will still die). Surface loudly so failures aren't+ # silently scored as 0 via the ``no_report`` fallback in+ # :func:`_run_harness`.+ logger.warning(+ "swebench.harness.modal_eval.run_evaluation_modal.set_cpu_quota "+ "is missing — cgroup-v2 patch could not be applied. If your "+ "Modal sandboxes are scoring 0 with `reason: no_report`, "+ "verify upstream swebench's Modal cgroup handling."+ )+ _m._hybrid_cgroup_patched = True # type: ignore[attr-defined]+ return++ def patched(*args, **kwargs): # type: ignore[no-untyped-def]+ try:+ return orig(*args, **kwargs)+ except FileNotFoundError:+ # cgroup v2 sandbox — path missing is expected, skip.+ return None+ except PermissionError:+ return None++ _m.set_cpu_quota = patched # type: ignore[assignment]+ _m._hybrid_cgroup_patched = True # type: ignore[attr-defined]+++def _apply_patches_once() -> None:+ global _PATCHES_APPLIED+ if _PATCHES_APPLIED:+ return+ _patch_modal_cgroup_v2()+ _PATCHES_APPLIED = True+++# ---------- Patch extraction ----------++_FENCE_PATTERNS = [+ re.compile(r"```(?:diff|patch)\n(.*?)```", re.DOTALL),+ re.compile(r"```\n(diff --git .*?)```", re.DOTALL),+]+++def extract_patch(text: str) -> Optional[str]:+ """Pull a unified diff out of agent output. ``None`` if not found."""+ if not text:+ return None+ for pat in _FENCE_PATTERNS:+ m = pat.search(text)+ if m:+ return m.group(1).strip() + "\n"+ if "diff --git" in text:+ start = text.index("diff --git")+ return text[start:].strip() + "\n"+ return None+++# ---------- Harness invocation ----------++def _harness_cache_dir() -> Path:+ """Where the swebench subprocess writes its report JSON + logs/ tree.++ Defaults to ``$OPENJARVIS_HOME/.swebench-cache`` if set, otherwise to a+ process-shared tempdir. Pin both so we don't pollute the project root.+ """+ home = os.environ.get("OPENJARVIS_HOME")+ if home:+ cache = Path(home) / ".swebench-cache"+ else:+ cache = Path(tempfile.gettempdir()) / "openjarvis-swebench-cache"+ cache.mkdir(parents=True, exist_ok=True)+ return cache+++def _find_report(cache: Path, instance_id: str, run_id: str) -> Optional[Dict[str, Any]]:+ """Find the harness's report JSON for one instance.++ swebench writes ``<model_name_or_path>.<run_id>.json`` inside the+ subprocess CWD. We use ``model_name_or_path="openjarvis-harness"``,+ ``run_id=f"oj-{instance_id}"`` in :func:`_run_harness`.+ """+ fname = f"openjarvis-harness.{run_id}.json"+ p = cache / fname+ if not p.exists():+ return None+ try:+ return json.loads(p.read_text())+ except Exception:+ return None+++def _run_harness(instance_id: str, patch: str, timeout_s: int) -> Dict[str, Any]:+ """Hand one prediction to ``python -m swebench.harness.run_evaluation``.++ Returns ``{"success": bool, "score": float, "details": dict}``.+ """+ _apply_patches_once()+ backend = os.environ.get("SWEBENCH_BACKEND", "modal").lower()+ cache = _harness_cache_dir()+ run_id = f"oj-{instance_id}"++ # Defend against stale reports: ``run_id`` is deterministic per+ # instance, the cache dir is shared across runs, and ``_find_report``+ # globs by filename. If a prior subprocess crashed mid-run (or was+ # killed by timeout) and left a stale JSON, we'd silently read that+ # old verdict as the current result. Delete it up front.+ stale = cache / f"openjarvis-harness.{run_id}.json"+ if stale.exists():+ try:+ stale.unlink()+ except OSError as exc:+ logger.warning("Could not remove stale report %s: %s", stale, exc)++ with tempfile.TemporaryDirectory() as tmp:+ tmp_path = Path(tmp)+ preds_path = tmp_path / "predictions.jsonl"+ preds_path.write_text(json.dumps({+ "instance_id": instance_id,+ "model_name_or_path": "openjarvis-harness",+ "model_patch": patch,+ }) + "\n")++ cmd = [+ sys.executable, "-m", "swebench.harness.run_evaluation",+ "--predictions_path", str(preds_path),+ "--max_workers", "1",+ "--run_id", run_id,+ "--dataset_name", "SWE-bench/SWE-bench_Verified",+ "--instance_ids", instance_id,+ ]+ if backend == "modal":+ cmd += ["--modal", "true"]++ proc = subprocess.run(+ cmd, capture_output=True, text=True,+ timeout=timeout_s, cwd=str(cache),+ )++ report = _find_report(cache, instance_id, run_id)+ if report is None:+ return {+ "success": False,+ "score": 0.0,+ "details": {+ "reason": "no_report",+ "stdout": proc.stdout[-2000:],+ "stderr": proc.stderr[-2000:],+ },+ }++ # The fix from `rescore.py`: read `resolved_ids` (current schema)+ # not `resolved_instances` (older). Older harness wrote lists into+ # `resolved_instances`; current swebench puts the count there and+ # the actual instance ids in `resolved_ids`.+ resolved_ids = report.get("resolved_ids") or []+ # Belt-and-suspenders: also accept the older list-typed field, in+ # case the user is on an older swebench install.+ if not resolved_ids:+ legacy = report.get("resolved_instances")+ if isinstance(legacy, list):+ resolved_ids = legacy+ resolved = instance_id in resolved_ids+ return {+ "success": resolved,+ "score": 1.0 if resolved else 0.0,+ "details": {"report": report},+ }+++# ---------- Scorer ----------++class SWEBenchHarnessScorer(Scorer):+ """SWE-bench Verified scorer that runs the official harness.++ ``score(record, model_answer)`` returns ``(is_correct, details)``:++ - ``is_correct = True`` if the harness marks the instance resolved.+ - ``is_correct = False`` on harness failure or unresolved.+ - ``details`` includes the raw harness report under ``["report"]`` plus+ a ``"patch"`` field with the extracted patch text.+ """++ scorer_id = "swebench_harness"++ def __init__(+ self,+ *,+ timeout_s: int = 1800,+ judge_backend: object = None, # noqa: ARG002 — CLI factory compat+ judge_model: str = "", # noqa: ARG002 — CLI factory compat+ ) -> None:+ self._timeout_s = int(timeout_s)++ def score(+ self,+ record: EvalRecord,+ model_answer: str,+ ) -> Tuple[Optional[bool], Dict[str, Any]]:+ if not model_answer or not model_answer.strip():+ return False, {"reason": "empty_response"}++ patch = extract_patch(model_answer)+ if patch is None:+ return False, {"reason": "no_patch_extracted"}++ instance_id = (+ record.metadata.get("instance_id")+ or record.record_id+ or ""+ )+ if not instance_id:+ return False, {"reason": "missing_instance_id"}++ result = _run_harness(instance_id, patch, self._timeout_s)+ details = dict(result.get("details", {}))+ details["patch"] = patch+ return bool(result["success"]), details+++__all__ = ["SWEBenchHarnessScorer", "extract_patch"]diff --git a/src/openjarvis/telemetry/instrumented_engine.py b/src/openjarvis/telemetry/instrumented_engine.pyindex 1e6491d8..762849df 100644--- a/src/openjarvis/telemetry/instrumented_engine.py+++ b/src/openjarvis/telemetry/instrumented_engine.py@@ -255,9 +255,11 @@ class InstrumentedEngine(InferenceEngine):"mean_itl_ms": mean_itl_ms,"energy_method": energy_method,"energy_vendor": energy_vendor,- # Rich trace data: model response content+ # Rich trace data: model response content + structured blocks"content": result.get("content", ""),"tool_calls": result.get("tool_calls", []),+ "tool_results": result.get("tool_results", []),+ "content_blocks": result.get("content_blocks", []),"finish_reason": result.get("finish_reason", ""),}diff --git a/src/openjarvis/traces/collector.py b/src/openjarvis/traces/collector.pyindex 96f17311..0eb4186e 100644--- a/src/openjarvis/traces/collector.py+++ b/src/openjarvis/traces/collector.py@@ -158,6 +158,8 @@ class TraceCollector:),"content": data.get("content", ""),"tool_calls": data.get("tool_calls", []),+ "tool_results": data.get("tool_results", []),+ "content_blocks": data.get("content_blocks", []),"finish_reason": data.get("finish_reason", ""),},metadata={