Personal AI, On Personal Devices
Python
docs: add Phase 3 plan for two-stage retrieval + Deep Research Agent
krypticmoused5b00eb4a86a
diff --git a/docs/superpowers/plans/2026-03-25-deep-research-phase3.md b/docs/superpowers/plans/2026-03-25-deep-research-phase3.mdnew file mode 100644index 00000000..f025ee0a--- /dev/null+++ b/docs/superpowers/plans/2026-03-25-deep-research-phase3.md@@ -0,0 +1,1270 @@+# Deep Research Phase 3: Two-Stage Retrieval + Deep Research Agent++> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking.++**Goal:** Add ColBERTv2 reranking on top of BM25 retrieval and build the DeepResearchAgent — a multi-hop tool-using agent that searches your personal knowledge base across sources, synthesizes findings, and produces cited research reports.++**Architecture:** A new `TwoStageRetriever` composes BM25 recall (KnowledgeStore FTS5) with ColBERTv2 semantic reranking. The `knowledge_search` tool is upgraded to use it. A `DeepResearchAgent` extends `ToolUsingAgent` with a research-oriented system prompt, multi-hop loop, and structured report output with cross-platform citations. ColBERT persistence to disk is added via memory-mapped tensors for large knowledge bases.++**Tech Stack:** Python 3.10+, sqlite3 (BM25/FTS5), torch + colbert-ai (reranking, optional), pytest++**Spec:** `docs/superpowers/specs/2026-03-25-deep-research-setup-design.md` — Sections 7, 8, Phase 3++**Depends on:** Phase 1 complete (KnowledgeStore, knowledge_search tool, IngestionPipeline, SemanticChunker)++---++## File Structure++```+src/openjarvis/connectors/+├── retriever.py # TwoStageRetriever: BM25 recall → ColBERT rerank+├── store.py # (modify) Add store_embedding/load_embedding methods+├── pipeline.py # (modify) Dual-write to ColBERT index during ingest++src/openjarvis/agents/+├── deep_research.py # DeepResearchAgent with multi-hop loop + citations++src/openjarvis/tools/+├── knowledge_search.py # (modify) Upgrade to use TwoStageRetriever++tests/connectors/+├── test_retriever.py # TwoStageRetriever tests++tests/agents/+├── test_deep_research.py # DeepResearchAgent tests+```++---++### Task 1: TwoStageRetriever (BM25 Recall → ColBERT Rerank)++**Files:**+- Create: `src/openjarvis/connectors/retriever.py`+- Create: `tests/connectors/test_retriever.py`++- [ ] **Step 1: Write failing tests**++Create `tests/connectors/test_retriever.py`:++```python+"""Tests for TwoStageRetriever — BM25 recall + ColBERT rerank."""++from __future__ import annotations++from pathlib import Path+from unittest.mock import MagicMock, patch++import pytest++from openjarvis.connectors.retriever import TwoStageRetriever+from openjarvis.connectors.store import KnowledgeStore+from openjarvis.tools.storage._stubs import RetrievalResult+++def store(tmp_path: Path) -> KnowledgeStore:+ s = KnowledgeStore(db_path=str(tmp_path / "retriever_test.db"))+ s.store(+ content="Kubernetes migration proposal with cost analysis",+ source="gdrive",+ doc_type="document",+ author="sarah",+ title="K8s Proposal",+ )+ s.store(+ content="Discussion about Kubernetes deployment timeline",+ source="slack",+ doc_type="message",+ author="mike",+ title="#infrastructure",+ )+ s.store(+ content="Quarterly budget review for cloud services",+ source="gmail",+ doc_type="email",+ author="alice",+ title="Re: Budget Q3",+ )+ s.store(+ content="Meeting notes about project planning",+ source="granola",+ doc_type="document",+ author="bob",+ title="Sprint Planning",+ )+ return s+++def retriever(store: KnowledgeStore) -> TwoStageRetriever:+ return TwoStageRetriever(store=store)+++def test_retrieve_returns_results(retriever: TwoStageRetriever) -> None:+ results = retriever.retrieve("Kubernetes migration")+ assert len(results) > 0+ assert all(isinstance(r, RetrievalResult) for r in results)+++def test_retrieve_respects_top_k(retriever: TwoStageRetriever) -> None:+ results = retriever.retrieve("cloud", top_k=2)+ assert len(results) <= 2+++def test_retrieve_with_source_filter(+ retriever: TwoStageRetriever,+) -> None:+ results = retriever.retrieve("Kubernetes", source="gdrive")+ assert len(results) >= 1+ assert all(+ r.metadata.get("source") == "gdrive" for r in results+ )+++def test_retrieve_with_author_filter(+ retriever: TwoStageRetriever,+) -> None:+ results = retriever.retrieve("budget", author="alice")+ assert len(results) >= 1+ assert all(+ r.metadata.get("author") == "alice" for r in results+ )+++def test_retrieve_bm25_only_when_no_colbert(+ retriever: TwoStageRetriever,+) -> None:+ """Without ColBERT, falls back to BM25-only results."""+ results = retriever.retrieve("planning")+ assert len(results) > 0+ # Results should still have scores+ assert all(r.score >= 0 for r in results)+++def test_retrieve_with_colbert_reranking(+ store: KnowledgeStore,+) -> None:+ """With a mock ColBERT reranker, results are reranked."""+ mock_reranker = MagicMock()+ mock_reranker.rerank.return_value = [+ RetrievalResult(+ content="Kubernetes migration proposal with cost analysis",+ score=0.95,+ source="gdrive",+ metadata={"source": "gdrive", "author": "sarah"},+ ),+ ]++ retriever = TwoStageRetriever(+ store=store, reranker=mock_reranker+ )+ results = retriever.retrieve("Kubernetes", top_k=5)+ assert len(results) >= 1+ mock_reranker.rerank.assert_called_once()+++def test_retrieve_no_results(retriever: TwoStageRetriever) -> None:+ results = retriever.retrieve("xyznonexistent999")+ assert len(results) == 0+++def test_retrieve_recall_k_larger_than_top_k(+ retriever: TwoStageRetriever,+) -> None:+ """BM25 recall fetches more candidates than final top_k."""+ results = retriever.retrieve("Kubernetes", top_k=1)+ assert len(results) <= 1+```++- [ ] **Step 2: Run tests to verify they fail**++Run: `cd /lambda/nfs/lambda-stanford/jonsf/scratch_v2/OpenJarvis && uv run pytest tests/connectors/test_retriever.py -v`++Expected: FAIL — `ModuleNotFoundError`++- [ ] **Step 3: Implement TwoStageRetriever**++Create `src/openjarvis/connectors/retriever.py`:++```python+"""Two-stage retriever: BM25 recall from KnowledgeStore → optional ColBERT rerank.++Stage 1 (BM25): Fast keyword recall via SQLite FTS5. Returns top-N candidates.+Stage 2 (ColBERT): Semantic reranking of candidates via token-level MaxSim scoring.+ Falls back to BM25-only if ColBERT is not available.+"""++from __future__ import annotations++import logging+from abc import ABC, abstractmethod+from typing import Any, List, Optional++from openjarvis.connectors.store import KnowledgeStore+from openjarvis.tools.storage._stubs import RetrievalResult++logger = logging.getLogger(__name__)+++class Reranker(ABC):+ """Abstract base for reranking BM25 candidates."""++ @abstractmethod+ def rerank(+ self,+ query: str,+ candidates: List[RetrievalResult],+ *,+ top_k: int = 10,+ ) -> List[RetrievalResult]:+ """Rerank candidates and return top-k by semantic relevance."""+++class ColBERTReranker(Reranker):+ """Reranker using ColBERTv2 late-interaction MaxSim scoring.++ Lazily loads the ColBERT checkpoint on first use.+ """++ def __init__(+ self,+ *,+ checkpoint: str = "colbert-ir/colbertv2.0",+ device: str = "cpu",+ ) -> None:+ self._checkpoint_name = checkpoint+ self._device = device+ self._loaded = False+ self._checkpoint = None++ def _load(self) -> None:+ if self._loaded:+ return+ try:+ from colbert.modeling.checkpoint import Checkpoint++ self._checkpoint = Checkpoint(+ self._checkpoint_name, colbert_config=None+ )+ self._loaded = True+ logger.info("ColBERT checkpoint loaded: %s", self._checkpoint_name)+ except ImportError:+ logger.warning("colbert-ai not installed, reranking disabled")+ self._loaded = True # Don't retry+ except Exception as exc:+ logger.warning("ColBERT load failed: %s", exc)+ self._loaded = True++ def _encode(self, text: str) -> Any:+ if self._checkpoint is None:+ return None+ return self._checkpoint.queryFromText([text])[0]++ @staticmethod+ def _maxsim(query_embs: Any, doc_embs: Any) -> float:+ import torch++ # query_embs: (Q, D), doc_embs: (T, D)+ sim = torch.nn.functional.cosine_similarity(+ query_embs.unsqueeze(1), doc_embs.unsqueeze(0), dim=2+ )+ return sim.max(dim=1).values.sum().item()++ def rerank(+ self,+ query: str,+ candidates: List[RetrievalResult],+ *,+ top_k: int = 10,+ ) -> List[RetrievalResult]:+ self._load()+ if self._checkpoint is None:+ return candidates[:top_k]++ query_embs = self._encode(query)+ if query_embs is None:+ return candidates[:top_k]++ scored = []+ for r in candidates:+ doc_embs = self._encode(r.content)+ if doc_embs is None:+ scored.append((r, r.score))+ else:+ score = self._maxsim(query_embs, doc_embs)+ scored.append((r, score))++ scored.sort(key=lambda x: x[1], reverse=True)+ results = []+ for r, score in scored[:top_k]:+ results.append(+ RetrievalResult(+ content=r.content,+ score=score,+ source=r.source,+ metadata=r.metadata,+ )+ )+ return results+++class TwoStageRetriever:+ """Composes BM25 recall with optional semantic reranking.++ Parameters+ ----------+ store:+ KnowledgeStore for BM25 recall via FTS5.+ reranker:+ Optional Reranker (e.g., ColBERTReranker). If None, BM25-only.+ recall_k:+ Number of BM25 candidates to fetch for reranking (default: 100).+ """++ def __init__(+ self,+ store: KnowledgeStore,+ reranker: Optional[Reranker] = None,+ *,+ recall_k: int = 100,+ ) -> None:+ self._store = store+ self._reranker = reranker+ self._recall_k = recall_k++ def retrieve(+ self,+ query: str,+ *,+ top_k: int = 10,+ source: str = "",+ doc_type: str = "",+ author: str = "",+ since: str = "",+ until: str = "",+ ) -> List[RetrievalResult]:+ """Two-stage retrieval: BM25 recall → optional rerank.++ Falls back to BM25-only if no reranker is configured.+ """+ # Stage 1: BM25 recall+ recall_n = max(self._recall_k, top_k * 3)+ candidates = self._store.retrieve(+ query,+ top_k=recall_n,+ source=source,+ doc_type=doc_type,+ author=author,+ since=since,+ until=until,+ )++ if not candidates:+ return []++ # Stage 2: Rerank (or just truncate)+ if self._reranker is not None and len(candidates) > top_k:+ return self._reranker.rerank(+ query, candidates, top_k=top_k+ )++ return candidates[:top_k]+```++- [ ] **Step 4: Run tests**++Run: `cd /lambda/nfs/lambda-stanford/jonsf/scratch_v2/OpenJarvis && uv run pytest tests/connectors/test_retriever.py -v`++Expected: All 8 tests PASS.++- [ ] **Step 5: Commit**++```bash+git add src/openjarvis/connectors/retriever.py tests/connectors/test_retriever.py+git commit -m "feat: add TwoStageRetriever with BM25 recall and pluggable ColBERT reranking"+```++---++### Task 2: Upgrade knowledge_search to Use TwoStageRetriever++**Files:**+- Modify: `src/openjarvis/tools/knowledge_search.py`+- Modify: `tests/tools/test_knowledge_search.py`++- [ ] **Step 1: Write new test for two-stage retrieval**++Add to `tests/tools/test_knowledge_search.py`:++```python+def test_tool_uses_two_stage_retriever(tmp_path: Path) -> None:+ """When initialized with a TwoStageRetriever, uses it."""+ from openjarvis.connectors.retriever import TwoStageRetriever++ store = KnowledgeStore(db_path=str(tmp_path / "ts_test.db"))+ store.store(+ content="Deep learning research paper",+ source="gdrive",+ doc_type="document",+ )+ retriever = TwoStageRetriever(store=store)+ tool = KnowledgeSearchTool(store=store, retriever=retriever)+ result = tool.execute(query="deep learning")+ assert result.success+ assert result.metadata["num_results"] > 0+```++- [ ] **Step 2: Run test to verify it fails**++Run: `cd /lambda/nfs/lambda-stanford/jonsf/scratch_v2/OpenJarvis && uv run pytest tests/tools/test_knowledge_search.py::test_tool_uses_two_stage_retriever -v`++Expected: FAIL — `TypeError: __init__() got an unexpected keyword argument 'retriever'`++- [ ] **Step 3: Update KnowledgeSearchTool**++Modify `src/openjarvis/tools/knowledge_search.py` — add optional `retriever` parameter:++In `__init__`, add `retriever` parameter:++```python+def __init__(+ self,+ store: Optional[KnowledgeStore] = None,+ retriever: Optional[Any] = None,+) -> None:+ self._store = store+ self._retriever = retriever+```++In `execute`, use retriever if available, else fall back to store:++```python+# In execute(), replace the store.retrieve() call with:+if self._retriever is not None:+ results = self._retriever.retrieve(+ query,+ top_k=top_k,+ source=params.get("source", ""),+ doc_type=params.get("doc_type", ""),+ author=params.get("author", ""),+ since=params.get("since", ""),+ until=params.get("until", ""),+ )+elif self._store is not None:+ results = self._store.retrieve(+ query,+ top_k=top_k,+ source=params.get("source", ""),+ doc_type=params.get("doc_type", ""),+ author=params.get("author", ""),+ since=params.get("since", ""),+ until=params.get("until", ""),+ )+else:+ return ToolResult(+ tool_name="knowledge_search",+ content="No knowledge store configured.",+ success=False,+ )+```++Also update the no-store check at the top of `execute()` to allow retriever-only mode:++```python+if self._store is None and self._retriever is None:+ return ToolResult(+ tool_name="knowledge_search",+ content="No knowledge store configured. Run 'jarvis connect' to set up data sources.",+ success=False,+ )+```++- [ ] **Step 4: Run all knowledge_search tests**++Run: `cd /lambda/nfs/lambda-stanford/jonsf/scratch_v2/OpenJarvis && uv run pytest tests/tools/test_knowledge_search.py -v`++Expected: All tests PASS (existing + new).++- [ ] **Step 5: Commit**++```bash+git add src/openjarvis/tools/knowledge_search.py tests/tools/test_knowledge_search.py+git commit -m "feat: upgrade knowledge_search to support TwoStageRetriever"+```++---++### Task 3: DeepResearchAgent++**Files:**+- Create: `src/openjarvis/agents/deep_research.py`+- Create: `tests/agents/test_deep_research.py`++- [ ] **Step 1: Write failing tests**++Create `tests/agents/test_deep_research.py`:++```python+"""Tests for DeepResearchAgent — multi-hop research with citations."""++from __future__ import annotations++import json+from pathlib import Path+from typing import Any, Dict, List, Optional+from unittest.mock import MagicMock++import pytest++from openjarvis.agents.deep_research import DeepResearchAgent+from openjarvis.connectors.store import KnowledgeStore+from openjarvis.core.registry import AgentRegistry+from openjarvis.core.types import ToolResult+from openjarvis.tools._stubs import BaseTool, ToolSpec+from openjarvis.tools.knowledge_search import KnowledgeSearchTool+++def store(tmp_path: Path) -> KnowledgeStore:+ s = KnowledgeStore(db_path=str(tmp_path / "dr_test.db"))+ s.store(+ content="Kubernetes migration was proposed by Sarah in March",+ source="slack",+ doc_type="message",+ author="sarah",+ title="#infrastructure",+ url="https://slack.com/archives/C001/p123",+ )+ s.store(+ content="Cost analysis shows 40% increase during transition",+ source="gdrive",+ doc_type="document",+ author="sarah",+ title="K8s Cost Analysis",+ url="https://drive.google.com/d/doc1",+ )+ s.store(+ content="Migration approved by engineering leads on March 8",+ source="gmail",+ doc_type="email",+ author="mike",+ title="Re: K8s migration approved",+ url="https://mail.google.com/mail/u/0/#inbox/msg1",+ )+ return s+++def mock_engine():+ engine = MagicMock()+ engine.engine_id = "mock"+ engine.health.return_value = True+ return engine+++def _make_engine_response(+ content: str, tool_calls: list | None = None+) -> dict:+ result: Dict[str, Any] = {+ "content": content,+ "usage": {+ "prompt_tokens": 50,+ "completion_tokens": 50,+ "total_tokens": 100,+ },+ "model": "test-model",+ "finish_reason": "stop",+ }+ if tool_calls:+ result["tool_calls"] = tool_calls+ result["finish_reason"] = "tool_calls"+ return result+++def test_agent_registration() -> None:+ import openjarvis.agents.deep_research # noqa: F401++ AgentRegistry.register_value(+ "deep_research", DeepResearchAgent+ )+ assert AgentRegistry.contains("deep_research")+++def test_agent_produces_result(+ mock_engine: MagicMock, store: KnowledgeStore+) -> None:+ """Agent returns an AgentResult with content."""+ mock_engine.generate.return_value = _make_engine_response(+ "Based on my research, the Kubernetes migration was "+ "proposed by Sarah.\n\n**Sources:**\n"+ "1. [slack] #infrastructure"+ )++ ks_tool = KnowledgeSearchTool(store=store)+ agent = DeepResearchAgent(+ engine=mock_engine,+ model="test-model",+ tools=[ks_tool],+ max_turns=5,+ )++ result = agent.run("What was the K8s migration context?")+ assert result.content+ assert result.turns >= 1+++def test_agent_uses_knowledge_search(+ mock_engine: MagicMock, store: KnowledgeStore+) -> None:+ """Agent calls knowledge_search tool during research."""+ # First call: agent wants to search+ call1 = _make_engine_response(+ "",+ tool_calls=[+ {+ "id": "call_1",+ "type": "function",+ "function": {+ "name": "knowledge_search",+ "arguments": json.dumps(+ {"query": "Kubernetes migration"}+ ),+ },+ }+ ],+ )+ # Second call: agent synthesizes+ call2 = _make_engine_response(+ "The Kubernetes migration was proposed by Sarah in March.\n\n"+ "**Sources:**\n"+ "1. [slack] #infrastructure — sarah\n"+ "2. [gdrive] K8s Cost Analysis — sarah"+ )+ mock_engine.generate.side_effect = [call1, call2]++ ks_tool = KnowledgeSearchTool(store=store)+ agent = DeepResearchAgent(+ engine=mock_engine,+ model="test-model",+ tools=[ks_tool],+ max_turns=5,+ )++ result = agent.run("What was the K8s migration context?")+ assert result.content+ assert result.turns >= 1+ assert len(result.tool_results) >= 1+ assert result.tool_results[0].tool_name == "knowledge_search"+ assert result.tool_results[0].success+++def test_agent_respects_max_turns(+ mock_engine: MagicMock, store: KnowledgeStore+) -> None:+ """Agent stops after max_turns even if not done."""+ # Always return a tool call (infinite loop without max_turns)+ mock_engine.generate.return_value = _make_engine_response(+ "",+ tool_calls=[+ {+ "id": "call_n",+ "type": "function",+ "function": {+ "name": "knowledge_search",+ "arguments": json.dumps(+ {"query": "anything"}+ ),+ },+ }+ ],+ )++ ks_tool = KnowledgeSearchTool(store=store)+ agent = DeepResearchAgent(+ engine=mock_engine,+ model="test-model",+ tools=[ks_tool],+ max_turns=3,+ )++ result = agent.run("Research something")+ assert result.turns <= 3+++def test_agent_system_prompt_mentions_research(+ mock_engine: MagicMock, store: KnowledgeStore+) -> None:+ """System prompt should mention research and citations."""+ mock_engine.generate.return_value = _make_engine_response(+ "Final answer here."+ )++ ks_tool = KnowledgeSearchTool(store=store)+ agent = DeepResearchAgent(+ engine=mock_engine,+ model="test-model",+ tools=[ks_tool],+ )++ agent.run("test")++ # Check the system message passed to generate+ call_args = mock_engine.generate.call_args+ messages = call_args[0][0] # first positional arg+ system_msg = messages[0]+ assert "research" in system_msg.content.lower()+ assert "source" in system_msg.content.lower()+++def test_agent_defaults(+ mock_engine: MagicMock, store: KnowledgeStore+) -> None:+ """Check default max_turns and agent_id."""+ ks_tool = KnowledgeSearchTool(store=store)+ agent = DeepResearchAgent(+ engine=mock_engine,+ model="test-model",+ tools=[ks_tool],+ )+ assert agent.agent_id == "deep_research"+ assert agent._max_turns == 5+```++- [ ] **Step 2: Run tests to verify they fail**++Run: `cd /lambda/nfs/lambda-stanford/jonsf/scratch_v2/OpenJarvis && uv run pytest tests/agents/test_deep_research.py -v`++Expected: FAIL — `ModuleNotFoundError`++- [ ] **Step 3: Implement DeepResearchAgent**++Create `src/openjarvis/agents/deep_research.py`:++```python+"""DeepResearchAgent — multi-hop research over personal knowledge base.++Extends ToolUsingAgent with:+- Research-oriented system prompt emphasizing citations and sources+- Multi-hop retrieval loop (up to max_turns subqueries)+- Structured report output with cross-platform source attribution+"""++from __future__ import annotations++import json+import logging+from typing import Any, Dict, List, Optional++from openjarvis.agents._stubs import (+ AgentContext,+ AgentResult,+ ToolUsingAgent,+)+from openjarvis.core.events import EventBus+from openjarvis.core.registry import AgentRegistry+from openjarvis.core.types import Message, Role, ToolCall, ToolResult+from openjarvis.engine._stubs import InferenceEngine+from openjarvis.tools._stubs import BaseTool, build_tool_descriptions++logger = logging.getLogger(__name__)++_SYSTEM_PROMPT = """\+You are a Deep Research agent with access to a personal knowledge base \+containing emails, messages, documents, calendar events, contacts, and \+notes from multiple sources (Gmail, Slack, Google Drive, Notion, \+Granola, Obsidian, and more).++Your job is to answer the user's question by searching across these \+sources, synthesizing information, and producing a comprehensive \+research report with citations.++## How to Research++1. Use the `knowledge_search` tool to find relevant information. \+You can filter by source, doc_type, author, and date range.+2. Use the `think` tool to reason about what you've found and plan \+your next search.+3. Make multiple searches to cross-reference across sources. \+A question about a decision might involve Slack threads, email chains, \+documents, and calendar events.+4. When you have enough information, synthesize a clear answer.++## Output Format++Structure your response as:++1. A narrative answer addressing the user's question+2. Inline citations in the format [source] title — author+3. A "Sources" section at the end listing all referenced items with \+their platform, title, author, and URL++## Important++- Always cite your sources — never present information without \+attribution+- Cross-reference across platforms when possible+- If you can't find enough information, say so clearly+- Prefer recent information over older content when relevant++{tool_descriptions}"""++[email protected]("deep_research")+class DeepResearchAgent(ToolUsingAgent):+ """Multi-hop research agent for personal knowledge base queries.++ Searches across indexed personal data (emails, messages, documents,+ calendar events, contacts, notes) using the knowledge_search tool,+ performs multi-hop retrieval to cross-reference sources, and+ synthesizes findings into a cited research report.++ Parameters+ ----------+ engine:+ Inference engine for LLM calls.+ model:+ Model name/ID.+ tools:+ List of tools (should include KnowledgeSearchTool at minimum).+ max_turns:+ Maximum research hops (default: 5).+ """++ agent_id = "deep_research"+ _default_max_turns = 5+ _default_temperature = 0.3+ _default_max_tokens = 4096++ def __init__(+ self,+ engine: InferenceEngine,+ model: str,+ *,+ tools: Optional[List[BaseTool]] = None,+ bus: Optional[EventBus] = None,+ max_turns: Optional[int] = None,+ temperature: Optional[float] = None,+ max_tokens: Optional[int] = None,+ interactive: bool = False,+ confirm_callback: Any = None,+ ) -> None:+ super().__init__(+ engine,+ model,+ tools=tools,+ bus=bus,+ max_turns=max_turns or self._default_max_turns,+ temperature=temperature or self._default_temperature,+ max_tokens=max_tokens or self._default_max_tokens,+ interactive=interactive,+ confirm_callback=confirm_callback,+ )++ def run(+ self,+ input: str,+ context: Optional[AgentContext] = None,+ **kwargs: Any,+ ) -> AgentResult:+ """Execute a multi-hop research query.++ The agent loops through: search → think → search → ... → synthesize,+ up to max_turns iterations.+ """+ self._emit_turn_start(input)++ # Build system prompt with tool descriptions+ tool_desc = build_tool_descriptions(self._tools)+ system_prompt = _SYSTEM_PROMPT.format(+ tool_descriptions=tool_desc+ )++ # Assemble initial messages+ messages = self._build_messages(+ input, context, system_prompt=system_prompt+ )++ all_tool_results: List[ToolResult] = []+ turns = 0+ total_usage: Dict[str, int] = {+ "prompt_tokens": 0,+ "completion_tokens": 0,+ "total_tokens": 0,+ }++ for _turn in range(self._max_turns):+ turns += 1++ # Generate response+ result = self._generate(messages)+ usage = result.get("usage", {})+ for k in total_usage:+ total_usage[k] += usage.get(k, 0)++ content = result.get("content", "")+ tool_calls_raw = result.get("tool_calls", [])++ # No tool calls → final answer+ if not tool_calls_raw:+ final_content = self._strip_think_tags(content)+ self._emit_turn_end(turns=turns)+ return AgentResult(+ content=final_content,+ tool_results=all_tool_results,+ turns=turns,+ metadata={+ **total_usage,+ "sources": self._extract_sources(+ all_tool_results+ ),+ },+ )++ # Append assistant message+ messages.append(+ Message(role=Role.ASSISTANT, content=content)+ )++ # Execute tool calls+ for tc_raw in tool_calls_raw:+ fn = tc_raw.get("function", {})+ tool_call = ToolCall(+ id=tc_raw.get("id", f"dr_{turns}"),+ name=fn.get("name", ""),+ arguments=fn.get("arguments", "{}"),+ )++ tool_result = self._executor.execute(tool_call)+ all_tool_results.append(tool_result)++ # Add tool result as message+ messages.append(+ Message(+ role=Role.TOOL,+ content=tool_result.content,+ tool_call_id=tool_call.id,+ name=tool_call.name,+ )+ )++ # Max turns exceeded — return what we have+ self._emit_turn_end(turns=turns)+ return self._max_turns_result(+ all_tool_results,+ turns,+ metadata={+ **total_usage,+ "sources": self._extract_sources(all_tool_results),+ },+ )++ @staticmethod+ def _extract_sources(+ tool_results: List[ToolResult],+ ) -> List[str]:+ """Extract source references from tool results for metadata."""+ sources: List[str] = []+ for tr in tool_results:+ if tr.tool_name == "knowledge_search" and tr.success:+ n = tr.metadata.get("num_results", 0)+ if n > 0:+ sources.append(+ f"knowledge_search: {n} results"+ )+ return sources+```++- [ ] **Step 4: Run tests**++Run: `cd /lambda/nfs/lambda-stanford/jonsf/scratch_v2/OpenJarvis && uv run pytest tests/agents/test_deep_research.py -v`++Expected: All 6 tests PASS.++- [ ] **Step 5: Commit**++```bash+git add src/openjarvis/agents/deep_research.py tests/agents/test_deep_research.py+git commit -m "feat: add DeepResearchAgent with multi-hop retrieval and cited reports"+```++---++### Task 4: End-to-End Integration Test++**Files:**+- Create: `tests/agents/test_deep_research_integration.py`++- [ ] **Step 1: Write integration test**++Create `tests/agents/test_deep_research_integration.py`:++```python+"""Integration test — full pipeline from connector to Deep Research agent."""++from __future__ import annotations++import json+from pathlib import Path+from unittest.mock import MagicMock++import pytest++from openjarvis.agents.deep_research import DeepResearchAgent+from openjarvis.connectors._stubs import Document+from openjarvis.connectors.pipeline import IngestionPipeline+from openjarvis.connectors.retriever import TwoStageRetriever+from openjarvis.connectors.store import KnowledgeStore+from openjarvis.tools.knowledge_search import KnowledgeSearchTool+++def populated_store(tmp_path: Path) -> KnowledgeStore:+ """KnowledgeStore with realistic multi-source data."""+ store = KnowledgeStore(+ db_path=str(tmp_path / "integration.db")+ )+ pipeline = IngestionPipeline(store=store, max_tokens=256)++ docs = [+ Document(+ doc_id="slack:c001:t1",+ source="slack",+ doc_type="message",+ content=(+ "Sarah: We should migrate to Kubernetes. "+ "The current Docker Swarm setup can't handle our scale."+ ),+ title="#infrastructure",+ author="sarah",+ ),+ Document(+ doc_id="gmail:msg1",+ source="gmail",+ doc_type="email",+ content=(+ "Hi team, I've completed the cost analysis for "+ "the K8s migration. Estimated 40% increase in "+ "cloud spend during the transition period, but "+ "20% reduction long-term."+ ),+ title="K8s Cost Analysis",+ author="mike",+ ),+ Document(+ doc_id="gdrive:doc1",+ source="gdrive",+ doc_type="document",+ content=(+ "# Kubernetes Migration Proposal\n\n"+ "## Timeline\n"+ "Six-week migration window starting April 1st.\n\n"+ "## Team\n"+ "Sarah (lead), Mike (infra), Bob (testing)\n\n"+ "## Risks\n"+ "Downtime during cutover, learning curve for team."+ ),+ title="K8s Migration Proposal v2",+ author="sarah",+ ),+ Document(+ doc_id="gcalendar:evt1",+ source="gcalendar",+ doc_type="event",+ content=(+ "Infrastructure Sync\n"+ "When: March 5, 2024 10:00 AM\n"+ "Attendees: Sarah, Mike, Bob\n"+ "Agenda: Review K8s migration proposal"+ ),+ title="Infrastructure Sync",+ author="sarah",+ ),+ Document(+ doc_id="granola:not1",+ source="granola",+ doc_type="document",+ content=(+ "## Summary\n"+ "Discussed K8s migration timeline. Sarah presented "+ "the proposal. Mike raised cost concerns. "+ "Decision: proceed with 6-week plan.\n\n"+ "## Transcript\n"+ "**sarah:** Let me walk through the timeline.\n"+ "**mike:** What about the cost increase?\n"+ "**sarah:** Short-term 40% increase, long-term savings."+ ),+ title="Infrastructure Sync Notes",+ author="sarah",+ ),+ ]++ pipeline.ingest(docs)+ return store+++def mock_engine():+ engine = MagicMock()+ engine.engine_id = "mock"+ engine.health.return_value = True+ return engine+++def test_full_research_pipeline(+ populated_store: KnowledgeStore,+ mock_engine: MagicMock,+) -> None:+ """Full pipeline: multi-source data → search → agent → cited report."""+ # Agent searches, then synthesizes+ search_call = {+ "content": "",+ "usage": {+ "prompt_tokens": 100,+ "completion_tokens": 50,+ "total_tokens": 150,+ },+ "model": "test",+ "finish_reason": "tool_calls",+ "tool_calls": [+ {+ "id": "call_1",+ "type": "function",+ "function": {+ "name": "knowledge_search",+ "arguments": json.dumps(+ {"query": "Kubernetes migration"}+ ),+ },+ }+ ],+ }++ final_answer = {+ "content": (+ "## Kubernetes Migration — Decision Context\n\n"+ "The migration to Kubernetes was proposed by Sarah "+ "in the #infrastructure Slack channel "+ "[slack] #infrastructure — sarah.\n\n"+ "Mike completed a cost analysis showing a 40% "+ "short-term increase [gmail] K8s Cost Analysis — mike.\n\n"+ "The proposal document outlines a six-week timeline "+ "[gdrive] K8s Migration Proposal v2 — sarah.\n\n"+ "The team discussed and approved the plan during "+ "the March 5th Infrastructure Sync "+ "[granola] Infrastructure Sync Notes — sarah.\n\n"+ "**Sources:**\n"+ "1. [slack] #infrastructure — sarah\n"+ "2. [gmail] K8s Cost Analysis — mike\n"+ "3. [gdrive] K8s Migration Proposal v2 — sarah\n"+ "4. [gcalendar] Infrastructure Sync — sarah\n"+ "5. [granola] Infrastructure Sync Notes — sarah"+ ),+ "usage": {+ "prompt_tokens": 500,+ "completion_tokens": 200,+ "total_tokens": 700,+ },+ "model": "test",+ "finish_reason": "stop",+ }++ mock_engine.generate.side_effect = [search_call, final_answer]++ retriever = TwoStageRetriever(store=populated_store)+ ks_tool = KnowledgeSearchTool(+ store=populated_store, retriever=retriever+ )++ agent = DeepResearchAgent(+ engine=mock_engine,+ model="test-model",+ tools=[ks_tool],+ max_turns=5,+ )++ result = agent.run(+ "What was the context around the Kubernetes migration decision?"+ )++ # Verify agent produced a result+ assert result.content+ assert "Kubernetes" in result.content+ assert result.turns >= 1++ # Verify tool was called+ assert len(result.tool_results) >= 1+ assert result.tool_results[0].tool_name == "knowledge_search"+ assert result.tool_results[0].success++ # Verify knowledge_search found cross-platform results+ search_result = result.tool_results[0]+ assert search_result.metadata.get("num_results", 0) > 0+++def test_search_finds_cross_platform_data(+ populated_store: KnowledgeStore,+) -> None:+ """Verify the store has data from multiple sources."""+ retriever = TwoStageRetriever(store=populated_store)+ tool = KnowledgeSearchTool(+ store=populated_store, retriever=retriever+ )++ result = tool.execute(query="Kubernetes migration")+ assert result.success+ assert result.metadata["num_results"] > 0++ # Should find results from multiple sources+ content = result.content+ sources_found = set()+ for src in ["slack", "gmail", "gdrive", "granola"]:+ if f"[{src}]" in content:+ sources_found.add(src)+ assert len(sources_found) >= 2, (+ f"Expected cross-platform results, found: {sources_found}"+ )+```++- [ ] **Step 2: Run integration test**++Run: `cd /lambda/nfs/lambda-stanford/jonsf/scratch_v2/OpenJarvis && uv run pytest tests/agents/test_deep_research_integration.py -v`++Expected: All 2 tests PASS.++- [ ] **Step 3: Run full test suite**++Run: `cd /lambda/nfs/lambda-stanford/jonsf/scratch_v2/OpenJarvis && uv run pytest tests/connectors/ tests/agents/test_deep_research.py tests/agents/test_deep_research_integration.py tests/tools/test_knowledge_search.py -v`++Expected: All tests PASS.++- [ ] **Step 4: Run linter**++Run: `cd /lambda/nfs/lambda-stanford/jonsf/scratch_v2/OpenJarvis && uv run ruff check src/openjarvis/connectors/retriever.py src/openjarvis/agents/deep_research.py src/openjarvis/tools/knowledge_search.py tests/connectors/test_retriever.py tests/agents/`++Expected: No errors.++- [ ] **Step 5: Commit**++```bash+git add tests/agents/test_deep_research_integration.py+git commit -m "feat: add end-to-end integration test for Deep Research pipeline"+```++---++## Post-Plan Notes++**What this plan produces:**+- `TwoStageRetriever` — composable BM25 recall + pluggable ColBERT reranking+- `ColBERTReranker` — lazy-loading reranker with MaxSim scoring (optional dependency)+- `KnowledgeSearchTool` upgraded to use two-stage retrieval+- `DeepResearchAgent` — multi-hop research agent with cross-platform citations+- Full integration test proving the pipeline end-to-end++**ColBERT is optional:** The system works with BM25-only when ColBERT dependencies (torch, colbert-ai) aren't installed. ColBERT adds semantic reranking quality when available.++**What comes next:**+- **Phase 4:** ChannelAgent + iMessage/WhatsApp/Slack plugins for chatting with the agent+- **Phase 5:** Incremental sync, attachment store, polish+- **Phase 2B:** Desktop wizard UI