Skip to content

Personal AI, On Personal Devices

Python

docs: add Phase 1 implementation plan for Deep Research connector foundation

krypticmousee9e4bb874b53
diff --git a/docs/superpowers/plans/2026-03-25-deep-research-phase1.md b/docs/superpowers/plans/2026-03-25-deep-research-phase1.md
new file mode 100644
index 00000000..5c1c84e4
--- /dev/null
+++ b/docs/superpowers/plans/2026-03-25-deep-research-phase1.md
@@ -0,0 +1,2895 @@
+# Deep Research Phase 1: Connector Foundation Implementation Plan
+
+> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking.
+
+**Goal:** Build the connector framework, ingestion pipeline, knowledge store, and two connectors (Gmail + Obsidian) that form the foundation for the full Deep Research feature.
+
+**Architecture:** A new `BaseConnector` ABC with `ConnectorRegistry` provides the plugin interface. Connectors yield normalized `Document` objects that flow through a pipeline (normalize → dedup → chunk → index) into an extended SQLite store with FTS5 and source-aware columns. A `knowledge_search` tool exposes filtered BM25 retrieval to agents. A `SyncEngine` orchestrates connectors with checkpoint/resume. CLI `jarvis connect` provides text-based setup.
+
+**Tech Stack:** Python 3.10+, SQLite/FTS5 (via existing Rust backend), httpx (for Gmail API), Click (for CLI), pytest + respx (for tests)
+
+**Spec:** `docs/superpowers/specs/2026-03-25-deep-research-setup-design.md` — Sections 5, 6, 7, 10, Phase 1
+
+---
+
+## File Structure
+
+```
+src/openjarvis/
+├── connectors/
+│ ├── __init__.py # Auto-imports, ConnectorRegistry re-export
+│ ├── _stubs.py # BaseConnector ABC, Document, Attachment, SyncStatus
+│ ├── store.py # KnowledgeStore (extended SQLite with source-aware schema)
+│ ├── chunker.py # Type-aware semantic chunker
+│ ├── pipeline.py # Ingestion pipeline: normalize → dedup → chunk → index
+│ ├── sync_engine.py # SyncEngine: orchestrate connectors with checkpoint/resume
+│ ├── oauth.py # Shared OAuth helper (localhost callback server)
+│ ├── gmail.py # Gmail connector
+│ └── obsidian.py # Obsidian/Markdown connector
+├── tools/
+│ └── knowledge_search.py # knowledge_search tool for agents
+├── core/
+│ └── registry.py # (modify) Add ConnectorRegistry
+├── cli/
+│ ├── __init__.py # (modify) Add connect command
+│ └── connect_cmd.py # jarvis connect CLI command
+tests/
+├── conftest.py # (modify) Clear ConnectorRegistry between tests
+├── connectors/
+│ ├── __init__.py
+│ ├── test_stubs.py # BaseConnector, Document, SyncStatus tests
+│ ├── test_store.py # KnowledgeStore tests
+│ ├── test_chunker.py # Semantic chunker tests
+│ ├── test_pipeline.py # Ingestion pipeline tests
+│ ├── test_sync_engine.py # SyncEngine tests
+│ ├── test_gmail.py # Gmail connector tests (mocked API)
+│ └── test_obsidian.py # Obsidian connector tests (temp dirs)
+├── tools/
+│ └── test_knowledge_search.py # knowledge_search tool tests
+├── cli/
+│ └── test_connect.py # CLI connect command tests
+```
+
+---
+
+### Task 1: ConnectorRegistry + Base Types
+
+**Files:**
+- Modify: `src/openjarvis/core/registry.py`
+- Create: `src/openjarvis/connectors/_stubs.py`
+- Create: `src/openjarvis/connectors/__init__.py`
+- Modify: `tests/conftest.py`
+- Create: `tests/connectors/__init__.py`
+- Create: `tests/connectors/test_stubs.py`
+
+- [ ] **Step 1: Add ConnectorRegistry to registry.py**
+
+Open `src/openjarvis/core/registry.py`. At the bottom, after the existing registry classes (after `CompressionRegistry`), add:
+
+```python
+class ConnectorRegistry(RegistryBase[Any]):
+ """Registry for data source connectors (Gmail, Slack, etc.)."""
+```
+
+Also add `ConnectorRegistry` to the imports in the file's `__all__` or wherever the other registries are exported. Find the existing pattern — the other registries are bare class definitions at module level.
+
+- [ ] **Step 2: Clear ConnectorRegistry in test conftest**
+
+Open `tests/conftest.py`. Add `ConnectorRegistry` to the import:
+
+```python
+from openjarvis.core.registry import (
+ AgentRegistry,
+ BenchmarkRegistry,
+ ChannelRegistry,
+ CompressionRegistry,
+ ConnectorRegistry, # ADD THIS
+ EngineRegistry,
+ MemoryRegistry,
+ ModelRegistry,
+ RouterPolicyRegistry,
+ SpeechRegistry,
+ ToolRegistry,
+)
+```
+
+In the `_clean_registries` fixture, add `ConnectorRegistry.clear()` after `CompressionRegistry.clear()`.
+
+- [ ] **Step 3: Create connectors _stubs.py with base types**
+
+Create `src/openjarvis/connectors/_stubs.py`:
+
+```python
+"""Base types for data source connectors."""
+
+from __future__ import annotations
+
+from abc import ABC, abstractmethod
+from dataclasses import dataclass, field
+from datetime import datetime
+from typing import Any, Dict, Iterator, List, Optional
+
+from openjarvis.tools._stubs import ToolSpec
+
+
+@dataclass(slots=True)
+class Attachment:
+ """A file attached to a document (email attachment, shared file, etc.)."""
+
+ filename: str
+ mime_type: str
+ size_bytes: int
+ sha256: str = ""
+ content: bytes = field(default=b"", repr=False)
+
+
+@dataclass(slots=True)
+class Document:
+ """Universal schema for data from any connector.
+
+ All connectors normalize their output to this format before ingestion.
+ """
+
+ doc_id: str
+ source: str
+ doc_type: str
+ content: str
+ title: str = ""
+ author: str = ""
+ participants: List[str] = field(default_factory=list)
+ timestamp: datetime = field(default_factory=datetime.now)
+ thread_id: Optional[str] = None
+ url: Optional[str] = None
+ attachments: List[Attachment] = field(default_factory=list)
+ metadata: Dict[str, Any] = field(default_factory=dict)
+
+
+@dataclass(slots=True)
+class SyncStatus:
+ """Progress of a connector's sync operation."""
+
+ state: str = "idle"
+ items_synced: int = 0
+ items_total: int = 0
+ last_sync: Optional[datetime] = None
+ cursor: Optional[str] = None
+ error: Optional[str] = None
+
+
+class BaseConnector(ABC):
+ """Abstract base for data source connectors.
+
+ Each connector knows how to authenticate with a service, bulk-sync
+ its data as ``Document`` objects, and optionally expose MCP tools
+ for real-time agent queries.
+ """
+
+ connector_id: str
+ display_name: str
+ auth_type: str # "oauth" | "local" | "bridge" | "filesystem"
+
+ @abstractmethod
+ def is_connected(self) -> bool:
+ """Return True if the connector has valid credentials."""
+
+ @abstractmethod
+ def disconnect(self) -> None:
+ """Revoke credentials and clean up."""
+
+ @abstractmethod
+ def sync(
+ self, *, since: Optional[datetime] = None, cursor: Optional[str] = None
+ ) -> Iterator[Document]:
+ """Yield documents from the data source.
+
+ If *since* is given, only return items created/modified after that time.
+ If *cursor* is given, resume from a previous checkpoint.
+ """
+
+ @abstractmethod
+ def sync_status(self) -> SyncStatus:
+ """Return current sync progress."""
+
+ def auth_url(self) -> str:
+ """Generate an OAuth consent URL. Only relevant for auth_type='oauth'."""
+ raise NotImplementedError(f"{self.connector_id} does not use OAuth")
+
+ def handle_callback(self, code: str) -> None:
+ """Handle the OAuth callback. Only relevant for auth_type='oauth'."""
+ raise NotImplementedError(f"{self.connector_id} does not use OAuth")
+
+ def mcp_tools(self) -> List[ToolSpec]:
+ """Return MCP tool specs for real-time agent queries. Optional."""
+ return []
+```
+
+- [ ] **Step 4: Create connectors __init__.py**
+
+Create `src/openjarvis/connectors/__init__.py`:
+
+```python
+"""Data source connectors for Deep Research."""
+
+from openjarvis.connectors._stubs import (
+ Attachment,
+ BaseConnector,
+ Document,
+ SyncStatus,
+)
+
+__all__ = ["Attachment", "BaseConnector", "Document", "SyncStatus"]
+```
+
+- [ ] **Step 5: Write tests for base types**
+
+Create `tests/connectors/__init__.py` (empty file).
+
+Create `tests/connectors/test_stubs.py`:
+
+```python
+"""Tests for connector base types and registry."""
+
+from __future__ import annotations
+
+from datetime import datetime
+from typing import Iterator, List, Optional
+
+from openjarvis.connectors._stubs import (
+ Attachment,
+ BaseConnector,
+ Document,
+ SyncStatus,
+)
+from openjarvis.core.registry import ConnectorRegistry
+from openjarvis.tools._stubs import ToolSpec
+
+
+class FakeConnector(BaseConnector):
+ connector_id = "fake"
+ display_name = "Fake"
+ auth_type = "filesystem"
+
+ def __init__(self) -> None:
+ self._connected = True
+
+ def is_connected(self) -> bool:
+ return self._connected
+
+ def disconnect(self) -> None:
+ self._connected = False
+
+ def sync(
+ self, *, since: Optional[datetime] = None, cursor: Optional[str] = None
+ ) -> Iterator[Document]:
+ yield Document(
+ doc_id="fake:1",
+ source="fake",
+ doc_type="note",
+ content="Hello world",
+ title="Test",
+ )
+
+ def sync_status(self) -> SyncStatus:
+ return SyncStatus(state="idle", items_synced=1, items_total=1)
+
+
+def test_document_creation() -> None:
+ doc = Document(
+ doc_id="gmail:abc123",
+ source="gmail",
+ doc_type="email",
+ content="Meeting tomorrow at 3pm",
+ title="Re: Project sync",
+ author="[email protected]",
+ participants=["[email protected]", "[email protected]"],
+ )
+ assert doc.source == "gmail"
+ assert doc.doc_type == "email"
+ assert doc.thread_id is None
+ assert doc.attachments == []
+
+
+def test_attachment_creation() -> None:
+ att = Attachment(
+ filename="report.pdf",
+ mime_type="application/pdf",
+ size_bytes=1024,
+ sha256="abcdef1234567890",
+ )
+ assert att.filename == "report.pdf"
+ assert att.content == b""
+
+
+def test_sync_status_defaults() -> None:
+ status = SyncStatus()
+ assert status.state == "idle"
+ assert status.items_synced == 0
+ assert status.cursor is None
+ assert status.error is None
+
+
+def test_base_connector_lifecycle() -> None:
+ conn = FakeConnector()
+ assert conn.is_connected()
+ docs = list(conn.sync())
+ assert len(docs) == 1
+ assert docs[0].doc_id == "fake:1"
+ assert conn.sync_status().state == "idle"
+ conn.disconnect()
+ assert not conn.is_connected()
+
+
+def test_connector_registry() -> None:
+ ConnectorRegistry.register_value("fake", FakeConnector)
+ assert ConnectorRegistry.contains("fake")
+ cls = ConnectorRegistry.get("fake")
+ instance = cls()
+ assert instance.connector_id == "fake"
+
+
+def test_mcp_tools_default_empty() -> None:
+ conn = FakeConnector()
+ assert conn.mcp_tools() == []
+```
+
+- [ ] **Step 6: Run tests**
+
+Run: `cd /lambda/nfs/lambda-stanford/jonsf/scratch_v2/OpenJarvis && uv run pytest tests/connectors/test_stubs.py -v`
+
+Expected: All 6 tests PASS.
+
+- [ ] **Step 7: Commit**
+
+```bash
+git add src/openjarvis/core/registry.py src/openjarvis/connectors/ tests/conftest.py tests/connectors/
+git commit -m "feat: add ConnectorRegistry and base connector types (Document, SyncStatus, BaseConnector)"
+```
+
+---
+
+### Task 2: KnowledgeStore (Extended SQLite Schema)
+
+**Files:**
+- Create: `src/openjarvis/connectors/store.py`
+- Create: `tests/connectors/test_store.py`
+
+- [ ] **Step 1: Write failing tests for KnowledgeStore**
+
+Create `tests/connectors/test_store.py`:
+
+```python
+"""Tests for KnowledgeStore — extended SQLite with source-aware columns."""
+
+from __future__ import annotations
+
+import sqlite3
+import tempfile
+from pathlib import Path
+
+import pytest
+
+from openjarvis.connectors.store import KnowledgeStore
+
+
+def store(tmp_path: Path) -> KnowledgeStore:
+ return KnowledgeStore(db_path=str(tmp_path / "test_knowledge.db"))
+
+
+def test_store_and_retrieve_basic(store: KnowledgeStore) -> None:
+ doc_id = store.store(
+ content="Meeting about Kubernetes migration",
+ source="gmail",
+ doc_type="email",
+ title="Re: K8s migration",
+ author="[email protected]",
+ timestamp="2024-03-15T10:00:00",
+ )
+ assert doc_id
+ results = store.retrieve("Kubernetes migration", top_k=5)
+ assert len(results) >= 1
+ assert "Kubernetes" in results[0].content
+
+
+def test_retrieve_filter_by_source(store: KnowledgeStore) -> None:
+ store.store(content="K8s discussion in Slack", source="slack", doc_type="message")
+ store.store(content="K8s email thread", source="gmail", doc_type="email")
+ results = store.retrieve("K8s", top_k=10, source="gmail")
+ assert all(r.metadata.get("source") == "gmail" for r in results)
+
+
+def test_retrieve_filter_by_doc_type(store: KnowledgeStore) -> None:
+ store.store(content="API design proposal", source="gdrive", doc_type="document")
+ store.store(content="API design discussion", source="slack", doc_type="message")
+ results = store.retrieve("API design", top_k=10, doc_type="document")
+ assert all(r.metadata.get("doc_type") == "document" for r in results)
+
+
+def test_retrieve_filter_by_author(store: KnowledgeStore) -> None:
+ store.store(content="Budget report", source="gmail", doc_type="email", author="alice")
+ store.store(content="Budget concerns", source="gmail", doc_type="email", author="bob")
+ results = store.retrieve("budget", top_k=10, author="alice")
+ assert all(r.metadata.get("author") == "alice" for r in results)
+
+
+def test_retrieve_filter_by_timestamp(store: KnowledgeStore) -> None:
+ store.store(
+ content="Old meeting notes",
+ source="slack",
+ doc_type="message",
+ timestamp="2024-01-01T00:00:00",
+ )
+ store.store(
+ content="Recent meeting notes",
+ source="slack",
+ doc_type="message",
+ timestamp="2024-06-01T00:00:00",
+ )
+ results = store.retrieve(
+ "meeting notes", top_k=10, since="2024-03-01T00:00:00"
+ )
+ assert len(results) == 1
+ assert "Recent" in results[0].content
+
+
+def test_delete_by_doc_id(store: KnowledgeStore) -> None:
+ chunk_id = store.store(content="Delete me", source="gmail", doc_type="email")
+ assert store.delete(chunk_id)
+ results = store.retrieve("Delete me", top_k=5)
+ assert len(results) == 0
+
+
+def test_clear(store: KnowledgeStore) -> None:
+ store.store(content="Item 1", source="gmail", doc_type="email")
+ store.store(content="Item 2", source="slack", doc_type="message")
+ store.clear()
+ results = store.retrieve("Item", top_k=10)
+ assert len(results) == 0
+
+
+def test_store_with_metadata(store: KnowledgeStore) -> None:
+ store.store(
+ content="Labeled email",
+ source="gmail",
+ doc_type="email",
+ metadata={"labels": ["important", "inbox"], "thread_id": "t123"},
+ )
+ results = store.retrieve("Labeled email", top_k=5)
+ assert results[0].metadata.get("labels") == ["important", "inbox"]
+
+
+def test_store_preserves_url(store: KnowledgeStore) -> None:
+ store.store(
+ content="Linked document",
+ source="gdrive",
+ doc_type="document",
+ url="https://drive.google.com/d/abc123",
+ )
+ results = store.retrieve("Linked document", top_k=5)
+ assert results[0].metadata.get("url") == "https://drive.google.com/d/abc123"
+```
+
+- [ ] **Step 2: Run tests to verify they fail**
+
+Run: `cd /lambda/nfs/lambda-stanford/jonsf/scratch_v2/OpenJarvis && uv run pytest tests/connectors/test_store.py -v`
+
+Expected: FAIL — `ModuleNotFoundError: No module named 'openjarvis.connectors.store'`
+
+- [ ] **Step 3: Implement KnowledgeStore**
+
+Create `src/openjarvis/connectors/store.py`:
+
+```python
+"""KnowledgeStore — extended SQLite backend with source-aware columns for Deep Research."""
+
+from __future__ import annotations
+
+import json
+import sqlite3
+import uuid
+from datetime import datetime
+from pathlib import Path
+from typing import Any, Dict, List, Optional
+
+from openjarvis.core.config import DEFAULT_CONFIG_DIR
+from openjarvis.core.events import EventType, get_event_bus
+from openjarvis.tools.storage._stubs import MemoryBackend, RetrievalResult
+
+_SCHEMA = """\
+CREATE TABLE IF NOT EXISTS documents (
+ id TEXT PRIMARY KEY,
+ doc_id TEXT NOT NULL DEFAULT '',
+ content TEXT NOT NULL,
+ source TEXT NOT NULL DEFAULT '',
+ doc_type TEXT NOT NULL DEFAULT '',
+ title TEXT DEFAULT '',
+ author TEXT DEFAULT '',
+ participants TEXT DEFAULT '[]',
+ timestamp TEXT NOT NULL DEFAULT '',
+ thread_id TEXT DEFAULT '',
+ url TEXT DEFAULT '',
+ metadata TEXT DEFAULT '{}',
+ chunk_index INTEGER DEFAULT 0,
+ created_at TEXT NOT NULL
+);
+
+CREATE VIRTUAL TABLE IF NOT EXISTS documents_fts USING fts5(
+ content, title, author,
+ content=documents,
+ content_rowid=rowid,
+ tokenize='porter unicode61'
+);
+
+CREATE TRIGGER IF NOT EXISTS documents_ai AFTER INSERT ON documents BEGIN
+ INSERT INTO documents_fts(rowid, content, title, author)
+ VALUES (new.rowid, new.content, new.title, new.author);
+END;
+
+CREATE TRIGGER IF NOT EXISTS documents_ad AFTER DELETE ON documents BEGIN
+ INSERT INTO documents_fts(documents_fts, rowid, content, title, author)
+ VALUES ('delete', old.rowid, old.content, old.title, old.author);
+END;
+
+CREATE INDEX IF NOT EXISTS idx_source ON documents(source);
+CREATE INDEX IF NOT EXISTS idx_doc_type ON documents(doc_type);
+CREATE INDEX IF NOT EXISTS idx_author ON documents(author);
+CREATE INDEX IF NOT EXISTS idx_timestamp ON documents(timestamp);
+CREATE INDEX IF NOT EXISTS idx_thread ON documents(thread_id);
+CREATE INDEX IF NOT EXISTS idx_doc_id ON documents(doc_id);
+"""
+
+
+class KnowledgeStore(MemoryBackend):
+ """SQLite + FTS5 backend with source-aware columns for Deep Research.
+
+ Extends the base MemoryBackend with filtered retrieval by source,
+ doc_type, author, and timestamp. Uses pure Python sqlite3 so we
+ don't depend on the Rust extension for the new schema.
+ """
+
+ backend_id: str = "knowledge"
+
+ def __init__(self, db_path: str = "") -> None:
+ if not db_path:
+ db_path = str(DEFAULT_CONFIG_DIR / "knowledge.db")
+ self._db_path = db_path
+ Path(db_path).parent.mkdir(parents=True, exist_ok=True)
+ self._conn = sqlite3.connect(db_path)
+ self._conn.execute("PRAGMA journal_mode=WAL")
+ self._conn.executescript(_SCHEMA)
+
+ def store(
+ self,
+ content: str,
+ *,
+ source: str = "",
+ doc_type: str = "",
+ doc_id: str = "",
+ title: str = "",
+ author: str = "",
+ participants: Optional[List[str]] = None,
+ timestamp: str = "",
+ thread_id: str = "",
+ url: str = "",
+ metadata: Optional[Dict[str, Any]] = None,
+ chunk_index: int = 0,
+ ) -> str:
+ chunk_id = uuid.uuid4().hex[:16]
+ now = datetime.now().isoformat()
+ if not timestamp:
+ timestamp = now
+ meta = metadata or {}
+ # Store source-level fields in metadata too so retrieve() can return them
+ meta.setdefault("source", source)
+ meta.setdefault("doc_type", doc_type)
+ meta.setdefault("author", author)
+ meta.setdefault("title", title)
+ if url:
+ meta.setdefault("url", url)
+ if thread_id:
+ meta.setdefault("thread_id", thread_id)
+
+ self._conn.execute(
+ """INSERT INTO documents
+ (id, doc_id, content, source, doc_type, title, author,
+ participants, timestamp, thread_id, url, metadata,
+ chunk_index, created_at)
+ VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)""",
+ (
+ chunk_id,
+ doc_id,
+ content,
+ source,
+ doc_type,
+ title,
+ author,
+ json.dumps(participants or []),
+ timestamp,
+ thread_id,
+ url,
+ json.dumps(meta),
+ chunk_index,
+ now,
+ ),
+ )
+ self._conn.commit()
+
+ bus = get_event_bus()
+ bus.publish(
+ EventType.MEMORY_STORE,
+ {"backend": "knowledge", "doc_id": chunk_id, "source": source},
+ )
+ return chunk_id
+
+ def retrieve(
+ self,
+ query: str,
+ *,
+ top_k: int = 5,
+ source: str = "",
+ doc_type: str = "",
+ author: str = "",
+ since: str = "",
+ until: str = "",
+ **kwargs: Any,
+ ) -> List[RetrievalResult]:
+ # Build FTS5 query
+ # Escape double quotes in user query for FTS5
+ safe_query = query.replace('"', '""')
+ fts_clause = f'documents_fts MATCH \\"{safe_query}\\"'
+
+ # Build WHERE filters on the main table
+ filters: list[str] = []
+ params: list[str] = []
+ if source:
+ filters.append("d.source = ?")
+ params.append(source)
+ if doc_type:
+ filters.append("d.doc_type = ?")
+ params.append(doc_type)
+ if author:
+ filters.append("d.author = ?")
+ params.append(author)
+ if since:
+ filters.append("d.timestamp >= ?")
+ params.append(since)
+ if until:
+ filters.append("d.timestamp <= ?")
+ params.append(until)
+
+ where = " AND ".join(filters) if filters else "1=1"
+
+ sql = f"""
+ SELECT d.content, d.metadata, bm25(documents_fts) AS score
+ FROM documents_fts
+ JOIN documents d ON d.rowid = documents_fts.rowid
+ WHERE documents_fts MATCH ? AND {where}
+ ORDER BY score
+ LIMIT ?
+ """
+ try:
+ rows = self._conn.execute(sql, [safe_query, *params, top_k]).fetchall()
+ except sqlite3.OperationalError:
+ return []
+
+ results = []
+ for content, meta_json, score in rows:
+ meta = json.loads(meta_json) if meta_json else {}
+ results.append(
+ RetrievalResult(
+ content=content,
+ score=abs(score), # bm25() returns negative scores
+ source=meta.get("source", ""),
+ metadata=meta,
+ )
+ )
+
+ bus = get_event_bus()
+ bus.publish(
+ EventType.MEMORY_RETRIEVE,
+ {"backend": "knowledge", "query": query, "num_results": len(results)},
+ )
+ return results
+
+ def delete(self, doc_id: str) -> bool:
+ cursor = self._conn.execute("DELETE FROM documents WHERE id = ?", (doc_id,))
+ self._conn.commit()
+ return cursor.rowcount > 0
+
+ def clear(self) -> None:
+ self._conn.execute("DELETE FROM documents")
+ self._conn.execute("INSERT INTO documents_fts(documents_fts) VALUES('rebuild')")
+ self._conn.commit()
+
+ def close(self) -> None:
+ self._conn.close()
+```
+
+- [ ] **Step 4: Run tests**
+
+Run: `cd /lambda/nfs/lambda-stanford/jonsf/scratch_v2/OpenJarvis && uv run pytest tests/connectors/test_store.py -v`
+
+Expected: All 10 tests PASS. If FTS5 syntax issues arise, adjust the MATCH query escaping — SQLite FTS5 can be picky about special characters.
+
+- [ ] **Step 5: Commit**
+
+```bash
+git add src/openjarvis/connectors/store.py tests/connectors/test_store.py
+git commit -m "feat: add KnowledgeStore with source-aware SQLite schema and filtered BM25 retrieval"
+```
+
+---
+
+### Task 3: Type-Aware Semantic Chunker
+
+**Files:**
+- Create: `src/openjarvis/connectors/chunker.py`
+- Create: `tests/connectors/test_chunker.py`
+
+- [ ] **Step 1: Write failing tests**
+
+Create `tests/connectors/test_chunker.py`:
+
+```python
+"""Tests for type-aware semantic chunker."""
+
+from __future__ import annotations
+
+from openjarvis.connectors.chunker import SemanticChunker, ChunkResult
+
+
+def test_short_message_not_split() -> None:
+ """Messages under chunk limit should stay as one chunk."""
+ chunker = SemanticChunker(max_tokens=512)
+ results = chunker.chunk("Hey, are we meeting tomorrow?", doc_type="message")
+ assert len(results) == 1
+ assert results[0].content == "Hey, are we meeting tomorrow?"
+
+
+def test_long_document_splits_on_sections() -> None:
+ """Documents should split on ## headings first."""
+ text = (
+ "## Introduction\n\n"
+ + "This is the intro paragraph. " * 50
+ + "\n\n## Methods\n\n"
+ + "This is the methods section. " * 50
+ )
+ chunker = SemanticChunker(max_tokens=100)
+ results = chunker.chunk(text, doc_type="document")
+ assert len(results) >= 2
+ assert results[0].metadata.get("section") == "Introduction"
+ assert results[1].metadata.get("section") == "Methods"
+
+
+def test_document_splits_on_paragraphs_within_section() -> None:
+ """Within a section, split on paragraph boundaries."""
+ text = "## Overview\n\n" + "\n\n".join(
+ [f"Paragraph {i}. " * 30 for i in range(5)]
+ )
+ chunker = SemanticChunker(max_tokens=80)
+ results = chunker.chunk(text, doc_type="document")
+ assert len(results) >= 3
+ for r in results:
+ assert r.metadata.get("section") == "Overview"
+
+
+def test_never_splits_mid_sentence() -> None:
+ """Chunks should end at sentence boundaries."""
+ text = "First sentence here. Second sentence here. Third sentence here. " * 30
+ chunker = SemanticChunker(max_tokens=20)
+ results = chunker.chunk(text, doc_type="document")
+ for r in results:
+ stripped = r.content.strip()
+ assert stripped.endswith(".") or stripped.endswith("?") or stripped.endswith("!"), (
+ f"Chunk does not end at sentence boundary: ...{stripped[-40:]}"
+ )
+
+
+def test_email_thread_splits_on_reply_boundaries() -> None:
+ """Email threads should split on reply markers."""
+ text = (
+ "Hi team, let's discuss.\n\n"
+ "On Mon, Jan 1, Alice wrote:\n> Previous message. " * 20
+ + "\n\nThat sounds good to me.\n\n"
+ "On Tue, Jan 2, Bob wrote:\n> Another reply. " * 20
+ )
+ chunker = SemanticChunker(max_tokens=50)
+ results = chunker.chunk(text, doc_type="email")
+ # Should split on "On ... wrote:" boundaries
+ assert len(results) >= 2
+
+
+def test_event_stays_as_single_chunk() -> None:
+ """Events should always be one chunk."""
+ text = "Team Standup\nAttendees: Alice, Bob, Carol\nLocation: Room 3\nAgenda: Sprint review"
+ chunker = SemanticChunker(max_tokens=512)
+ results = chunker.chunk(text, doc_type="event")
+ assert len(results) == 1
+
+
+def test_contact_stays_as_single_chunk() -> None:
+ """Contacts should always be one chunk."""
+ text = "Alice Smith\[email protected]\n+1-555-0100\nAcme Corp, VP Engineering"
+ chunker = SemanticChunker(max_tokens=512)
+ results = chunker.chunk(text, doc_type="contact")
+ assert len(results) == 1
+
+
+def test_metadata_inherited_to_chunks() -> None:
+ """Parent metadata should carry to all chunks."""
+ text = "## Part 1\n\n" + "Content. " * 100 + "\n\n## Part 2\n\n" + "More content. " * 100
+ chunker = SemanticChunker(max_tokens=60)
+ parent_meta = {"title": "My Doc", "author": "Alice", "source": "gdrive"}
+ results = chunker.chunk(text, doc_type="document", metadata=parent_meta)
+ for r in results:
+ assert r.metadata["title"] == "My Doc"
+ assert r.metadata["author"] == "Alice"
+
+
+def test_chunk_result_has_index() -> None:
+ """Each chunk should have a sequential index."""
+ text = "\n\n".join([f"Paragraph {i}. " * 30 for i in range(5)])
+ chunker = SemanticChunker(max_tokens=50)
+ results = chunker.chunk(text, doc_type="document")
+ for i, r in enumerate(results):
+ assert r.index == i
+```
+
+- [ ] **Step 2: Run tests to verify they fail**
+
+Run: `cd /lambda/nfs/lambda-stanford/jonsf/scratch_v2/OpenJarvis && uv run pytest tests/connectors/test_chunker.py -v`
+
+Expected: FAIL — `ModuleNotFoundError`
+
+- [ ] **Step 3: Implement SemanticChunker**
+
+Create `src/openjarvis/connectors/chunker.py`:
+
+```python
+"""Type-aware semantic chunker for Deep Research ingestion.
+
+Splits documents respecting structural boundaries:
+- Sections (## headings) → paragraphs → sentences.
+- Never splits mid-sentence.
+- Chunk size defaults to reranker max context (512 for ColBERT, 256 for MiniLM).
+"""
+
+from __future__ import annotations
+
+import re
+from dataclasses import dataclass, field
+from typing import Any, Dict, List, Optional
+
+_SENTENCE_RE = re.compile(r"(?<=[.!?])\s+(?=[A-Z\"])")
+_SECTION_RE = re.compile(r"(?m)^##\s+(.+)$")
+_EMAIL_REPLY_RE = re.compile(r"(?m)^On .+wrote:\s*$")
+
+
+def _count_tokens(text: str) -> int:
+ """Approximate token count via whitespace split."""
+ return len(text.split())
+
+
+def _split_sentences(text: str) -> List[str]:
+ """Split text on sentence boundaries."""
+ parts = _SENTENCE_RE.split(text)
+ return [p.strip() for p in parts if p.strip()]
+
+
+@dataclass(slots=True)
+class ChunkResult:
+ """A single chunk produced by the semantic chunker."""
+
+ content: str
+ index: int = 0
+ metadata: Dict[str, Any] = field(default_factory=dict)
+
+
+class SemanticChunker:
+ """Type-aware chunker that respects structural boundaries."""
+
+ def __init__(self, max_tokens: int = 512) -> None:
+ self.max_tokens = max_tokens
+
+ def chunk(
+ self,
+ text: str,
+ *,
+ doc_type: str = "document",
+ metadata: Optional[Dict[str, Any]] = None,
+ ) -> List[ChunkResult]:
+ parent_meta = metadata or {}
+
+ # Events and contacts: always single chunk
+ if doc_type in ("event", "contact"):
+ return [ChunkResult(content=text.strip(), index=0, metadata=dict(parent_meta))]
+
+ # Short text: keep as single chunk
+ if _count_tokens(text) <= self.max_tokens:
+ return [ChunkResult(content=text.strip(), index=0, metadata=dict(parent_meta))]
+
+ # Route to type-specific splitter
+ if doc_type == "email":
+ raw_chunks = self._split_email(text)
+ elif doc_type == "message":
+ raw_chunks = self._split_messages(text)
+ else:
+ raw_chunks = self._split_document(text)
+
+ # Assign indexes and inherit parent metadata
+ results = []
+ for i, (content, chunk_meta) in enumerate(raw_chunks):
+ merged = {**parent_meta, **chunk_meta}
+ results.append(ChunkResult(content=content.strip(), index=i, metadata=merged))
+ return results
+
+ def _split_document(self, text: str) -> List[tuple[str, dict]]:
+ """Split on section headings, then paragraphs, then sentences."""
+ sections = self._extract_sections(text)
+ chunks: list[tuple[str, dict]] = []
+ for section_title, section_text in sections:
+ meta = {"section": section_title} if section_title else {}
+ paragraphs = [p.strip() for p in section_text.split("\n\n") if p.strip()]
+ buffer = ""
+ for para in paragraphs:
+ candidate = (buffer + "\n\n" + para).strip() if buffer else para
+ if _count_tokens(candidate) <= self.max_tokens:
+ buffer = candidate
+ else:
+ if buffer:
+ chunks.append((buffer, dict(meta)))
+ if _count_tokens(para) <= self.max_tokens:
+ buffer = para
+ else:
+ # Paragraph too big — split on sentences
+ for sent_chunk in self._split_by_sentences(para):
+ chunks.append((sent_chunk, dict(meta)))
+ buffer = ""
+ if buffer:
+ chunks.append((buffer, dict(meta)))
+ return chunks
+
+ def _split_email(self, text: str) -> List[tuple[str, dict]]:
+ """Split email threads on reply boundaries."""
+ parts = _EMAIL_REPLY_RE.split(text)
+ chunks: list[tuple[str, dict]] = []
+ for part in parts:
+ part = part.strip()
+ if not part:
+ continue
+ if _count_tokens(part) <= self.max_tokens:
+ chunks.append((part, {}))
+ else:
+ chunks.extend(self._split_document(part))
+ return chunks
+
+ def _split_messages(self, text: str) -> List[tuple[str, dict]]:
+ """Split message threads — treat each double-newline block as a message."""
+ messages = [m.strip() for m in text.split("\n\n") if m.strip()]
+ chunks: list[tuple[str, dict]] = []
+ buffer = ""
+ for msg in messages:
+ candidate = (buffer + "\n\n" + msg).strip() if buffer else msg
+ if _count_tokens(candidate) <= self.max_tokens:
+ buffer = candidate
+ else:
+ if buffer:
+ chunks.append((buffer, {}))
+ if _count_tokens(msg) <= self.max_tokens:
+ buffer = msg
+ else:
+ for sent_chunk in self._split_by_sentences(msg):
+ chunks.append((sent_chunk, {}))
+ buffer = ""
+ if buffer:
+ chunks.append((buffer, {}))
+ return chunks
+
+ def _extract_sections(self, text: str) -> List[tuple[str, str]]:
+ """Split text into (section_title, section_content) pairs."""
+ matches = list(_SECTION_RE.finditer(text))
+ if not matches:
+ return [("", text)]
+ sections: list[tuple[str, str]] = []
+ # Content before first heading
+ before = text[: matches[0].start()].strip()
+ if before:
+ sections.append(("", before))
+ for i, match in enumerate(matches):
+ title = match.group(1).strip()
+ start = match.end()
+ end = matches[i + 1].start() if i + 1 < len(matches) else len(text)
+ body = text[start:end].strip()
+ if body:
+ sections.append((title, body))
+ return sections
+
+ def _split_by_sentences(self, text: str) -> List[str]:
+ """Split text into sentence-boundary chunks that fit within max_tokens."""
+ sentences = _split_sentences(text)
+ if not sentences:
+ return [text]
+ chunks: list[str] = []
+ buffer = ""
+ for sent in sentences:
+ candidate = (buffer + " " + sent).strip() if buffer else sent
+ if _count_tokens(candidate) <= self.max_tokens:
+ buffer = candidate
+ else:
+ if buffer:
+ chunks.append(buffer)
+ buffer = sent
+ if buffer:
+ chunks.append(buffer)
+ return chunks
+```
+
+- [ ] **Step 4: Run tests**
+
+Run: `cd /lambda/nfs/lambda-stanford/jonsf/scratch_v2/OpenJarvis && uv run pytest tests/connectors/test_chunker.py -v`
+
+Expected: All 10 tests PASS. The sentence-boundary test may need tuning if some chunks end up without terminal punctuation due to the final chunk of a section — adjust the test assertion to allow the last chunk in a sequence to not end with punctuation.
+
+- [ ] **Step 5: Commit**
+
+```bash
+git add src/openjarvis/connectors/chunker.py tests/connectors/test_chunker.py
+git commit -m "feat: add type-aware semantic chunker with section/paragraph/sentence splitting"
+```
+
+---
+
+### Task 4: Ingestion Pipeline
+
+**Files:**
+- Create: `src/openjarvis/connectors/pipeline.py`
+- Create: `tests/connectors/test_pipeline.py`
+
+- [ ] **Step 1: Write failing tests**
+
+Create `tests/connectors/test_pipeline.py`:
+
+```python
+"""Tests for the ingestion pipeline: Document → chunk → store."""
+
+from __future__ import annotations
+
+from datetime import datetime
+from pathlib import Path
+
+import pytest
+
+from openjarvis.connectors._stubs import Attachment, Document
+from openjarvis.connectors.pipeline import IngestionPipeline
+from openjarvis.connectors.store import KnowledgeStore
+
+
+def store(tmp_path: Path) -> KnowledgeStore:
+ return KnowledgeStore(db_path=str(tmp_path / "test.db"))
+
+
+def pipeline(store: KnowledgeStore) -> IngestionPipeline:
+ return IngestionPipeline(store=store)
+
+
+def test_ingest_single_document(pipeline: IngestionPipeline, store: KnowledgeStore) -> None:
+ doc = Document(
+ doc_id="gmail:abc",
+ source="gmail",
+ doc_type="email",
+ content="Meeting tomorrow at 3pm to discuss the roadmap.",
+ title="Re: Roadmap",
+ author="[email protected]",
+ participants=["[email protected]", "[email protected]"],
+ timestamp=datetime(2024, 3, 15, 10, 0),
+ )
+ count = pipeline.ingest([doc])
+ assert count == 1
+ results = store.retrieve("roadmap meeting", top_k=5)
+ assert len(results) >= 1
+ assert "roadmap" in results[0].content.lower()
+
+
+def test_ingest_deduplicates_by_doc_id(
+ pipeline: IngestionPipeline, store: KnowledgeStore
+) -> None:
+ doc = Document(
+ doc_id="gmail:dup1",
+ source="gmail",
+ doc_type="email",
+ content="Duplicate email content.",
+ )
+ pipeline.ingest([doc])
+ pipeline.ingest([doc]) # Ingest same doc_id again
+ results = store.retrieve("Duplicate email", top_k=10)
+ assert len(results) == 1
+
+
+def test_ingest_long_document_creates_multiple_chunks(
+ pipeline: IngestionPipeline, store: KnowledgeStore
+) -> None:
+ long_content = "\n\n".join([f"Paragraph {i}. " * 60 for i in range(10)])
+ doc = Document(
+ doc_id="gdrive:longdoc",
+ source="gdrive",
+ doc_type="document",
+ content=long_content,
+ title="Long Report",
+ author="carol",
+ )
+ count = pipeline.ingest([doc])
+ assert count > 1
+ results = store.retrieve("Paragraph", top_k=20)
+ assert len(results) > 1
+ # All chunks should inherit parent metadata
+ for r in results:
+ assert r.metadata.get("source") == "gdrive"
+ assert r.metadata.get("author") == "carol"
+
+
+def test_ingest_event_single_chunk(
+ pipeline: IngestionPipeline, store: KnowledgeStore
+) -> None:
+ doc = Document(
+ doc_id="calendar:evt1",
+ source="calendar",
+ doc_type="event",
+ content="Sprint Review\nAttendees: Alice, Bob\nRoom 3",
+ title="Sprint Review",
+ )
+ count = pipeline.ingest([doc])
+ assert count == 1
+
+
+def test_ingest_multiple_sources(
+ pipeline: IngestionPipeline, store: KnowledgeStore
+) -> None:
+ docs = [
+ Document(doc_id="gmail:1", source="gmail", doc_type="email", content="Email about budget"),
+ Document(doc_id="slack:1", source="slack", doc_type="message", content="Slack msg about budget"),
+ ]
+ count = pipeline.ingest(docs)
+ assert count == 2
+ # Can filter by source
+ gmail_results = store.retrieve("budget", top_k=10, source="gmail")
+ assert len(gmail_results) == 1
+ slack_results = store.retrieve("budget", top_k=10, source="slack")
+ assert len(slack_results) == 1
+```
+
+- [ ] **Step 2: Run tests to verify they fail**
+
+Run: `cd /lambda/nfs/lambda-stanford/jonsf/scratch_v2/OpenJarvis && uv run pytest tests/connectors/test_pipeline.py -v`
+
+Expected: FAIL — `ModuleNotFoundError`
+
+- [ ] **Step 3: Implement IngestionPipeline**
+
+Create `src/openjarvis/connectors/pipeline.py`:
+
+```python
+"""Ingestion pipeline: normalize → deduplicate → chunk → index."""
+
+from __future__ import annotations
+
+import logging
+from typing import Iterable
+
+from openjarvis.connectors._stubs import Document
+from openjarvis.connectors.chunker import SemanticChunker
+from openjarvis.connectors.store import KnowledgeStore
+
+logger = logging.getLogger(__name__)
+
+
+class IngestionPipeline:
+ """Processes Documents from connectors into the KnowledgeStore.
+
+ Handles deduplication (by doc_id), chunking (type-aware),
+ and indexing (dual-write to SQLite/FTS5).
+ """
+
+ def __init__(
+ self,
+ store: KnowledgeStore,
+ *,
+ max_tokens: int = 512,
+ ) -> None:
+ self._store = store
+ self._chunker = SemanticChunker(max_tokens=max_tokens)
+ self._seen_doc_ids: set[str] = set()
+ self._load_existing_doc_ids()
+
+ def _load_existing_doc_ids(self) -> None:
+ """Load already-ingested doc_ids from the store to support dedup."""
+ try:
+ rows = self._store._conn.execute(
+ "SELECT DISTINCT doc_id FROM documents"
+ ).fetchall()
+ self._seen_doc_ids = {r[0] for r in rows}
+ except Exception:
+ self._seen_doc_ids = set()
+
+ def ingest(self, documents: Iterable[Document]) -> int:
+ """Ingest documents into the knowledge store.
+
+ Returns the number of chunks stored.
+ """
+ total_chunks = 0
+ for doc in documents:
+ if doc.doc_id in self._seen_doc_ids:
+ logger.debug("Skipping duplicate doc_id=%s", doc.doc_id)
+ continue
+ self._seen_doc_ids.add(doc.doc_id)
+
+ parent_meta = {
+ "title": doc.title,
+ "author": doc.author,
+ "source": doc.source,
+ "doc_type": doc.doc_type,
+ }
+ if doc.url:
+ parent_meta["url"] = doc.url
+ if doc.thread_id:
+ parent_meta["thread_id"] = doc.thread_id
+ if doc.metadata:
+ parent_meta.update(doc.metadata)
+
+ chunks = self._chunker.chunk(
+ doc.content,
+ doc_type=doc.doc_type,
+ metadata=parent_meta,
+ )
+
+ for chunk in chunks:
+ self._store.store(
+ content=chunk.content,
+ source=doc.source,
+ doc_type=doc.doc_type,
+ doc_id=doc.doc_id,
+ title=doc.title,
+ author=doc.author,
+ participants=doc.participants,
+ timestamp=doc.timestamp.isoformat()
+ if hasattr(doc.timestamp, "isoformat")
+ else str(doc.timestamp),
+ thread_id=doc.thread_id or "",
+ url=doc.url or "",
+ metadata=chunk.metadata,
+ chunk_index=chunk.index,
+ )
+ total_chunks += 1
+
+ return total_chunks
+```
+
+- [ ] **Step 4: Run tests**
+
+Run: `cd /lambda/nfs/lambda-stanford/jonsf/scratch_v2/OpenJarvis && uv run pytest tests/connectors/test_pipeline.py -v`
+
+Expected: All 6 tests PASS.
+
+- [ ] **Step 5: Commit**
+
+```bash
+git add src/openjarvis/connectors/pipeline.py tests/connectors/test_pipeline.py
+git commit -m "feat: add ingestion pipeline with dedup, type-aware chunking, and indexed storage"
+```
+
+---
+
+### Task 5: SyncEngine (Checkpoint/Resume + Orchestration)
+
+**Files:**
+- Create: `src/openjarvis/connectors/sync_engine.py`
+- Create: `tests/connectors/test_sync_engine.py`
+
+- [ ] **Step 1: Write failing tests**
+
+Create `tests/connectors/test_sync_engine.py`:
+
+```python
+"""Tests for SyncEngine — orchestrates connectors with checkpoint/resume."""
+
+from __future__ import annotations
+
+import sqlite3
+from datetime import datetime
+from pathlib import Path
+from typing import Iterator, Optional
+from unittest.mock import MagicMock
+
+import pytest
+
+from openjarvis.connectors._stubs import BaseConnector, Document, SyncStatus
+from openjarvis.connectors.pipeline import IngestionPipeline
+from openjarvis.connectors.store import KnowledgeStore
+from openjarvis.connectors.sync_engine import SyncEngine
+
+
+class StubConnector(BaseConnector):
+ connector_id = "stub"
+ display_name = "Stub"
+ auth_type = "filesystem"
+
+ def __init__(self, docs: list[Document] | None = None) -> None:
+ self._docs = docs or []
+ self._connected = True
+ self._synced = 0
+
+ def is_connected(self) -> bool:
+ return self._connected
+
+ def disconnect(self) -> None:
+ self._connected = False
+
+ def sync(
+ self, *, since: Optional[datetime] = None, cursor: Optional[str] = None
+ ) -> Iterator[Document]:
+ start = int(cursor) if cursor else 0
+ for i, doc in enumerate(self._docs[start:], start=start):
+ self._synced = i + 1
+ yield doc
+
+ def sync_status(self) -> SyncStatus:
+ return SyncStatus(
+ state="idle",
+ items_synced=self._synced,
+ items_total=len(self._docs),
+ )
+
+
+def store(tmp_path: Path) -> KnowledgeStore:
+ return KnowledgeStore(db_path=str(tmp_path / "sync_test.db"))
+
+
+def engine(store: KnowledgeStore, tmp_path: Path) -> SyncEngine:
+ pipeline = IngestionPipeline(store=store)
+ return SyncEngine(pipeline=pipeline, state_db=str(tmp_path / "sync_state.db"))
+
+
+def _make_docs(n: int, source: str = "stub") -> list[Document]:
+ return [
+ Document(
+ doc_id=f"{source}:{i}",
+ source=source,
+ doc_type="message",
+ content=f"Message number {i}",
+ )
+ for i in range(n)
+ ]
+
+
+def test_sync_connector(engine: SyncEngine, store: KnowledgeStore) -> None:
+ conn = StubConnector(docs=_make_docs(5))
+ engine.sync(conn)
+ results = store.retrieve("Message", top_k=10)
+ assert len(results) == 5
+
+
+def test_sync_saves_checkpoint(engine: SyncEngine) -> None:
+ conn = StubConnector(docs=_make_docs(3))
+ engine.sync(conn)
+ cp = engine.get_checkpoint("stub")
+ assert cp is not None
+ assert cp["items_synced"] == 3
+
+
+def test_sync_status_for_unsynced(engine: SyncEngine) -> None:
+ status = engine.get_checkpoint("nonexistent")
+ assert status is None
+
+
+def test_sync_multiple_connectors(engine: SyncEngine, store: KnowledgeStore) -> None:
+ conn_a = StubConnector(docs=_make_docs(3, source="a"))
+ conn_a.connector_id = "a"
+ conn_b = StubConnector(docs=_make_docs(2, source="b"))
+ conn_b.connector_id = "b"
+ engine.sync(conn_a)
+ engine.sync(conn_b)
+ a_results = store.retrieve("Message", top_k=10, source="a")
+ b_results = store.retrieve("Message", top_k=10, source="b")
+ assert len(a_results) == 3
+ assert len(b_results) == 2
+```
+
+- [ ] **Step 2: Run tests to verify they fail**
+
+Run: `cd /lambda/nfs/lambda-stanford/jonsf/scratch_v2/OpenJarvis && uv run pytest tests/connectors/test_sync_engine.py -v`
+
+Expected: FAIL — `ModuleNotFoundError`
+
+- [ ] **Step 3: Implement SyncEngine**
+
+Create `src/openjarvis/connectors/sync_engine.py`:
+
+```python
+"""SyncEngine — orchestrates connector sync with checkpoint/resume."""
+
+from __future__ import annotations
+
+import json
+import logging
+import sqlite3
+from datetime import datetime
+from pathlib import Path
+from typing import Any, Dict, Optional
+
+from openjarvis.connectors._stubs import BaseConnector
+from openjarvis.connectors.pipeline import IngestionPipeline
+from openjarvis.core.config import DEFAULT_CONFIG_DIR
+
+logger = logging.getLogger(__name__)
+
+_STATE_SCHEMA = """\
+CREATE TABLE IF NOT EXISTS sync_state (
+ connector_id TEXT PRIMARY KEY,
+ items_synced INTEGER NOT NULL DEFAULT 0,
+ cursor TEXT,
+ last_sync TEXT,
+ error TEXT
+);
+"""
+
+
+class SyncEngine:
+ """Orchestrates data source sync with checkpointing.
+
+ Tracks per-connector progress in a small SQLite state database,
+ separate from the knowledge store. Supports resume after interruption.
+ """
+
+ def __init__(
+ self,
+ pipeline: IngestionPipeline,
+ *,
+ state_db: str = "",
+ ) -> None:
+ self._pipeline = pipeline
+ if not state_db:
+ state_db = str(DEFAULT_CONFIG_DIR / "sync_state.db")
+ Path(state_db).parent.mkdir(parents=True, exist_ok=True)
+ self._state_conn = sqlite3.connect(state_db)
+ self._state_conn.executescript(_STATE_SCHEMA)
+
+ def sync(self, connector: BaseConnector) -> int:
+ """Run sync for a single connector. Returns items ingested."""
+ cid = connector.connector_id
+ checkpoint = self.get_checkpoint(cid)
+ cursor = checkpoint["cursor"] if checkpoint else None
+
+ logger.info("Starting sync for %s (cursor=%s)", cid, cursor)
+ items = 0
+ try:
+ docs = connector.sync(cursor=cursor)
+ batch: list = []
+ for doc in docs:
+ batch.append(doc)
+ items += 1
+ # Flush in batches of 100
+ if len(batch) >= 100:
+ self._pipeline.ingest(batch)
+ self._save_checkpoint(cid, items, cursor=str(items))
+ batch = []
+ # Flush remaining
+ if batch:
+ self._pipeline.ingest(batch)
+
+ self._save_checkpoint(cid, items, cursor=str(items))
+ logger.info("Sync complete for %s: %d items", cid, items)
+ except Exception as exc:
+ logger.error("Sync error for %s: %s", cid, exc)
+ self._save_checkpoint(cid, items, cursor=str(items), error=str(exc))
+ raise
+
+ return items
+
+ def get_checkpoint(self, connector_id: str) -> Optional[Dict[str, Any]]:
+ """Get the last checkpoint for a connector, or None if never synced."""
+ row = self._state_conn.execute(
+ "SELECT items_synced, cursor, last_sync, error FROM sync_state WHERE connector_id = ?",
+ (connector_id,),
+ ).fetchone()
+ if row is None:
+ return None
+ return {
+ "items_synced": row[0],
+ "cursor": row[1],
+ "last_sync": row[2],
+ "error": row[3],
+ }
+
+ def _save_checkpoint(
+ self,
+ connector_id: str,
+ items_synced: int,
+ *,
+ cursor: Optional[str] = None,
+ error: Optional[str] = None,
+ ) -> None:
+ now = datetime.now().isoformat()
+ self._state_conn.execute(
+ """INSERT INTO sync_state (connector_id, items_synced, cursor, last_sync, error)
+ VALUES (?, ?, ?, ?, ?)
+ ON CONFLICT(connector_id)
+ DO UPDATE SET items_synced=?, cursor=?, last_sync=?, error=?""",
+ (
+ connector_id,
+ items_synced,
+ cursor,
+ now,
+ error,
+ items_synced,
+ cursor,
+ now,
+ error,
+ ),
+ )
+ self._state_conn.commit()
+
+ def close(self) -> None:
+ self._state_conn.close()
+```
+
+- [ ] **Step 4: Run tests**
+
+Run: `cd /lambda/nfs/lambda-stanford/jonsf/scratch_v2/OpenJarvis && uv run pytest tests/connectors/test_sync_engine.py -v`
+
+Expected: All 4 tests PASS.
+
+- [ ] **Step 5: Commit**
+
+```bash
+git add src/openjarvis/connectors/sync_engine.py tests/connectors/test_sync_engine.py
+git commit -m "feat: add SyncEngine with checkpoint/resume for connector orchestration"
+```
+
+---
+
+### Task 6: Obsidian/Markdown Connector
+
+**Files:**
+- Create: `src/openjarvis/connectors/obsidian.py`
+- Create: `tests/connectors/test_obsidian.py`
+
+- [ ] **Step 1: Write failing tests**
+
+Create `tests/connectors/test_obsidian.py`:
+
+```python
+"""Tests for the Obsidian/Markdown connector."""
+
+from __future__ import annotations
+
+from datetime import datetime
+from pathlib import Path
+from typing import Optional
+
+import pytest
+
+from openjarvis.connectors.obsidian import ObsidianConnector
+from openjarvis.core.registry import ConnectorRegistry
+
+
+def vault(tmp_path: Path) -> Path:
+ vault_dir = tmp_path / "my-vault"
+ vault_dir.mkdir()
+ (vault_dir / "note1.md").write_text(
+ "---\ntitle: Meeting Notes\ntags: [work, meetings]\n---\n\n# Meeting Notes\n\nDiscussed Q3 roadmap."
+ )
+ (vault_dir / "note2.md").write_text("# Ideas\n\nBrainstorm for new feature.")
+ (vault_dir / "subdir").mkdir()
+ (vault_dir / "subdir" / "deep.md").write_text("# Deep Note\n\nNested vault content.")
+ (vault_dir / ".obsidian").mkdir() # Should be skipped
+ (vault_dir / ".obsidian" / "config.json").write_text("{}")
+ (vault_dir / "image.png").write_bytes(b"\x89PNG") # Should be skipped
+ return vault_dir
+
+
+def connector(vault: Path) -> ObsidianConnector:
+ return ObsidianConnector(vault_path=str(vault))
+
+
+def test_is_connected(connector: ObsidianConnector) -> None:
+ assert connector.is_connected()
+
+
+def test_not_connected_bad_path() -> None:
+ conn = ObsidianConnector(vault_path="/nonexistent/path")
+ assert not conn.is_connected()
+
+
+def test_sync_yields_markdown_files(connector: ObsidianConnector) -> None:
+ docs = list(connector.sync())
+ assert len(docs) == 3 # note1, note2, deep
+ sources = {d.doc_id for d in docs}
+ assert any("note1" in s for s in sources)
+ assert any("deep" in s for s in sources)
+
+
+def test_sync_skips_hidden_dirs(connector: ObsidianConnector) -> None:
+ docs = list(connector.sync())
+ paths = [d.doc_id for d in docs]
+ assert not any(".obsidian" in p for p in paths)
+
+
+def test_sync_skips_binary_files(connector: ObsidianConnector) -> None:
+ docs = list(connector.sync())
+ types = [d.title for d in docs]
+ assert not any("image" in t for t in types)
+
+
+def test_sync_parses_frontmatter(connector: ObsidianConnector) -> None:
+ docs = list(connector.sync())
+ note1 = next(d for d in docs if "note1" in d.doc_id)
+ assert note1.title == "Meeting Notes"
+ assert note1.metadata.get("tags") == ["work", "meetings"]
+
+
+def test_sync_sets_doc_type_note(connector: ObsidianConnector) -> None:
+ docs = list(connector.sync())
+ assert all(d.doc_type == "note" for d in docs)
+ assert all(d.source == "obsidian" for d in docs)
+
+
+def test_disconnect(connector: ObsidianConnector) -> None:
+ connector.disconnect()
+ assert not connector.is_connected()
+
+
+def test_registry() -> None:
+ import openjarvis.connectors.obsidian # noqa: F401
+ assert ConnectorRegistry.contains("obsidian")
+```
+
+- [ ] **Step 2: Run tests to verify they fail**
+
+Run: `cd /lambda/nfs/lambda-stanford/jonsf/scratch_v2/OpenJarvis && uv run pytest tests/connectors/test_obsidian.py -v`
+
+Expected: FAIL — `ModuleNotFoundError`
+
+- [ ] **Step 3: Implement ObsidianConnector**
+
+Create `src/openjarvis/connectors/obsidian.py`:
+
+```python
+"""Obsidian/Markdown vault connector — reads .md files from a local directory."""
+
+from __future__ import annotations
+
+import os
+import re
+from datetime import datetime
+from pathlib import Path
+from typing import Any, Dict, Iterator, List, Optional
+
+from openjarvis.connectors._stubs import BaseConnector, Document, SyncStatus
+from openjarvis.core.registry import ConnectorRegistry
+from openjarvis.tools._stubs import ToolSpec
+
+_FRONTMATTER_RE = re.compile(r"^---\s*\n(.*?)\n---\s*\n", re.DOTALL)
+_SKIP_DIRS = {".obsidian", ".trash", ".git", "__pycache__", "node_modules", ".venv"}
+_MD_EXTENSIONS = {".md", ".markdown", ".txt"}
+
+
+def _parse_frontmatter(text: str) -> tuple[Dict[str, Any], str]:
+ """Extract YAML frontmatter and return (metadata, body)."""
+ match = _FRONTMATTER_RE.match(text)
+ if not match:
+ return {}, text
+ raw = match.group(1)
+ body = text[match.end() :]
+ meta: Dict[str, Any] = {}
+ for line in raw.strip().split("\n"):
+ if ":" not in line:
+ continue
+ key, _, val = line.partition(":")
+ key = key.strip()
+ val = val.strip()
+ # Simple YAML list: [a, b, c]
+ if val.startswith("[") and val.endswith("]"):
+ meta[key] = [v.strip() for v in val[1:-1].split(",")]
+ else:
+ meta[key] = val
+ return meta, body
+
+
[email protected]("obsidian")
+class ObsidianConnector(BaseConnector):
+ """Connector for Obsidian vaults and plain markdown directories."""
+
+ connector_id = "obsidian"
+ display_name = "Obsidian / Markdown"
+ auth_type = "filesystem"
+
+ def __init__(self, vault_path: str = "") -> None:
+ self._vault_path = vault_path
+ self._connected = bool(vault_path) and Path(vault_path).is_dir()
+ self._synced = 0
+ self._total = 0
+
+ def is_connected(self) -> bool:
+ return self._connected and Path(self._vault_path).is_dir()
+
+ def disconnect(self) -> None:
+ self._connected = False
+ self._vault_path = ""
+
+ def sync(
+ self, *, since: Optional[datetime] = None, cursor: Optional[str] = None
+ ) -> Iterator[Document]:
+ if not self.is_connected():
+ return
+ vault = Path(self._vault_path)
+ md_files = sorted(self._walk_md_files(vault))
+ self._total = len(md_files)
+ self._synced = 0
+
+ for filepath in md_files:
+ try:
+ text = filepath.read_text(encoding="utf-8")
+ except (UnicodeDecodeError, OSError):
+ continue
+
+ meta, body = _parse_frontmatter(text)
+ title = meta.pop("title", filepath.stem)
+ rel_path = str(filepath.relative_to(vault))
+ mtime = datetime.fromtimestamp(filepath.stat().st_mtime)
+
+ if since and mtime < since:
+ self._synced += 1
+ continue
+
+ self._synced += 1
+ yield Document(
+ doc_id=f"obsidian:{rel_path}",
+ source="obsidian",
+ doc_type="note",
+ content=body.strip(),
+ title=title,
+ author="",
+ timestamp=mtime,
+ url=f"obsidian://open?vault={vault.name}&file={rel_path}",
+ metadata=meta,
+ )
+
+ def sync_status(self) -> SyncStatus:
+ return SyncStatus(
+ state="idle",
+ items_synced=self._synced,
+ items_total=self._total,
+ )
+
+ def mcp_tools(self) -> List[ToolSpec]:
+ return [
+ ToolSpec(
+ name="obsidian_search_notes",
+ description="Search Obsidian vault notes by keyword.",
+ parameters={
+ "type": "object",
+ "properties": {
+ "query": {"type": "string", "description": "Search query"},
+ },
+ "required": ["query"],
+ },
+ category="connector",
+ ),
+ ]
+
+ def _walk_md_files(self, root: Path) -> List[Path]:
+ """Walk the vault, skipping hidden/config dirs and non-markdown files."""
+ results: List[Path] = []
+ for dirpath, dirnames, filenames in os.walk(root):
+ # Prune hidden/config directories
+ dirnames[:] = [d for d in dirnames if d not in _SKIP_DIRS and not d.startswith(".")]
+ for fname in filenames:
+ if Path(fname).suffix.lower() in _MD_EXTENSIONS:
+ results.append(Path(dirpath) / fname)
+ return results
+```
+
+- [ ] **Step 4: Run tests**
+
+Run: `cd /lambda/nfs/lambda-stanford/jonsf/scratch_v2/OpenJarvis && uv run pytest tests/connectors/test_obsidian.py -v`
+
+Expected: All 9 tests PASS.
+
+- [ ] **Step 5: Commit**
+
+```bash
+git add src/openjarvis/connectors/obsidian.py tests/connectors/test_obsidian.py
+git commit -m "feat: add Obsidian/Markdown vault connector with frontmatter parsing"
+```
+
+---
+
+### Task 7: Gmail Connector (Mocked OAuth + API)
+
+**Files:**
+- Create: `src/openjarvis/connectors/oauth.py`
+- Create: `src/openjarvis/connectors/gmail.py`
+- Create: `tests/connectors/test_gmail.py`
+
+- [ ] **Step 1: Write failing tests**
+
+Create `tests/connectors/test_gmail.py`:
+
+```python
+"""Tests for the Gmail connector (mocked API calls)."""
+
+from __future__ import annotations
+
+from datetime import datetime
+from typing import Optional
+from unittest.mock import MagicMock, patch
+
+import pytest
+
+from openjarvis.connectors.gmail import GmailConnector
+from openjarvis.core.registry import ConnectorRegistry
+
+
+def connector(tmp_path) -> GmailConnector:
+ return GmailConnector(
+ credentials_path=str(tmp_path / "gmail_creds.json"),
+ )
+
+
+def test_not_connected_without_credentials(connector: GmailConnector) -> None:
+ assert not connector.is_connected()
+
+
+def test_auth_type_is_oauth(connector: GmailConnector) -> None:
+ assert connector.auth_type == "oauth"
+
+
+def test_auth_url_returns_string(connector: GmailConnector) -> None:
+ url = connector.auth_url()
+ assert url.startswith("https://accounts.google.com/o/oauth2/v2/auth")
+ assert "gmail.readonly" in url
+
+
+@patch("openjarvis.connectors.gmail._gmail_api_list_messages")
+@patch("openjarvis.connectors.gmail._gmail_api_get_message")
+def test_sync_yields_documents(mock_get, mock_list, connector, tmp_path) -> None:
+ # Simulate having valid credentials
+ creds_path = tmp_path / "gmail_creds.json"
+ creds_path.write_text('{"token": "fake", "refresh_token": "fake"}')
+ connector._credentials_path = str(creds_path)
+
+ mock_list.return_value = {
+ "messages": [{"id": "msg1"}, {"id": "msg2"}],
+ "nextPageToken": None,
+ }
+ mock_get.side_effect = [
+ {
+ "id": "msg1",
+ "threadId": "t1",
+ "labelIds": ["INBOX"],
+ "payload": {
+ "headers": [
+ {"name": "From", "value": "[email protected]"},
+ {"name": "Subject", "value": "Q3 Planning"},
+ {"name": "Date", "value": "Mon, 15 Mar 2024 10:00:00 +0000"},
+ {"name": "To", "value": "[email protected]"},
+ ],
+ "body": {"data": "SGVsbG8gd29ybGQ="}, # "Hello world" base64
+ },
+ },
+ {
+ "id": "msg2",
+ "threadId": "t2",
+ "labelIds": ["SENT"],
+ "payload": {
+ "headers": [
+ {"name": "From", "value": "[email protected]"},
+ {"name": "Subject", "value": "Re: Budget"},
+ {"name": "Date", "value": "Tue, 16 Mar 2024 11:00:00 +0000"},
+ {"name": "To", "value": "[email protected]"},
+ ],
+ "body": {"data": "QnVkZ2V0IHJlcGx5"}, # "Budget reply" base64
+ },
+ },
+ ]
+
+ docs = list(connector.sync())
+ assert len(docs) == 2
+ assert docs[0].doc_id == "gmail:msg1"
+ assert docs[0].source == "gmail"
+ assert docs[0].doc_type == "email"
+ assert docs[0].author == "[email protected]"
+ assert docs[0].title == "Q3 Planning"
+ assert docs[0].thread_id == "t1"
+ assert "Hello world" in docs[0].content
+
+
+def test_disconnect(connector: GmailConnector, tmp_path) -> None:
+ creds_path = tmp_path / "gmail_creds.json"
+ creds_path.write_text('{"token": "fake"}')
+ connector._credentials_path = str(creds_path)
+ connector.disconnect()
+ assert not connector.is_connected()
+
+
+def test_mcp_tools(connector: GmailConnector) -> None:
+ tools = connector.mcp_tools()
+ names = [t.name for t in tools]
+ assert "gmail_search_emails" in names
+ assert "gmail_get_thread" in names
+
+
+def test_registry() -> None:
+ import openjarvis.connectors.gmail # noqa: F401
+ assert ConnectorRegistry.contains("gmail")
+```
+
+- [ ] **Step 2: Run tests to verify they fail**
+
+Run: `cd /lambda/nfs/lambda-stanford/jonsf/scratch_v2/OpenJarvis && uv run pytest tests/connectors/test_gmail.py -v`
+
+Expected: FAIL — `ModuleNotFoundError`
+
+- [ ] **Step 3: Create OAuth helper**
+
+Create `src/openjarvis/connectors/oauth.py`:
+
+```python
+"""Shared OAuth helper for connector authentication.
+
+Handles the OAuth2 authorization code flow:
+1. Generate consent URL with scopes
+2. Start a temporary localhost server to receive the callback
+3. Exchange the authorization code for tokens
+4. Save tokens to credentials file
+"""
+
+from __future__ import annotations
+
+import json
+import logging
+from pathlib import Path
+from typing import Any, Dict, Optional
+from urllib.parse import urlencode
+
+logger = logging.getLogger(__name__)
+
+
+def build_google_auth_url(
+ client_id: str,
+ redirect_uri: str = "http://localhost:8789/callback",
+ scopes: Optional[list[str]] = None,
+) -> str:
+ """Build a Google OAuth2 consent URL."""
+ params = {
+ "client_id": client_id,
+ "redirect_uri": redirect_uri,
+ "response_type": "code",
+ "scope": " ".join(scopes or []),
+ "access_type": "offline",
+ "prompt": "consent",
+ }
+ return f"https://accounts.google.com/o/oauth2/v2/auth?{urlencode(params)}"
+
+
+def load_tokens(path: str) -> Optional[Dict[str, Any]]:
+ """Load OAuth tokens from a JSON file."""
+ p = Path(path)
+ if not p.exists():
+ return None
+ try:
+ data = json.loads(p.read_text())
+ if data.get("token") or data.get("access_token"):
+ return data
+ except (json.JSONDecodeError, OSError):
+ pass
+ return None
+
+
+def save_tokens(path: str, tokens: Dict[str, Any]) -> None:
+ """Save OAuth tokens to a JSON file with restrictive permissions."""
+ p = Path(path)
+ p.parent.mkdir(parents=True, exist_ok=True)
+ p.write_text(json.dumps(tokens, indent=2))
+ try:
+ p.chmod(0o600)
+ except OSError:
+ pass
+
+
+def delete_tokens(path: str) -> None:
+ """Delete a credentials file."""
+ p = Path(path)
+ if p.exists():
+ p.unlink()
+```
+
+- [ ] **Step 4: Implement GmailConnector**
+
+Create `src/openjarvis/connectors/gmail.py`:
+
+```python
+"""Gmail connector — OAuth + Gmail API for bulk email sync."""
+
+from __future__ import annotations
+
+import base64
+import logging
+from datetime import datetime
+from email.utils import parsedate_to_datetime
+from pathlib import Path
+from typing import Any, Dict, Iterator, List, Optional
+
+import httpx
+
+from openjarvis.connectors._stubs import BaseConnector, Document, SyncStatus
+from openjarvis.connectors.oauth import (
+ build_google_auth_url,
+ delete_tokens,
+ load_tokens,
+ save_tokens,
+)
+from openjarvis.core.config import DEFAULT_CONFIG_DIR
+from openjarvis.core.registry import ConnectorRegistry
+from openjarvis.tools._stubs import ToolSpec
+
+logger = logging.getLogger(__name__)
+
+_GMAIL_API = "https://gmail.googleapis.com/gmail/v1/users/me"
+# Placeholder — in production, use OpenJarvis's registered OAuth app
+_CLIENT_ID = "openjarvis-gmail.apps.googleusercontent.com"
+_SCOPES = ["https://www.googleapis.com/auth/gmail.readonly"]
+
+
+def _gmail_api_list_messages(
+ token: str, *, page_token: Optional[str] = None, query: str = ""
+) -> Dict[str, Any]:
+ """Call Gmail messages.list API."""
+ params: dict[str, Any] = {"maxResults": 100}
+ if page_token:
+ params["pageToken"] = page_token
+ if query:
+ params["q"] = query
+ resp = httpx.get(
+ f"{_GMAIL_API}/messages",
+ headers={"Authorization": f"Bearer {token}"},
+ params=params,
+ timeout=30.0,
+ )
+ resp.raise_for_status()
+ return resp.json()
+
+
+def _gmail_api_get_message(token: str, msg_id: str) -> Dict[str, Any]:
+ """Call Gmail messages.get API."""
+ resp = httpx.get(
+ f"{_GMAIL_API}/messages/{msg_id}",
+ headers={"Authorization": f"Bearer {token}"},
+ params={"format": "full"},
+ timeout=30.0,
+ )
+ resp.raise_for_status()
+ return resp.json()
+
+
+def _extract_header(headers: List[Dict[str, str]], name: str) -> str:
+ """Extract a header value from Gmail API payload headers."""
+ for h in headers:
+ if h.get("name", "").lower() == name.lower():
+ return h.get("value", "")
+ return ""
+
+
+def _decode_body(payload: Dict[str, Any]) -> str:
+ """Decode the email body from Gmail API payload."""
+ # Try top-level body
+ body_data = payload.get("body", {}).get("data", "")
+ if body_data:
+ return base64.urlsafe_b64decode(body_data + "==").decode("utf-8", errors="replace")
+ # Try multipart parts
+ for part in payload.get("parts", []):
+ mime = part.get("mimeType", "")
+ if mime == "text/plain":
+ data = part.get("body", {}).get("data", "")
+ if data:
+ return base64.urlsafe_b64decode(data + "==").decode("utf-8", errors="replace")
+ # Fallback: try first part with data
+ for part in payload.get("parts", []):
+ data = part.get("body", {}).get("data", "")
+ if data:
+ return base64.urlsafe_b64decode(data + "==").decode("utf-8", errors="replace")
+ return ""
+
+
+def _parse_date(date_str: str) -> datetime:
+ """Parse an email Date header."""
+ try:
+ return parsedate_to_datetime(date_str)
+ except Exception:
+ return datetime.now()
+
+
+class GmailConnector(BaseConnector):
+ """Gmail connector — syncs email via the Gmail API."""
+
+ connector_id = "gmail"
+ display_name = "Gmail"
+ auth_type = "oauth"
+
+ def __init__(self, credentials_path: str = "") -> None:
+ if not credentials_path:
+ credentials_path = str(DEFAULT_CONFIG_DIR / "connectors" / "gmail.json")
+ self._credentials_path = credentials_path
+ self._synced = 0
+ self._total = 0
+
+ def is_connected(self) -> bool:
+ tokens = load_tokens(self._credentials_path)
+ return tokens is not None
+
+ def disconnect(self) -> None:
+ delete_tokens(self._credentials_path)
+
+ def auth_url(self) -> str:
+ return build_google_auth_url(client_id=_CLIENT_ID, scopes=_SCOPES)
+
+ def handle_callback(self, code: str) -> None:
+ # In production: exchange code for tokens via Google's token endpoint
+ save_tokens(self._credentials_path, {"token": code, "refresh_token": code})
+
+ def sync(
+ self, *, since: Optional[datetime] = None, cursor: Optional[str] = None
+ ) -> Iterator[Document]:
+ tokens = load_tokens(self._credentials_path)
+ if not tokens:
+ return
+ token = tokens.get("token", "")
+
+ page_token = cursor
+ while True:
+ data = _gmail_api_list_messages(token, page_token=page_token)
+ messages = data.get("messages", [])
+ self._total += len(messages)
+
+ for msg_ref in messages:
+ msg = _gmail_api_get_message(token, msg_ref["id"])
+ payload = msg.get("payload", {})
+ headers = payload.get("headers", [])
+
+ from_addr = _extract_header(headers, "From")
+ subject = _extract_header(headers, "Subject")
+ date_str = _extract_header(headers, "Date")
+ to_addr = _extract_header(headers, "To")
+ body = _decode_body(payload)
+ timestamp = _parse_date(date_str)
+
+ if since and timestamp < since:
+ continue
+
+ self._synced += 1
+ yield Document(
+ doc_id=f"gmail:{msg['id']}",
+ source="gmail",
+ doc_type="email",
+ content=body,
+ title=subject,
+ author=from_addr,
+ participants=[a.strip() for a in to_addr.split(",") if a.strip()],
+ timestamp=timestamp,
+ thread_id=msg.get("threadId", ""),
+ url=f"https://mail.google.com/mail/u/0/#inbox/{msg['id']}",
+ metadata={
+ "labels": msg.get("labelIds", []),
+ },
+ )
+
+ page_token = data.get("nextPageToken")
+ if not page_token:
+ break
+
+ def sync_status(self) -> SyncStatus:
+ return SyncStatus(
+ state="idle",
+ items_synced=self._synced,
+ items_total=self._total,
+ )
+
+ def mcp_tools(self) -> List[ToolSpec]:
+ return [
+ ToolSpec(
+ name="gmail_search_emails",
+ description="Search Gmail messages by query string.",
+ parameters={
+ "type": "object",
+ "properties": {
+ "query": {"type": "string", "description": "Gmail search query (e.g., 'from:alice subject:budget')"},
+ "max_results": {"type": "integer", "description": "Max results to return", "default": 10},
+ },
+ "required": ["query"],
+ },
+ category="connector",
+ ),
+ ToolSpec(
+ name="gmail_get_thread",
+ description="Get all messages in a Gmail thread by thread ID.",
+ parameters={
+ "type": "object",
+ "properties": {
+ "thread_id": {"type": "string", "description": "Gmail thread ID"},
+ },
+ "required": ["thread_id"],
+ },
+ category="connector",
+ ),
+ ToolSpec(
+ name="gmail_list_unread",
+ description="List recent unread emails.",
+ parameters={
+ "type": "object",
+ "properties": {
+ "max_results": {"type": "integer", "description": "Max results", "default": 10},
+ },
+ },
+ category="connector",
+ ),
+ ]
+```
+
+- [ ] **Step 5: Run tests**
+
+Run: `cd /lambda/nfs/lambda-stanford/jonsf/scratch_v2/OpenJarvis && uv run pytest tests/connectors/test_gmail.py -v`
+
+Expected: All 7 tests PASS.
+
+- [ ] **Step 6: Commit**
+
+```bash
+git add src/openjarvis/connectors/oauth.py src/openjarvis/connectors/gmail.py tests/connectors/test_gmail.py
+git commit -m "feat: add Gmail connector with OAuth and mocked API sync"
+```
+
+---
+
+### Task 8: knowledge_search Tool
+
+**Files:**
+- Create: `src/openjarvis/tools/knowledge_search.py`
+- Create: `tests/tools/test_knowledge_search.py`
+
+- [ ] **Step 1: Write failing tests**
+
+Create `tests/tools/test_knowledge_search.py`:
+
+```python
+"""Tests for the knowledge_search tool."""
+
+from __future__ import annotations
+
+from pathlib import Path
+
+import pytest
+
+from openjarvis.connectors.store import KnowledgeStore
+from openjarvis.core.registry import ToolRegistry
+from openjarvis.tools.knowledge_search import KnowledgeSearchTool
+
+
+def store(tmp_path: Path) -> KnowledgeStore:
+ s = KnowledgeStore(db_path=str(tmp_path / "ks_test.db"))
+ s.store(content="Kubernetes migration proposal", source="gdrive", doc_type="document", author="sarah")
+ s.store(content="Budget discussion for Q3", source="gmail", doc_type="email", author="mike")
+ s.store(content="Sprint standup notes", source="slack", doc_type="message", author="alice")
+ return s
+
+
+def tool(store: KnowledgeStore) -> KnowledgeSearchTool:
+ return KnowledgeSearchTool(store=store)
+
+
+def test_basic_search(tool: KnowledgeSearchTool) -> None:
+ result = tool.execute(query="Kubernetes migration")
+ assert result.success
+ assert "Kubernetes" in result.content
+
+
+def test_filter_by_source(tool: KnowledgeSearchTool) -> None:
+ result = tool.execute(query="discussion", source="gmail")
+ assert result.success
+ assert "Budget" in result.content
+
+
+def test_filter_by_author(tool: KnowledgeSearchTool) -> None:
+ result = tool.execute(query="proposal", author="sarah")
+ assert result.success
+ assert "Kubernetes" in result.content
+
+
+def test_no_results(tool: KnowledgeSearchTool) -> None:
+ result = tool.execute(query="xyznonexistent")
+ assert result.success
+ assert "No relevant results" in result.content
+
+
+def test_empty_query(tool: KnowledgeSearchTool) -> None:
+ result = tool.execute(query="")
+ assert not result.success
+
+
+def test_no_store() -> None:
+ tool = KnowledgeSearchTool(store=None)
+ result = tool.execute(query="anything")
+ assert not result.success
+
+
+def test_spec_has_filter_params(tool: KnowledgeSearchTool) -> None:
+ spec = tool.spec
+ props = spec.parameters.get("properties", {})
+ assert "query" in props
+ assert "source" in props
+ assert "doc_type" in props
+ assert "author" in props
+ assert "since" in props
+ assert "top_k" in props
+
+
+def test_registry() -> None:
+ import openjarvis.tools.knowledge_search # noqa: F401
+ assert ToolRegistry.contains("knowledge_search")
+```
+
+- [ ] **Step 2: Run tests to verify they fail**
+
+Run: `cd /lambda/nfs/lambda-stanford/jonsf/scratch_v2/OpenJarvis && uv run pytest tests/tools/test_knowledge_search.py -v`
+
+Expected: FAIL — `ModuleNotFoundError`
+
+- [ ] **Step 3: Implement KnowledgeSearchTool**
+
+Create `src/openjarvis/tools/knowledge_search.py`:
+
+```python
+"""knowledge_search — filtered BM25 retrieval over the personal knowledge base."""
+
+from __future__ import annotations
+
+from typing import Any, Optional
+
+from openjarvis.connectors.store import KnowledgeStore
+from openjarvis.core.registry import ToolRegistry
+from openjarvis.core.types import ToolResult
+from openjarvis.tools._stubs import BaseTool, ToolSpec
+
+
[email protected]("knowledge_search")
+class KnowledgeSearchTool(BaseTool):
+ """Search the personal knowledge base with optional filters.
+
+ Supports filtering by source (gmail, slack, gdrive, etc.),
+ doc_type (email, message, document, etc.), author, and time range.
+ """
+
+ tool_id = "knowledge_search"
+
+ def __init__(self, store: Optional[KnowledgeStore] = None) -> None:
+ self._store = store
+
+ @property
+ def spec(self) -> ToolSpec:
+ return ToolSpec(
+ name="knowledge_search",
+ description=(
+ "Search your personal knowledge base (emails, messages, documents, "
+ "calendar events, contacts, notes) with optional filters. Returns "
+ "relevant results with source attribution and deep links."
+ ),
+ parameters={
+ "type": "object",
+ "properties": {
+ "query": {
+ "type": "string",
+ "description": "Search query to find relevant information.",
+ },
+ "source": {
+ "type": "string",
+ "description": 'Filter by source: "gmail", "slack", "gdrive", "obsidian", etc.',
+ },
+ "doc_type": {
+ "type": "string",
+ "description": 'Filter by type: "email", "message", "document", "event", "contact", "note".',
+ },
+ "author": {
+ "type": "string",
+ "description": "Filter by author/sender name or email.",
+ },
+ "since": {
+ "type": "string",
+ "description": "Only return results after this ISO 8601 timestamp.",
+ },
+ "until": {
+ "type": "string",
+ "description": "Only return results before this ISO 8601 timestamp.",
+ },
+ "top_k": {
+ "type": "integer",
+ "description": "Number of results to return (default: 10).",
+ },
+ },
+ "required": ["query"],
+ },
+ category="knowledge",
+ )
+
+ def execute(self, **params: Any) -> ToolResult:
+ if self._store is None:
+ return ToolResult(
+ tool_name="knowledge_search",
+ content="No knowledge store configured. Run 'jarvis connect' to set up data sources.",
+ success=False,
+ )
+
+ query = params.get("query", "")
+ if not query:
+ return ToolResult(
+ tool_name="knowledge_search",
+ content="No query provided.",
+ success=False,
+ )
+
+ top_k = params.get("top_k", 10)
+ try:
+ results = self._store.retrieve(
+ query,
+ top_k=top_k,
+ source=params.get("source", ""),
+ doc_type=params.get("doc_type", ""),
+ author=params.get("author", ""),
+ since=params.get("since", ""),
+ until=params.get("until", ""),
+ )
+ except Exception as exc:
+ return ToolResult(
+ tool_name="knowledge_search",
+ content=f"Search error: {exc}",
+ success=False,
+ )
+
+ if not results:
+ return ToolResult(
+ tool_name="knowledge_search",
+ content="No relevant results found.",
+ success=True,
+ metadata={"num_results": 0},
+ )
+
+ # Format results with source attribution
+ lines = []
+ for i, r in enumerate(results, 1):
+ meta = r.metadata
+ source_tag = meta.get("source", "unknown")
+ author = meta.get("author", "")
+ title = meta.get("title", "")
+ url = meta.get("url", "")
+ header_parts = [f"[{source_tag}]"]
+ if title:
+ header_parts.append(title)
+ if author:
+ header_parts.append(f"by {author}")
+ if url:
+ header_parts.append(f"({url})")
+ header = " ".join(header_parts)
+ lines.append(f"**Result {i}:** {header}\n{r.content}\n")
+
+ return ToolResult(
+ tool_name="knowledge_search",
+ content="\n".join(lines),
+ success=True,
+ metadata={"num_results": len(results)},
+ )
+```
+
+- [ ] **Step 4: Run tests**
+
+Run: `cd /lambda/nfs/lambda-stanford/jonsf/scratch_v2/OpenJarvis && uv run pytest tests/tools/test_knowledge_search.py -v`
+
+Expected: All 8 tests PASS.
+
+- [ ] **Step 5: Commit**
+
+```bash
+git add src/openjarvis/tools/knowledge_search.py tests/tools/test_knowledge_search.py
+git commit -m "feat: add knowledge_search tool with filtered BM25 retrieval and source attribution"
+```
+
+---
+
+### Task 9: CLI `jarvis connect` Command
+
+**Files:**
+- Create: `src/openjarvis/cli/connect_cmd.py`
+- Modify: `src/openjarvis/cli/__init__.py`
+- Create: `tests/cli/test_connect.py`
+
+- [ ] **Step 1: Write failing tests**
+
+Create `tests/cli/test_connect.py`:
+
+```python
+"""Tests for the jarvis connect CLI command."""
+
+from __future__ import annotations
+
+from pathlib import Path
+from unittest.mock import patch
+
+import pytest
+from click.testing import CliRunner
+
+from openjarvis.cli.connect_cmd import connect
+
+
+def runner() -> CliRunner:
+ return CliRunner()
+
+
+def test_connect_list_no_connectors(runner: CliRunner) -> None:
+ result = runner.invoke(connect, ["--list"])
+ assert result.exit_code == 0
+ assert "No connectors" in result.output or "connected" in result.output.lower()
+
+
+def test_connect_list_with_connector(runner: CliRunner, tmp_path: Path) -> None:
+ # Create a fake credential file so Gmail appears connected
+ creds = tmp_path / "gmail.json"
+ creds.write_text('{"token": "fake"}')
+ with patch("openjarvis.connectors.gmail.DEFAULT_CONFIG_DIR", tmp_path / "config"):
+ result = runner.invoke(connect, ["--list"])
+ assert result.exit_code == 0
+
+
+def test_connect_help(runner: CliRunner) -> None:
+ result = runner.invoke(connect, ["--help"])
+ assert result.exit_code == 0
+ assert "connect" in result.output.lower()
+
+
+def test_connect_specific_source(runner: CliRunner) -> None:
+ result = runner.invoke(connect, ["obsidian", "--path", "/nonexistent"])
+ # Should fail gracefully — path doesn't exist
+ assert result.exit_code == 0 or "not found" in result.output.lower() or "does not exist" in result.output.lower()
+
+
+def test_connect_disconnect(runner: CliRunner) -> None:
+ result = runner.invoke(connect, ["--disconnect", "gmail"])
+ assert result.exit_code == 0
+```
+
+- [ ] **Step 2: Run tests to verify they fail**
+
+Run: `cd /lambda/nfs/lambda-stanford/jonsf/scratch_v2/OpenJarvis && uv run pytest tests/cli/test_connect.py -v`
+
+Expected: FAIL — `ModuleNotFoundError`
+
+- [ ] **Step 3: Implement connect command**
+
+Create `src/openjarvis/cli/connect_cmd.py`:
+
+```python
+"""CLI command: jarvis connect — manage data source connections."""
+
+from __future__ import annotations
+
+import click
+from rich.console import Console
+from rich.table import Table
+
+console = Console()
+
+
[email protected](invoke_without_command=True)
[email protected]("source", required=False)
[email protected]("--list", "list_sources", is_flag=True, help="List connected sources and sync status.")
[email protected]("--sync", "trigger_sync", is_flag=True, help="Trigger incremental sync for all sources.")
[email protected]("--disconnect", "disconnect_source", default="", help="Disconnect a source.")
[email protected]("--path", default="", help="Path for filesystem connectors (e.g., Obsidian vault).")
+def connect(
+ ctx: click.Context,
+ source: str | None,
+ list_sources: bool,
+ trigger_sync: bool,
+ disconnect_source: str,
+ path: str,
+) -> None:
+ """Connect data sources for Deep Research.
+
+ \b
+ Examples:
+ jarvis connect # Interactive source picker
+ jarvis connect gmail # Connect Gmail
+ jarvis connect obsidian --path ~/vault
+ jarvis connect --list # Show connected sources
+ jarvis connect --disconnect gmail # Disconnect Gmail
+ """
+ # Import connectors to trigger registration
+ import openjarvis.connectors.gmail # noqa: F401
+ import openjarvis.connectors.obsidian # noqa: F401
+ from openjarvis.core.registry import ConnectorRegistry
+
+ if list_sources:
+ _list_sources(ConnectorRegistry)
+ return
+
+ if disconnect_source:
+ _disconnect_source(ConnectorRegistry, disconnect_source)
+ return
+
+ if trigger_sync:
+ console.print("[yellow]Sync not yet implemented in CLI — use the desktop app.[/yellow]")
+ return
+
+ if source:
+ _connect_source(ConnectorRegistry, source, path=path)
+ return
+
+ # No args — show help
+ if not ctx.invoked_subcommand:
+ console.print(ctx.get_help())
+
+
+def _list_sources(registry: type) -> None:
+ """List all available connectors and their status."""
+ table = Table(title="Data Sources")
+ table.add_column("Source", style="cyan")
+ table.add_column("Type", style="dim")
+ table.add_column("Status", style="green")
+
+ for key in sorted(registry.keys()):
+ cls = registry.get(key)
+ try:
+ instance = cls()
+ connected = instance.is_connected()
+ status = "[green]Connected[/green]" if connected else "[dim]Not connected[/dim]"
+ except Exception:
+ status = "[dim]Not configured[/dim]"
+ auth = getattr(cls, "auth_type", "unknown")
+ table.add_row(key, auth, status)
+
+ if not registry.keys():
+ console.print("[dim]No connectors registered.[/dim]")
+ return
+
+ console.print(table)
+
+
+def _disconnect_source(registry: type, source: str) -> None:
+ """Disconnect a specific source."""
+ if not registry.contains(source):
+ console.print(f"[red]Unknown source: {source}[/red]")
+ return
+ cls = registry.get(source)
+ try:
+ instance = cls()
+ instance.disconnect()
+ console.print(f"[green]Disconnected {source}.[/green]")
+ except Exception as exc:
+ console.print(f"[red]Error disconnecting {source}: {exc}[/red]")
+
+
+def _connect_source(registry: type, source: str, *, path: str = "") -> None:
+ """Connect a specific source."""
+ if not registry.contains(source):
+ console.print(f"[red]Unknown source: {source}[/red]")
+ console.print(f"Available: {', '.join(sorted(registry.keys()))}")
+ return
+
+ cls = registry.get(source)
+ auth_type = getattr(cls, "auth_type", "unknown")
+
+ if auth_type == "filesystem":
+ if not path:
+ console.print(f"[red]{source} requires --path argument.[/red]")
+ return
+ from pathlib import Path as P
+
+ if not P(path).is_dir():
+ console.print(f"[red]Path does not exist: {path}[/red]")
+ return
+ instance = cls(vault_path=path)
+ if instance.is_connected():
+ console.print(f"[green]Connected to {source} at {path}[/green]")
+ else:
+ console.print(f"[red]Could not connect to {source} at {path}[/red]")
+
+ elif auth_type == "oauth":
+ instance = cls()
+ if instance.is_connected():
+ console.print(f"[green]{source} is already connected.[/green]")
+ return
+ url = instance.auth_url()
+ console.print(f"Open this URL in your browser to authorize {source}:")
+ console.print(f"[link={url}]{url}[/link]")
+ console.print("[dim]After authorizing, paste the code here.[/dim]")
+
+ else:
+ console.print(f"[yellow]Auth type '{auth_type}' not yet supported in CLI.[/yellow]")
+```
+
+- [ ] **Step 4: Register connect command in CLI __init__.py**
+
+Open `src/openjarvis/cli/__init__.py`. Find where other commands are added (look for `cli.add_command` calls). Add:
+
+```python
+from openjarvis.cli.connect_cmd import connect
+
+cli.add_command(connect, "connect")
+```
+
+Place this alongside the existing `cli.add_command` calls.
+
+- [ ] **Step 5: Run tests**
+
+Run: `cd /lambda/nfs/lambda-stanford/jonsf/scratch_v2/OpenJarvis && uv run pytest tests/cli/test_connect.py -v`
+
+Expected: All 5 tests PASS.
+
+- [ ] **Step 6: Commit**
+
+```bash
+git add src/openjarvis/cli/connect_cmd.py src/openjarvis/cli/__init__.py tests/cli/test_connect.py
+git commit -m "feat: add 'jarvis connect' CLI command for managing data source connections"
+```
+
+---
+
+### Task 10: Integration Test + Wire Up Connectors __init__.py
+
+**Files:**
+- Modify: `src/openjarvis/connectors/__init__.py`
+- Create: `tests/connectors/test_integration.py`
+
+- [ ] **Step 1: Update connectors __init__.py to auto-register**
+
+Open `src/openjarvis/connectors/__init__.py` and replace with:
+
+```python
+"""Data source connectors for Deep Research."""
+
+from openjarvis.connectors._stubs import (
+ Attachment,
+ BaseConnector,
+ Document,
+ SyncStatus,
+)
+
+__all__ = ["Attachment", "BaseConnector", "Document", "SyncStatus"]
+
+# Auto-register built-in connectors
+import openjarvis.connectors.obsidian # noqa: F401
+
+try:
+ import openjarvis.connectors.gmail # noqa: F401
+except ImportError:
+ pass # httpx may not be installed
+```
+
+- [ ] **Step 2: Write integration test**
+
+Create `tests/connectors/test_integration.py`:
+
+```python
+"""Integration test — full pipeline from connector to retrieval."""
+
+from __future__ import annotations
+
+from pathlib import Path
+
+import pytest
+
+from openjarvis.connectors._stubs import Document
+from openjarvis.connectors.obsidian import ObsidianConnector
+from openjarvis.connectors.pipeline import IngestionPipeline
+from openjarvis.connectors.store import KnowledgeStore
+from openjarvis.connectors.sync_engine import SyncEngine
+from openjarvis.tools.knowledge_search import KnowledgeSearchTool
+
+
+def vault(tmp_path: Path) -> Path:
+ vault_dir = tmp_path / "test-vault"
+ vault_dir.mkdir()
+ (vault_dir / "project-notes.md").write_text(
+ "# Project Alpha\n\nWe decided to migrate to Kubernetes in March.\n\n"
+ "## Cost Analysis\n\nEstimated 40% increase in cloud spend during transition.\n\n"
+ "## Timeline\n\nSix-week migration window starting April 1st."
+ )
+ (vault_dir / "meeting.md").write_text(
+ "# Sprint Review\n\nDiscussed budget concerns with Mike and Sarah.\n"
+ "Action item: Sarah to prepare cost comparison document."
+ )
+ return vault_dir
+
+
+def test_full_pipeline_obsidian_to_search(vault: Path, tmp_path: Path) -> None:
+ """End-to-end: Obsidian vault → SyncEngine → KnowledgeStore → knowledge_search."""
+ # 1. Setup
+ store = KnowledgeStore(db_path=str(tmp_path / "integration.db"))
+ pipeline = IngestionPipeline(store=store)
+ engine = SyncEngine(pipeline=pipeline, state_db=str(tmp_path / "state.db"))
+ connector = ObsidianConnector(vault_path=str(vault))
+
+ # 2. Sync
+ engine.sync(connector)
+
+ # 3. Verify checkpoint
+ cp = engine.get_checkpoint("obsidian")
+ assert cp is not None
+ assert cp["items_synced"] == 2
+
+ # 4. Search via knowledge_search tool
+ tool = KnowledgeSearchTool(store=store)
+
+ # Search for Kubernetes content
+ result = tool.execute(query="Kubernetes migration")
+ assert result.success
+ assert "Kubernetes" in result.content
+
+ # Search for budget discussion
+ result = tool.execute(query="budget concerns")
+ assert result.success
+ assert "budget" in result.content.lower()
+
+ # Search with source filter
+ result = tool.execute(query="migration", source="obsidian")
+ assert result.success
+
+ # No results for nonexistent source
+ result = tool.execute(query="migration", source="gmail")
+ assert "No relevant results" in result.content
+```
+
+- [ ] **Step 3: Run the full integration test**
+
+Run: `cd /lambda/nfs/lambda-stanford/jonsf/scratch_v2/OpenJarvis && uv run pytest tests/connectors/test_integration.py -v`
+
+Expected: PASS — the full pipeline works end-to-end.
+
+- [ ] **Step 4: Run ALL connector and tool tests together**
+
+Run: `cd /lambda/nfs/lambda-stanford/jonsf/scratch_v2/OpenJarvis && uv run pytest tests/connectors/ tests/tools/test_knowledge_search.py tests/cli/test_connect.py -v`
+
+Expected: All tests PASS.
+
+- [ ] **Step 5: Run linter**
+
+Run: `cd /lambda/nfs/lambda-stanford/jonsf/scratch_v2/OpenJarvis && uv run ruff check src/openjarvis/connectors/ src/openjarvis/tools/knowledge_search.py src/openjarvis/cli/connect_cmd.py tests/connectors/ tests/tools/test_knowledge_search.py tests/cli/test_connect.py`
+
+Expected: No errors. If there are import ordering or unused import warnings, fix them.
+
+- [ ] **Step 6: Commit**
+
+```bash
+git add src/openjarvis/connectors/__init__.py tests/connectors/test_integration.py
+git commit -m "feat: add integration test and auto-registration for built-in connectors"
+```
+
+---
+
+## Post-Plan Notes
+
+**What this plan produces:** A working connector framework with two connectors (Gmail + Obsidian), a knowledge store with filtered BM25 retrieval, a `knowledge_search` tool for agents, a SyncEngine with checkpointing, and a `jarvis connect` CLI. This is the foundation that Phases 2-5 build on.
+
+**What comes next (separate plans):**
+- **Phase 2:** Desktop setup wizard UI + Slack, Google Drive, Calendar, Contacts, iMessage connectors
+- **Phase 3:** ColBERTv2 disk persistence + DeepResearchAgent + research report UI
+- **Phase 4:** ChannelAgent + iMessage/WhatsApp/Slack plugins
+- **Phase 5:** Incremental sync, attachment store, settings page polish