Self-evolving Context Database for AI Agents. Unify Agent Memory, Knowledge RAG and Skills.
Python agent-contextagent-memoryagent-pluginsagentic-ragcontext-databasedsh-pluginknowledge-basememorymemory-managementself-evolvingvector-database
feat(ingest): add a WorkBuddy log source (#5084)
zhangxin2be6814d44d0
diff --git a/docs/en/agent-integrations/09-log-ingestion.md b/docs/en/agent-integrations/09-log-ingestion.mdindex 2860d93ca..a0a84d404 100644--- a/docs/en/agent-integrations/09-log-ingestion.md+++ b/docs/en/agent-integrations/09-log-ingestion.md@@ -1,6 +1,6 @@# Import Local Agent Logs (openviking-server ingest)-`openviking-server ingest` parses the conversation logs that AI coding / agent harnesses (Claude Code, Codex, OpenCode, Hermes, OpenClaw) already leave on your machine, then "replays" them through OpenViking's existing session pipeline (`create session → batch add messages → commit`, where commit triggers memory extraction). This turns both your historical and newly written conversations into long-term memory. It complements the per-harness memory plugins: a plugin captures **while a conversation is happening**, whereas this tool is for **importing existing logs** and **watching for new logs offline** — no plugin required and no change to the harness itself.+`openviking-server ingest` parses the conversation logs that AI coding / agent harnesses (Claude Code, Codex, WorkBuddy, OpenCode, Hermes, OpenClaw) already leave on your machine, then "replays" them through OpenViking's existing session pipeline (`create session → batch add messages → commit`, where commit triggers memory extraction). This turns both your historical and newly written conversations into long-term memory. It complements the per-harness memory plugins: a plugin captures **while a conversation is happening**, whereas this tool is for **importing existing logs** and **watching for new logs offline** — no plugin required and no change to the harness itself.Key difference from the plugins: this tool is an OpenViking **client**. It runs where the logs live and points at a local or remote server via the SDK, and it is **off by default** — installing OpenViking does not silently scan your local files.@@ -20,6 +20,7 @@ The feature is doubly disabled and must be turned on explicitly:|---|---|---|---|| `claude_code` | Supported | `~/.claude/projects/*/*.jsonl` | append-only JSONL, byte-offset cursor || `codex` | Supported | `~/.codex/sessions/YYYY/MM/DD/rollout-*.jsonl` | append-only JSONL |+| `workbuddy` | Supported | `~/.workbuddy/projects/*/*.jsonl` | append-only JSONL; injects system reminders into user turns — the adapter strips them and keeps only `<user_query>` || `hermes` | Supported | `~/.hermes/sessions/*.jsonl` | group-chat agent; user peer = original username || `openclaw` | Supported | `~/.openclaw/agents/*/sessions/*.jsonl` | group-chat agent; user peer = original username || `opencode` | Experimental | `~/.local/share/opencode/opencode.db` | SQLite, polled by `(time, id)`; the legacy file-store is not supported |@@ -42,6 +43,7 @@ Add an `ingest` section to `ov.conf`, listing the harnesses to import and their"harnesses": {"claude_code": { "enabled": true, "mode": "both" },"codex": { "enabled": true, "mode": "backfill" },+ "workbuddy": { "enabled": true, "mode": "both" },"opencode": { "enabled": false, "mode": "watch", "experimental": true },"hermes": { "enabled": false, "mode": "both", "user_field": "sender" },"openclaw": { "enabled": false, "mode": "both", "user_field": "sender" }diff --git a/docs/zh/agent-integrations/09-log-ingestion.md b/docs/zh/agent-integrations/09-log-ingestion.mdindex 21a945f10..4e4d01902 100644--- a/docs/zh/agent-integrations/09-log-ingestion.md+++ b/docs/zh/agent-integrations/09-log-ingestion.md@@ -1,6 +1,6 @@# 导入本地 Agent 日志(openviking-server ingest)-`openviking-server ingest` 把你本地已有的 AI 编码 / agent harness 的对话日志(Claude Code、Codex、OpenCode、Hermes、OpenClaw)解析成标准消息,再通过 OpenViking 既有的会话管线“重放”进去(`创建会话 → 批量追加消息 → 提交`,提交时触发记忆抽取),从而把这些历史与新增对话沉淀为长期记忆。它与各 harness 的“记忆插件”互补:插件在对话**进行时**实时挂载捕获,而本工具用于**导入既有日志**与**离线监听新增日志**,无需插件、也无需改动对应 harness。+`openviking-server ingest` 把你本地已有的 AI 编码 / agent harness 的对话日志(Claude Code、Codex、WorkBuddy、OpenCode、Hermes、OpenClaw)解析成标准消息,再通过 OpenViking 既有的会话管线“重放”进去(`创建会话 → 批量追加消息 → 提交`,提交时触发记忆抽取),从而把这些历史与新增对话沉淀为长期记忆。它与各 harness 的“记忆插件”互补:插件在对话**进行时**实时挂载捕获,而本工具用于**导入既有日志**与**离线监听新增日志**,无需插件、也无需改动对应 harness。与插件方案的关键区别:本工具是 OpenViking 的**客户端**,跑在日志所在的机器上,通过 SDK 指向本地或远端 server;它默认**完全关闭**,不会“装上就扫你本地文件”。@@ -20,6 +20,7 @@|---|---|---|---|| `claude_code` | 支持 | `~/.claude/projects/*/*.jsonl` | append-only JSONL,字节偏移游标 || `codex` | 支持 | `~/.codex/sessions/YYYY/MM/DD/rollout-*.jsonl` | append-only JSONL |+| `workbuddy` | 支持 | `~/.workbuddy/projects/*/*.jsonl` | append-only JSONL;会把系统提示等注入到 user 轮里,适配器会剥离并只保留 `<user_query>` || `hermes` | 支持 | `~/.hermes/sessions/*.jsonl` | 群聊 agent,user 取原始用户名 || `openclaw` | 支持 | `~/.openclaw/agents/*/sessions/*.jsonl` | 群聊 agent,user 取原始用户名 || `opencode` | 实验性 | `~/.local/share/opencode/opencode.db` | SQLite,按 `(time, id)` 轮询;旧版文件存储暂不支持 |@@ -42,6 +43,7 @@"harnesses": {"claude_code": { "enabled": true, "mode": "both" },"codex": { "enabled": true, "mode": "backfill" },+ "workbuddy": { "enabled": true, "mode": "both" },"opencode": { "enabled": false, "mode": "watch", "experimental": true },"hermes": { "enabled": false, "mode": "both", "user_field": "sender" },"openclaw": { "enabled": false, "mode": "both", "user_field": "sender" }diff --git a/openviking/ingest/sources/__init__.py b/openviking/ingest/sources/__init__.pyindex 7e6ddc659..c006a80f8 100644--- a/openviking/ingest/sources/__init__.py+++ b/openviking/ingest/sources/__init__.py@@ -10,6 +10,7 @@ from openviking.ingest.sources import ( # noqa: F401hermes,openclaw,opencode,+ workbuddy,)-__all__ = ["claude_code", "codex", "cursor", "hermes", "opencode", "openclaw"]+__all__ = ["claude_code", "codex", "cursor", "hermes", "opencode", "openclaw", "workbuddy"]diff --git a/openviking/ingest/sources/workbuddy.py b/openviking/ingest/sources/workbuddy.pynew file mode 100644index 000000000..91c73e256--- /dev/null+++ b/openviking/ingest/sources/workbuddy.py@@ -0,0 +1,257 @@+# Copyright (c) 2026 Beijing Volcano Engine Technology Co., Ltd.+# SPDX-License-Identifier: AGPL-3.0+"""WorkBuddy (Tencent agentic coding assistant) adapter.++Logs: ``~/.workbuddy/projects/<project-slug>/<session-uuid>.jsonl`` (append-only JSONL).++The on-disk layout mirrors Claude Code's, but the record schema does not:++* turns are ``type == "message"`` with a **top-level** ``role``;+* text lives in flat ``content[]`` blocks -- ``input_text`` for user turns,+ ``output_text`` for assistant turns;+* timestamps are epoch **milliseconds**;+* the session title comes from a separate ``type == "ai-title"`` record+ (``aiTitle``), and ``cwd`` / ``sessionId`` ride along on every record.++WorkBuddy also composes each user turn out of host-injected blocks -- system prompt,+project context, quoted history and reminders -- and wraps the human ask in+``<user_query>``. A naive adapter that maps the whole turn to memory therefore+archives the agent's own instructions: on a real corpus the injected block is ~10 KB+per turn, so recall degrades into extracts of the system prompt. ``parse_line``+strips the host blocks, keeps only the ``<user_query>`` body, and drops the turn when+nothing survives (compaction summaries, continuation prompts).+"""++from __future__ import annotations++import json+import re+from pathlib import Path+from typing import Any, Dict, List, Optional, Tuple++from openviking.ingest.models import NormalizedMessage, SessionRef, iso_from_epoch_ms+from openviking.ingest.registry import register_source+from openviking.ingest.sources.base import JsonlLogSource++_TEXT_BLOCK_TYPES = ("input_text", "output_text")++# Host-injected blocks inside a user turn. Stripping them before looking for+# ``<user_query>`` matters because some of them *quote* earlier user turns+# (``<previous_user_message>`` wraps a full ``<user_query>``), and a quoted history+# block must never be mistaken for the current ask.+_HOST_BLOCK_TAGS = (+ "system-reminder",+ "memory_and_skills_reminder",+ "automation_system_reminder",+ "previous_user_message",+ "previous_assistant_message",+ "previous_tool_call",+ "cb_summary",+ "conversation_history_summary",+ "additional_data",+ "user_info",+ "identity_context",+ "craft_mode",+ "connector-status",+)+_HOST_BLOCK_RE = re.compile(+ r"<(?P<tag>" + "|".join(_HOST_BLOCK_TAGS) + r")(?:\s[^>]*)?>.*?</(?P=tag)\s*>",+ re.DOTALL | re.IGNORECASE,+)+_USER_QUERY_RE = re.compile(r"<user_query\s*>(.*?)</user_query\s*>", re.DOTALL | re.IGNORECASE)++# Discovery runs on every poll, so the metadata is cached on (mtime, size). It is read+# from two bounded windows rather than one, because the fields do not live in the same+# place: ``sessionId`` / ``cwd`` / ``timestamp`` ride along on every record (always+# line 1), while the title is *appended* -- see ``_TITLE_TAIL_BYTES``.+_PEEK_MAX_LINES = 2000+_PEEK_MAX_BYTES = 4 * 1024 * 1024+_PEEK_CACHE_MAX_ENTRIES = 4096+_PEEK_CACHE: Dict[str, Tuple[Tuple[float, int], Dict[str, Any]]] = {}++# WorkBuddy re-emits an ``ai-title`` record every time it re-titles the session, so a log+# can hold several and only the **last** is the settled name -- the earlier ones are the+# name the session carried before it found its topic. On a real corpus the newest title sat+# at a 7.3 MB / 8.2 MB offset of a 9 MB log, far outside the head budget above, so the+# title is read from a second bounded window at the *tail*: the log is append-only, so the+# newest title is the last one written. Neither window alone suffices -- of 40 titled+# sessions, 36 had the settled title inside both, 2 only in the head (never re-titled), and+# 2 only in the tail -- hence the tail result wins and the head result stands as fallback.+# Logs smaller than the window are covered whole.+_TITLE_TAIL_BYTES = 4 * 1024 * 1024+++def _blocks_text(content: Any) -> str:+ """Join the text blocks of a ``content`` array (or accept a bare string)."""+ if isinstance(content, str):+ return content.strip()+ if not isinstance(content, list):+ return ""+ chunks: List[str] = []+ for block in content:+ if isinstance(block, dict) and block.get("type") in _TEXT_BLOCK_TYPES:+ text = block.get("text")+ if isinstance(text, str) and text.strip():+ chunks.append(text.strip())+ return "\n".join(chunks)+++def user_turn_text(raw: str) -> str:+ """Return the human ask from a host-composed user turn, or ``""`` if there is none.++ WorkBuddy wraps the actual prompt in ``<user_query>`` and surrounds it with+ injected context. Everything else is host-authored, so a turn that has no+ ``<user_query>`` at all (compaction summary, "continue from the summary") carries+ no human text and must be dropped instead of being stored as a memory.+ """+ if not raw:+ return ""+ stripped = _HOST_BLOCK_RE.sub(" ", raw)+ queries = [match.strip() for match in _USER_QUERY_RE.findall(stripped)]+ queries = [query for query in queries if query]+ if len(queries) == 1:+ return queries[0]+ if not queries:+ return ""+ # Unexpected shape (the corpus never hits it): prefer the longest candidate rather+ # than silently dropping a turn that probably carries the real ask.+ return max(queries, key=len)+++@register_source("workbuddy")+class WorkBuddySource(JsonlLogSource):+ file_glob = "*/*.jsonl"++ def default_paths(self) -> List[Path]:+ return [Path.home() / ".workbuddy" / "projects"]++ def session_ref_for_file(self, path: Path) -> SessionRef:+ meta = self._peek_meta(path)+ return SessionRef(+ harness=self.name,+ native_session_id=meta.get("session_id") or path.stem,+ locator=str(path),+ title=meta.get("title"),+ started_at=meta.get("started_at"),+ meta={"model": meta.get("model"), "cwd": meta.get("cwd")},+ )++ @staticmethod+ def _peek_meta(path: Path) -> Dict[str, Any]:+ """Best-effort session metadata, cached on ``(mtime, size)``."""+ try:+ st = path.stat()+ except OSError:+ return {}+ key = str(path)+ signature = (st.st_mtime, st.st_size)+ cached = _PEEK_CACHE.get(key)+ if cached is not None and cached[0] == signature:+ return cached[1]++ meta: Dict[str, Any] = {}+ consumed = 0+ try:+ with open(path, "rb") as handle:+ for index, raw in enumerate(handle):+ if index >= _PEEK_MAX_LINES or consumed >= _PEEK_MAX_BYTES:+ break+ consumed += len(raw)+ stripped = raw.strip()+ if not stripped:+ continue+ try:+ obj = json.loads(stripped)+ except ValueError:+ continue+ if not isinstance(obj, dict):+ continue+ if not meta.get("session_id") and isinstance(obj.get("sessionId"), str):+ meta["session_id"] = obj["sessionId"]+ if not meta.get("cwd") and isinstance(obj.get("cwd"), str):+ meta["cwd"] = obj["cwd"]+ if meta.get("started_at") is None and obj.get("timestamp") is not None:+ meta["started_at"] = iso_from_epoch_ms(obj.get("timestamp"))+ if meta.get("title") is None and obj.get("type") == "ai-title":+ # Provisional: superseded below if the log holds a newer title.+ candidate = obj.get("aiTitle")+ if isinstance(candidate, str) and candidate.strip():+ meta["title"] = candidate.strip()+ if meta.get("model") is None and obj.get("role") == "assistant":+ provider_data = obj.get("providerData")+ if isinstance(provider_data, dict) and provider_data.get("model"):+ meta["model"] = provider_data["model"]+ if all(+ meta.get(field)+ for field in ("session_id", "cwd", "started_at", "title", "model")+ ):+ break+ except OSError:+ return {}++ tail_title = WorkBuddySource._peek_title(path, signature[1])+ if tail_title is not None:+ meta["title"] = tail_title++ if len(_PEEK_CACHE) >= _PEEK_CACHE_MAX_ENTRIES:+ _PEEK_CACHE.clear()+ _PEEK_CACHE[key] = (signature, meta)+ return meta++ @staticmethod+ def _peek_title(path: Path, size: int) -> Optional[str]:+ """The last non-empty ``ai-title`` in a bounded window at the tail of the log."""+ start = max(0, size - _TITLE_TAIL_BYTES)+ title: Optional[str] = None+ try:+ with open(path, "rb") as handle:+ if start:+ handle.seek(start)+ # The window can open in the middle of a record, so drop that first+ # partial line instead of trying to salvage it.+ handle.readline()+ for raw in handle:+ stripped = raw.strip()+ if not stripped:+ continue+ try:+ obj = json.loads(stripped)+ except ValueError:+ continue+ if not isinstance(obj, dict) or obj.get("type") != "ai-title":+ continue+ candidate = obj.get("aiTitle")+ if isinstance(candidate, str) and candidate.strip():+ title = candidate.strip()+ except OSError:+ return None+ return title++ def parse_line(self, obj: Dict[str, Any], ref: SessionRef) -> List[NormalizedMessage]:+ if obj.get("type") != "message":+ return []+ role = obj.get("role")+ if role not in ("user", "assistant"):+ return []+ text = _blocks_text(obj.get("content"))+ if not text:+ return []+ if role == "user":+ # Keep only what the human actually typed; host-composed turns yield "".+ text = user_turn_text(text)+ if not text:+ return []++ provider_data = obj.get("providerData")+ model = provider_data.get("model") if isinstance(provider_data, dict) else None+ cwd = obj.get("cwd") or ref.meta.get("cwd")+ peer = self.assistant_peer(model) if role == "assistant" else self.user_peer(cwd=cwd)+ return [+ NormalizedMessage(+ role=role,+ text=text,+ created_at=iso_from_epoch_ms(obj.get("timestamp")),+ peer_id=peer,+ meta={"model": model, "cwd": cwd},+ )+ ]diff --git a/tests/ingest/test_parsers.py b/tests/ingest/test_parsers.pyindex 3b7d83e60..aa9ba6803 100644--- a/tests/ingest/test_parsers.py+++ b/tests/ingest/test_parsers.py@@ -10,6 +10,7 @@ from openviking.ingest.sources.codex import CodexSourcefrom openviking.ingest.sources.hermes import HermesSourcefrom openviking.ingest.sources.openclaw import OpenClawSourcefrom openviking.ingest.sources.opencode import OpenCodeSource+from openviking.ingest.sources.workbuddy import WorkBuddySource, user_turn_textfrom openviking_cli.utils.config.ingest_config import IngestHarnessConfig@@ -117,6 +118,147 @@ def test_codex(tmp_path):assert msgs[1].peer_id == "codex__openai"+def _workbuddy_turn(role, text, ts_ms, model=None):+ record = {+ "id": f"m-{ts_ms}",+ "timestamp": ts_ms,+ "type": "message",+ "role": role,+ "content": [{"type": "input_text" if role == "user" else "output_text", "text": text}],+ "sessionId": "wb-session",+ "cwd": "/tmp/wb-project",+ }+ if model:+ record["providerData"] = {"model": model}+ return record+++def test_workbuddy_keeps_only_the_user_query(tmp_path):+ """Host-injected context and quoted history must not become memories."""+ root = tmp_path / "projects"+ _write_jsonl(+ root / "proj" / "wb-session.jsonl",+ [+ {"timestamp": 1757000000000, "type": "file-history-snapshot", "cwd": "/tmp/wb-project"},+ {"timestamp": 1757000001000, "type": "ai-title", "aiTitle": "Refactor the parser"},+ _workbuddy_turn(+ "user",+ '<system-reminder data-role="user-context">\n'+ "You are a coding agent. Project Context: ... <user_query>example</user_query>\n"+ "</system-reminder>\n"+ "<memory_and_skills_reminder>prefer the repo skill</memory_and_skills_reminder>\n"+ "<previous_user_message><user_query>an older ask</user_query></previous_user_message>\n"+ "<current_time>2026-09-05T10:00:00Z</current_time>\n"+ "<user_query>rename the ingest cursor helper</user_query>",+ 1757000002000,+ ),+ _workbuddy_turn("assistant", "Done — renamed it.", 1757000003000, model="glm-5.2"),+ ],+ )+ src = WorkBuddySource(_cfg(root), fallback_user="tester")+ ref, msgs, _ = _read_all(src)+ assert ref.native_session_id == "wb-session"+ assert ref.title == "Refactor the parser"+ assert ref.meta["cwd"] == "/tmp/wb-project"+ assert [(m.role, m.text) for m in msgs] == [+ ("user", "rename the ingest cursor helper"),+ ("assistant", "Done — renamed it."),+ ]+ # epoch milliseconds are converted to ISO-8601 UTC+ assert msgs[0].created_at == "2025-09-04T15:33:22+00:00"+ assert msgs[1].peer_id == "workbuddy__glm-5.2"+++def test_workbuddy_drops_host_generated_turns(tmp_path):+ """A turn with no <user_query> carries no human text and must be dropped."""+ root = tmp_path / "projects"+ _write_jsonl(+ root / "proj" / "wb-session.jsonl",+ [+ _workbuddy_turn(+ "user",+ "<conversation_history_summary>Summary of the conversation so far.</conversation_history_summary>"+ "<additional_data>noise</additional_data>",+ 1757000000000,+ ),+ _workbuddy_turn(+ "user",+ "Please continue with the conversation based on the summarized context above.",+ 1757000001000,+ ),+ _workbuddy_turn("user", "<user_query>the real ask</user_query>", 1757000002000),+ ],+ )+ src = WorkBuddySource(_cfg(root), fallback_user="tester")+ _, msgs, _ = _read_all(src)+ assert [(m.role, m.text) for m in msgs] == [("user", "the real ask")]+++def test_workbuddy_quoted_user_query_does_not_win():+ """A quoted <user_query> inside <previous_user_message> is history, not the ask."""+ quoted = "<previous_user_message><user_query>the first thing I ever asked</user_query></previous_user_message>"+ assert user_turn_text(f"{quoted}<user_query>the current ask</user_query>") == "the current ask"+ # nothing but injected blocks -> no human text at all+ assert user_turn_text("<system-reminder>ctx</system-reminder>") == ""+ assert user_turn_text("") == ""+++def test_workbuddy_unknown_records_are_ignored(tmp_path):+ root = tmp_path / "projects"+ _write_jsonl(+ root / "proj" / "wb-session.jsonl",+ [+ {"timestamp": 1757000000000, "type": "reasoning", "content": "thinking"},+ {"timestamp": 1757000001000, "type": "function_call", "name": "read_file"},+ {+ "timestamp": 1757000002000,+ "type": "function_call_result",+ "name": "read_file",+ "output": "file body",+ },+ _workbuddy_turn("user", "<user_query>ship it</user_query>", 1757000003000),+ ],+ )+ src = WorkBuddySource(_cfg(root), fallback_user="tester")+ _, msgs, _ = _read_all(src)+ assert [(m.role, m.text) for m in msgs] == [("user", "ship it")]+++def test_workbuddy_finds_title_beyond_the_head(tmp_path):+ """``ai-title`` can be written far into the file, not just in the first record."""+ root = tmp_path / "projects"+ filler = [+ _workbuddy_turn("assistant", f"filler {i}", 1757000000000 + i, model="hy3")+ for i in range(600)+ ]+ _write_jsonl(+ root / "proj" / "wb-session.jsonl",+ filler + [{"timestamp": 1757009999999, "type": "ai-title", "aiTitle": "Late title"}],+ )+ src = WorkBuddySource(_cfg(root), fallback_user="tester")+ refs = list(src.discover_sessions())+ assert [r.title for r in refs] == ["Late title"]+++def test_workbuddy_latest_ai_title_wins(tmp_path):+ """The host re-emits ``ai-title`` when it re-titles a session; the last one is the name."""+ root = tmp_path / "projects"+ _write_jsonl(+ root / "proj" / "wb-session.jsonl",+ [+ {"timestamp": 1757000000000, "type": "ai-title", "aiTitle": "First guess"},+ _workbuddy_turn(+ "user", "<user_query>look at the installer</user_query>", 1757000001000+ ),+ _workbuddy_turn("assistant", "on it", 1757000002000, model="hy3"),+ {"timestamp": 1757000003000, "type": "ai-title", "aiTitle": "Settled name"},+ ],+ )+ src = WorkBuddySource(_cfg(root), fallback_user="tester")+ refs = list(src.discover_sessions())+ assert [r.title for r in refs] == ["Settled name"]++def test_hermes_group_username(tmp_path):root = tmp_path / "sessions"_write_jsonl(