Personal AI, On Personal Devices
Python
fix: wire skills and traces into jarvis ask (#920)
Elliot Slusky4916c296eb53
diff --git a/docs/tutorials/skills-workflow.md b/docs/tutorials/skills-workflow.mdindex 77fd3251..e3ffdc7b 100644--- a/docs/tutorials/skills-workflow.md+++ b/docs/tutorials/skills-workflow.md@@ -16,8 +16,8 @@ OpenJarvis can import skills from the [Hermes Agent](https://github.com/NousRese```bash# Install individual skills+jarvis skill install hermes:llm-wikijarvis skill install hermes:arxiv-jarvis skill install hermes:github-pr-workflow# Or bulk install an entire categoryjarvis skill sync hermes --category research@@ -35,10 +35,10 @@ You should see a table with each skill's name, description, version, and tags.## Step 2: Inspect an Installed Skill-Let's look at what the `arxiv` skill contains:+Let's look at what the `llm-wiki` skill contains:```bash-jarvis skill info arxiv+jarvis skill info llm-wiki```This shows the skill's metadata — author, description, tags, capabilities, whether it has structured steps or markdown instructions, and its invocation flags.@@ -46,40 +46,40 @@ This shows the skill's metadata — author, description, tags, capabilities, wheYou can also inspect the raw SKILL.md:```bash-cat ~/.openjarvis/skills/hermes/arxiv/SKILL.md | head -40+cat ~/.openjarvis/skills/hermes/llm-wiki/SKILL.md | head -40```The `.source` file records provenance:```bash-cat ~/.openjarvis/skills/hermes/arxiv/.source+cat ~/.openjarvis/skills/hermes/llm-wiki/.source```-This shows the source (`hermes:arxiv`), the git commit it was imported from, which tool names were translated (e.g., `Edit→file_edit`), and the install timestamp.+This shows the source (`hermes:llm-wiki`), the git commit it was imported from, which tool names were translated (e.g., `Edit→file_edit`), and the install timestamp.## Step 3: Use Skills with an AgentNow let's ask the agent a question that should trigger skill usage:```bash-jarvis ask "Use the code-explainer skill to explain this Python code: for i in range(5): print(i*2)" \+jarvis ask --agent orchestrator "Use the llm-wiki skill to explain how model entries should be organized" \--engine ollama --model qwen3.5:9b```The agent will:-1. See the skill catalog in its system prompt-2. Decide to invoke `skill_code-explainer`+1. See the skill catalog and `skill_llm-wiki` tool definition+2. Decide to invoke `skill_llm-wiki`3. Receive the markdown instructions from the skill-4. Follow the 5-step pattern to explain the code+4. Follow those instructions in its answer-Try a pipeline skill too:+Try the other installed skill too:```bash-jarvis ask "Use the math-solver skill to compute 17 * 23" \+jarvis ask --agent orchestrator "Use the arxiv skill to outline how you would research transformer efficiency" \--engine ollama --model qwen3.5:9b```-This time the agent invokes `skill_math-solver`, which executes a deterministic pipeline (calling the `calculator` tool internally) and returns the computed result directly.+The `simple` agent intentionally cannot call tools. Use `orchestrator` or another tool-capable agent whenever you want a skill to be invoked.## Step 4: Create Your Own Skill@@ -124,7 +124,7 @@ jarvis skill listYou should see `my-reviewer` in the table. Try it:```bash-jarvis ask "Use the my-reviewer skill to review this function: def login(user, pwd): return db.query(f'SELECT * FROM users WHERE name={user} AND pass={pwd}')" \+jarvis ask --agent orchestrator "Use the my-reviewer skill to review this function: def login(user, pwd): return db.query(f'SELECT * FROM users WHERE name={user} AND pass={pwd}')" \--engine ollama --model qwen3.5:9b```@@ -135,15 +135,19 @@ The agent should follow the security-first approach and flag the SQL injection vFor the learning loop to work, we need traces. Run several queries that use skills:```bash+# Tracing is disabled in the generated config until you opt in+jarvis config set traces.enabled true+# Generate a few traces-jarvis ask "Use math-solver to compute 100 / 7"-jarvis ask "Use code-explainer to explain: lambda x: x**2"-jarvis ask "Use my-reviewer to review: def add(a,b): return a+b"-jarvis ask "Use math-solver to compute 2**10"-jarvis ask "Use code-explainer to explain: [x for x in range(10) if x % 2 == 0]"+jarvis ask --agent orchestrator "Use llm-wiki to describe a model entry"+jarvis ask --agent orchestrator "Use my-reviewer to review: def add(a,b): return a+b"+jarvis ask --agent orchestrator "Use llm-wiki to describe an evaluation entry"+jarvis ask --agent orchestrator "Use my-reviewer to review: lambda x: x**2"+jarvis ask --agent orchestrator "Use llm-wiki to describe a dataset entry"+jarvis ask --agent orchestrator "Use my-reviewer to review: def square(x): return x*x"```-Each query produces a trace in `~/.openjarvis/traces.db` with skill metadata tags (`skill`, `skill_source`, `skill_kind`).+Each query produces one trace in `~/.openjarvis/traces.db`. An invoked skill's tool-call step includes the `skill`, `skill_source`, and `skill_kind` metadata tags.## Step 6: Discover Patterns from Traces@@ -151,21 +155,23 @@ Mine the trace store for recurring tool sequences:```bash# Preview without writing-jarvis skill discover --dry-run --min-frequency 2+jarvis skill discover --dry-run --min-frequency 2 --min-outcome 0# Write discovered patterns as skill manifests-jarvis skill discover --min-frequency 2+jarvis skill discover --min-frequency 2 --min-outcome 0```Discovered skills land in `~/.openjarvis/skills/discovered/` and automatically appear in `jarvis skill list` on the next session.+Discovery requires recurring sequences of at least two tool calls. `--min-outcome 0` includes new traces that have not yet been scored.+## Step 7: Optimize Skills with DSPyOnce you have enough traces (at least 3-5 per skill), run the optimizer:```bash# Preview what would be optimized-jarvis optimize skills --dry-run+jarvis optimize skills --dry-run --min-traces 3# Run DSPy optimizationjarvis optimize skills --policy dspy --min-traces 3@@ -176,8 +182,8 @@ This produces overlay files at `~/.openjarvis/learning/skills/<skill-name>/optimInspect what was produced:```bash-jarvis skill show-overlay math-solver-jarvis skill show-overlay code-explainer+jarvis skill show-overlay llm-wiki+jarvis skill show-overlay my-reviewer```The next time you run a query, the agent sees the optimized descriptions and few-shot examples in its system prompt.@@ -220,9 +226,9 @@ Now skills are automatically synced from Hermes on session start, and the optimi| Concept | What you did ||---------|-------------|| **Installing skills** | `jarvis skill install hermes:arxiv` — imported from public sources |-| **Using skills** | `jarvis ask "Use the code-explainer skill..."` — agent invokes skills as tools |+| **Using skills** | `jarvis ask --agent orchestrator "Use the llm-wiki skill..."` — a tool-capable agent invokes skills || **Creating skills** | Wrote a `SKILL.md` with YAML frontmatter and markdown instructions |-| **Generating traces** | Ran skill-using queries to populate the trace store |+| **Generating traces** | Enabled tracing and ran skill-using queries to populate the trace store || **Discovering patterns** | `jarvis skill discover` — mined traces for recurring tool sequences || **Optimizing skills** | `jarvis optimize skills --policy dspy` — improved descriptions + few-shot examples || **Benchmarking** | `jarvis bench skills` — measured the impact across 4 conditions |diff --git a/src/openjarvis/agents/orchestrator.py b/src/openjarvis/agents/orchestrator.pyindex 22ee17fb..9d1e5744 100644--- a/src/openjarvis/agents/orchestrator.py+++ b/src/openjarvis/agents/orchestrator.py@@ -65,6 +65,7 @@ class OrchestratorAgent(ToolUsingAgent):parallel_tools: bool = True,interactive: bool = False,confirm_callback=None,+ capability_policy: Optional[Any] = None,before_tool_call: Optional[Callable[[str, dict[str, Any]], bool]] = None,) -> None:super().__init__(@@ -77,6 +78,7 @@ class OrchestratorAgent(ToolUsingAgent):max_tokens=max_tokens,interactive=interactive,confirm_callback=confirm_callback,+ capability_policy=capability_policy,prompt_builder=prompt_builder,)self._mode = modediff --git a/src/openjarvis/cli/ask.py b/src/openjarvis/cli/ask.pyindex 7c06e71b..464f590c 100644--- a/src/openjarvis/cli/ask.py+++ b/src/openjarvis/cli/ask.py@@ -369,6 +369,7 @@ def _run_agent()agent_cls = AgentRegistry.get(agent_name)+ accepts_tools = getattr(agent_cls, "accepts_tools", False)# Build tools — local registry tools + MCP server tools from config# (#461 — MCP tools were silently dropped because ask.py only used@@ -400,13 +401,58 @@ def _run_agent(tools.append(t)existing.add(t.spec.name)+ skill_manager = None+ skill_catalog_xml = None+ skill_few_shot_examples: list[str] = []+ skill_tools_added = False+ if accepts_tools and config.skills.enabled:+ try:+ from pathlib import Path++ from openjarvis.skills.manager import SkillManager+ from openjarvis.tools._stubs import ToolExecutor++ pipeline_executor = ToolExecutor(+ tools,+ bus,+ capability_policy=capability_policy,+ agent_id=getattr(agent_cls, "agent_id", agent_name),+ interactive=True,+ confirm_callback=lambda prompt: True,+ )+ skill_manager = SkillManager(+ bus,+ capability_policy=capability_policy,+ )+ skill_paths = [Path(config.skills.skills_dir).expanduser()]+ workspace_skills = Path("./skills")+ if workspace_skills.exists():+ skill_paths.insert(0, workspace_skills)+ skill_manager.discover(paths=skill_paths)+ skill_manager.set_tool_executor(pipeline_executor)++ existing = {tool.spec.name for tool in tools}+ for skill_tool in skill_manager.get_skill_tools(+ tool_executor=pipeline_executor,+ ):+ if skill_tool.spec.name not in existing:+ tools.append(skill_tool)+ existing.add(skill_tool.spec.name)+ skill_tools_added = True++ if skill_tools_added:+ skill_catalog_xml = skill_manager.get_catalog_xml()+ skill_few_shot_examples = skill_manager.get_few_shot_examples()+ except Exception as exc:+ logger.warning("Failed to initialize skills: %s", exc)+# Build agent with appropriate kwargsagent_kwargs = {"bus": bus,"temperature": temperature,"max_tokens": max_tokens,}- if getattr(agent_cls, "accepts_tools", False):+ if accepts_tools:agent_kwargs["tools"] = toolsagent_kwargs["max_turns"] = config.agent.max_turnsagent_kwargs["interactive"] = True@@ -427,6 +473,8 @@ def _run_agent(agent_template=config.agent.default_system_prompt or "",memory_files_config=memory_files_config or config.memory_files,system_prompt_config=config.system_prompt,+ skill_catalog_xml=skill_catalog_xml,+ skill_few_shot_examples=skill_few_shot_examples,)agent = agent_cls(engine, model_name, **agent_kwargs)@@ -436,6 +484,8 @@ def _run_agent(# adversarial review caught this).if mcp_clients:agent._mcp_clients = mcp_clients+ if skill_manager is not None:+ agent._skill_manager = skill_managerctx = AgentContext()# Inject memory context into conversation if available@@ -463,7 +513,27 @@ def _run_agent(except Exception as exc:logger.warning("Failed to inject memory context for agent: %s", exc)- return agent.run(query_text, context=ctx)+ trace_store = None+ if config.traces.enabled:+ try:+ from openjarvis.traces.store import TraceStore++ trace_store = TraceStore(config.traces.db_path)+ except Exception as exc:+ logger.warning("Failed to initialize TraceStore: %s", exc)++ try:+ if trace_store is not None:+ from openjarvis.traces.collector import TraceCollector++ return TraceCollector(agent, store=trace_store, bus=bus).run(+ query_text,+ context=ctx,+ )+ return agent.run(query_text, context=ctx)+ finally:+ if trace_store is not None:+ trace_store.close()def _print_profile(diff --git a/src/openjarvis/skills/manager.py b/src/openjarvis/skills/manager.pyindex 32eba401..955bc52d 100644--- a/src/openjarvis/skills/manager.py+++ b/src/openjarvis/skills/manager.py@@ -193,6 +193,8 @@ class SkillManager:tools: List[BaseTool] = []for manifest in self._skills.values():+ if manifest.disable_model_invocation:+ continuereal_executor = executor or _NullToolExecutor()skill_exec = SkillExecutor(real_executor, bus=self._bus)@@ -250,6 +252,8 @@ class SkillManager:"""examples: List[str] = []for name, manifest in self._skills.items():+ if manifest.disable_model_invocation:+ continueoj = manifest.metadata.get("openjarvis", {}) if manifest.metadata else {}few_shot = oj.get("few_shot", []) or []for ex in few_shot:diff --git a/src/openjarvis/traces/store.py b/src/openjarvis/traces/store.pyindex 8527e051..493ebd9d 100644--- a/src/openjarvis/traces/store.py+++ b/src/openjarvis/traces/store.py@@ -81,7 +81,11 @@ class TraceStore:"""Append-only SQLite store for interaction traces."""def __init__(self, db_path: str | Path) -> None:- self._db_path = str(db_path)+ self._db_path = (+ ":memory:"+ if str(db_path) == ":memory:"+ else str(Path(db_path).expanduser())+ )if self._db_path != ":memory:":from openjarvis.security.file_utils import secure_creatediff --git a/tests/agents/test_orchestrator.py b/tests/agents/test_orchestrator.pyindex 124c569f..7d677284 100644--- a/tests/agents/test_orchestrator.py+++ b/tests/agents/test_orchestrator.py@@ -151,6 +151,15 @@ class TestOrchestratorAgent:agent = OrchestratorAgent(engine, "test-model")assert agent.agent_id == "orchestrator"+ def test_capability_policy_is_forwarded_to_tool_executor(self):+ policy = MagicMock()+ agent = OrchestratorAgent(+ _make_engine_no_tools(),+ "test-model",+ capability_policy=policy,+ )+ assert agent._executor._capability_policy is policy+def test_no_tools_single_turn(self):engine = _make_engine_no_tools("Hello!")agent = OrchestratorAgent(engine, "test-model")diff --git a/tests/cli/test_ask_agent.py b/tests/cli/test_ask_agent.pyindex 73abdaf9..49d7480c 100644--- a/tests/cli/test_ask_agent.py+++ b/tests/cli/test_ask_agent.py@@ -11,7 +11,7 @@ from click.testing import CliRunnerfrom openjarvis.agents._stubs import AgentContext, AgentResult, ToolUsingAgentfrom openjarvis.cli import cli-from openjarvis.core.types import ToolCall, ToolResult+from openjarvis.core.types import Role, StepType, ToolCall, ToolResultfrom openjarvis.tools._stubs import BaseTool, ToolSpec_ask_mod = importlib.import_module("openjarvis.cli.ask")@@ -114,6 +114,7 @@ def agent_setup():config = JarvisConfig()config.intelligence.default_model = "test-model"config.agent.max_turns = 3+ config.traces.enabled = FalseAgentRegistry.register_value("confirming_agent", _ConfirmingAgent)ToolRegistry.register_value("dangerous", _DangerousTool)@@ -154,7 +155,9 @@ def mock_setup():):from openjarvis.core.config import JarvisConfig- mock_cfg.return_value = JarvisConfig()+ config = JarvisConfig()+ config.traces.enabled = False+ mock_cfg.return_value = configmock_ge.return_value = ("mock", engine)mock_de.return_value = [("mock", engine)]mock_dm.return_value = {"mock": ["test-model"]}@@ -239,6 +242,7 @@ class TestAskAgentOption:from openjarvis.core.config import JarvisConfigcfg = JarvisConfig()+ cfg.traces.enabled = Falsecfg.agent.default_agent = ""monkeypatch.setattr(_ask_mod, "load_config", lambda *a, **kw: cfg)result = runner.invoke(cli, ["ask", "Hello"])@@ -287,6 +291,112 @@ class TestAskAgentOption:agent_setup.engine.generate.assert_not_called()+class TestAskSkillsAndTraces:+ def test_orchestrator_discovers_invokes_and_traces_skill(+ self,+ runner,+ tmp_path,+ ):+ from openjarvis.core.config import JarvisConfig+ from openjarvis.traces.store import TraceStore++ skill_dir = tmp_path / "skills" / "hermes" / "llm-wiki"+ skill_dir.mkdir(parents=True)+ sentinel = "LLM_WIKI_INSTRUCTIONS_SENTINEL"+ (skill_dir / "SKILL.md").write_text(+ "---\n"+ "name: llm-wiki\n"+ "description: Maintain a local LLM knowledge wiki\n"+ "---\n"+ f"{sentinel}: update the wiki carefully.\n",+ encoding="utf-8",+ )+ (skill_dir / ".source").write_text(+ 'source = "hermes:llm-wiki"\n',+ encoding="utf-8",+ )++ config = JarvisConfig()+ config.intelligence.default_model = "test-model"+ config.agent.context_from_memory = False+ config.security.enabled = False+ config.telemetry.enabled = False+ config.skills.enabled = True+ config.skills.skills_dir = str(tmp_path / "skills")+ config.traces.enabled = True+ config.traces.db_path = str(tmp_path / "traces.db")++ engine = _mock_engine()+ engine.generate.side_effect = [+ {+ "content": "",+ "tool_calls": [+ {+ "id": "call_skill",+ "name": "skill_llm-wiki",+ "arguments": '{"task": "Update the model page"}',+ }+ ],+ "usage": {"total_tokens": 8},+ },+ {+ "content": "Wiki updated",+ "tool_calls": [],+ "usage": {"total_tokens": 8},+ },+ ]+ _register_agents()++ with (+ patch.object(_ask_mod, "load_config", return_value=config),+ patch.object(_ask_mod, "get_engine", return_value=("mock", engine)),+ patch.object(+ _ask_mod,+ "discover_engines",+ return_value=[("mock", engine)],+ ),+ patch.object(+ _ask_mod,+ "discover_models",+ return_value={"mock": ["test-model"]},+ ),+ patch.object(_ask_mod, "register_builtin_models"),+ patch.object(_ask_mod, "merge_discovered_models"),+ ):+ result = runner.invoke(+ cli,+ ["ask", "--agent", "orchestrator", "Update my LLM wiki"],+ )++ assert result.exit_code == 0, result.output+ first_call = engine.generate.call_args_list[0]+ assert any(+ schema["function"]["name"] == "skill_llm-wiki"+ for schema in first_call.kwargs["tools"]+ )+ assert "<available_skills>" in first_call.args[0][0].content++ tool_messages = [+ message+ for message in engine.generate.call_args_list[1].args[0]+ if message.role == Role.TOOL+ ]+ assert any(sentinel in message.content for message in tool_messages)++ store = TraceStore(config.traces.db_path)+ traces = store.list_traces()+ store.close()+ assert len(traces) == 1+ assert [step.step_type for step in traces[0].steps] == [+ StepType.GENERATE,+ StepType.TOOL_CALL,+ StepType.GENERATE,+ StepType.RESPOND,+ ]+ assert traces[0].steps[1].metadata["skill"] == "llm-wiki"+ assert traces[0].steps[1].metadata["skill_source"] == "hermes"++class TestBuildTools:def test_build_calculator(self, mock_setup):from openjarvis.cli.ask import _build_tools@@ -357,6 +467,7 @@ class TestPersonaFilesReachModel:user.write_text("USER_SENTINEL", encoding="utf-8")cfg = JarvisConfig()+ cfg.traces.enabled = Falsecfg.memory_files.soul_path = str(soul)cfg.memory_files.memory_path = str(memory)cfg.memory_files.user_path = str(user)diff --git a/tests/cli/test_ask_e2e.py b/tests/cli/test_ask_e2e.pyindex 3f833d14..e2a9e50d 100644--- a/tests/cli/test_ask_e2e.py+++ b/tests/cli/test_ask_e2e.py@@ -43,6 +43,7 @@ def _patch_ask(monkeypatch, tmp_path, *, engine_result=None, no_engine=False):cfg = JarvisConfig()cfg.telemetry.db_path = str(tmp_path / "telemetry.db")+ cfg.traces.enabled = Falsemonkeypatch.setattr(_ask_mod, "load_config", lambda: cfg)diff --git a/tests/skills/test_manager.py b/tests/skills/test_manager.pyindex c43d2a80..7595d9d1 100644--- a/tests/skills/test_manager.py+++ b/tests/skills/test_manager.py@@ -310,6 +310,29 @@ class TestSkillManagerTools:tools = mgr.get_skill_tools()assert any(t.spec.name == "skill_my_skill" for t in tools)+ def test_model_disabled_skill_is_not_exposed(self, tmp_path: Path) -> None:+ skill_dir = tmp_path / "hidden"+ skill_dir.mkdir()+ (skill_dir / "SKILL.md").write_text(+ "---\n"+ "name: hidden\n"+ "disable_model_invocation: true\n"+ "metadata:\n"+ " openjarvis:\n"+ " few_shot:\n"+ " - input: secret\n"+ " output: secret\n"+ "---\n"+ "Hidden instructions\n",+ encoding="utf-8",+ )++ mgr = SkillManager(bus=EventBus())+ mgr.discover(paths=[tmp_path])++ assert mgr.get_skill_tools() == []+ assert mgr.get_few_shot_examples() == []+# ---------------------------------------------------------------------------# TestSkillManagerResolvediff --git a/tests/traces/test_store.py b/tests/traces/test_store.pyindex b2863ede..138f0471 100644--- a/tests/traces/test_store.py+++ b/tests/traces/test_store.py@@ -55,6 +55,23 @@ class TestTraceStore:assert store.count() == 0store.close()+ def test_expands_user_in_db_path(+ self,+ tmp_path: Path,+ monkeypatch,+ ) -> None:+ home = tmp_path / "home"+ cwd = tmp_path / "cwd"+ cwd.mkdir()+ monkeypatch.setenv("HOME", str(home))+ monkeypatch.chdir(cwd)++ store = TraceStore("~/.openjarvis/traces.db")+ store.close()++ assert (home / ".openjarvis" / "traces.db").exists()+ assert not (cwd / "~").exists()+def test_save_and_get(self, tmp_path: Path) -> None:store = TraceStore(tmp_path / "test.db")trace = _make_trace()