Skip to content

Personal AI, On Personal Devices

Python

fix: wire skills and traces into jarvis ask (#920)

Elliot Slusky4916c296eb53
diff --git a/docs/tutorials/skills-workflow.md b/docs/tutorials/skills-workflow.md
index 77fd3251..e3ffdc7b 100644
--- a/docs/tutorials/skills-workflow.md
+++ b/docs/tutorials/skills-workflow.md
@@ -16,8 +16,8 @@ OpenJarvis can import skills from the [Hermes Agent](https://github.com/NousRese
```bash
# Install individual skills
+jarvis skill install hermes:llm-wiki
jarvis skill install hermes:arxiv
-jarvis skill install hermes:github-pr-workflow
# Or bulk install an entire category
jarvis skill sync hermes --category research
@@ -35,10 +35,10 @@ You should see a table with each skill's name, description, version, and tags.
## Step 2: Inspect an Installed Skill
-Let's look at what the `arxiv` skill contains:
+Let's look at what the `llm-wiki` skill contains:
```bash
-jarvis skill info arxiv
+jarvis skill info llm-wiki
```
This shows the skill's metadata — author, description, tags, capabilities, whether it has structured steps or markdown instructions, and its invocation flags.
@@ -46,40 +46,40 @@ This shows the skill's metadata — author, description, tags, capabilities, whe
You can also inspect the raw SKILL.md:
```bash
-cat ~/.openjarvis/skills/hermes/arxiv/SKILL.md | head -40
+cat ~/.openjarvis/skills/hermes/llm-wiki/SKILL.md | head -40
```
The `.source` file records provenance:
```bash
-cat ~/.openjarvis/skills/hermes/arxiv/.source
+cat ~/.openjarvis/skills/hermes/llm-wiki/.source
```
-This shows the source (`hermes:arxiv`), the git commit it was imported from, which tool names were translated (e.g., `Edit→file_edit`), and the install timestamp.
+This shows the source (`hermes:llm-wiki`), the git commit it was imported from, which tool names were translated (e.g., `Edit→file_edit`), and the install timestamp.
## Step 3: Use Skills with an Agent
Now let's ask the agent a question that should trigger skill usage:
```bash
-jarvis ask "Use the code-explainer skill to explain this Python code: for i in range(5): print(i*2)" \
+jarvis ask --agent orchestrator "Use the llm-wiki skill to explain how model entries should be organized" \
--engine ollama --model qwen3.5:9b
```
The agent will:
-1. See the skill catalog in its system prompt
-2. Decide to invoke `skill_code-explainer`
+1. See the skill catalog and `skill_llm-wiki` tool definition
+2. Decide to invoke `skill_llm-wiki`
3. Receive the markdown instructions from the skill
-4. Follow the 5-step pattern to explain the code
+4. Follow those instructions in its answer
-Try a pipeline skill too:
+Try the other installed skill too:
```bash
-jarvis ask "Use the math-solver skill to compute 17 * 23" \
+jarvis ask --agent orchestrator "Use the arxiv skill to outline how you would research transformer efficiency" \
--engine ollama --model qwen3.5:9b
```
-This time the agent invokes `skill_math-solver`, which executes a deterministic pipeline (calling the `calculator` tool internally) and returns the computed result directly.
+The `simple` agent intentionally cannot call tools. Use `orchestrator` or another tool-capable agent whenever you want a skill to be invoked.
## Step 4: Create Your Own Skill
@@ -124,7 +124,7 @@ jarvis skill list
You should see `my-reviewer` in the table. Try it:
```bash
-jarvis ask "Use the my-reviewer skill to review this function: def login(user, pwd): return db.query(f'SELECT * FROM users WHERE name={user} AND pass={pwd}')" \
+jarvis ask --agent orchestrator "Use the my-reviewer skill to review this function: def login(user, pwd): return db.query(f'SELECT * FROM users WHERE name={user} AND pass={pwd}')" \
--engine ollama --model qwen3.5:9b
```
@@ -135,15 +135,19 @@ The agent should follow the security-first approach and flag the SQL injection v
For the learning loop to work, we need traces. Run several queries that use skills:
```bash
+# Tracing is disabled in the generated config until you opt in
+jarvis config set traces.enabled true
+
# Generate a few traces
-jarvis ask "Use math-solver to compute 100 / 7"
-jarvis ask "Use code-explainer to explain: lambda x: x**2"
-jarvis ask "Use my-reviewer to review: def add(a,b): return a+b"
-jarvis ask "Use math-solver to compute 2**10"
-jarvis ask "Use code-explainer to explain: [x for x in range(10) if x % 2 == 0]"
+jarvis ask --agent orchestrator "Use llm-wiki to describe a model entry"
+jarvis ask --agent orchestrator "Use my-reviewer to review: def add(a,b): return a+b"
+jarvis ask --agent orchestrator "Use llm-wiki to describe an evaluation entry"
+jarvis ask --agent orchestrator "Use my-reviewer to review: lambda x: x**2"
+jarvis ask --agent orchestrator "Use llm-wiki to describe a dataset entry"
+jarvis ask --agent orchestrator "Use my-reviewer to review: def square(x): return x*x"
```
-Each query produces a trace in `~/.openjarvis/traces.db` with skill metadata tags (`skill`, `skill_source`, `skill_kind`).
+Each query produces one trace in `~/.openjarvis/traces.db`. An invoked skill's tool-call step includes the `skill`, `skill_source`, and `skill_kind` metadata tags.
## Step 6: Discover Patterns from Traces
@@ -151,21 +155,23 @@ Mine the trace store for recurring tool sequences:
```bash
# Preview without writing
-jarvis skill discover --dry-run --min-frequency 2
+jarvis skill discover --dry-run --min-frequency 2 --min-outcome 0
# Write discovered patterns as skill manifests
-jarvis skill discover --min-frequency 2
+jarvis skill discover --min-frequency 2 --min-outcome 0
```
Discovered skills land in `~/.openjarvis/skills/discovered/` and automatically appear in `jarvis skill list` on the next session.
+Discovery requires recurring sequences of at least two tool calls. `--min-outcome 0` includes new traces that have not yet been scored.
+
## Step 7: Optimize Skills with DSPy
Once you have enough traces (at least 3-5 per skill), run the optimizer:
```bash
# Preview what would be optimized
-jarvis optimize skills --dry-run
+jarvis optimize skills --dry-run --min-traces 3
# Run DSPy optimization
jarvis optimize skills --policy dspy --min-traces 3
@@ -176,8 +182,8 @@ This produces overlay files at `~/.openjarvis/learning/skills/<skill-name>/optim
Inspect what was produced:
```bash
-jarvis skill show-overlay math-solver
-jarvis skill show-overlay code-explainer
+jarvis skill show-overlay llm-wiki
+jarvis skill show-overlay my-reviewer
```
The next time you run a query, the agent sees the optimized descriptions and few-shot examples in its system prompt.
@@ -220,9 +226,9 @@ Now skills are automatically synced from Hermes on session start, and the optimi
| Concept | What you did |
|---------|-------------|
| **Installing skills** | `jarvis skill install hermes:arxiv` — imported from public sources |
-| **Using skills** | `jarvis ask "Use the code-explainer skill..."` — agent invokes skills as tools |
+| **Using skills** | `jarvis ask --agent orchestrator "Use the llm-wiki skill..."` — a tool-capable agent invokes skills |
| **Creating skills** | Wrote a `SKILL.md` with YAML frontmatter and markdown instructions |
-| **Generating traces** | Ran skill-using queries to populate the trace store |
+| **Generating traces** | Enabled tracing and ran skill-using queries to populate the trace store |
| **Discovering patterns** | `jarvis skill discover` — mined traces for recurring tool sequences |
| **Optimizing skills** | `jarvis optimize skills --policy dspy` — improved descriptions + few-shot examples |
| **Benchmarking** | `jarvis bench skills` — measured the impact across 4 conditions |
diff --git a/src/openjarvis/agents/orchestrator.py b/src/openjarvis/agents/orchestrator.py
index 22ee17fb..9d1e5744 100644
--- a/src/openjarvis/agents/orchestrator.py
+++ b/src/openjarvis/agents/orchestrator.py
@@ -65,6 +65,7 @@ class OrchestratorAgent(ToolUsingAgent):
parallel_tools: bool = True,
interactive: bool = False,
confirm_callback=None,
+ capability_policy: Optional[Any] = None,
before_tool_call: Optional[Callable[[str, dict[str, Any]], bool]] = None,
) -> None:
super().__init__(
@@ -77,6 +78,7 @@ class OrchestratorAgent(ToolUsingAgent):
max_tokens=max_tokens,
interactive=interactive,
confirm_callback=confirm_callback,
+ capability_policy=capability_policy,
prompt_builder=prompt_builder,
)
self._mode = mode
diff --git a/src/openjarvis/cli/ask.py b/src/openjarvis/cli/ask.py
index 7c06e71b..464f590c 100644
--- a/src/openjarvis/cli/ask.py
+++ b/src/openjarvis/cli/ask.py
@@ -369,6 +369,7 @@ def _run_agent(
)
agent_cls = AgentRegistry.get(agent_name)
+ accepts_tools = getattr(agent_cls, "accepts_tools", False)
# Build tools — local registry tools + MCP server tools from config
# (#461 — MCP tools were silently dropped because ask.py only used
@@ -400,13 +401,58 @@ def _run_agent(
tools.append(t)
existing.add(t.spec.name)
+ skill_manager = None
+ skill_catalog_xml = None
+ skill_few_shot_examples: list[str] = []
+ skill_tools_added = False
+ if accepts_tools and config.skills.enabled:
+ try:
+ from pathlib import Path
+
+ from openjarvis.skills.manager import SkillManager
+ from openjarvis.tools._stubs import ToolExecutor
+
+ pipeline_executor = ToolExecutor(
+ tools,
+ bus,
+ capability_policy=capability_policy,
+ agent_id=getattr(agent_cls, "agent_id", agent_name),
+ interactive=True,
+ confirm_callback=lambda prompt: True,
+ )
+ skill_manager = SkillManager(
+ bus,
+ capability_policy=capability_policy,
+ )
+ skill_paths = [Path(config.skills.skills_dir).expanduser()]
+ workspace_skills = Path("./skills")
+ if workspace_skills.exists():
+ skill_paths.insert(0, workspace_skills)
+ skill_manager.discover(paths=skill_paths)
+ skill_manager.set_tool_executor(pipeline_executor)
+
+ existing = {tool.spec.name for tool in tools}
+ for skill_tool in skill_manager.get_skill_tools(
+ tool_executor=pipeline_executor,
+ ):
+ if skill_tool.spec.name not in existing:
+ tools.append(skill_tool)
+ existing.add(skill_tool.spec.name)
+ skill_tools_added = True
+
+ if skill_tools_added:
+ skill_catalog_xml = skill_manager.get_catalog_xml()
+ skill_few_shot_examples = skill_manager.get_few_shot_examples()
+ except Exception as exc:
+ logger.warning("Failed to initialize skills: %s", exc)
+
# Build agent with appropriate kwargs
agent_kwargs = {
"bus": bus,
"temperature": temperature,
"max_tokens": max_tokens,
}
- if getattr(agent_cls, "accepts_tools", False):
+ if accepts_tools:
agent_kwargs["tools"] = tools
agent_kwargs["max_turns"] = config.agent.max_turns
agent_kwargs["interactive"] = True
@@ -427,6 +473,8 @@ def _run_agent(
agent_template=config.agent.default_system_prompt or "",
memory_files_config=memory_files_config or config.memory_files,
system_prompt_config=config.system_prompt,
+ skill_catalog_xml=skill_catalog_xml,
+ skill_few_shot_examples=skill_few_shot_examples,
)
agent = agent_cls(engine, model_name, **agent_kwargs)
@@ -436,6 +484,8 @@ def _run_agent(
# adversarial review caught this).
if mcp_clients:
agent._mcp_clients = mcp_clients
+ if skill_manager is not None:
+ agent._skill_manager = skill_manager
ctx = AgentContext()
# Inject memory context into conversation if available
@@ -463,7 +513,27 @@ def _run_agent(
except Exception as exc:
logger.warning("Failed to inject memory context for agent: %s", exc)
- return agent.run(query_text, context=ctx)
+ trace_store = None
+ if config.traces.enabled:
+ try:
+ from openjarvis.traces.store import TraceStore
+
+ trace_store = TraceStore(config.traces.db_path)
+ except Exception as exc:
+ logger.warning("Failed to initialize TraceStore: %s", exc)
+
+ try:
+ if trace_store is not None:
+ from openjarvis.traces.collector import TraceCollector
+
+ return TraceCollector(agent, store=trace_store, bus=bus).run(
+ query_text,
+ context=ctx,
+ )
+ return agent.run(query_text, context=ctx)
+ finally:
+ if trace_store is not None:
+ trace_store.close()
def _print_profile(
diff --git a/src/openjarvis/skills/manager.py b/src/openjarvis/skills/manager.py
index 32eba401..955bc52d 100644
--- a/src/openjarvis/skills/manager.py
+++ b/src/openjarvis/skills/manager.py
@@ -193,6 +193,8 @@ class SkillManager:
tools: List[BaseTool] = []
for manifest in self._skills.values():
+ if manifest.disable_model_invocation:
+ continue
real_executor = executor or _NullToolExecutor()
skill_exec = SkillExecutor(real_executor, bus=self._bus)
@@ -250,6 +252,8 @@ class SkillManager:
"""
examples: List[str] = []
for name, manifest in self._skills.items():
+ if manifest.disable_model_invocation:
+ continue
oj = manifest.metadata.get("openjarvis", {}) if manifest.metadata else {}
few_shot = oj.get("few_shot", []) or []
for ex in few_shot:
diff --git a/src/openjarvis/traces/store.py b/src/openjarvis/traces/store.py
index 8527e051..493ebd9d 100644
--- a/src/openjarvis/traces/store.py
+++ b/src/openjarvis/traces/store.py
@@ -81,7 +81,11 @@ class TraceStore:
"""Append-only SQLite store for interaction traces."""
def __init__(self, db_path: str | Path) -> None:
- self._db_path = str(db_path)
+ self._db_path = (
+ ":memory:"
+ if str(db_path) == ":memory:"
+ else str(Path(db_path).expanduser())
+ )
if self._db_path != ":memory:":
from openjarvis.security.file_utils import secure_create
diff --git a/tests/agents/test_orchestrator.py b/tests/agents/test_orchestrator.py
index 124c569f..7d677284 100644
--- a/tests/agents/test_orchestrator.py
+++ b/tests/agents/test_orchestrator.py
@@ -151,6 +151,15 @@ class TestOrchestratorAgent:
agent = OrchestratorAgent(engine, "test-model")
assert agent.agent_id == "orchestrator"
+ def test_capability_policy_is_forwarded_to_tool_executor(self):
+ policy = MagicMock()
+ agent = OrchestratorAgent(
+ _make_engine_no_tools(),
+ "test-model",
+ capability_policy=policy,
+ )
+ assert agent._executor._capability_policy is policy
+
def test_no_tools_single_turn(self):
engine = _make_engine_no_tools("Hello!")
agent = OrchestratorAgent(engine, "test-model")
diff --git a/tests/cli/test_ask_agent.py b/tests/cli/test_ask_agent.py
index 73abdaf9..49d7480c 100644
--- a/tests/cli/test_ask_agent.py
+++ b/tests/cli/test_ask_agent.py
@@ -11,7 +11,7 @@ from click.testing import CliRunner
from openjarvis.agents._stubs import AgentContext, AgentResult, ToolUsingAgent
from openjarvis.cli import cli
-from openjarvis.core.types import ToolCall, ToolResult
+from openjarvis.core.types import Role, StepType, ToolCall, ToolResult
from openjarvis.tools._stubs import BaseTool, ToolSpec
_ask_mod = importlib.import_module("openjarvis.cli.ask")
@@ -114,6 +114,7 @@ def agent_setup():
config = JarvisConfig()
config.intelligence.default_model = "test-model"
config.agent.max_turns = 3
+ config.traces.enabled = False
AgentRegistry.register_value("confirming_agent", _ConfirmingAgent)
ToolRegistry.register_value("dangerous", _DangerousTool)
@@ -154,7 +155,9 @@ def mock_setup():
):
from openjarvis.core.config import JarvisConfig
- mock_cfg.return_value = JarvisConfig()
+ config = JarvisConfig()
+ config.traces.enabled = False
+ mock_cfg.return_value = config
mock_ge.return_value = ("mock", engine)
mock_de.return_value = [("mock", engine)]
mock_dm.return_value = {"mock": ["test-model"]}
@@ -239,6 +242,7 @@ class TestAskAgentOption:
from openjarvis.core.config import JarvisConfig
cfg = JarvisConfig()
+ cfg.traces.enabled = False
cfg.agent.default_agent = ""
monkeypatch.setattr(_ask_mod, "load_config", lambda *a, **kw: cfg)
result = runner.invoke(cli, ["ask", "Hello"])
@@ -287,6 +291,112 @@ class TestAskAgentOption:
agent_setup.engine.generate.assert_not_called()
+class TestAskSkillsAndTraces:
+ def test_orchestrator_discovers_invokes_and_traces_skill(
+ self,
+ runner,
+ tmp_path,
+ ):
+ from openjarvis.core.config import JarvisConfig
+ from openjarvis.traces.store import TraceStore
+
+ skill_dir = tmp_path / "skills" / "hermes" / "llm-wiki"
+ skill_dir.mkdir(parents=True)
+ sentinel = "LLM_WIKI_INSTRUCTIONS_SENTINEL"
+ (skill_dir / "SKILL.md").write_text(
+ "---\n"
+ "name: llm-wiki\n"
+ "description: Maintain a local LLM knowledge wiki\n"
+ "---\n"
+ f"{sentinel}: update the wiki carefully.\n",
+ encoding="utf-8",
+ )
+ (skill_dir / ".source").write_text(
+ 'source = "hermes:llm-wiki"\n',
+ encoding="utf-8",
+ )
+
+ config = JarvisConfig()
+ config.intelligence.default_model = "test-model"
+ config.agent.context_from_memory = False
+ config.security.enabled = False
+ config.telemetry.enabled = False
+ config.skills.enabled = True
+ config.skills.skills_dir = str(tmp_path / "skills")
+ config.traces.enabled = True
+ config.traces.db_path = str(tmp_path / "traces.db")
+
+ engine = _mock_engine()
+ engine.generate.side_effect = [
+ {
+ "content": "",
+ "tool_calls": [
+ {
+ "id": "call_skill",
+ "name": "skill_llm-wiki",
+ "arguments": '{"task": "Update the model page"}',
+ }
+ ],
+ "usage": {"total_tokens": 8},
+ },
+ {
+ "content": "Wiki updated",
+ "tool_calls": [],
+ "usage": {"total_tokens": 8},
+ },
+ ]
+ _register_agents()
+
+ with (
+ patch.object(_ask_mod, "load_config", return_value=config),
+ patch.object(_ask_mod, "get_engine", return_value=("mock", engine)),
+ patch.object(
+ _ask_mod,
+ "discover_engines",
+ return_value=[("mock", engine)],
+ ),
+ patch.object(
+ _ask_mod,
+ "discover_models",
+ return_value={"mock": ["test-model"]},
+ ),
+ patch.object(_ask_mod, "register_builtin_models"),
+ patch.object(_ask_mod, "merge_discovered_models"),
+ ):
+ result = runner.invoke(
+ cli,
+ ["ask", "--agent", "orchestrator", "Update my LLM wiki"],
+ )
+
+ assert result.exit_code == 0, result.output
+ first_call = engine.generate.call_args_list[0]
+ assert any(
+ schema["function"]["name"] == "skill_llm-wiki"
+ for schema in first_call.kwargs["tools"]
+ )
+ assert "<available_skills>" in first_call.args[0][0].content
+
+ tool_messages = [
+ message
+ for message in engine.generate.call_args_list[1].args[0]
+ if message.role == Role.TOOL
+ ]
+ assert any(sentinel in message.content for message in tool_messages)
+
+ store = TraceStore(config.traces.db_path)
+ traces = store.list_traces()
+ store.close()
+ assert len(traces) == 1
+ assert [step.step_type for step in traces[0].steps] == [
+ StepType.GENERATE,
+ StepType.TOOL_CALL,
+ StepType.GENERATE,
+ StepType.RESPOND,
+ ]
+ assert traces[0].steps[1].metadata["skill"] == "llm-wiki"
+ assert traces[0].steps[1].metadata["skill_source"] == "hermes"
+
+
class TestBuildTools:
def test_build_calculator(self, mock_setup):
from openjarvis.cli.ask import _build_tools
@@ -357,6 +467,7 @@ class TestPersonaFilesReachModel:
user.write_text("USER_SENTINEL", encoding="utf-8")
cfg = JarvisConfig()
+ cfg.traces.enabled = False
cfg.memory_files.soul_path = str(soul)
cfg.memory_files.memory_path = str(memory)
cfg.memory_files.user_path = str(user)
diff --git a/tests/cli/test_ask_e2e.py b/tests/cli/test_ask_e2e.py
index 3f833d14..e2a9e50d 100644
--- a/tests/cli/test_ask_e2e.py
+++ b/tests/cli/test_ask_e2e.py
@@ -43,6 +43,7 @@ def _patch_ask(monkeypatch, tmp_path, *, engine_result=None, no_engine=False):
cfg = JarvisConfig()
cfg.telemetry.db_path = str(tmp_path / "telemetry.db")
+ cfg.traces.enabled = False
monkeypatch.setattr(_ask_mod, "load_config", lambda: cfg)
diff --git a/tests/skills/test_manager.py b/tests/skills/test_manager.py
index c43d2a80..7595d9d1 100644
--- a/tests/skills/test_manager.py
+++ b/tests/skills/test_manager.py
@@ -310,6 +310,29 @@ class TestSkillManagerTools:
tools = mgr.get_skill_tools()
assert any(t.spec.name == "skill_my_skill" for t in tools)
+ def test_model_disabled_skill_is_not_exposed(self, tmp_path: Path) -> None:
+ skill_dir = tmp_path / "hidden"
+ skill_dir.mkdir()
+ (skill_dir / "SKILL.md").write_text(
+ "---\n"
+ "name: hidden\n"
+ "disable_model_invocation: true\n"
+ "metadata:\n"
+ " openjarvis:\n"
+ " few_shot:\n"
+ " - input: secret\n"
+ " output: secret\n"
+ "---\n"
+ "Hidden instructions\n",
+ encoding="utf-8",
+ )
+
+ mgr = SkillManager(bus=EventBus())
+ mgr.discover(paths=[tmp_path])
+
+ assert mgr.get_skill_tools() == []
+ assert mgr.get_few_shot_examples() == []
+
# ---------------------------------------------------------------------------
# TestSkillManagerResolve
diff --git a/tests/traces/test_store.py b/tests/traces/test_store.py
index b2863ede..138f0471 100644
--- a/tests/traces/test_store.py
+++ b/tests/traces/test_store.py
@@ -55,6 +55,23 @@ class TestTraceStore:
assert store.count() == 0
store.close()
+ def test_expands_user_in_db_path(
+ self,
+ tmp_path: Path,
+ monkeypatch,
+ ) -> None:
+ home = tmp_path / "home"
+ cwd = tmp_path / "cwd"
+ cwd.mkdir()
+ monkeypatch.setenv("HOME", str(home))
+ monkeypatch.chdir(cwd)
+
+ store = TraceStore("~/.openjarvis/traces.db")
+ store.close()
+
+ assert (home / ".openjarvis" / "traces.db").exists()
+ assert not (cwd / "~").exists()
+
def test_save_and_get(self, tmp_path: Path) -> None:
store = TraceStore(tmp_path / "test.db")
trace = _make_trace()