Skip to content

Create, Evaluate, and Connect AI Skills

update analyze

LiamLYYY206d7411bea6
diff --git a/README.md b/README.md
index 4b78fdd..2911d5b 100644
--- a/README.md
+++ b/README.md
@@ -41,6 +41,7 @@ SkillNet is an open infrastructure for creating, evaluating, and organizing AI s
- Office documents (PDF, PPT, Word)
- Direct text prompts
- **πŸ“Š Evaluation**: Evaluate and score skills for quality assurance (Safety, Completeness, Excutability, Modifiability, Cost-Aware).
+- **πŸ•ΈοΈ Relationship Analysis**: Automatically map the connections between skills in your local library, identifying structural relationships between skills (similar_to, belong_to, compose_with, depend_on).
# 🌐 API Access
@@ -222,6 +223,23 @@ result = client.evaluate(target=target_skill)
print(f"Evaluation Result: {result}")
```
+#### 6. Skill Relationship Analysis
+Analyze a local directory containing multiple skills to infer a relationship graph. It identifies relationships like dependencies (depend_on), collaboration (compose_with), hierarchy (belong_to), and alternatives (similar_to).
+
+```python
+# Directory containing multiple skill folders
+skills_directory = "./my_agent_skills"
+
+# Analyze relationships between skills
+# This will also save a 'relationships.json' in the directory by default
+relationships = client.analyze(skills_dir=skills_directory)
+
+# Display the relationships
+for rel in relationships:
+ print(f"{rel['source']} --[{rel['type']}]--> {rel['target']}")
+ # Output: PDF_Parser --[compose_with]--> Text_Summarizer
+```
+
### πŸ’» CLI Usage
@@ -307,8 +325,24 @@ skillnet evaluate ./my_skills/web_search
skillnet evaluate ./my_skills/tool --category "DevOps" --model gpt-4o
```
+#### 5. Analyze Relationships (`analyze`)
+Scan a local directory of skills to analyze their connections using AI.
+
+Requirement: Ensure API_KEY is set in your environment variables.
+
+```bash
+# Analyze a directory containing multiple skill folders
+skillnet analyze ./my_agent_skills
+
+# Analyze without saving the result file (just print to console)
+skillnet analyze ./my_agent_skills --no-save
+
+# Specify a model for the analysis
+skillnet analyze ./my_agent_skills --model gpt-4o
+```
+
### Environment Configuration
-To use **Creation** or **Evaluation** features, set your environment variables:
+To use **Creation**, **Evaluation**, or **Analyze** features, set your environment variables:
```bash
export API_KEY="your_api_key"
diff --git a/skillnet-ai/README.md b/skillnet-ai/README.md
index 1f0822b..7168404 100644
--- a/skillnet-ai/README.md
+++ b/skillnet-ai/README.md
@@ -18,6 +18,7 @@
- Office documents (PDF, PPT, Word)
- Direct text prompts
- **πŸ“Š Evaluation**: Evaluate and score skills for quality assurance (Safety, Completeness, Excutability, Modifiability, Cost-Aware).
+- **πŸ•ΈοΈ Relationship Analysis**: Automatically map the connections between skills in your local library, identifying structural relationships between skills (similar_to, belong_to, compose_with, depend_on).
---
@@ -163,6 +164,23 @@ result = client.evaluate(target=target_skill)
print(f"Evaluation Result: {result}")
```
+### 6. Skill Relationship Analysis
+Analyze a local directory containing multiple skills to infer a relationship graph. It identifies relationships like dependencies (depend_on), collaboration (compose_with), hierarchy (belong_to), and alternatives (similar_to).
+
+```python
+# Directory containing multiple skill folders
+skills_directory = "./my_agent_skills"
+
+# Analyze relationships between skills
+# This will also save a 'relationships.json' in the directory by default
+relationships = client.analyze(skills_dir=skills_directory)
+
+# Display the relationships
+for rel in relationships:
+ print(f"{rel['source']} --[{rel['type']}]--> {rel['target']}")
+ # Output: PDF_Parser --[compose_with]--> Text_Summarizer
+```
+
---
## πŸ’» CLI Usage
@@ -256,11 +274,27 @@ skillnet evaluate ./my_skills/web_search
skillnet evaluate ./my_skills/tool --category "DevOps" --model gpt-4o
```
+### 5. Analyze Relationships (`analyze`)
+Scan a directory of skills to analyze their connections using AI.
+
+Requirement: Ensure API_KEY is set in your environment variables.
+
+```bash
+# Analyze a directory containing multiple skill folders
+skillnet analyze ./my_agent_skills
+
+# Analyze without saving the result file (just print to console)
+skillnet analyze ./my_agent_skills --no-save
+
+# Specify a model for the analysis
+skillnet analyze ./my_agent_skills --model gpt-4o
+```
+
## βš™οΈ Configuration
### Environment Variables
-If you are using the **Skill Creation** or **Skill Evaluation** feature, you must configure your LLM provider.
+If you are using the **Creation**, **Evaluation**, or **Analyze** feature, you must configure your LLM provider.
| Variable | Description | Default |
| :--- | :--- | :--- |
diff --git a/skillnet-ai/src/skillnet_ai/__init__.py b/skillnet-ai/src/skillnet_ai/__init__.py
index bd4d9e6..0c8038a 100644
--- a/skillnet-ai/src/skillnet_ai/__init__.py
+++ b/skillnet-ai/src/skillnet_ai/__init__.py
@@ -10,6 +10,7 @@ from skillnet_ai.creator import SkillCreator
from skillnet_ai.downloader import SkillDownloader
from skillnet_ai.evaluator import SkillEvaluator, EvaluatorConfig
from skillnet_ai.searcher import SkillNetSearcher
+from skillnet_ai.analyzer import SkillRelationshipAnalyzer
__all__ = [
"SkillNetClient",
@@ -17,5 +18,6 @@ __all__ = [
"SkillDownloader",
"SkillEvaluator",
"EvaluatorConfig",
- "SkillNetSearcher"
+ "SkillNetSearcher",
+ "SkillRelationshipAnalyzer"
]
\ No newline at end of file
diff --git a/skillnet-ai/src/skillnet_ai/analyzer.py b/skillnet-ai/src/skillnet_ai/analyzer.py
new file mode 100644
index 0000000..ba7c6db
--- /dev/null
+++ b/skillnet-ai/src/skillnet_ai/analyzer.py
@@ -0,0 +1,178 @@
+import os
+import json
+import logging
+import re
+from typing import List, Dict, Any, Optional
+
+from openai import OpenAI
+from skillnet_ai.prompts import (
+ RELATIONSHIP_ANALYSIS_SYSTEM_PROMPT,
+ RELATIONSHIP_ANALYSIS_USER_PROMPT_TEMPLATE
+)
+
+logger = logging.getLogger(__name__)
+
+class SkillRelationshipAnalyzer:
+ """
+ Analyzes a directory of skills to determine relationships between them.
+
+ Relationships determined:
+ - similar_to: A and B are functionally similar and interchangeable.
+ - belong_to: A is a sub-task/part of B (B is the larger scope).
+ - compose_with: A and B are independent but often used together.
+ - depend_on: A requires B to execute (prerequisite).
+ """
+
+ def __init__(self, api_key: Optional[str] = None, base_url: Optional[str] = None, model: str = "gpt-4o"):
+ self.api_key = api_key or os.getenv("API_KEY")
+ self.base_url = base_url or os.getenv("BASE_URL") or "https://api.openai.com/v1"
+ self.model = model
+
+ if not self.api_key:
+ raise ValueError("API Key is missing. Please provide it in init or set API_KEY environment variable.")
+
+ self.client = OpenAI(api_key=self.api_key, base_url=self.base_url)
+
+ def analyze_local_skills(self, skills_dir: str, save_to_file: bool = True) -> List[Dict[str, Any]]:
+ """
+ Main entry point: Scans a directory for skills and maps their relationships.
+
+ Args:
+ skills_dir: Path to the directory containing skill folders.
+ save_to_file: Whether to save the result as 'relationships.json' in the dir.
+
+ Returns:
+ A list of relationship dictionaries:
+ [
+ {
+ "source": "skill_a",
+ "target": "skill_b",
+ "type": "depend_on",
+ "reason": "Skill A imports modules provided by Skill B"
+ },
+ ...
+ ]
+ """
+ logger.info(f"Starting relationship analysis in: {skills_dir}")
+
+ if not os.path.exists(skills_dir):
+ raise FileNotFoundError(f"Directory not found: {skills_dir}")
+
+ # 1. Load Skill Metadata
+ skills_metadata = self._load_skills_metadata(skills_dir)
+ if len(skills_metadata) < 2:
+ logger.warning("Not enough skills found to analyze relationships (need at least 2).")
+ return []
+
+ logger.info(f"Found {len(skills_metadata)} skills. Analyzing potential connections...")
+
+ # 2. Analyze with LLM
+ relationships = self._generate_relationship_graph(skills_metadata)
+
+ # 3. Save Results
+ if save_to_file and relationships:
+ output_path = os.path.join(skills_dir, "relationships.json")
+ try:
+ with open(output_path, 'w', encoding='utf-8') as f:
+ json.dump(relationships, f, indent=2, ensure_ascii=False)
+ logger.info(f"Relationships saved to: {output_path}")
+ except IOError as e:
+ logger.error(f"Failed to save relationships file: {e}")
+
+ return relationships
+
+ def _load_skills_metadata(self, root_dir: str) -> List[Dict[str, str]]:
+ """
+ Walks the directory to extract 'name' and 'description' from SKILL.md or README.md.
+ """
+ skills = []
+
+ # Assuming typical structure: root_dir/skill_name/SKILL.md
+ for entry in os.scandir(root_dir):
+ if entry.is_dir():
+ skill_path = entry.path
+ skill_name = entry.name
+ description = "No description provided."
+
+ # Try to read SKILL.md
+ content_file = None
+ if os.path.exists(os.path.join(skill_path, "SKILL.md")):
+ content_file = os.path.join(skill_path, "SKILL.md")
+
+ if content_file:
+ try:
+ with open(content_file, 'r', encoding='utf-8') as f:
+ raw_content = f.read()
+ # Extract description from Frontmatter or first paragraph
+ description = self._extract_description(raw_content)
+ except Exception as e:
+ logger.warning(f"Could not read content for {skill_name}: {e}")
+
+ skills.append({
+ "name": skill_name,
+ "description": description
+ })
+
+ return skills
+
+ def _extract_description(self, content: str) -> str:
+ """Helper to parse description from markdown frontmatter or body."""
+ # 1. Try YAML Frontmatter
+ frontmatter_match = re.search(r'^---\n(.*?)\n---', content, re.DOTALL)
+ if frontmatter_match:
+ fm_text = frontmatter_match.group(1)
+ # Simple regex search for description: ... line
+ desc_match = re.search(r'description:\s*(.+)$', fm_text, re.MULTILINE)
+ if desc_match:
+ return desc_match.group(1).strip().strip('"').strip("'")
+
+ # 2. Fallback: Use the first non-header text block
+ # Remove headers
+ clean_text = re.sub(r'#+\s.*', '', content)
+ # Remove code blocks
+ clean_text = re.sub(r'```.*?```', '', clean_text, flags=re.DOTALL)
+ lines = [line.strip() for line in clean_text.split('\n') if line.strip()]
+
+ if lines:
+ return lines[0]
+
+ return "No description available."
+
+ def _generate_relationship_graph(self, skills: List[Dict]) -> List[Dict]:
+ """Calls LLM to infer edges between nodes."""
+
+ skills_json = json.dumps(skills, indent=2)
+
+ messages = [
+ {"role": "system", "content": RELATIONSHIP_ANALYSIS_SYSTEM_PROMPT},
+ {"role": "user", "content": RELATIONSHIP_ANALYSIS_USER_PROMPT_TEMPLATE.format(
+ skills_list=skills_json
+ )}
+ ]
+
+ try:
+ response = self.client.chat.completions.create(
+ model=self.model,
+ messages=messages,
+ )
+ content = response.choices[0].message.content
+
+ parsed = json.loads(content)
+ edges = parsed.get("relationships", [])
+
+ # Validate edges structure
+ valid_edges = []
+ valid_names = {s['name'] for s in skills}
+
+ for edge in edges:
+ if (edge.get('source') in valid_names and
+ edge.get('target') in valid_names and
+ edge.get('type') in ['similar_to', 'belong_to', 'compose_with', 'depend_on']):
+ valid_edges.append(edge)
+
+ logger.info(f"Identified {len(valid_edges)} valid relationships.")
+ return valid_edges
+
+ except Exception as e:
+ logger.error(f"Failed to analyze relationships: {e}")
+ return []
\ No newline at end of file
diff --git a/skillnet-ai/src/skillnet_ai/cli.py b/skillnet-ai/src/skillnet_ai/cli.py
index 9f8dfc4..a8af7ed 100644
--- a/skillnet-ai/src/skillnet_ai/cli.py
+++ b/skillnet-ai/src/skillnet_ai/cli.py
@@ -9,6 +9,7 @@ from skillnet_ai.creator import SkillCreator
from skillnet_ai.downloader import SkillDownloader
from skillnet_ai.evaluator import SkillEvaluator, EvaluatorConfig
from skillnet_ai.searcher import SkillNetSearcher
+from skillnet_ai.analyzer import SkillRelationshipAnalyzer
app = typer.Typer(help="SkillNet AI CLI Tool")
console = Console()
@@ -397,7 +398,7 @@ def evaluate(
description: str = typer.Option(None, help="Short description of what the skill does."),
# Config options
- model: str = typer.Option("gpt-4o", help="The LLM model used for evaluation."),
+ model: str = typer.Option("gpt-4o", "--model", "-m", help="LLM model to use."),
max_workers: int = typer.Option(5, help="Concurrency for batch operations (not used for single eval)."),
):
"""
@@ -487,5 +488,90 @@ def _display_evaluation_report(target_name: str, data: dict):
if summary:
console.print(Panel(summary, title="Executive Summary", border_style="cyan"))
+def analyze(
+ skills_dir: Path = typer.Argument(..., exists=True, file_okay=False, help="Directory containing multiple skill folders to analyze."),
+ save: bool = typer.Option(True, "--save/--no-save", help="Save the result to relationships.json in the directory."),
+ model: str = typer.Option("gpt-4o", "--model", "-m", help="LLM model to use."),
+):
+ """
+ Analyze and map relationships (similar_to, belong_to, compose_with, depend_on) between local skills.
+
+ This command scans all subdirectories in the target folder, reads their descriptions,
+ and uses AI to build a knowledge graph of how the skills relate to each other.
+ """
+ # 1. Validate Environment
+ if not API_KEY:
+ console.print("[bold red]Error:[/bold red] API_KEY environment variable is not set.")
+ raise typer.Exit(code=1)
+
+ try:
+ # 2. Initialize Analyzer
+ analyzer = SkillRelationshipAnalyzer(
+ api_key=API_KEY,
+ base_url=BASE_URL,
+ model=model
+ )
+
+ # 3. Visual Feedback & Execution
+ console.print(f"[dim]Scanning directory: {os.path.abspath(skills_dir)}[/dim]")
+
+ results = []
+ with console.status("[bold green]Reading skills and analyzing relationships...[/bold green]", spinner="earth"):
+ results = analyzer.analyze_local_skills(
+ skills_dir=str(skills_dir),
+ save_to_file=save
+ )
+
+ # 4. Handle Empty Results
+ if not results:
+ console.print("\n[yellow]No strong relationships detected among the skills found.[/yellow]")
+ console.print("[dim]Make sure the directory contains subfolders with valid SKILL.md or README.md files.[/dim]")
+ return
+
+ # 5. Render Results Table
+ console.print(f"\n[bold green]Analysis Complete! Found {len(results)} relationships:[/bold green]\n")
+
+ table = Table(show_header=True, header_style="bold magenta", title="Skill Relationship Graph")
+ table.add_column("Source Skill", style="cyan", no_wrap=True)
+ table.add_column("Relationship", style="bold white", justify="center")
+ table.add_column("Target Skill", style="cyan", no_wrap=True)
+ table.add_column("Reasoning", style="dim")
+
+ for edge in results:
+ # Color code the relationship types for better readability
+ rel_type = edge.get('type', 'unknown')
+ rel_style = "white"
+ arrow = "->"
+
+ if rel_type == "depend_on":
+ rel_style = "red" # Critical dependency
+ arrow = "DEPEND ON"
+ elif rel_type == "belong_to":
+ rel_style = "blue" # Hierarchy
+ arrow = "BELONG TO"
+ elif rel_type == "compose_with": # pairs_with
+ rel_style = "green" # Collaboration
+ arrow = "COMPOSE WITH"
+ elif rel_type == "similar_to":
+ rel_style = "yellow" # Alternative
+ arrow = "SIMILAR TO"
+
+ table.add_row(
+ edge.get('source', 'Unknown'),
+ f"[{rel_style}]{arrow}[/{rel_style}]",
+ edge.get('target', 'Unknown'),
+ edge.get('reason', '')
+ )
+
+ console.print(table)
+
+ if save:
+ console.print(f"\n[dim]Relationship data saved to: {os.path.join(skills_dir, 'relationships.json')}[/dim]")
+
+ except Exception as e:
+ console.print(f"\n[bold red]Analysis Failed:[/bold red] {str(e)}")
+ raise typer.Exit(code=1)
+
if __name__ == "__main__":
app()
diff --git a/skillnet-ai/src/skillnet_ai/client.py b/skillnet-ai/src/skillnet_ai/client.py
index ee548d1..46b3f64 100644
--- a/skillnet-ai/src/skillnet_ai/client.py
+++ b/skillnet-ai/src/skillnet_ai/client.py
@@ -6,6 +6,7 @@ from skillnet_ai.creator import SkillCreator
from skillnet_ai.downloader import SkillDownloader
from skillnet_ai.evaluator import SkillEvaluator, EvaluatorConfig
from skillnet_ai.searcher import SkillNetSearcher
+from skillnet_ai.analyzer import SkillRelationshipAnalyzer
class SkillNetError(Exception):
"""Custom exception class for SkillNet Client errors."""
@@ -273,4 +274,42 @@ class SkillNetClient:
return result
except Exception as e:
- raise SkillNetError(f"Evaluation process failed: {str(e)}") from e
\ No newline at end of file
+ raise SkillNetError(f"Evaluation process failed: {str(e)}") from e
+
+ def analyze(
+ self,
+ skills_dir: Union[str, Path],
+ save_to_file: bool = True,
+ model: str = "gpt-4o"
+ ) -> List[Dict[str, Any]]:
+ """
+ Analyze a local directory containing multiple skills to infer relationships between them.
+
+ This builds a knowledge graph (edges) between skills based on their names and descriptions.
+ Relationships detected: similar_to, belong_to, pairs_with (compose_with), depend_on.
+
+ Args:
+ skills_dir: Path to the directory containing skill folders.
+ save_to_file: If True, saves a 'relationships.json' file in the skills_dir.
+ model: The LLM model to use for analysis.
+
+ Returns:
+ List[Dict[str, Any]]: A list of relationship edges (source, target, type, reason).
+ """
+ if not self.api_key:
+ raise SkillNetError("API_KEY is required for relationship analysis.")
+
+ try:
+ analyzer = SkillRelationshipAnalyzer(
+ api_key=self.api_key,
+ base_url=self.base_url,
+ model=model
+ )
+
+ results = analyzer.analyze_local_skills(
+ skills_dir=str(skills_dir),
+ save_to_file=save_to_file
+ )
+ return results
+ except Exception as e:
+ raise SkillNetError(f"Relationship analysis failed: {str(e)}") from e
\ No newline at end of file
diff --git a/skillnet-ai/src/skillnet_ai/prompts.py b/skillnet-ai/src/skillnet_ai/prompts.py
index 88be845..e57398a 100644
--- a/skillnet-ai/src/skillnet_ai/prompts.py
+++ b/skillnet-ai/src/skillnet_ai/prompts.py
@@ -775,4 +775,68 @@ For every file, use this exact pattern:
```
Now, generate a complete, high-quality skill package based on the user's request.
-Be comprehensive and practical - create a skill that an AI agent would find genuinely useful."""
\ No newline at end of file
+Be comprehensive and practical - create a skill that an AI agent would find genuinely useful."""
+
+
+RELATIONSHIP_ANALYSIS_SYSTEM_PROMPT = """
+You are the SkillNet Architect.
+"""
+
+RELATIONSHIP_ANALYSIS_USER_PROMPT_TEMPLATE = """
+Your task is to map logical relationships between the provided skills based on their names and descriptions.
+
+You must strictly identify ONLY the following 4 types of relationships:
+
+1. similar_to
+ - A and B perform functionally equivalent tasks (e.g., "Google Search" and "Bing Search").
+ - Users can replace A with B.
+
+2. belong_to
+ - A is a sub-component or specific step within B.
+ - B represents a larger workflow or agent, and A is just one part of it.
+ - Direction: Child -> belong_to -> Parent.
+
+3. compose_with
+ - A and B are independent but often used together in a workflow.
+ - One usually produces data that the other consumes, or they are logically paired.
+ - Example: "PDF Parser" compose_with "Text Summarizer".
+
+4. depend_on
+ - A CANNOT execute without B.
+ - B is a hard dependency (e.g., Environment setup, API Key loader, or a core library skill).
+ - Direction: Dependent -> depend_on -> Prerequisite.
+
+Here is the list of Skills in the user's local environment. Please analyze them and generate the relationships.
+
+Skills List:
+{skills_list}
+
+Remember:
+- Be conservative. Only create a relationship if there is a logical connection based on the name and description.
+- Do not hallucinate skills not in the list.
+
+Output Format:
+Return a JSON object with a single key "relationships", which is a list of objects.
+Each object must have: "source", "target", "type", and "reason" (short explanation).
+
+Output Example:
+{{
+ "relationships": [
+ {{
+ "source": "google_search_tool",
+ "target": "bing_search_tool",
+ "type": "similar_to",
+ "reason": "Both provide web search capabilities and are interchangeable."
+ }},
+ {{
+ "source": "analyze_stock_chart",
+ "target": "get_stock_data",
+ "type": "depend_on",
+ "reason": "Analysis requires the raw data fetcher to run first."
+ }},
+ ...
+ ]
+}}
+
+Now, analyze the skills and provide the relationships in the valid JSON format.
+"""
\ No newline at end of file