Create, Evaluate, and Connect AI Skills
Python ai-agentsai-skillsartificial-intelligenceknowledge-graphlarge-language-modelsnatural-language-processingopenkgskillnetskills
update analyze
LiamLYYY206d7411bea6
diff --git a/README.md b/README.mdindex 4b78fdd..2911d5b 100644--- a/README.md+++ b/README.md@@ -41,6 +41,7 @@ SkillNet is an open infrastructure for creating, evaluating, and organizing AI s- Office documents (PDF, PPT, Word)- Direct text prompts- **π Evaluation**: Evaluate and score skills for quality assurance (Safety, Completeness, Excutability, Modifiability, Cost-Aware).+- **πΈοΈ Relationship Analysis**: Automatically map the connections between skills in your local library, identifying structural relationships between skills (similar_to, belong_to, compose_with, depend_on).# π API Access@@ -222,6 +223,23 @@ result = client.evaluate(target=target_skill)print(f"Evaluation Result: {result}")```+#### 6. Skill Relationship Analysis+Analyze a local directory containing multiple skills to infer a relationship graph. It identifies relationships like dependencies (depend_on), collaboration (compose_with), hierarchy (belong_to), and alternatives (similar_to).++```python+# Directory containing multiple skill folders+skills_directory = "./my_agent_skills"++# Analyze relationships between skills+# This will also save a 'relationships.json' in the directory by default+relationships = client.analyze(skills_dir=skills_directory)++# Display the relationships+for rel in relationships:+ print(f"{rel['source']} --[{rel['type']}]--> {rel['target']}")+ # Output: PDF_Parser --[compose_with]--> Text_Summarizer+```+### π» CLI Usage@@ -307,8 +325,24 @@ skillnet evaluate ./my_skills/web_searchskillnet evaluate ./my_skills/tool --category "DevOps" --model gpt-4o```+#### 5. Analyze Relationships (`analyze`)+Scan a local directory of skills to analyze their connections using AI.++Requirement: Ensure API_KEY is set in your environment variables.++```bash+# Analyze a directory containing multiple skill folders+skillnet analyze ./my_agent_skills++# Analyze without saving the result file (just print to console)+skillnet analyze ./my_agent_skills --no-save++# Specify a model for the analysis+skillnet analyze ./my_agent_skills --model gpt-4o+```+### Environment Configuration-To use **Creation** or **Evaluation** features, set your environment variables:+To use **Creation**, **Evaluation**, or **Analyze** features, set your environment variables:```bashexport API_KEY="your_api_key"diff --git a/skillnet-ai/README.md b/skillnet-ai/README.mdindex 1f0822b..7168404 100644--- a/skillnet-ai/README.md+++ b/skillnet-ai/README.md@@ -18,6 +18,7 @@- Office documents (PDF, PPT, Word)- Direct text prompts- **π Evaluation**: Evaluate and score skills for quality assurance (Safety, Completeness, Excutability, Modifiability, Cost-Aware).+- **πΈοΈ Relationship Analysis**: Automatically map the connections between skills in your local library, identifying structural relationships between skills (similar_to, belong_to, compose_with, depend_on).---@@ -163,6 +164,23 @@ result = client.evaluate(target=target_skill)print(f"Evaluation Result: {result}")```+### 6. Skill Relationship Analysis+Analyze a local directory containing multiple skills to infer a relationship graph. It identifies relationships like dependencies (depend_on), collaboration (compose_with), hierarchy (belong_to), and alternatives (similar_to).++```python+# Directory containing multiple skill folders+skills_directory = "./my_agent_skills"++# Analyze relationships between skills+# This will also save a 'relationships.json' in the directory by default+relationships = client.analyze(skills_dir=skills_directory)++# Display the relationships+for rel in relationships:+ print(f"{rel['source']} --[{rel['type']}]--> {rel['target']}")+ # Output: PDF_Parser --[compose_with]--> Text_Summarizer+```+---## π» CLI Usage@@ -256,11 +274,27 @@ skillnet evaluate ./my_skills/web_searchskillnet evaluate ./my_skills/tool --category "DevOps" --model gpt-4o```+### 5. Analyze Relationships (`analyze`)+Scan a directory of skills to analyze their connections using AI.++Requirement: Ensure API_KEY is set in your environment variables.++```bash+# Analyze a directory containing multiple skill folders+skillnet analyze ./my_agent_skills++# Analyze without saving the result file (just print to console)+skillnet analyze ./my_agent_skills --no-save++# Specify a model for the analysis+skillnet analyze ./my_agent_skills --model gpt-4o+```+## βοΈ Configuration### Environment Variables-If you are using the **Skill Creation** or **Skill Evaluation** feature, you must configure your LLM provider.+If you are using the **Creation**, **Evaluation**, or **Analyze** feature, you must configure your LLM provider.| Variable | Description | Default || :--- | :--- | :--- |diff --git a/skillnet-ai/src/skillnet_ai/__init__.py b/skillnet-ai/src/skillnet_ai/__init__.pyindex bd4d9e6..0c8038a 100644--- a/skillnet-ai/src/skillnet_ai/__init__.py+++ b/skillnet-ai/src/skillnet_ai/__init__.py@@ -10,6 +10,7 @@ from skillnet_ai.creator import SkillCreatorfrom skillnet_ai.downloader import SkillDownloaderfrom skillnet_ai.evaluator import SkillEvaluator, EvaluatorConfigfrom skillnet_ai.searcher import SkillNetSearcher+from skillnet_ai.analyzer import SkillRelationshipAnalyzer__all__ = ["SkillNetClient",@@ -17,5 +18,6 @@ __all__ = ["SkillDownloader","SkillEvaluator","EvaluatorConfig",- "SkillNetSearcher"+ "SkillNetSearcher",+ "SkillRelationshipAnalyzer"]\ No newline at end of filediff --git a/skillnet-ai/src/skillnet_ai/analyzer.py b/skillnet-ai/src/skillnet_ai/analyzer.pynew file mode 100644index 0000000..ba7c6db--- /dev/null+++ b/skillnet-ai/src/skillnet_ai/analyzer.py@@ -0,0 +1,178 @@+import os+import json+import logging+import re+from typing import List, Dict, Any, Optional++from openai import OpenAI+from skillnet_ai.prompts import (+ RELATIONSHIP_ANALYSIS_SYSTEM_PROMPT,+ RELATIONSHIP_ANALYSIS_USER_PROMPT_TEMPLATE+)++logger = logging.getLogger(__name__)++class SkillRelationshipAnalyzer:+ """+ Analyzes a directory of skills to determine relationships between them.++ Relationships determined:+ - similar_to: A and B are functionally similar and interchangeable.+ - belong_to: A is a sub-task/part of B (B is the larger scope).+ - compose_with: A and B are independent but often used together.+ - depend_on: A requires B to execute (prerequisite).+ """++ def __init__(self, api_key: Optional[str] = None, base_url: Optional[str] = None, model: str = "gpt-4o"):+ self.api_key = api_key or os.getenv("API_KEY")+ self.base_url = base_url or os.getenv("BASE_URL") or "https://api.openai.com/v1"+ self.model = model++ if not self.api_key:+ raise ValueError("API Key is missing. Please provide it in init or set API_KEY environment variable.")++ self.client = OpenAI(api_key=self.api_key, base_url=self.base_url)++ def analyze_local_skills(self, skills_dir: str, save_to_file: bool = True) -> List[Dict[str, Any]]:+ """+ Main entry point: Scans a directory for skills and maps their relationships.++ Args:+ skills_dir: Path to the directory containing skill folders.+ save_to_file: Whether to save the result as 'relationships.json' in the dir.++ Returns:+ A list of relationship dictionaries:+ [+ {+ "source": "skill_a",+ "target": "skill_b",+ "type": "depend_on",+ "reason": "Skill A imports modules provided by Skill B"+ },+ ...+ ]+ """+ logger.info(f"Starting relationship analysis in: {skills_dir}")++ if not os.path.exists(skills_dir):+ raise FileNotFoundError(f"Directory not found: {skills_dir}")++ # 1. Load Skill Metadata+ skills_metadata = self._load_skills_metadata(skills_dir)+ if len(skills_metadata) < 2:+ logger.warning("Not enough skills found to analyze relationships (need at least 2).")+ return []++ logger.info(f"Found {len(skills_metadata)} skills. Analyzing potential connections...")++ # 2. Analyze with LLM+ relationships = self._generate_relationship_graph(skills_metadata)++ # 3. Save Results+ if save_to_file and relationships:+ output_path = os.path.join(skills_dir, "relationships.json")+ try:+ with open(output_path, 'w', encoding='utf-8') as f:+ json.dump(relationships, f, indent=2, ensure_ascii=False)+ logger.info(f"Relationships saved to: {output_path}")+ except IOError as e:+ logger.error(f"Failed to save relationships file: {e}")++ return relationships++ def _load_skills_metadata(self, root_dir: str) -> List[Dict[str, str]]:+ """+ Walks the directory to extract 'name' and 'description' from SKILL.md or README.md.+ """+ skills = []++ # Assuming typical structure: root_dir/skill_name/SKILL.md+ for entry in os.scandir(root_dir):+ if entry.is_dir():+ skill_path = entry.path+ skill_name = entry.name+ description = "No description provided."++ # Try to read SKILL.md+ content_file = None+ if os.path.exists(os.path.join(skill_path, "SKILL.md")):+ content_file = os.path.join(skill_path, "SKILL.md")++ if content_file:+ try:+ with open(content_file, 'r', encoding='utf-8') as f:+ raw_content = f.read()+ # Extract description from Frontmatter or first paragraph+ description = self._extract_description(raw_content)+ except Exception as e:+ logger.warning(f"Could not read content for {skill_name}: {e}")++ skills.append({+ "name": skill_name,+ "description": description+ })++ return skills++ def _extract_description(self, content: str) -> str:+ """Helper to parse description from markdown frontmatter or body."""+ # 1. Try YAML Frontmatter+ frontmatter_match = re.search(r'^---\n(.*?)\n---', content, re.DOTALL)+ if frontmatter_match:+ fm_text = frontmatter_match.group(1)+ # Simple regex search for description: ... line+ desc_match = re.search(r'description:\s*(.+)$', fm_text, re.MULTILINE)+ if desc_match:+ return desc_match.group(1).strip().strip('"').strip("'")++ # 2. Fallback: Use the first non-header text block+ # Remove headers+ clean_text = re.sub(r'#+\s.*', '', content)+ # Remove code blocks+ clean_text = re.sub(r'```.*?```', '', clean_text, flags=re.DOTALL)+ lines = [line.strip() for line in clean_text.split('\n') if line.strip()]++ if lines:+ return lines[0]++ return "No description available."++ def _generate_relationship_graph(self, skills: List[Dict]) -> List[Dict]:+ """Calls LLM to infer edges between nodes."""++ skills_json = json.dumps(skills, indent=2)++ messages = [+ {"role": "system", "content": RELATIONSHIP_ANALYSIS_SYSTEM_PROMPT},+ {"role": "user", "content": RELATIONSHIP_ANALYSIS_USER_PROMPT_TEMPLATE.format(+ skills_list=skills_json+ )}+ ]++ try:+ response = self.client.chat.completions.create(+ model=self.model,+ messages=messages,+ )+ content = response.choices[0].message.content++ parsed = json.loads(content)+ edges = parsed.get("relationships", [])++ # Validate edges structure+ valid_edges = []+ valid_names = {s['name'] for s in skills}++ for edge in edges:+ if (edge.get('source') in valid_names and+ edge.get('target') in valid_names and+ edge.get('type') in ['similar_to', 'belong_to', 'compose_with', 'depend_on']):+ valid_edges.append(edge)++ logger.info(f"Identified {len(valid_edges)} valid relationships.")+ return valid_edges++ except Exception as e:+ logger.error(f"Failed to analyze relationships: {e}")+ return []\ No newline at end of filediff --git a/skillnet-ai/src/skillnet_ai/cli.py b/skillnet-ai/src/skillnet_ai/cli.pyindex 9f8dfc4..a8af7ed 100644--- a/skillnet-ai/src/skillnet_ai/cli.py+++ b/skillnet-ai/src/skillnet_ai/cli.py@@ -9,6 +9,7 @@ from skillnet_ai.creator import SkillCreatorfrom skillnet_ai.downloader import SkillDownloaderfrom skillnet_ai.evaluator import SkillEvaluator, EvaluatorConfigfrom skillnet_ai.searcher import SkillNetSearcher+from skillnet_ai.analyzer import SkillRelationshipAnalyzerapp = typer.Typer(help="SkillNet AI CLI Tool")console = Console()@@ -397,7 +398,7 @@ def evaluate(description: str = typer.Option(None, help="Short description of what the skill does."),# Config options- model: str = typer.Option("gpt-4o", help="The LLM model used for evaluation."),+ model: str = typer.Option("gpt-4o", "--model", "-m", help="LLM model to use."),max_workers: int = typer.Option(5, help="Concurrency for batch operations (not used for single eval)."),):"""@@ -487,5 +488,90 @@ def _display_evaluation_report(target_name: str, data: dict):if summary:console.print(Panel(summary, title="Executive Summary", border_style="cyan"))+def analyze(+ skills_dir: Path = typer.Argument(..., exists=True, file_okay=False, help="Directory containing multiple skill folders to analyze."),+ save: bool = typer.Option(True, "--save/--no-save", help="Save the result to relationships.json in the directory."),+ model: str = typer.Option("gpt-4o", "--model", "-m", help="LLM model to use."),+):+ """+ Analyze and map relationships (similar_to, belong_to, compose_with, depend_on) between local skills.++ This command scans all subdirectories in the target folder, reads their descriptions,+ and uses AI to build a knowledge graph of how the skills relate to each other.+ """+ # 1. Validate Environment+ if not API_KEY:+ console.print("[bold red]Error:[/bold red] API_KEY environment variable is not set.")+ raise typer.Exit(code=1)++ try:+ # 2. Initialize Analyzer+ analyzer = SkillRelationshipAnalyzer(+ api_key=API_KEY,+ base_url=BASE_URL,+ model=model+ )++ # 3. Visual Feedback & Execution+ console.print(f"[dim]Scanning directory: {os.path.abspath(skills_dir)}[/dim]")++ results = []+ with console.status("[bold green]Reading skills and analyzing relationships...[/bold green]", spinner="earth"):+ results = analyzer.analyze_local_skills(+ skills_dir=str(skills_dir),+ save_to_file=save+ )++ # 4. Handle Empty Results+ if not results:+ console.print("\n[yellow]No strong relationships detected among the skills found.[/yellow]")+ console.print("[dim]Make sure the directory contains subfolders with valid SKILL.md or README.md files.[/dim]")+ return++ # 5. Render Results Table+ console.print(f"\n[bold green]Analysis Complete! Found {len(results)} relationships:[/bold green]\n")++ table = Table(show_header=True, header_style="bold magenta", title="Skill Relationship Graph")+ table.add_column("Source Skill", style="cyan", no_wrap=True)+ table.add_column("Relationship", style="bold white", justify="center")+ table.add_column("Target Skill", style="cyan", no_wrap=True)+ table.add_column("Reasoning", style="dim")++ for edge in results:+ # Color code the relationship types for better readability+ rel_type = edge.get('type', 'unknown')+ rel_style = "white"+ arrow = "->"++ if rel_type == "depend_on":+ rel_style = "red" # Critical dependency+ arrow = "DEPEND ON"+ elif rel_type == "belong_to":+ rel_style = "blue" # Hierarchy+ arrow = "BELONG TO"+ elif rel_type == "compose_with": # pairs_with+ rel_style = "green" # Collaboration+ arrow = "COMPOSE WITH"+ elif rel_type == "similar_to":+ rel_style = "yellow" # Alternative+ arrow = "SIMILAR TO"++ table.add_row(+ edge.get('source', 'Unknown'),+ f"[{rel_style}]{arrow}[/{rel_style}]",+ edge.get('target', 'Unknown'),+ edge.get('reason', '')+ )++ console.print(table)++ if save:+ console.print(f"\n[dim]Relationship data saved to: {os.path.join(skills_dir, 'relationships.json')}[/dim]")++ except Exception as e:+ console.print(f"\n[bold red]Analysis Failed:[/bold red] {str(e)}")+ raise typer.Exit(code=1)+if __name__ == "__main__":app()diff --git a/skillnet-ai/src/skillnet_ai/client.py b/skillnet-ai/src/skillnet_ai/client.pyindex ee548d1..46b3f64 100644--- a/skillnet-ai/src/skillnet_ai/client.py+++ b/skillnet-ai/src/skillnet_ai/client.py@@ -6,6 +6,7 @@ from skillnet_ai.creator import SkillCreatorfrom skillnet_ai.downloader import SkillDownloaderfrom skillnet_ai.evaluator import SkillEvaluator, EvaluatorConfigfrom skillnet_ai.searcher import SkillNetSearcher+from skillnet_ai.analyzer import SkillRelationshipAnalyzerclass SkillNetError(Exception):"""Custom exception class for SkillNet Client errors."""@@ -273,4 +274,42 @@ class SkillNetClient:return resultexcept Exception as e:- raise SkillNetError(f"Evaluation process failed: {str(e)}") from e\ No newline at end of file+ raise SkillNetError(f"Evaluation process failed: {str(e)}") from e++ def analyze(+ self,+ skills_dir: Union[str, Path],+ save_to_file: bool = True,+ model: str = "gpt-4o"+ ) -> List[Dict[str, Any]]:+ """+ Analyze a local directory containing multiple skills to infer relationships between them.++ This builds a knowledge graph (edges) between skills based on their names and descriptions.+ Relationships detected: similar_to, belong_to, pairs_with (compose_with), depend_on.++ Args:+ skills_dir: Path to the directory containing skill folders.+ save_to_file: If True, saves a 'relationships.json' file in the skills_dir.+ model: The LLM model to use for analysis.++ Returns:+ List[Dict[str, Any]]: A list of relationship edges (source, target, type, reason).+ """+ if not self.api_key:+ raise SkillNetError("API_KEY is required for relationship analysis.")++ try:+ analyzer = SkillRelationshipAnalyzer(+ api_key=self.api_key,+ base_url=self.base_url,+ model=model+ )++ results = analyzer.analyze_local_skills(+ skills_dir=str(skills_dir),+ save_to_file=save_to_file+ )+ return results+ except Exception as e:+ raise SkillNetError(f"Relationship analysis failed: {str(e)}") from e\ No newline at end of filediff --git a/skillnet-ai/src/skillnet_ai/prompts.py b/skillnet-ai/src/skillnet_ai/prompts.pyindex 88be845..e57398a 100644--- a/skillnet-ai/src/skillnet_ai/prompts.py+++ b/skillnet-ai/src/skillnet_ai/prompts.py@@ -775,4 +775,68 @@ For every file, use this exact pattern:```Now, generate a complete, high-quality skill package based on the user's request.-Be comprehensive and practical - create a skill that an AI agent would find genuinely useful."""\ No newline at end of file+Be comprehensive and practical - create a skill that an AI agent would find genuinely useful."""+++RELATIONSHIP_ANALYSIS_SYSTEM_PROMPT = """+You are the SkillNet Architect.+"""++RELATIONSHIP_ANALYSIS_USER_PROMPT_TEMPLATE = """+Your task is to map logical relationships between the provided skills based on their names and descriptions.++You must strictly identify ONLY the following 4 types of relationships:++1. similar_to+ - A and B perform functionally equivalent tasks (e.g., "Google Search" and "Bing Search").+ - Users can replace A with B.++2. belong_to+ - A is a sub-component or specific step within B.+ - B represents a larger workflow or agent, and A is just one part of it.+ - Direction: Child -> belong_to -> Parent.++3. compose_with+ - A and B are independent but often used together in a workflow.+ - One usually produces data that the other consumes, or they are logically paired.+ - Example: "PDF Parser" compose_with "Text Summarizer".++4. depend_on+ - A CANNOT execute without B.+ - B is a hard dependency (e.g., Environment setup, API Key loader, or a core library skill).+ - Direction: Dependent -> depend_on -> Prerequisite.++Here is the list of Skills in the user's local environment. Please analyze them and generate the relationships.++Skills List:+{skills_list}++Remember:+- Be conservative. Only create a relationship if there is a logical connection based on the name and description.+- Do not hallucinate skills not in the list.++Output Format:+Return a JSON object with a single key "relationships", which is a list of objects.+Each object must have: "source", "target", "type", and "reason" (short explanation).++Output Example:+{{+ "relationships": [+ {{+ "source": "google_search_tool",+ "target": "bing_search_tool",+ "type": "similar_to",+ "reason": "Both provide web search capabilities and are interchangeable."+ }},+ {{+ "source": "analyze_stock_chart",+ "target": "get_stock_data",+ "type": "depend_on",+ "reason": "Analysis requires the raw data fetcher to run first."+ }},+ ...+ ]+}}++Now, analyze the skills and provide the relationships in the valid JSON format.+"""\ No newline at end of file