DSPy: The framework for programming—not prompting—language models
Python
add llms.txt generation tutorial (#8421)
TomuHirata94299c7edbbe
diff --git a/docs/docs/tutorials/llms_txt_generation/index.md b/docs/docs/tutorials/llms_txt_generation/index.mdnew file mode 100644index 000000000..6e2872a15--- /dev/null+++ b/docs/docs/tutorials/llms_txt_generation/index.md@@ -0,0 +1,244 @@+# Generating llms.txt for Code Documentation with DSPy++This tutorial demonstrates how to use DSPy to automatically generate an `llms.txt` file for the DSPy repository itself. The `llms.txt` standard provides LLM-friendly documentation that helps AI systems better understand codebases.++## What is llms.txt?++`llms.txt` is a proposed standard for providing structured, LLM-friendly documentation about a project. It typically includes:++- Project overview and purpose+- Key concepts and terminology+- Architecture and structure+- Usage examples+- Important files and directories++## Building a DSPy Program for llms.txt Generation++Let's create a DSPy program that analyzes a repository and generates comprehensive `llms.txt` documentation.++### Step 1: Define Our Signatures++First, we'll define signatures for different aspects of documentation generation:++```python+import dspy+from typing import List++class AnalyzeRepository(dspy.Signature):+ """Analyze a repository structure and identify key components."""+ repo_url: str = dspy.InputField(desc="GitHub repository URL")+ file_tree: str = dspy.InputField(desc="Repository file structure")+ readme_content: str = dspy.InputField(desc="README.md content")++ project_purpose: str = dspy.OutputField(desc="Main purpose and goals of the project")+ key_concepts: List[str] = dspy.OutputField(desc="List of important concepts and terminology")+ architecture_overview: str = dspy.OutputField(desc="High-level architecture description")++class AnalyzeCodeStructure(dspy.Signature):+ """Analyze code structure to identify important directories and files."""+ file_tree: str = dspy.InputField(desc="Repository file structure")+ package_files: str = dspy.InputField(desc="Key package and configuration files")++ important_directories: List[str] = dspy.OutputField(desc="Key directories and their purposes")+ entry_points: List[str] = dspy.OutputField(desc="Main entry points and important files")+ development_info: str = dspy.OutputField(desc="Development setup and workflow information")++class GenerateLLMsTxt(dspy.Signature):+ """Generate a comprehensive llms.txt file from analyzed repository information."""+ project_purpose: str = dspy.InputField()+ key_concepts: List[str] = dspy.InputField()+ architecture_overview: str = dspy.InputField()+ important_directories: List[str] = dspy.InputField()+ entry_points: List[str] = dspy.InputField()+ development_info: str = dspy.InputField()+ usage_examples: str = dspy.InputField(desc="Common usage patterns and examples")++ llms_txt_content: str = dspy.OutputField(desc="Complete llms.txt file content following the standard format")+```++### Step 2: Create the Repository Analyzer Module++```python+class RepositoryAnalyzer(dspy.Module):+ def __init__(self):+ super().__init__()+ self.analyze_repo = dspy.ChainOfThought(AnalyzeRepository)+ self.analyze_structure = dspy.ChainOfThought(AnalyzeCodeStructure)+ self.generate_examples = dspy.ChainOfThought("repo_info -> usage_examples")+ self.generate_llms_txt = dspy.ChainOfThought(GenerateLLMsTxt)++ def forward(self, repo_url, file_tree, readme_content, package_files):+ # Analyze repository purpose and concepts+ repo_analysis = self.analyze_repo(+ repo_url=repo_url,+ file_tree=file_tree,+ readme_content=readme_content+ )++ # Analyze code structure+ structure_analysis = self.analyze_structure(+ file_tree=file_tree,+ package_files=package_files+ )++ # Generate usage examples+ usage_examples = self.generate_examples(+ repo_info=f"Purpose: {repo_analysis.project_purpose}\nConcepts: {repo_analysis.key_concepts}"+ )++ # Generate final llms.txt+ llms_txt = self.generate_llms_txt(+ project_purpose=repo_analysis.project_purpose,+ key_concepts=repo_analysis.key_concepts,+ architecture_overview=repo_analysis.architecture_overview,+ important_directories=structure_analysis.important_directories,+ entry_points=structure_analysis.entry_points,+ development_info=structure_analysis.development_info,+ usage_examples=usage_examples.usage_examples+ )++ return dspy.Prediction(+ llms_txt_content=llms_txt.llms_txt_content,+ analysis=repo_analysis,+ structure=structure_analysis+ )+```++### Step 3: Gather Repository Information++Let's create helper functions to extract repository information:++```python+import requests+import os+from pathlib import Path++def get_github_file_tree(repo_url):+ """Get repository file structure from GitHub API."""+ # Extract owner/repo from URL+ parts = repo_url.rstrip('/').split('/')+ owner, repo = parts[-2], parts[-1]++ api_url = f"https://api.github.com/repos/{owner}/{repo}/git/trees/main?recursive=1"+ response = requests.get(api_url)++ if response.status_code == 200:+ tree_data = response.json()+ file_paths = [item['path'] for item in tree_data['tree'] if item['type'] == 'blob']+ return '\n'.join(sorted(file_paths))+ else:+ raise Exception(f"Failed to fetch repository tree: {response.status_code}")++def get_github_file_content(repo_url, file_path):+ """Get specific file content from GitHub."""+ parts = repo_url.rstrip('/').split('/')+ owner, repo = parts[-2], parts[-1]++ api_url = f"https://api.github.com/repos/{owner}/{repo}/contents/{file_path}"+ response = requests.get(api_url)++ if response.status_code == 200:+ import base64+ content = base64.b64decode(response.json()['content']).decode('utf-8')+ return content+ else:+ return f"Could not fetch {file_path}"++def gather_repository_info(repo_url):+ """Gather all necessary repository information."""+ file_tree = get_github_file_tree(repo_url)+ readme_content = get_github_file_content(repo_url, "README.md")++ # Get key package files+ package_files = []+ for file_path in ["pyproject.toml", "setup.py", "requirements.txt", "package.json"]:+ try:+ content = get_github_file_content(repo_url, file_path)+ if "Could not fetch" not in content:+ package_files.append(f"=== {file_path} ===\n{content}")+ except:+ continue++ package_files_content = "\n\n".join(package_files)++ return file_tree, readme_content, package_files_content+```++### Step 4: Configure DSPy and Generate llms.txt++```python+def generate_llms_txt_for_dspy():+ # Configure DSPy (use your preferred LM)+ lm = dspy.LM(model="gpt-4o-mini")+ dspy.configure(lm=lm)+ os.environ["OPENAI_API_KEY"] = "<YOUR OPENAI KEY>"++ # Initialize our analyzer+ analyzer = RepositoryAnalyzer()++ # Gather DSPy repository information+ repo_url = "https://github.com/stanfordnlp/dspy"+ file_tree, readme_content, package_files = gather_repository_info(repo_url)++ # Generate llms.txt+ result = analyzer(+ repo_url=repo_url,+ file_tree=file_tree,+ readme_content=readme_content,+ package_files=package_files+ )++ return result++# Run the generation+if __name__ == "__main__":+ result = generate_llms_txt_for_dspy()++ # Save the generated llms.txt+ with open("llms.txt", "w") as f:+ f.write(result.llms_txt_content)++ print("Generated llms.txt file!")+ print("\nPreview:")+ print(result.llms_txt_content[:500] + "...")+```++## Expected Output Structure++The generated `llms.txt` for DSPy would follow this structure:++```+# DSPy: Programming Language Models++## Project Overview+DSPy is a framework for programming—rather than prompting—language models...++## Key Concepts+- **Modules**: Building blocks for LM programs+- **Signatures**: Input/output specifications+- **Teleprompters**: Optimization algorithms+- **Predictors**: Core reasoning components++## Architecture+- `/dspy/`: Main package directory+ - `/adapters/`: Input/output format handlers+ - `/clients/`: LM client interfaces+ - `/predict/`: Core prediction modules+ - `/teleprompt/`: Optimization algorithms++## Usage Examples+1. **Building a Classifier**: Using DSPy, a user can define a modular classifier that takes in text data and categorizes it into predefined classes. The user can specify the classification logic declaratively, allowing for easy adjustments and optimizations.+2. **Creating a RAG Pipeline**: A developer can implement a retrieval-augmented generation pipeline that first retrieves relevant documents based on a query and then generates a coherent response using those documents. DSPy facilitates the integration of retrieval and generation components seamlessly.+3. **Optimizing Prompts**: Users can leverage DSPy to create a system that automatically optimizes prompts for language models based on performance metrics, improving the quality of responses over time without manual intervention.+4. **Implementing Agent Loops**: A user can design an agent loop that continuously interacts with users, learns from feedback, and refines its responses, showcasing the self-improving capabilities of the DSPy framework.+5. **Compositional Code**: Developers can write compositional code that allows different modules of the AI system to interact with each other, enabling complex workflows that can be easily modified and extended.+```++The resulting `llms.txt` file provides a comprehensive, LLM-friendly overview of the DSPy repository that can help other AI systems better understand and work with the codebase.++## Next Steps++- Extend the program to analyze multiple repositories+- Add support for different documentation formats+- Create metrics for documentation quality assessment+- Build a web interface for interactive repository analysisdiff --git a/docs/mkdocs.yml b/docs/mkdocs.ymlindex 8ab5dfae9..fcb2128b9 100644--- a/docs/mkdocs.yml+++ b/docs/mkdocs.yml@@ -59,6 +59,8 @@ nav:- Tracking DSPy Optimizers: tutorials/optimizer_tracking/index.md- Streaming: tutorials/streaming/index.md- Async: tutorials/async/index.md+ - Real-World Examples:+ - Generating llms.txt: tutorials/llms_txt_generation/index.md- DSPy in Production: production/index.md- Community:- Community Resources: community/community-resources.md