Building AI agents, atomically
docs: replace single llms.txt with modular generation system
KennyVaneetvelde83ea634c23ff
diff --git a/.github/workflows/docs.yml b/.github/workflows/docs.ymlindex 813a187..453a023 100644--- a/.github/workflows/docs.yml+++ b/.github/workflows/docs.yml@@ -42,9 +42,15 @@ jobs:poetry run make singlehtmlpoetry run make html- - name: Generate llms.txt+ - name: Generate all llms.txt filesrun: |- poetry run python scripts/generate_llms_txt.py+ poetry run python scripts/generate_llms_files.py++ - name: Copy llms files to build directory+ run: |+ cp docs/llms-*.txt docs/_build/html/+ echo "Generated llms files:"+ ls -lh docs/_build/html/llms-*.txt- name: Upload artifactuses: actions/upload-pages-artifact@v3diff --git a/.gitignore b/.gitignoreindex f0a0c5e..3afa735 100644--- a/.gitignore+++ b/.gitignore@@ -78,6 +78,9 @@ instance/# Sphinx documentationdocs/_build/+docs/llms-*.txt+docs/llms.txt+docs/examples/source/# PyBuildertarget/@@ -150,4 +153,5 @@ debug.log# AI tools.claudeCLAUDE.md+CLAUDE.local.md.serena\ No newline at end of filediff --git a/docs/examples/index.md b/docs/examples/index.mdindex ca44bc7..70c9448 100644--- a/docs/examples/index.md+++ b/docs/examples/index.md@@ -2,6 +2,12 @@This section contains detailed examples of using Atomic Agents in various scenarios.+```{note}+All examples are available in optimized formats for AI assistants:+- **[Examples with documentation](../llms-examples.txt)** - All examples with source code and READMEs+- **[Full framework package](../llms-full.txt)** - Complete documentation, source, and examples+```+## Quickstart ExamplesSimple examples to get started with the framework:@@ -12,7 +18,7 @@ Simple examples to get started with the framework:- Custom input/output schemas- Multiple provider support-[View Quickstart Examples](https://github.com/BrainBlend-AI/atomic-agents/tree/main/atomic-examples/quickstart)+π **[View on GitHub](https://github.com/BrainBlend-AI/atomic-monorepo/tree/main/atomic-examples/quickstart)** - Browse the complete source code and run the examples## Basic Multimodal@@ -23,7 +29,7 @@ Examples of working with images and text:- Visual content generation- Multi-image comparisons-[View Basic Multimodal Examples](https://github.com/BrainBlend-AI/atomic-agents/tree/main/atomic-examples/basic-multimodal)+π **[View on GitHub](https://github.com/BrainBlend-AI/atomic-monorepo/tree/main/atomic-examples/basic-multimodal)** - Browse the complete source code and run the examples## RAG Chatbot@@ -35,7 +41,7 @@ Build context-aware chatbots with retrieval-augmented generation:- Source attribution- Follow-up suggestions-[View RAG Chatbot Examples](https://github.com/BrainBlend-AI/atomic-agents/tree/main/atomic-examples/rag-chatbot)+π **[View on GitHub](https://github.com/BrainBlend-AI/atomic-monorepo/tree/main/atomic-examples/rag-chatbot)** - Browse the complete source code and run the examples## Web Search Agent@@ -47,7 +53,7 @@ Create agents that can search and analyze web content:- Multi-source research- Citation tracking-[View Web Search Examples](https://github.com/BrainBlend-AI/atomic-agents/tree/main/atomic-examples/web-search-agent)+π **[View on GitHub](https://github.com/BrainBlend-AI/atomic-monorepo/tree/main/atomic-examples/web-search-agent)** - Browse the complete source code and run the examples## Deep Research@@ -59,7 +65,7 @@ Perform comprehensive research tasks:- Structured output generation- Citation management-[View Deep Research Examples](https://github.com/BrainBlend-AI/atomic-agents/tree/main/atomic-examples/deep-research)+π **[View on GitHub](https://github.com/BrainBlend-AI/atomic-monorepo/tree/main/atomic-examples/deep-research)** - Browse the complete source code and run the examples## YouTube Summarizer@@ -71,7 +77,7 @@ Extract and analyze information from videos:- Timestamp linking- Chapter generation-[View YouTube Summarizer Examples](https://github.com/BrainBlend-AI/atomic-agents/tree/main/atomic-examples/youtube-summarizer)+π **[View on GitHub](https://github.com/BrainBlend-AI/atomic-monorepo/tree/main/atomic-examples/youtube-summarizer)** - Browse the complete source code and run the examples## YouTube to Recipe@@ -83,7 +89,7 @@ Convert cooking videos into structured recipes:- Step-by-step instructions- Time and temperature conversion-[View YouTube Recipe Examples](https://github.com/BrainBlend-AI/atomic-agents/tree/main/atomic-examples/youtube-to-recipe)+π **[View on GitHub](https://github.com/BrainBlend-AI/atomic-monorepo/tree/main/atomic-examples/youtube-to-recipe)** - Browse the complete source code and run the examples## Orchestration Agent@@ -95,7 +101,7 @@ Coordinate multiple agents for complex tasks:- Error handling- Result aggregation-[View Orchestration Examples](https://github.com/BrainBlend-AI/atomic-agents/tree/main/atomic-examples/orchestration-agent)+π **[View on GitHub](https://github.com/BrainBlend-AI/atomic-monorepo/tree/main/atomic-examples/orchestration-agent)** - Browse the complete source code and run the examples## MCP Agent@@ -108,4 +114,4 @@ Build intelligent agents using the Model Context Protocol:- Extensible tool architecture[View MCP Agent Documentation](mcp_agent.md)-[View MCP Agent Examples](https://github.com/BrainBlend-AI/atomic-agents/tree/main/atomic-examples/mcp-agent)+π **[View on GitHub](https://github.com/BrainBlend-AI/atomic-monorepo/tree/main/atomic-examples/mcp-agent)** - Browse the complete source code and run the examplesdiff --git a/docs/index.md b/docs/index.mdindex 62b69ec..05151f6 100644--- a/docs/index.md+++ b/docs/index.md@@ -14,6 +14,21 @@ contributing+```{admonition} AI Assistant Resources+:class: tip++π₯ **Download Documentation for AI Assistants and LLMs**++Choose the resource that best fits your needs:++- **[π Full Package](llms-full.txt)** - Complete documentation, source code, and examples in one file+- **[π Documentation Only](llms-docs.txt)** - API documentation, guides, and references+- **[π» Source Code Only](llms-source.txt)** - Complete atomic-agents framework source code+- **[π― Examples Only](llms-examples.txt)** - All example implementations with READMEs++All files are optimized for AI assistants and Large Language Models, with clear structure and formatting for easy parsing.+```+The Atomic Agents framework is designed around the concept of atomicity to be an extremely lightweight and modular framework for building Agentic AI pipelines and applications without sacrificing developer experience and maintainability. The framework provides a set of tools and agents that can be combined to create powerful applications. It is built on top of [Instructor](https://github.com/jxnl/instructor) and leverages the power of [Pydantic](https://docs.pydantic.dev/latest/) for data and schema validation and serialization.All logic and control flows are written in Python, enabling developers to apply familiar best practices and workflows from traditional software development without compromising flexibility or clarity.diff --git a/scripts/generate_llms_files.py b/scripts/generate_llms_files.pynew file mode 100644index 0000000..09509be--- /dev/null+++ b/scripts/generate_llms_files.py@@ -0,0 +1,285 @@+#!/usr/bin/env python3+"""+Generate separate and combined llms.txt files:+- llms-docs.txt: Documentation only+- llms-source.txt: Source code only+- llms-examples.txt: Examples only+- llms-full.txt: Everything combined+"""++import os+import sys+from pathlib import Path+from typing import List, Tuple+from bs4 import BeautifulSoup+from markdownify import markdownify as md++# Define paths+PROJECT_ROOT = Path(__file__).parent.parent+DOCS_DIR = PROJECT_ROOT / "docs"+ATOMIC_AGENTS_DIR = PROJECT_ROOT / "atomic-agents"+ATOMIC_EXAMPLES_DIR = PROJECT_ROOT / "atomic-examples"+SINGLEHTML_DIR = DOCS_DIR / "_build" / "singlehtml"++# Output files+OUTPUT_DOCS = DOCS_DIR / "llms-docs.txt"+OUTPUT_SOURCE = DOCS_DIR / "llms-source.txt"+OUTPUT_EXAMPLES = DOCS_DIR / "llms-examples.txt"+OUTPUT_FULL = DOCS_DIR / "llms-full.txt"++# File extensions to include in source code+SOURCE_CODE_EXTENSIONS = {'.py', '.md', '.txt', '.yml', '.yaml', '.toml', '.cfg', '.ini', '.json'}++# Directories to exclude+EXCLUDE_DIRS = {'__pycache__', '.git', '.venv', 'venv', 'env', '.env', 'node_modules',+ 'build', 'dist', '.egg-info', '.pytest_cache', '.mypy_cache', '.ruff_cache'}++def create_section_divider(title: str, level: int = 1) -> str:+ """Create a clear section divider."""+ border = "=" * 80 if level == 1 else "-" * 80+ return f"\n\n{border}\n{title}\n{border}\n\n"++def get_files_to_include(directory: Path, extensions: set) -> List[Tuple[Path, str]]:+ """+ Get all files to include from a directory, respecting exclusions.+ Returns list of (file_path, relative_path_string) tuples.+ """+ files_to_include = []++ for root, dirs, files in os.walk(directory):+ # Remove excluded directories from dirs to prevent walking into them+ dirs[:] = [d for d in dirs if d not in EXCLUDE_DIRS]++ root_path = Path(root)+ for file in files:+ file_path = root_path / file+ if file_path.suffix.lower() in extensions:+ relative_path = file_path.relative_to(directory)+ files_to_include.append((file_path, str(relative_path)))++ return sorted(files_to_include, key=lambda x: x[1])++def extract_documentation_from_html() -> str:+ """Extract documentation from built HTML."""+ html_file = SINGLEHTML_DIR / "index.html"++ if not html_file.exists():+ print("Warning: singlehtml output not found. Build with 'make singlehtml' first.")+ return "Documentation not yet built. Please run 'make singlehtml' in the docs directory.\n"++ with open(html_file, encoding="utf-8") as f:+ html = f.read()++ soup = BeautifulSoup(html, "html.parser")+ doc_main = soup.find("div", {"role": "main"}) or soup.find("div", class_="document")+ content_html = str(doc_main) if doc_main else html++ # Convert to Markdown+ md_text = md(content_html)+ return md_text++def process_source_file(file_path: Path, relative_path: str) -> str:+ """Process a single source file for inclusion."""+ output = f"\n### File: {relative_path}\n\n"++ try:+ with open(file_path, encoding="utf-8", errors='ignore') as f:+ content = f.read()++ # Add code fence for syntax highlighting+ if file_path.suffix == '.py':+ output += "```python\n"+ elif file_path.suffix in ['.yml', '.yaml']:+ output += "```yaml\n"+ elif file_path.suffix == '.json':+ output += "```json\n"+ elif file_path.suffix == '.toml':+ output += "```toml\n"+ elif file_path.suffix == '.md':+ output += "" # Markdown doesn't need code fence+ else:+ output += "```\n"++ output += content++ if file_path.suffix != '.md':+ output += "\n```\n"++ except Exception as e:+ output += f"Error reading file: {e}\n"++ return output++def generate_documentation_only() -> str:+ """Generate documentation-only content."""+ content = create_section_divider("ATOMIC AGENTS DOCUMENTATION", 1)+ content += "This file contains the complete documentation for the Atomic Agents framework.\n"+ content += "Generated for use with Large Language Models and AI assistants.\n"+ content += f"Project Repository: https://github.com/BrainBlend-AI/atomic-monorepo\n"++ content += create_section_divider("DOCUMENTATION", 1)+ content += extract_documentation_from_html()++ content += create_section_divider("END OF DOCUMENT", 1)+ return content++def generate_source_code_only() -> str:+ """Generate source code-only content."""+ content = create_section_divider("ATOMIC AGENTS SOURCE CODE", 1)+ content += "This file contains the complete source code for the Atomic Agents framework.\n"+ content += "Generated for use with Large Language Models and AI assistants.\n"+ content += f"Project Repository: https://github.com/BrainBlend-AI/atomic-monorepo\n"++ files = get_files_to_include(ATOMIC_AGENTS_DIR, SOURCE_CODE_EXTENSIONS)++ for file_path, relative_path in files:+ content += process_source_file(file_path, f"atomic-agents/{relative_path}")++ content += create_section_divider("END OF DOCUMENT", 1)+ return content++def generate_examples_only() -> str:+ """Generate examples-only content."""+ content = create_section_divider("ATOMIC AGENTS EXAMPLES", 1)+ content += "This file contains all example implementations using the Atomic Agents framework.\n"+ content += "Each example includes its README documentation and complete source code.\n"+ content += f"Project Repository: https://github.com/BrainBlend-AI/atomic-monorepo\n"++ # Get all example directories+ example_dirs = [d for d in ATOMIC_EXAMPLES_DIR.iterdir() if d.is_dir() and d.name not in EXCLUDE_DIRS]+ example_dirs.sort()++ for example_dir in example_dirs:+ content += create_section_divider(f"Example: {example_dir.name}", 2)++ # Add GitHub link+ github_url = f"https://github.com/BrainBlend-AI/atomic-monorepo/tree/main/atomic-examples/{example_dir.name}"+ content += f"**View on GitHub:** {github_url}\n\n"++ # Process README first if it exists+ readme_files = list(example_dir.glob("README.*"))+ if readme_files:+ readme_file = readme_files[0]+ content += "## Documentation\n\n"+ try:+ with open(readme_file, encoding="utf-8", errors='ignore') as f:+ content += f.read() + "\n\n"+ except Exception as e:+ content += f"Error reading README: {e}\n\n"++ # Process all source files in the example+ content += "## Source Code\n\n"+ files = get_files_to_include(example_dir, SOURCE_CODE_EXTENSIONS)++ for file_path, relative_path in files:+ # Skip README files as we've already processed them+ if not file_path.name.startswith("README"):+ content += process_source_file(file_path, f"atomic-examples/{example_dir.name}/{relative_path}")++ content += create_section_divider("END OF DOCUMENT", 1)+ return content++def generate_full_content() -> str:+ """Generate the comprehensive llms-full.txt content."""+ content = create_section_divider("ATOMIC AGENTS - COMPREHENSIVE DOCUMENTATION, SOURCE CODE, AND EXAMPLES", 1)+ content += "This file contains the complete documentation, source code, and examples for the Atomic Agents framework.\n"+ content += "Generated for use with Large Language Models and AI assistants.\n"+ content += f"Project Repository: https://github.com/BrainBlend-AI/atomic-monorepo\n"+ content += f"\nTable of Contents:\n"+ content += "1. Documentation\n"+ content += "2. Atomic Agents Source Code\n"+ content += "3. Atomic Examples\n"++ # Section 1: Documentation+ content += create_section_divider("DOCUMENTATION", 1)+ content += "This section contains the full documentation built from the docs folder.\n\n"+ content += extract_documentation_from_html()++ # Section 2: Source Code+ content += create_section_divider("ATOMIC AGENTS SOURCE CODE", 1)+ content += "This section contains the complete source code for the Atomic Agents framework.\n\n"++ files = get_files_to_include(ATOMIC_AGENTS_DIR, SOURCE_CODE_EXTENSIONS)+ for file_path, relative_path in files:+ content += process_source_file(file_path, f"atomic-agents/{relative_path}")++ # Section 3: Examples+ content += create_section_divider("ATOMIC EXAMPLES", 1)+ content += "This section contains all example implementations using the Atomic Agents framework.\n"+ content += "Each example includes its README documentation and complete source code.\n\n"++ example_dirs = [d for d in ATOMIC_EXAMPLES_DIR.iterdir() if d.is_dir() and d.name not in EXCLUDE_DIRS]+ example_dirs.sort()++ for example_dir in example_dirs:+ content += create_section_divider(f"Example: {example_dir.name}", 2)++ github_url = f"https://github.com/BrainBlend-AI/atomic-monorepo/tree/main/atomic-examples/{example_dir.name}"+ content += f"**View on GitHub:** {github_url}\n\n"++ readme_files = list(example_dir.glob("README.*"))+ if readme_files:+ readme_file = readme_files[0]+ content += "## Documentation\n\n"+ try:+ with open(readme_file, encoding="utf-8", errors='ignore') as f:+ content += f.read() + "\n\n"+ except Exception as e:+ content += f"Error reading README: {e}\n\n"++ content += "## Source Code\n\n"+ files = get_files_to_include(example_dir, SOURCE_CODE_EXTENSIONS)++ for file_path, relative_path in files:+ if not file_path.name.startswith("README"):+ content += process_source_file(file_path, f"atomic-examples/{example_dir.name}/{relative_path}")++ content += create_section_divider("END OF DOCUMENT", 1)+ content += "This comprehensive documentation was generated for use with AI assistants and LLMs.\n"+ content += "For the latest version, please visit: https://github.com/BrainBlend-AI/atomic-monorepo\n"++ return content++def write_to_file_and_copy(output_file: Path, content: str):+ """Write content to file and copy to HTML build directory if it exists."""+ with open(output_file, "w", encoding="utf-8") as f:+ f.write(content)+ print(f"Generated {output_file.name} ({len(content):,} characters)")++ # Also copy to the HTML build directory if it exists+ html_output = DOCS_DIR / "_build" / "html" / output_file.name+ if html_output.parent.exists():+ with open(html_output, "w", encoding="utf-8") as f:+ f.write(content)+ print(f" Copied to HTML build directory")++def main():+ """Generate all llms.txt files."""+ print("Generating llms.txt files...")+ print("-" * 40)++ # Generate documentation only+ print("Generating llms-docs.txt...")+ docs_content = generate_documentation_only()+ write_to_file_and_copy(OUTPUT_DOCS, docs_content)++ # Generate source code only+ print("\nGenerating llms-source.txt...")+ source_content = generate_source_code_only()+ write_to_file_and_copy(OUTPUT_SOURCE, source_content)++ # Generate examples only+ print("\nGenerating llms-examples.txt...")+ examples_content = generate_examples_only()+ write_to_file_and_copy(OUTPUT_EXAMPLES, examples_content)++ # Generate full combined file+ print("\nGenerating llms-full.txt...")+ full_content = generate_full_content()+ write_to_file_and_copy(OUTPUT_FULL, full_content)++ print("\n" + "=" * 40)+ print("Successfully generated all llms.txt files!")++if __name__ == "__main__":+ main()\ No newline at end of filediff --git a/scripts/generate_llms_txt.py b/scripts/generate_llms_txt.pydeleted file mode 100644index 1a751c7..0000000--- a/scripts/generate_llms_txt.py+++ /dev/null@@ -1,31 +0,0 @@-import os-from bs4 import BeautifulSoup-from markdownify import markdownify as md--# Paths-docs_dir = os.path.abspath(os.path.join(os.path.dirname(__file__), "..", "docs"))-single_dir = os.path.join(docs_dir, "_build", "singlehtml")-llms_txt_path = os.path.join(docs_dir, "llms.txt")--# Read single HTML index-html_file = os.path.join(single_dir, "index.html")-if not os.path.exists(html_file):- print("Error: singlehtml output not found. Build with sphinx-build -b singlehtml.")- exit(1)--with open(html_file, encoding="utf-8") as f:- html = f.read()--soup = BeautifulSoup(html, "html.parser")-# Extract main documentation content-doc_main = soup.find("div", {"role": "main"}) or soup.find("div", class_="document")-content_html = str(doc_main) if doc_main else html--# Convert to Markdown-md_text = md(content_html)--# Write full documentation to llms.txt-with open(llms_txt_path, "w", encoding="utf-8") as out_f:- out_f.write(md_text)--print(f"Generated full llms.txt at {llms_txt_path}")