Skip to content

armgit / dspy

Public Imported from GitHub
0

DSPy: The framework for programming—not prompting—language models

Python

[Feature] Add automatic llms.txt generation via mkdocs-llmstxt plugin (#8562)

Abraham Ingersolle29d05475589
diff --git a/docs/README.md b/docs/README.md
index df486afba..48c9ed782 100644
--- a/docs/README.md
+++ b/docs/README.md
@@ -72,4 +72,9 @@ This guide is for contributors looking to make changes to the documentation in t
2. **Push your new changes on a new branch**: Feel free to add or edit existing documentation and open a PR for your changes. Once your PR is reviewed and approved, the changes will be ready to merge into main.
-3. **Updating the website**: Once your changes are merged to main, the changes would be reflected on live websites usually in 5-15 mins.
\ No newline at end of file
+3. **Updating the website**: Once your changes are merged to main, the changes would be reflected on live websites usually in 5-15 mins.
+
+## LLMs.txt
+
+The build process generates an `/llms.txt` file for LLM consumption using [mkdocs-llmstxt](https://github.com/pawamoy/mkdocs-llmstxt). Configure sections in `mkdocs.yml` under the `llmstxt` plugin.
+
diff --git a/docs/mkdocs.yml b/docs/mkdocs.yml
index 28ce2b35e..bbab99710 100644
--- a/docs/mkdocs.yml
+++ b/docs/mkdocs.yml
@@ -245,6 +245,39 @@ plugins:
"docs/quick-start/getting-started-02.md": "tutorials/rag/index.ipynb"
"quick-start/getting-started-01.md": "tutorials/rag/index.ipynb"
"quick-start/getting-started-02.md": "tutorials/rag/index.ipynb"
+ - llmstxt:
+ markdown_description: >
+ DSPy is the framework for programming—rather than prompting—language models.
+ DSPy unifies techniques for prompting, fine-tuning, reasoning, tool use, and evaluation of LMs.
+ It provides a systematic approach to building AI applications through composable modules,
+ optimization techniques, and evaluation frameworks.
+ sections:
+ Getting Started:
+ - index.md: DSPy overview and quick start guide
+ - cheatsheet.md: DSPy cheatsheet for quick reference
+ Core Concepts:
+ - learn/programming/overview.md: Programming paradigm and philosophy
+ - learn/programming/signatures.md: Signatures - declarative input/output specifications
+ - learn/programming/modules.md: Modules - composable AI components
+ - learn/programming/language_models.md: Language model interfaces and configuration
+ Essential Tutorials:
+ - tutorials/rag/index.ipynb: Retrieval-Augmented Generation (RAG) tutorial
+ - tutorials/classification/index.md: Classification with DSPy
+ - tutorials/agents/index.ipynb: Building AI agents with DSPy
+ Optimization:
+ - learn/optimization/overview.md: Optimization techniques overview
+ - tutorials/optimize_ai_program/index.md: Guide to optimizing AI programs
+ - api/optimizers/BootstrapFewShot.md: Bootstrap few-shot optimizer
+ Key Modules API:
+ - api/modules/Predict.md: Basic prediction module
+ - api/modules/ChainOfThought.md: Chain of thought reasoning
+ - api/modules/ReAct.md: ReAct agent module
+ Core API Reference:
+ - api/signatures/Signature.md: Signature system documentation
+ - api/primitives/Example.md: Example primitive for training data
+ Production:
+ - tutorials/deployment/index.md: Production deployment guide
+ - tutorials/observability/index.md: Debugging and observability
extra:
social:
diff --git a/docs/requirements.txt b/docs/requirements.txt
index 8083e88fe..36badd232 100644
--- a/docs/requirements.txt
+++ b/docs/requirements.txt
@@ -5,5 +5,6 @@ mkdocs-material[imaging]
mkdocs-redirects
mkdocstrings
mkdocstrings-python
+mkdocs-llmstxt>=0.3.0
urllib3==1.26.6
mistune==3.0.2
diff --git a/docs/vercel.json b/docs/vercel.json
index 1e9c7bc39..9e4754e93 100644
--- a/docs/vercel.json
+++ b/docs/vercel.json
@@ -1,3 +1,14 @@
{
- "trailingSlash": true
+ "trailingSlash": true,
+ "headers": [
+ {
+ "source": "/(.*).md",
+ "headers": [
+ {
+ "key": "Content-Type",
+ "value": "text/markdown; charset=utf-8"
+ }
+ ]
+ }
+ ]
}