Build AI methods with your coding agent, then run them anywhere: as an MCP for chatbots, as a webapp for people, or via API for your software.
Enhance sitemap generation and clarify deployment behavior
Louis Choquelaf00badf2d66
diff --git a/Makefile b/Makefileindex e90a018e5..027895cc1 100644--- a/Makefile+++ b/Makefile@@ -890,7 +890,13 @@ endifcp docs/404.html "$$TMPDIR/404.html" && \echo "$$ROOT_ROBOTS_TXT" > "$$TMPDIR/robots.txt" && \echo "$$ROOT_INDEX_HTML" > "$$TMPDIR/index.html" && \- sed 's|/$(DOCS_VERSION)/|/latest/|g' "$$TMPDIR/$(DOCS_VERSION)/sitemap.xml" > "$$TMPDIR/sitemap.xml" && \+# Generate root sitemap with /latest/ URLs.+# Source from latest/ (not $(DOCS_VERSION)/) so that pre-release deploys don't+# overwrite the root sitemap with pages that aren't served at /latest/.+# Mike does NOT rewrite sitemap URLs for aliases — latest/sitemap.xml contains+# versioned URLs like /0.20.9/page/, identical to the version directory. The sed+# rewrites any semver path segment (including pre-release suffixes like -rc1) to /latest/.+ sed 's|/[0-9][0-9]*\.[0-9][0-9]*\.[0-9][0-9]*[^/]*/|/latest/|g' "$$TMPDIR/latest/sitemap.xml" > "$$TMPDIR/sitemap.xml" && \cd "$$TMPDIR" && \git add 404.html robots.txt index.html sitemap.xml && \(git diff --cached --quiet || git commit -m "Update root assets (404.html, robots.txt, index.html, sitemap.xml)") && \diff --git a/docs/CLAUDE.md b/docs/CLAUDE.mdindex a4b5cb453..fcad8e6b2 100644--- a/docs/CLAUDE.md+++ b/docs/CLAUDE.md@@ -38,11 +38,11 @@ There must be **NO** `docs/404.md` — a Markdown 404 would be treated as conten## robots.txt: Two Files, Different Purposes- **`docs/robots.txt`** — Lands at `/latest/robots.txt` inside the versioned site. Crawlers ignore it (RFC 9309: only domain-root robots.txt is authoritative). Exists as a courtesy/fallback.-- **`Makefile` `ROOT_ROBOTS_TXT`** — The authoritative robots.txt deployed to domain root by `docs-deploy-root`. Allows only `/latest/`, disallows everything else, points sitemap to root `/sitemap.xml` (which contains `/latest/` URLs, generated by `docs-deploy-root` from the versioned sitemap via sed rewrite).+- **`Makefile` `ROOT_ROBOTS_TXT`** — The authoritative robots.txt deployed to domain root by `docs-deploy-root`. Allows only `/latest/`, disallows everything else, points sitemap to root `/sitemap.xml` (which contains `/latest/` URLs, generated by `docs-deploy-root` from the `latest/` alias sitemap via sed version-to-`latest` rewrite).## Deployment (`docs-deploy-root`)-The `docs-deploy-root` Makefile target deploys root assets (`404.html`, `robots.txt`, `index.html`, `sitemap.xml`) directly to the `gh-pages` branch via a temporary git worktree. The root `sitemap.xml` is generated by sed-rewriting the versioned sitemap to use `/latest/` URLs.+The `docs-deploy-root` Makefile target deploys root assets (`404.html`, `robots.txt`, `index.html`, `sitemap.xml`) directly to the `gh-pages` branch via a temporary git worktree. The root `sitemap.xml` is generated from `latest/sitemap.xml` on gh-pages (the alias directory) by sed-rewriting version numbers to `/latest/`. This ensures the root sitemap always reflects what `/latest/` actually serves, even after pre-release deploys — because mike only updates the `latest/` alias directory during stable deploys, not pre-release ones.- Called automatically after `docs-deploy-stable` and `docs-deploy-specific-version-pre-release`- The root `index.html` is a meta-refresh redirect to `/latest/`@@ -62,6 +62,18 @@ To widen the sidebar you must override **two** things:1. `.md-sidebar { width: ... }` — the container itself2. `.md-sidebar .md-sidebar__inner { padding-right: 4px }` — the theme dynamically inflates `padding-right` on `.md-sidebar__inner` to absorb extra container width, keeping the nav at its original size. Pin this padding to prevent that.+## Mike Versioning Behavior (critical — non-obvious)++Mike (`mike deploy X.Y.Z latest`) creates two directories on gh-pages: `X.Y.Z/` and `latest/`. Understanding what mike does and does NOT rewrite is essential:++**What mike rewrites:** HTML files — mike inserts the version prefix into relative URLs, asset paths, and navigation links so pages work from both `/X.Y.Z/` and `/latest/`.++**What mike does NOT rewrite:** Sitemap URLs. `latest/sitemap.xml` is a copy of `X.Y.Z/sitemap.xml` with the same versioned URLs (e.g., `/X.Y.Z/page/`). Mike does NOT replace the version with the alias name. This means you cannot simply copy `latest/sitemap.xml` and expect `/latest/` URLs — you must sed-rewrite them.++**What mike does NOT rewrite:** Canonical tags, OG URLs, JSON-LD — these come from `site_url` + page path at build time. Mike does not touch them during deploy. That's why we override them in `docs/overrides/main.html`.++**The double-path trap:** If `site_url` is set to `https://docs.pipelex.com/latest/`, MkDocs builds URLs like `/latest/page/`. Then mike inserts the version prefix, producing `/latest/X.Y.Z/page/` — a broken double path. This is why `site_url` must be the bare domain.+## Common Mistakes to Avoid- Do NOT create `docs/404.md` — it poisons the sitemap with an indexable `/latest/404/` URL@@ -70,6 +82,9 @@ To widen the sidebar you must override **two** things:- Do NOT treat `docs/robots.txt` as the production robots policy — the authoritative one is `ROOT_ROBOTS_TXT` in the Makefile- When changing URL paths, update BOTH `mkdocs.yml` `redirect_maps` AND `docs/404.html` JS rewrite rules- Always add `mkdocs-redirects` to CI pip install when `redirect_maps` exist+- Do NOT set `site_url` to include `/latest/` — mike will create double paths like `/latest/X.Y.Z/page/`+- Do NOT assume `latest/sitemap.xml` has `/latest/` URLs — mike copies sitemaps verbatim from the version directory without rewriting+- Do NOT source the root sitemap from `$(DOCS_VERSION)/` — use `latest/` so pre-release deploys don't overwrite it with pages not served at `/latest/`## Spec vs Blueprint in Documentationdiff --git a/docs/overrides/404.html b/docs/overrides/404.htmlindex e3f25f51b..4b78dbef4 100644--- a/docs/overrides/404.html+++ b/docs/overrides/404.html@@ -1,9 +1,21 @@{% extends "main.html" %}+{# Override site_meta because main.html puts the canonical tag there (inherited from+ Material's base.html). Without this override, the 404 page would get main.html's+ canonical for the page path AND 404.html's own canonical — two conflicting tags. #}+{% block site_meta %}+ <meta charset="utf-8">+ <meta name="viewport" content="width=device-width,initial-scale=1">+ {% if config.site_description %}+ <meta name="description" content="{{ config.site_description }}">+ {% endif %}+ <meta name="robots" content="noindex, nofollow">+ <link rel="canonical" href="https://docs.pipelex.com/latest/">+{% endblock %}++{# Empty extrahead to suppress OG tags, Twitter cards, and JSON-LD from main.html.+ 404 pages should not carry social metadata. #}{% block extrahead %}-{{ super() }}-<meta name="robots" content="noindex, nofollow">-<link rel="canonical" href="{{ config.site_url }}">{% endblock %}{% block content %}diff --git a/docs/overrides/main.html b/docs/overrides/main.htmlindex 619d4cc10..016883b0d 100644--- a/docs/overrides/main.html+++ b/docs/overrides/main.html@@ -1,10 +1,16 @@{# overrides/main.html #}{% extends "base.html" %}-{# Force canonical URL to /latest/ path regardless of deployed version.- site_url is https://docs.pipelex.com/ (bare domain) so that mike can- properly prefix version paths in sitemaps. We override canonical here- to always point crawlers at /latest/. #}+{# We override the site_meta block (not just extrahead) because Material's base.html+ emits <link rel="canonical"> inside site_meta. If we only added our canonical in+ extrahead, every page would get TWO conflicting canonical tags.++ Why this is needed: site_url is the bare domain (https://docs.pipelex.com/) so that+ mike can properly prefix version paths in sitemaps without creating double paths.+ But we want all canonical URLs to point to /latest/, so we hardcode that prefix here.++ Note: docs/overrides/404.html also overrides this block to emit its own canonical+ (pointing to /latest/ homepage) and suppress the page-specific one. #}{% block site_meta %}<meta charset="utf-8">diff --git a/mkdocs.yml b/mkdocs.ymlindex f6aa204b0..b7e0e2cfb 100644--- a/mkdocs.yml+++ b/mkdocs.yml@@ -1,4 +1,8 @@site_name: Pipelex Documentation+# IMPORTANT: site_url must be the bare domain, NOT https://docs.pipelex.com/latest/.+# Mike inserts the version prefix into sitemap URLs during deployment. If site_url+# already contains /latest/, mike produces broken double paths like /latest/0.20.9/page/.+# Canonical URLs are forced to /latest/ via a template override in docs/overrides/main.html.site_url: https://docs.pipelex.com/site_description: "Official documentation for Pipelex, an open-source library/CLI for building and running executable AI methods."docs_dir: docs