Skip to content

Build AI methods with your coding agent, then run them anywhere: as an MCP for chatbots, as a webapp for people, or via API for your software.

Enhance sitemap generation and clarify deployment behavior

Louis Choquelaf00badf2d66
diff --git a/Makefile b/Makefile
index e90a018e5..027895cc1 100644
--- a/Makefile
+++ b/Makefile
@@ -890,7 +890,13 @@ endif
cp docs/404.html "$$TMPDIR/404.html" && \
echo "$$ROOT_ROBOTS_TXT" > "$$TMPDIR/robots.txt" && \
echo "$$ROOT_INDEX_HTML" > "$$TMPDIR/index.html" && \
- sed 's|/$(DOCS_VERSION)/|/latest/|g' "$$TMPDIR/$(DOCS_VERSION)/sitemap.xml" > "$$TMPDIR/sitemap.xml" && \
+# Generate root sitemap with /latest/ URLs.
+# Source from latest/ (not $(DOCS_VERSION)/) so that pre-release deploys don't
+# overwrite the root sitemap with pages that aren't served at /latest/.
+# Mike does NOT rewrite sitemap URLs for aliases — latest/sitemap.xml contains
+# versioned URLs like /0.20.9/page/, identical to the version directory. The sed
+# rewrites any semver path segment (including pre-release suffixes like -rc1) to /latest/.
+ sed 's|/[0-9][0-9]*\.[0-9][0-9]*\.[0-9][0-9]*[^/]*/|/latest/|g' "$$TMPDIR/latest/sitemap.xml" > "$$TMPDIR/sitemap.xml" && \
cd "$$TMPDIR" && \
git add 404.html robots.txt index.html sitemap.xml && \
(git diff --cached --quiet || git commit -m "Update root assets (404.html, robots.txt, index.html, sitemap.xml)") && \
diff --git a/docs/CLAUDE.md b/docs/CLAUDE.md
index a4b5cb453..fcad8e6b2 100644
--- a/docs/CLAUDE.md
+++ b/docs/CLAUDE.md
@@ -38,11 +38,11 @@ There must be **NO** `docs/404.md` — a Markdown 404 would be treated as conten
## robots.txt: Two Files, Different Purposes
- **`docs/robots.txt`** — Lands at `/latest/robots.txt` inside the versioned site. Crawlers ignore it (RFC 9309: only domain-root robots.txt is authoritative). Exists as a courtesy/fallback.
-- **`Makefile` `ROOT_ROBOTS_TXT`** — The authoritative robots.txt deployed to domain root by `docs-deploy-root`. Allows only `/latest/`, disallows everything else, points sitemap to root `/sitemap.xml` (which contains `/latest/` URLs, generated by `docs-deploy-root` from the versioned sitemap via sed rewrite).
+- **`Makefile` `ROOT_ROBOTS_TXT`** — The authoritative robots.txt deployed to domain root by `docs-deploy-root`. Allows only `/latest/`, disallows everything else, points sitemap to root `/sitemap.xml` (which contains `/latest/` URLs, generated by `docs-deploy-root` from the `latest/` alias sitemap via sed version-to-`latest` rewrite).
## Deployment (`docs-deploy-root`)
-The `docs-deploy-root` Makefile target deploys root assets (`404.html`, `robots.txt`, `index.html`, `sitemap.xml`) directly to the `gh-pages` branch via a temporary git worktree. The root `sitemap.xml` is generated by sed-rewriting the versioned sitemap to use `/latest/` URLs.
+The `docs-deploy-root` Makefile target deploys root assets (`404.html`, `robots.txt`, `index.html`, `sitemap.xml`) directly to the `gh-pages` branch via a temporary git worktree. The root `sitemap.xml` is generated from `latest/sitemap.xml` on gh-pages (the alias directory) by sed-rewriting version numbers to `/latest/`. This ensures the root sitemap always reflects what `/latest/` actually serves, even after pre-release deploys — because mike only updates the `latest/` alias directory during stable deploys, not pre-release ones.
- Called automatically after `docs-deploy-stable` and `docs-deploy-specific-version-pre-release`
- The root `index.html` is a meta-refresh redirect to `/latest/`
@@ -62,6 +62,18 @@ To widen the sidebar you must override **two** things:
1. `.md-sidebar { width: ... }` — the container itself
2. `.md-sidebar .md-sidebar__inner { padding-right: 4px }` — the theme dynamically inflates `padding-right` on `.md-sidebar__inner` to absorb extra container width, keeping the nav at its original size. Pin this padding to prevent that.
+## Mike Versioning Behavior (critical — non-obvious)
+
+Mike (`mike deploy X.Y.Z latest`) creates two directories on gh-pages: `X.Y.Z/` and `latest/`. Understanding what mike does and does NOT rewrite is essential:
+
+**What mike rewrites:** HTML files — mike inserts the version prefix into relative URLs, asset paths, and navigation links so pages work from both `/X.Y.Z/` and `/latest/`.
+
+**What mike does NOT rewrite:** Sitemap URLs. `latest/sitemap.xml` is a copy of `X.Y.Z/sitemap.xml` with the same versioned URLs (e.g., `/X.Y.Z/page/`). Mike does NOT replace the version with the alias name. This means you cannot simply copy `latest/sitemap.xml` and expect `/latest/` URLs — you must sed-rewrite them.
+
+**What mike does NOT rewrite:** Canonical tags, OG URLs, JSON-LD — these come from `site_url` + page path at build time. Mike does not touch them during deploy. That's why we override them in `docs/overrides/main.html`.
+
+**The double-path trap:** If `site_url` is set to `https://docs.pipelex.com/latest/`, MkDocs builds URLs like `/latest/page/`. Then mike inserts the version prefix, producing `/latest/X.Y.Z/page/` — a broken double path. This is why `site_url` must be the bare domain.
+
## Common Mistakes to Avoid
- Do NOT create `docs/404.md` — it poisons the sitemap with an indexable `/latest/404/` URL
@@ -70,6 +82,9 @@ To widen the sidebar you must override **two** things:
- Do NOT treat `docs/robots.txt` as the production robots policy — the authoritative one is `ROOT_ROBOTS_TXT` in the Makefile
- When changing URL paths, update BOTH `mkdocs.yml` `redirect_maps` AND `docs/404.html` JS rewrite rules
- Always add `mkdocs-redirects` to CI pip install when `redirect_maps` exist
+- Do NOT set `site_url` to include `/latest/` — mike will create double paths like `/latest/X.Y.Z/page/`
+- Do NOT assume `latest/sitemap.xml` has `/latest/` URLs — mike copies sitemaps verbatim from the version directory without rewriting
+- Do NOT source the root sitemap from `$(DOCS_VERSION)/` — use `latest/` so pre-release deploys don't overwrite it with pages not served at `/latest/`
## Spec vs Blueprint in Documentation
diff --git a/docs/overrides/404.html b/docs/overrides/404.html
index e3f25f51b..4b78dbef4 100644
--- a/docs/overrides/404.html
+++ b/docs/overrides/404.html
@@ -1,9 +1,21 @@
{% extends "main.html" %}
+{# Override site_meta because main.html puts the canonical tag there (inherited from
+ Material's base.html). Without this override, the 404 page would get main.html's
+ canonical for the page path AND 404.html's own canonical — two conflicting tags. #}
+{% block site_meta %}
+ <meta charset="utf-8">
+ <meta name="viewport" content="width=device-width,initial-scale=1">
+ {% if config.site_description %}
+ <meta name="description" content="{{ config.site_description }}">
+ {% endif %}
+ <meta name="robots" content="noindex, nofollow">
+ <link rel="canonical" href="https://docs.pipelex.com/latest/">
+{% endblock %}
+
+{# Empty extrahead to suppress OG tags, Twitter cards, and JSON-LD from main.html.
+ 404 pages should not carry social metadata. #}
{% block extrahead %}
-{{ super() }}
-<meta name="robots" content="noindex, nofollow">
-<link rel="canonical" href="{{ config.site_url }}">
{% endblock %}
{% block content %}
diff --git a/docs/overrides/main.html b/docs/overrides/main.html
index 619d4cc10..016883b0d 100644
--- a/docs/overrides/main.html
+++ b/docs/overrides/main.html
@@ -1,10 +1,16 @@
{# overrides/main.html #}
{% extends "base.html" %}
-{# Force canonical URL to /latest/ path regardless of deployed version.
- site_url is https://docs.pipelex.com/ (bare domain) so that mike can
- properly prefix version paths in sitemaps. We override canonical here
- to always point crawlers at /latest/. #}
+{# We override the site_meta block (not just extrahead) because Material's base.html
+ emits <link rel="canonical"> inside site_meta. If we only added our canonical in
+ extrahead, every page would get TWO conflicting canonical tags.
+
+ Why this is needed: site_url is the bare domain (https://docs.pipelex.com/) so that
+ mike can properly prefix version paths in sitemaps without creating double paths.
+ But we want all canonical URLs to point to /latest/, so we hardcode that prefix here.
+
+ Note: docs/overrides/404.html also overrides this block to emit its own canonical
+ (pointing to /latest/ homepage) and suppress the page-specific one. #}
{% block site_meta %}
<meta charset="utf-8">
diff --git a/mkdocs.yml b/mkdocs.yml
index f6aa204b0..b7e0e2cfb 100644
--- a/mkdocs.yml
+++ b/mkdocs.yml
@@ -1,4 +1,8 @@
site_name: Pipelex Documentation
+# IMPORTANT: site_url must be the bare domain, NOT https://docs.pipelex.com/latest/.
+# Mike inserts the version prefix into sitemap URLs during deployment. If site_url
+# already contains /latest/, mike produces broken double paths like /latest/0.20.9/page/.
+# Canonical URLs are forced to /latest/ via a template override in docs/overrides/main.html.
site_url: https://docs.pipelex.com/
site_description: "Official documentation for Pipelex, an open-source library/CLI for building and running executable AI methods."
docs_dir: docs