Skip to content

Apache DataFusion SQL Query Engine

docs: add version picker and manual release snapshots (#24792)

Ilya Krasnov5d9b7fd45ba0
diff --git a/dev/release/README.md b/dev/release/README.md
index 42af909549..477e01a068 100644
--- a/dev/release/README.md
+++ b/dev/release/README.md
@@ -313,6 +313,74 @@ git tag 55.2.0
git push apache 55.2.0
```
+#### Publish the versioned documentation
+
+**1. Build the documentation from the release tag** using the configuration
+overlay (`docs/scripts/release/conf.py`) from the main checkout. For example,
+for `55.0.0`, from the repository root:
+
+```shell
+export DATAFUSION_DOCS_OVERLAY="$PWD/docs/scripts/release"
+git fetch apache tag 55.0.0
+git worktree add --detach /tmp/datafusion-55.0.0 55.0.0
+cd /tmp/datafusion-55.0.0/docs
+DATAFUSION_DOCS_SOURCE="$PWD/source" DATAFUSION_DOCS_VERSION=55.0.0 \
+ SPHINXOPTS="-W -c $DATAFUSION_DOCS_OVERLAY" \
+ uv run --package datafusion-docs --with sphinx-sitemap ./build.sh
+cd -
+```
+
+Notes:
+
+1. You need to use an absolute path to the overlay in your current checkout.
+2. The `--with sphinx-sitemap` option supplies the sitemap extension for older tags.
+
+To preview the result, build the docs from your main checkout
+(`uv run --package datafusion-docs ./build.sh` in `docs/`), copy the release
+build in under `versions/<version>/`, and serve the tree over HTTP (the version
+picker does not work when the pages are opened directly as files):
+
+```shell
+mkdir -p docs/build/html/versions/55.0.0
+rsync -a /tmp/datafusion-55.0.0/docs/build/html/ docs/build/html/versions/55.0.0/
+python3 -m http.server --directory docs/build/html 8000
+```
+
+Then open http://localhost:8000/ and http://localhost:8000/versions/55.0.0/
+and check the version picker in the navigation bar.
+
+**2. Add the built site to the `asf-site` branch.** Open a PR against
+`asf-site` that adds the release under `versions/<version>/` (for example
+[#25881](https://github.com/apache/datafusion/pull/25881)):
+
+```shell
+git fetch apache asf-site
+git worktree add -b docs/publish-55.0.0 /tmp/datafusion-asf-site apache/asf-site
+rsync -a --exclude '/.buildinfo' \
+ /tmp/datafusion-55.0.0/docs/build/html/ \
+ /tmp/datafusion-asf-site/versions/55.0.0/
+cd /tmp/datafusion-asf-site
+git add versions/55.0.0
+git commit -m "Publish documentation for 55.0.0"
+git push origin docs/publish-55.0.0
+```
+
+The release is live at `https://datafusion.apache.org/versions/55.0.0/` once
+the PR is merged.
+
+**3. Add the release to the version picker.** Open a PR against `main` that
+adds the release to `docs/source/_static/versions.json`. Put the new release
+directly after the `Development (main)` entry and move `"preferred": true` to it, so that
+it is treated as the current stable release. Only add releases that are already
+published, since the picker links to them directly.
+
+Finally, remove the temporary worktrees:
+
+```shell
+git worktree remove /tmp/datafusion-55.0.0
+git worktree remove /tmp/datafusion-asf-site
+```
+
### 10. Publish on Crates.io
Only approved releases of the tarball should be published to
diff --git a/docs/README.md b/docs/README.md
index 48fdcefdea..b55d770cdc 100644
--- a/docs/README.md
+++ b/docs/README.md
@@ -20,8 +20,9 @@
# DataFusion Documentation
This folder contains the source content of the [User Guide](./source/user-guide)
-and [Contributor Guide](./source/contributor-guide). These are both published to
-https://datafusion.apache.org/ as part of the release process.
+and [Contributor Guide](./source/contributor-guide). The site root shows the
+development documentation built from `main`. Released versions of the complete
+site are available under `/versions/<version>/`.
## Dependencies
@@ -48,9 +49,7 @@ Run the provided script to build the HTML pages.
```
The HTML will be generated into a `build` directory. Open `build/html/index.html`
-in your preferred browser, e.g.
-
-Preview the site on Linux by running this command.
+in your preferred browser, for example on Linux via
```bash
# On macOS
@@ -59,6 +58,16 @@ open build/html/index.html
firefox build/html/index.html
```
+Note that some features of the site, such as the version picker, do not work
+when the pages are opened directly as files. To preview such features, serve
+the site over HTTP with sphinx-autobuild:
+
+```bash
+uv run --with sphinx-autobuild sphinx-autobuild source build/html --port 8000
+```
+
+And then open http://localhost:8000/ in your browser
+
## Making Changes
To make changes to the docs, simply make a Pull Request with your
@@ -69,7 +78,7 @@ automatically updated.
This documentation is hosted at https://datafusion.apache.org/
-When the PR is merged to the `main` branch of the DataFusion
+When a PR is merged to the `main` branch of the DataFusion
repository, a [github workflow](https://github.com/apache/datafusion/blob/main/.github/workflows/docs.yaml) which:
1. Builds the html content
diff --git a/docs/scripts/release/conf.py b/docs/scripts/release/conf.py
new file mode 100644
index 0000000000..8776449cb9
--- /dev/null
+++ b/docs/scripts/release/conf.py
@@ -0,0 +1,70 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+"""Sphinx configuration overlay for building docs from a release tag.
+
+Loads ``conf.py`` from a``docs/source`` directory located at
+``DATAFUSION_DOCS_SOURCE`` and overrides the following settings:
+- the version string (``DATAFUSION_DOCS_VERSION``),
+- the site base URL used for the sitemap
+- the version switcher.
+
+Why?
+
+Documentation from older release tags do not have the version picker code and
+expect their base URL to be ``/`` rather than ```versions/<version>/``, so we can
+not publish docs built for the archived version without modification.
+
+Rather than modifying the documentation in the old release branch,
+the release process builds docs with the following command from a main
+checkout:
+
+sphinx-build -c docs/scripts/release
+"""
+
+import os
+from pathlib import Path
+import sys
+
+source = Path(os.environ["DATAFUSION_DOCS_SOURCE"]).resolve()
+sys.path.insert(0, str(source.parent))
+exec(compile((source / "conf.py").read_bytes(), str(source / "conf.py"), "exec"))
+
+# Resolve paths relative to the tag, not this configuration overlay.
+templates_path = [str(source / path) for path in templates_path]
+html_static_path = [str(source / path) for path in html_static_path]
+html_extra_path = [str(source / path) for path in html_extra_path]
+html_logo = str(source / html_logo)
+html_favicon = str(source / html_favicon)
+
+version = release = os.environ["DATAFUSION_DOCS_VERSION"]
+html_baseurl = f"https://datafusion.apache.org/versions/{version}/"
+sitemap_url_scheme = "{link}"
+if "sphinx_sitemap" not in extensions:
+ extensions.append("sphinx_sitemap")
+html_context = {**html_context, "github_repo": "datafusion", "github_version": version}
+html_theme_options = {
+ **html_theme_options,
+ "navbar_end": ["version-switcher", "theme-switcher"],
+ "check_switcher": False,
+ "switcher": {
+ "json_url": "/_static/versions.json",
+ "version_match": version,
+ },
+}
+# The old tag's absolute redirect would leave the release documentation.
+redirects = {**redirects, "library-user-guide/upgrading": "upgrading/index.html"}
diff --git a/docs/source/_static/versions.json b/docs/source/_static/versions.json
new file mode 100644
index 0000000000..5da1af20cb
--- /dev/null
+++ b/docs/source/_static/versions.json
@@ -0,0 +1,18 @@
+[
+ {
+ "name": "Development (main)",
+ "url": "/",
+ "version": "main"
+ },
+ {
+ "name": "55.1.0",
+ "url": "/versions/55.1.0/",
+ "version": "55.1.0",
+ "preferred": true
+ },
+ {
+ "name": "55.0.0",
+ "url": "/versions/55.0.0/",
+ "version": "55.0.0"
+ }
+]
diff --git a/docs/source/conf.py b/docs/source/conf.py
index 68ba628bba..7df696576a 100644
--- a/docs/source/conf.py
+++ b/docs/source/conf.py
@@ -37,6 +37,7 @@ sys.path.insert(0, os.path.abspath(".."))
project = "Apache DataFusion"
copyright = "2019-2025, Apache Software Foundation"
author = "Apache Software Foundation"
+version = release = "main"
# -- General configuration ---------------------------------------------------
@@ -95,7 +96,13 @@ html_theme_options = {
},
"use_edit_page_button": True,
"navbar_center": [],
- "navbar_end": ["theme-switcher"],
+ "navbar_end": ["version-switcher", "theme-switcher"],
+ # Release docs are published separately; the manifest may not be live yet.
+ "check_switcher": False,
+ "switcher": {
+ "json_url": "/_static/versions.json",
+ "version_match": version,
+ },
}
html_context = {