Apache DataFusion SQL Query Engine
Add release management page to the documentation (#21001)
Andrew Lambf99ba69c6189
diff --git a/dev/release/README.md b/dev/release/README.mdindex 533bfc20ee..2f99d52b20 100644--- a/dev/release/README.md+++ b/dev/release/README.md@@ -19,59 +19,11 @@# Release Process-DataFusion typically has major releases around once per month, including breaking API changes.+For contributor-facing guidance on release branches and backports, see the+[Contributor Guide Release Management page](../../docs/source/contributor-guide/release_management.md).-Patch releases are made on an adhoc basis, but we try and avoid them given the frequent major releases.--## Release Process Overview--New development happens on the `main` branch.-Releases are made from branches, e.g. `branch-50` for the `50.x.y` release series.--To prepare for a new release series, we:--- Create a new branch from `main`, such as `branch-50` in the Apache repository (not in a fork)-- Continue merging new features changes to `main` branch-- Prepare the release branch for release:- - Update version numbers in `Cargo.toml` files and create `CHANGELOG.md`- - Add additional changes to the release branch as needed-- When the code is ready, create GitHub tags release candidate (rc) artifacts from the release branch.-- After the release is approved, publish to [crates.io], the ASF distribution servers, and GitHub tags.--To add changes to the release branch, depending on the change we either:--- Fix the issue on `main` and then backport the change to the release branch (e.g. [#18129])-- Fix the issue on the release branch and then forward-port the change back to `main` (e.g.[#18057])--[crates.io]: https://crates.io/crates/datafusion-[#18129]: https://github.com/apache/datafusion/pull/18129-[#18057]: https://github.com/apache/datafusion/pull/18057--## Backporting (add changes) to `branch-*` branch--If you would like to propose your change for inclusion in a patch release, the-change must be applied to the relevant release branch. To do so please follow-these steps:--1. Find (or create) the issue for the incremental release ([example release issue]) and discuss the proposed change there with the maintainers.-2. Follow normal workflow to create PR to `main` branch and wait for its approval and merge.-3. After PR is squash merged to `main`, branch from most recent release branch (e.g. `branch-50`), cherry-pick the commit and create a PR targeting the release branch [example backport PR].--For example, to backport commit `12345` from `main` to `branch-50`:--```shell-git checkout branch-50-git checkout -b backport_to_50-git cherry-pick 12345 # your git commit hash-git push -u <your fork>-# make a PR as normal targeting branch-50, prefixed with [branch-50]-```--It is also acceptable to fix the issue directly on the release branch first-and then cherry-pick the change back to `main` branch in a new PR.--[example release issue]: https://github.com/apache/datafusion/issues/18072-[example backport pr]: https://github.com/apache/datafusion/pull/18131+This guide is for maintainers to create release candidates and run the release+process.## Release Prerequisites@@ -160,33 +112,35 @@ branch-50:First, prepare a PR to update the changelog and versions to reflect the plannedrelease. See [#18173](https://github.com/apache/datafusion/pull/18173) for an example.-#### Update Version Numbers--Manually update the DataFusion version in the root `Cargo.toml` to reflect the new release version.+- This PR should be the last one merged before creating a release+ candidate, so the changelog includes all changes.+- If there are code changes between RCs, create and merge a new PR to update+ the changelog before creating the next RC.-Ensure Cargo.lock is updated accordingly by running:+**Step 1**: Manually update the DataFusion version in the root `Cargo.toml` to+reflect the new release version. Ensure `Cargo.lock` is updated accordingly by+running:```shellcargo check -p datafusion```-#### Changelog Generation--We maintain a [changelog] so our users know what has been changed between releases.--[changelog]: ../changelog--The changelog is generated using a Python script.+**Step 2**: Update the [changelog] in `dev/changelog/`. Each release has its+own file, such as `dev/changelog/50.0.0.md`, which should include all changes+since the previous release.-To run the script, you will need a GitHub Personal Access Token (described in the prerequisites section) and the `PyGitHub` library. First install the dev dependencies via `uv`:+The changelog is generated using a Python script, which requires a GitHub+Personal Access Token (described in the prerequisites section) and the+`PyGitHub` library. First install the dev dependencies via `uv`:```shelluv sync```-To generate the changelog, set the `GITHUB_TOKEN` environment variable and then run `./dev/release/generate-changelog.py`-providing two commit ids or tags followed by the version number of the release being created. For example,-to generate a change log of all changes between the `50.3.0` tag and `branch-51`, in preparation for release `51.0.0`:+To generate the changelog, set the `GITHUB_TOKEN` environment variable and run+`./dev/release/generate-changelog.py` with two commit IDs or tags followed by+the release version. For example, to generate a changelog of all changes+between the `50.3.0` tag and `branch-51` for release `51.0.0`:> [!NOTE]>@@ -205,52 +159,52 @@ uv run ./dev/release/generate-changelog.py 50.3.0 branch-51 51.0.0 > dev/changelThis script creates a changelog from GitHub PRs based on the labels associated with them as well as looking fortitles starting with `feat:`, `fix:`, or `docs:`.-Once the change log is generated, run `prettier` to format the document:+Once the changelog is generated, run `prettier` to format the document:```shellprettier -w dev/changelog/51.0.0.md```-#### Commit and PR-Then commit the changes and create a PR targeting the release branch.```shellgit commit -a -m 'Update version'```-Remember to merge any fixes back to `main` branch as well.+Remember to merge any fixes back to the `main` branch as well.### 4. Prepare Release Candidate ArtifactsAfter the PR gets merged, you are ready to create release artifacts based off themerged commit.-(Note you need to be a committer to run these scripts as they upload to the apache svn distribution servers)+- You must be a committer to run these scripts because they upload to the+ Apache SVN distribution servers.+- If there are code changes between RCs, create and merge a changelog PR before+ creating the next RC.#### Pick a Release Candidate (RC) numberPick numbers in sequential order, with `1` for `rc1`, `2` for `rc2`, etc.-#### Create git Tag for the Release:+#### Create Git tag for the Release CandidateWhile the official release artifacts are signed tarballs and zip files, we also-tag the commit it was created for convenience and code archaeology. Release tags-have the format `<version>` (e.g. `38.0.0`), and release candidates have the-format `<version>-rc<rc>` (e.g. `38.0.0-rc0`). See [the list of existing+tag the commit it was created from for convenience and code archaeology. Release tags+look like `38.0.0`, and release candidate tags look like `38.0.0-rc0`. See [the list of existingtags].[the list of existing tags]: https://github.com/apache/datafusion/tags-Using a string such as `38.0.0` as the `<version>`, create and push the rc tag by running these commands:+Create and push the RC tag:```shellgit fetch apache-git tag <version>-<rc> apache/branch-X # create tag from the release branch-git push apache <version>-<rc> # push tag to Github remote+git tag <version>-rc<rc> apache/branch-X+git push apache <version>-rc<rc>```-For example, to create the `50.3.0-rc1 tag from `branch-50`:+For example, to create the `50.3.0-rc1` tag from `branch-50`, use:```shellgit fetch apache@@ -260,7 +214,7 @@ git push apache 50.3.0-rc1#### Create, Sign, and Upload Artifacts-Run the `create-tarball.sh` script with the `<version>` tag and `<rc>` and you determined in previous steps:+Run the `create-tarball.sh` script with the `<version>` tag and `<rc>` number you determined in previous steps:For example, to create the `50.3.0-rc1` artifacts:@@ -275,18 +229,20 @@ The `create-tarball.sh` scriptapache distribution SVN server2. Provides you an email template to- send to [email protected] for release voting.+ send to `[email protected]` for release voting.### 5. Vote on Release Candidate Artifacts-Send the email output from the script to [email protected].+Send the email output from the script to `[email protected]`.-In order to publish the release on crates.io, it must be "official". To become-official it needs at least three PMC members to vote +1 on it.+In order to publish the release on crates.io, it must be "official." To become+official, it needs at least three PMC members to vote +1 on it and no -1 votes.+The vote must remain open for at least 72 hours to give everyone a chance to+review the release candidate.#### Verifying Release Candidates-The `dev/release/verify-release-candidate.sh` is a script in this repository that can assist in the verification process. Run it like:+`dev/release/verify-release-candidate.sh` is a script in this repository that can assist in the verification process. Run it like this:```shell./dev/release/verify-release-candidate.sh 50.3.0 1@@ -294,15 +250,18 @@ The `dev/release/verify-release-candidate.sh` is a script in this repository tha#### If the Release is not Approved-If the release is not approved, fix whatever the problem is, merge changelog-changes into the release branch and try again with the next RC number.+If the release is not approved, fix whatever the problem is, make a new release+candidate and try again.++Reminders:-Remember to merge any fixes back to `main` branch as well.+1. Update the changelog and create a new PR if there are any code changes, and merge it before creating the next RC.+2. Merge any fixes back to the `main` branch as well.-#### If the Release is Approved: Call the Vote+#### If the Vote Passes: Announce the ResultCall the vote on the Arrow dev list by replying to the RC voting thread. The-reply should have a new subject constructed by adding `[RESULT]` prefix to the+reply should have a new subject constructed by adding the `[RESULT]` prefix to theold subject line.Sample announcement template:@@ -415,39 +374,46 @@ the examples from previous releases.The release information is used to generate a template for a board report (see example from Apache Arrow project[here](https://github.com/apache/arrow/pull/14357)).-### 10: Delete old RCs and Releases+### 10: Delete Old RCs and ReleasesSee the ASF documentation on [when to archive](https://www.apache.org/legal/release-policy.html#when-to-archive)for more information.-#### Deleting old release candidates from `dev` svn-Release candidates should be deleted once the release is published.-Get a list of DataFusion release candidates:+To get a list of DataFusion release candidates:```shellsvn ls https://dist.apache.org/repos/dist/dev/datafusion```-Delete a release candidate:+To delete a release candidate:```shellsvn delete -m "delete old DataFusion RC" https://dist.apache.org/repos/dist/dev/datafusion/apache-datafusion-50.0.0-rc1/```-#### Deleting old releases from `release` svn+#### Delete old releases from `release` SVN-Only the latest release should be available. Delete old releases after publishing the new release.+Only the latest release should be available. Delete old releases after+publishing the new release.-Get a list of DataFusion releases:+To get a list of DataFusion releases:```shellsvn ls https://dist.apache.org/repos/dist/release/datafusion```-Delete a release:+To delete a release:```shellsvn delete -m "delete old DataFusion release" https://dist.apache.org/repos/dist/release/datafusion/datafusion-50.0.0```++### 11. Forward-port changelog to `main`++After the release is published, forward port the version update and changelog to+`main` so that it is up to date for the next release. See [#21053] for an+example PR that forward-ports the changelog to `main`.++[#21053]: https://github.com/apache/datafusion/pull/21053diff --git a/docs/source/contributor-guide/index.md b/docs/source/contributor-guide/index.mdindex 4ace4be494..6ec1efa4d9 100644--- a/docs/source/contributor-guide/index.md+++ b/docs/source/contributor-guide/index.md@@ -101,6 +101,11 @@ If you are concerned that a larger design will be lost in a string of small PRs,Note all commits in a PR are squashed when merged to the `main` branch so there is one commit per PR after merge.+## Release Management and Backports++Contributor-facing guidance for release branches, patch releases, and backports+is documented in the [Release Management](release_management.md) guide.+## Before Submitting a PRBefore submitting a PR, run the standard non-functional checks. PRs must passdiff --git a/docs/source/contributor-guide/release_management.md b/docs/source/contributor-guide/release_management.mdnew file mode 100644index 0000000000..0515204a5e--- /dev/null+++ b/docs/source/contributor-guide/release_management.md@@ -0,0 +1,123 @@+<!---+ Licensed to the Apache Software Foundation (ASF) under one+ or more contributor license agreements. See the NOTICE file+ distributed with this work for additional information+ regarding copyright ownership. The ASF licenses this file+ to you under the Apache License, Version 2.0 (the+ "License"); you may not use this file except in compliance+ with the License. You may obtain a copy of the License at++ http://www.apache.org/licenses/LICENSE-2.0++ Unless required by applicable law or agreed to in writing,+ software distributed under the License is distributed on an+ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY+ KIND, either express or implied. See the License for the+ specific language governing permissions and limitations+ under the License.+-->++# Release Management++This page describes DataFusion release branches and backports. For the+maintainer release guide, including release candidate artifacts, voting, and+publication, see [the release process README in `dev/release`].++## Overview++DataFusion typically has a major release about once per month, including+breaking API changes. Patch releases are made on an ad hoc basis, but we try to+avoid them because major releases are frequent.++New development happens on the [`main` branch]. Releases are made from release+branches named `branch-NN`, such as [`branch-50`] for the `50.x.y` release+series.++In general:++- New features land on [`main`]+- Patch releases are cut from the corresponding `branch-NN`+- Only targeted, low-risk fixes should be added to a release branch++Changes reach a release branch in one of two ways:++- (Most common) Fix the issue on `main` and then backport the merged change to the release branch+- Fix the issue on the release branch and then forward-port the change to `main`++Releases are coordinated in a GitHub issue, such as the+[release issue for 50.3.0]. If you think a fix should be included in a patch+release, discuss it on the relevant tracking issue first. You can also open the+backport PR first and then link it from the tracking issue.++To prepare for a new release series, maintainers:++- Create a new branch from `main`, such as `branch-50`, in the Apache repository+- Continue merging new features to `main`+- Prepare the release branch for release by updating versions, changelog content,+ and any additional release-specific fixes via the+ [Backport Workflow](#backport-workflow)+- Create release candidate artifacts from the release branch+- After approval, publish to crates.io, ASF distribution servers, and Git tags++## Backport Workflow++The usual workflow is:++1. Fix on `main` first, and merge the fix via a normal PR workflow.+2. Cherry-pick the merged commit onto the release branch.+3. Open a backport PR targeting the release branch (examples below).++- [Example backport PR]+- [Additional backport PR example]++### Inputs++To backport a change, gather the following information:++- Target branch, such as `apache/branch-52`+- The release tracking issue URL, such as https://github.com/apache/datafusion/issues/19692+- The original PR URL, such as https://github.com/apache/datafusion/pull/20192+- Optional explicit commit SHA to backport++### Apply the Backport++Start from the target release branch, create a dedicated backport branch, and+use `git cherry-pick`. For example, to backport PR #1234 to `branch-52` when+the commit SHA is `abc123`, run:++```bash+git checkout apache/branch-52+git checkout -b alamb/backport_1234+git cherry-pick abc123+```++### Test++Run tests as described in the [testing documentation].++### Open the PR++Create a PR against the release branch, not `main`, and prefix it with+`[branch-NN]` to show which release branch the backport targets. For example:++- `[branch-52] fix: validate inter-file ordering in eq_properties() (#20329)`++Use a PR description that links the tracking issue, original PR, and target+branch, for example:++```markdown+- Part of <tracking-issue-url>+- Closes <backport-issue-url> on <branch-name>++This PR:++- Backports <original-pr-url> from @<author> to the <branch-name> line+```++[`main` branch]: https://github.com/apache/datafusion/tree/main+[`branch-50`]: https://github.com/apache/datafusion/tree/branch-50+[the release process readme in `dev/release`]: https://github.com/apache/datafusion/blob/main/dev/release/README.md+[release issue for 50.3.0]: https://github.com/apache/datafusion/issues/18072+[example backport pr]: https://github.com/apache/datafusion/pull/18131+[additional backport pr example]: https://github.com/apache/datafusion/pull/20792+[testing documentation]: testing.mddiff --git a/docs/source/index.rst b/docs/source/index.rstindex 4d57faa0cb..cc0da3c444 100644--- a/docs/source/index.rst+++ b/docs/source/index.rst@@ -163,6 +163,7 @@ To get started, seecontributor-guide/testingcontributor-guide/api-healthcontributor-guide/howtos+ contributor-guide/release_managementcontributor-guide/roadmapcontributor-guide/governancecontributor-guide/inviting