Apache DataFusion SQL Query Engine
Update Release README.md with latest process (#18549)
Andrew Lambd57e215f95fb
diff --git a/dev/release/README.md b/dev/release/README.mdindex 1b78f8d13b..264982091b 100644--- a/dev/release/README.md+++ b/dev/release/README.md@@ -23,55 +23,91 @@ DataFusion typically has major releases around once per month, including breakinPatch releases are made on an adhoc basis, but we try and avoid them given the frequent major releases.-## Branching Policy+## Release Process Overview-- When we prepare a new release, we create a release branch, such as `branch-37` in the Apache repository (not in a fork)-- We update the crate version and generate the changelog in this branch and create a PR against the main branch-- Once the PR is approved and merged, we tag the rc in the release branch, and release from the release branch-- Bug fixes can be merged to the release branch and patch releases can be created from the release branch+New development happens on the `main` branch.+Releases are made from branches, e.g. `branch-37` for the `37.x.y` release series.-#### How to backport (add changes) to `branch-*` branch+To prepare for a new release series, we:-If you would like to propose your change for inclusion in a release branch for a-patch release:+- Create a new branch from `main`, such as `branch-37` in the Apache repository (not in a fork)+- Continue merging new features changes to `main` branch+- Prepare the release branch for release:+ - Update version numbers in `Cargo.toml` files and create `CHANGELOG.md`+ - Add additional changes to the release branch as needed+- When the code is ready, create GitHub tags release candidate (rc) artifacts from the release branch.+- After the release is approved, publish to [crates.io], the ASF distribution servers, and GitHub tags.++To add changes to the release branch, depending on the change we either:++- Fix the issue on `main` and then backport the change to the release branch (e.g. [#18129])+- Fix the issue on the release branch and then forward-port the change back to `main` (e.g.[#18057])++[crates.io]: https://crates.io/crates/datafusion+[#18129]: https://github.com/apache/datafusion/pull/18129+[#18057]: https://github.com/apache/datafusion/pull/18057++## Backporting (add changes) to `branch-*` branch++If you would like to propose your change for inclusion in a patch release, the+change must be applied to the relevant release branch. To do so please follow+these steps:1. Find (or create) the issue for the incremental release ([example release issue]) and discuss the proposed change there with the maintainers.2. Follow normal workflow to create PR to `main` branch and wait for its approval and merge.3. After PR is squash merged to `main`, branch from most recent release branch (e.g. `branch-37`), cherry-pick the commit and create a PR targeting the release branch [example backport PR].-For example, to backport commit `12345` from `main` to `branch-43`:+For example, to backport commit `12345` from `main` to `branch-50`:```shell-git checkout branch-43-git checkout -b backport_to_43-git cherry-pick 12345+git checkout branch-50+git checkout -b backport_to_50+git cherry-pick 12345 # your git commit hashgit push -u <your fork>-# make a PR as normal+# make a PR as normal targeting branch-50, prefixed with [branch-50]+```++It is also acceptable to fix the issue directly on the release branch first+and then cherry-pick the change back to `main` branch in a new PR.++[example release issue]: https://github.com/apache/datafusion/issues/18072+[example backport pr]: https://github.com/apache/datafusion/pull/18131++## Release Prerequisites++### Add git remote for `apache` repo++The instructions below assume the upstream git repo `[email protected]:apache/datafusion.git` in remote `apache`.++```shell+git remote add apache [email protected]:apache/datafusion.git```-[example release issue]: https://github.com/apache/datafusion/issues/9904-[example backport pr]: https://github.com/apache/datafusion/pull/10123+### Create GitHub Personal Access Token (PAT)++A personal access token (PAT) is needed for changelog automation script. If you+do not already have one, create a token with the `repo` access by navigating to+[GitHub Developer Settings] page, and [follow these steps].++[github developer settings]: https://github.com/settings/developers+[follow these steps]: https://docs.github.com/en/authentication/keeping-your-account-and-data-secure/creating-a-personal-access-token-## Release Prerequisite+### Add GPG Public Key to SVN `KEYS` file-- Have upstream git repo `[email protected]:apache/datafusion.git` add as git remote `apache`.-- Created a personal access token in GitHub for changelog automation script.- - Github PAT should be created with `repo` access-- Make sure your signing key is added to the following files in SVN:- - https://dist.apache.org/repos/dist/dev/datafusion/KEYS- - https://dist.apache.org/repos/dist/release/datafusion/KEYS+If you will be releasing the final tarball, your GPG public key must be present in the following SVN files:-### How to add signing key+- https://dist.apache.org/repos/dist/dev/datafusion/KEYS+- https://dist.apache.org/repos/dist/release/datafusion/KEYSSee instructions at https://infra.apache.org/release-signing.html#generate for generating keys.-Committers can add signing keys in Subversion client with their ASF account. e.g.:+Committers can add signing keys using the Subversion client and their ASF account:```shell$ svn co https://dist.apache.org/repos/dist/dev/datafusion$ cd datafusion-$ editor KEYS-$ svn ci KEYS+$ editor KEYS # add your key here+$ svn ci KEYS # commit changes```Follow the instructions in the header of the KEYS file to append your key. Here is an example:@@ -81,35 +117,61 @@ Follow the instructions in the header of the KEYS file to append your key. Heresvn commit KEYS -m "Add key for John Doe"```-## Process Overview+## Release Process: Step by StepAs part of the Apache governance model, official releases consist of signedsource tarballs approved by the PMC.+We then publish the code in the approved artifacts to crates.io.++### 1. Create Release Branch++First create a new release branch from `main` in the apache repository.++For example, to create the `branch-51` branch for the `51.x.y` release series:++```shell+git fetch apache # make sure we are up to date+git checkout apache/main # checkout current latest development branch+git checkout -b branch-51 # create local branch+git push -u apache branch-51 # push branch to apache remote+```++### 2. Prepare PR to Update Changelog and the Release Version++First, prepare a PR to update the changelog and versions to reflect the planned+release. See [#18173](https://github.com/apache/datafusion/pull/18173) for an example.++#### Update Version Numbers-We then use the code in the approved artifacts to release to crates.io and-PyPI.+Manually update the DataFusion version in the root `Cargo.toml` to reflect the new release version.-### Change Log+Ensure Cargo.lock is updated accordingly by running:-We maintain a `CHANGELOG.md` so our users know what has been changed between releases.+```shell+cargo check -p datafusion+```++#### Changelog Generation-You will need a GitHub Personal Access Token for the following steps. Follow-[these instructions](https://docs.github.com/en/authentication/keeping-your-account-and-data-secure/creating-a-personal-access-token)-to generate one if you do not already have one.+We maintain a [changelog] so our users know what has been changed between releases.-The changelog is generated using a Python script. There is a dependency on `PyGitHub`, which can be installed using pip:+[changelog]: ../changelog++The changelog is generated using a Python script.++To run the script, you will need a GitHub Personal Access Token (described in the prerequisites section) and the `PyGitHub` library. First install the `PyGitHub` dependency via `pip`:```shellpip3 install PyGitHub```-To generate the changelog, set the `GITHUB_TOKEN` environment variable to a valid token and then run the script-providing two commit ids or tags followed by the version number of the release being created. The following-example generates a change log of all changes between the first commit and the current HEAD revision.+To generate the changelog, set the `GITHUB_TOKEN` environment variable and then run `./dev/release/generate-changelog.py`+providing two commit ids or tags followed by the version number of the release being created. For example,+to generate a change log of all changes between the `50.3.0` tag and `branch-51`, in preparation for release `51.0.0`:```shellexport GITHUB_TOKEN=<your-token-here>-./dev/release/generate-changelog.py 24.0.0 HEAD 25.0.0 > dev/changelog/25.0.0.md+./dev/release/generate-changelog.py 50.3.0 branch-51 51.0.0 > dev/changelog/51.0.0.md```This script creates a changelog from GitHub PRs based on the labels associated with them as well as looking for@@ -118,15 +180,20 @@ titles starting with `feat:`, `fix:`, or `docs:`.Once the change log is generated, run `prettier` to format the document:```shell-prettier -w dev/changelog/25.0.0md+prettier -w dev/changelog/51.0.0.md```-## Prepare release commits and PR+#### Commit and PR++Then commit the changes and create a PR targeting the release branch.-Prepare a PR to update `CHANGELOG.md` and versions to reflect the planned-release.+```shell+git commit -a -m 'Update version'+```-See [#9697](https://github.com/apache/datafusion/pull/9697) for an example.+Remember to merge any fixes back to `main` branch as well.++### 3. Prepare a PR to Modify `asf.yaml`Modify `asf.yaml` to protect future release candidate branch to prevent accidental merges:@@ -137,119 +204,124 @@ branch-50:required_approving_review_count: 1```-Here are the commands that could be used to prepare the `38.0.0` release:--### Update Version--Checkout the main commit to be released--```shell-git fetch apache-git checkout apache/main-```--Manually update the datafusion version in the root `Cargo.toml` to `38.0.0`.--Run `cargo test` to re-generate some example files:--```shell-cargo test-```--Lastly commit the version change:--```shell-git commit -a -m 'Update version'-```--## Prepare release candidate artifacts+### 4. Prepare Release Candidate ArtifactsAfter the PR gets merged, you are ready to create release artifacts based off themerged commit.(Note you need to be a committer to run these scripts as they upload to the apache svn distribution servers)-### Pick a Release Candidate (RC) number+#### Pick a Release Candidate (RC) number-Pick numbers in sequential order, with `0` for `rc0`, `1` for `rc1`, etc.+Pick numbers in sequential order, with `1` for `rc1`, `2` for `rc2`, etc.-### Create git tag for the release:+#### Create git Tag for the Release:While the official release artifacts are signed tarballs and zip files, we also-tag the commit it was created for convenience and code archaeology.+tag the commit it was created for convenience and code archaeology. Release tags+have the format `<version>` (e.g. `38.0.0`), and release candidates have the+format `<version>-rc<rc>` (e.g. `38.0.0-rc0`). See [the list of existing+tags].++[the list of existing tags]: https://github.com/apache/datafusion/tags++Using a string such as `38.0.0` as the `<version>`, create and push the rc tag by running these commands:++```shell+git fetch apache+git tag <version>-<rc> apache/branch-X # create tag from the release branch+git push apache <version>-<rc> # push tag to Github remote+```-Using a string such as `38.0.0` as the `<version>`, create and push the tag by running these commands:+For example, to create the `50.3.0-rc1 tag from `branch-50`:```shellgit fetch apache-git tag <version>-<rc> apache/main-# push tag to Github remote-git push apache <version>+git tag 50.3.0-rc1 apache/branch-50+git push apache 50.3.0-rc1```-### Create, sign, and upload artifacts+#### Create, Sign, and Upload Artifacts-Run `create-tarball.sh` with the `<version>` tag and `<rc>` and you found in previous steps:+Run the `create-tarball.sh` script with the `<version>` tag and `<rc>` and you determined in previous steps:++For example, to create the `50.3.0-rc1` artifacts:```shell-GH_TOKEN=<TOKEN> ./dev/release/create-tarball.sh 38.0.0 0+GH_TOKEN=<TOKEN> ./dev/release/create-tarball.sh 50.3.0 1```The `create-tarball.sh` script-1. creates and uploads all release candidate artifacts to the [datafusion+1. Creates and uploads all release candidate artifacts to the [datafusiondev](https://dist.apache.org/repos/dist/dev/datafusion) location on the- apache distribution svn server+ apache distribution SVN server-2. provide you an email template to+2. Provides you an email template tosend to [email protected] for release voting.-### Vote on Release Candidate artifacts+### 5. Vote on Release Candidate ArtifactsSend the email output from the script to [email protected].-For the release to become "official" it needs at least three PMC members to vote +1 on it.+In order to publish the release on crates.io, it must be "official". To become+official it needs at least three PMC members to vote +1 on it.-### Verifying Release Candidates+#### Verifying Release CandidatesThe `dev/release/verify-release-candidate.sh` is a script in this repository that can assist in the verification process. Run it like:```shell-./dev/release/verify-release-candidate.sh 38.0.0 0+./dev/release/verify-release-candidate.sh 50.3.0 1```-#### If the release is not approved+#### If the Release is not ApprovedIf the release is not approved, fix whatever the problem is, merge changelog-changes into main if there is any and try again with the next RC number.+changes into the release branch and try again with the next RC number.++Remember to merge any fixes back to `main` branch as well.++#### If the Release is Approved: Call the Vote-## Finalize the release+Call the vote on the Arrow dev list by replying to the RC voting thread. The+reply should have a new subject constructed by adding `[RESULT]` prefix to the+old subject line.++Sample announcement template:++```+The vote has passed with <NUMBER> +1 votes. Thank you to all who helped+with the release verification.+```++### 6. Finalize the ReleaseNOTE: steps in this section can only be done by PMC members.-### After the release is approved+#### After the release is approvedMove artifacts to the release location in SVN, e.g.-https://dist.apache.org/repos/dist/release/datafusion/datafusion-38.0.0/, using+https://dist.apache.org/repos/dist/release/datafusion/datafusion-50.3.0/, usingthe `release-tarball.sh` script:```shell-./dev/release/release-tarball.sh 38.0.0 0+./dev/release/release-tarball.sh 50.3.0 1```Congratulations! The release is now official!-### Create release git tags+### 7. Create Release git tagsTag the same release candidate commit with the final release tag```shell-git co apache/38.0.0-rc0-git tag 38.0.0-git push apache 38.0.0+git co apache/50.3.0-rc1+git tag 50.3.0+git push apache 50.3.0```-### Publish on Crates.io+### 8. Publish on Crates.ioOnly approved releases of the tarball should be published tocrates.io, in order to conform to Apache Software Foundation@@ -261,7 +333,7 @@ been made to crates.io using the following instructions.Follow [theseinstructions](https://doc.rust-lang.org/cargo/reference/publishing.html) tocreate an account and login to crates.io before asking to be added as an owner-to all of the DataFusion crates.+to all DataFusion crates.Download and unpack the official release tarball@@ -312,30 +384,21 @@ Verify that the Cargo.toml in the tarball contains the correct version### Publish datafusion-cli on Homebrew-[`datafusion` formula](https://formulae.brew.sh/formula/datafusion) is [updated automatically](https://github.com/Homebrew/homebrew-core/pulls?q=is%3Apr+datafusion+is%3Aclosed),+Note: [`datafusion` formula](https://formulae.brew.sh/formula/datafusion) is [updated automatically](https://github.com/Homebrew/homebrew-core/pulls?q=is%3Apr+datafusion+is%3Aclosed),so no action is needed.-### Call the vote+### 9: Add the release to Apache Reporter-Call the vote on the Arrow dev list by replying to the RC voting thread. The-reply should have a new subject constructed by adding `[RESULT]` prefix to the-old subject line.--Sample announcement template:--```-The vote has passed with <NUMBER> +1 votes. Thank you to all who helped-with the release verification.-```--### Add the release to Apache Reporter--Add the release to https://reporter.apache.org/addrelease.html?datafusion using the version number e.g. 38.0.0.+When you have published the release, please help the project by adding the release to+[Apache Reporter](https://reporter.apache.org/). The reporter system should+send you a reminder email, but in case you miss it, you can add+the release to https://reporter.apache.org/addrelease.html?datafusion following+the examples from previous releases.The release information is used to generate a template for a board report (see example from Apache Arrow project[here](https://github.com/apache/arrow/pull/14357)).-### Delete old RCs and Releases+### 10: Delete old RCs and ReleasesSee the ASF documentation on [when to archive](https://www.apache.org/legal/release-policy.html#when-to-archive)for more information.@@ -371,29 +434,3 @@ Delete a release:```shellsvn delete -m "delete old DataFusion release" https://dist.apache.org/repos/dist/release/datafusion/datafusion-37.0.0```--### Optional: Write a blog post announcing the release--We typically crowd source release announcements by collaborating on a Google document, usually starting-with a copy of the previous release announcement.--Run the following commands to get the number of commits and number of unique contributors for inclusion in the blog post.--```shell-git log --pretty=oneline 37.0.0..38.0.0 datafusion datafusion-cli datafusion-examples | wc -l-git shortlog -sn 37.0.0..38.0.0 datafusion datafusion-cli datafusion-examples | wc -l-```--Once there is consensus on the contents of the post, create a PR to add a blog post to the-[arrow-site](https://github.com/apache/arrow-site) repository. Note that there is no need for a formal-PMC vote on the blog post contents since this isn't considered to be a "release".--Here is an example blog post PR:--- https://github.com/apache/arrow-site/pull/217--Once the PR is merged, a GitHub action will publish the new blog post to https://arrow.apache.org/blog/.--### Update the version on the download page--Update the version on the [download page](https://datafusion.apache.org/download) to point to the latest release [here](../../docs/source/download.md).