Skip to content

Commit e263eb7

Browse files
andygroveandygrove
authored
Implement new release process (apache#149)
* document the new release process * Update release notes
1 parent 6eb42e7 commit e263eb7

2 files changed

Lines changed: 50 additions & 27 deletions

File tree

‎.github/workflows/build.yml‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -21,7 +21,7 @@ on:
2121
branches: ["main"]
2222
push:
2323
tags: ["*-rc*"]
24-
branches: ["main"]
24+
branches: ["branch-*"]
2525

2626
jobs:
2727
generate-license:

‎dev/release/README.md‎

Lines changed: 49 additions & 26 deletions
Original file line numberDiff line numberDiff line change
@@ -19,22 +19,51 @@
1919

2020
# DataFusion Python Release Process
2121

22-
## Update Version
22+
Development happens on the `main` branch, and most of the time, we depend on DataFusion using GitHub dependencies
23+
rather than using an official release from crates.io. This allows us to pick up new features and bug fixes frequently
24+
by creating PRs to move to a later revision of the code. It also means we can incrementally make updates that are
25+
required due to changes in DataFusion rather than having a large amount of work to do when the next official release
26+
is available.
2327

24-
The version number in Cargo.toml should be increased, according to semver.
28+
When there is a new official release of DataFusion, we update the `main` branch to point to that, update the version
29+
number, and create a new release branch, such as `branch-0.8`. Once this branch is created, we switch the `main` branch
30+
back to using GitHub dependencies. The release activity (such as generating the changelog) can then happen on the
31+
release branch without blocking ongoing development in the `main` branch.
2532

26-
## Update CHANGELOG.md
33+
We can cherry-pick commits from the `main` branch into `branch-0.8` as needed and then create new patch releases
34+
from that branch.
2735

28-
Define release branch (e.g. `main`), base version tag (e.g. `0.6.0`) and future version tag (e.g. `0.7.0`). Commits
29-
between the base version tag and the release branch will be used to populate the changelog content.
36+
## Detailed Guide
37+
38+
### Pre-requisites
3039

31-
You will need a GitHub Personal Access Token for the following steps. Follow
40+
Releases can currently only be created by PMC members due to the permissions needed.
41+
42+
You will need a GitHub Personal Access Token. Follow
3243
[these instructions](https://docs.github.com/en/authentication/keeping-your-account-and-data-secure/creating-a-personal-access-token)
3344
to generate one if you do not already have one.
3445

46+
You will need a PyPI API token. Create one at https://test.pypi.org/manage/account/#api-tokens, setting the “Scope” to
47+
“Entire account”.
48+
49+
You will also need access to the [datafusion](https://test.pypi.org/project/datafusion/) project on testpypi.
50+
51+
### Preparing the `main` Branch
52+
53+
Before creating a new release:
54+
55+
- We need to ensure that the main branch does not have any GitHub dependencies
56+
- a PR should be created and merged to update the major version number of the project
57+
- A new release branch should be created, such as `branch-0.8`
58+
59+
### Update CHANGELOG.md
60+
61+
Define release branch (e.g. `branch-0.8`), base version tag (e.g. `0.7.0`) and future version tag (e.g. `0.9.0`). Commits
62+
between the base version tag and the release branch will be used to populate the changelog content.
63+
3564
```bash
3665
# create the changelog
37-
CHANGELOG_GITHUB_TOKEN=<TOKEN> ./dev/release/update_change_log-datafusion-python.sh main 0.7.0 0.6.0
66+
CHANGELOG_GITHUB_TOKEN=<TOKEN> ./dev/release/update_change_log-datafusion-python.sh main 0.8.0 0.7.0
3867
# review change log / edit issues and labels if needed, rerun until you are happy with the result
3968
git commit -a -m 'Create changelog for release'
4069
```
@@ -46,24 +75,23 @@ value of the `--cpus` argument in the `update_change_log.sh` script._
4675
You can add `invalid` or `development-process` label to exclude items from
4776
release notes.
4877

49-
Send a PR to get these changes merged into `main` branch. If new commits that
50-
could change the change log content landed in the `main` branch before you
51-
could merge the PR, you need to rerun the changelog update script to regenerate
52-
the changelog and update the PR accordingly.
78+
Send a PR to get these changes merged into the release branch (e.g. `branch-0.8`). If new commits that could change the
79+
change log content landed in the release branch before you could merge the PR, you need to rerun the changelog update
80+
script to regenerate the changelog and update the PR accordingly.
5381

54-
## Preparing a Release Candidate
82+
### Preparing a Release Candidate
5583

5684
### Tag the Repository
5785

5886
```bash
59-
git tag 0.7.0-rc1
60-
git push apache 0.7.0-rc1
87+
git tag 0.8.0-rc1
88+
git push apache 0.8.0-rc1
6189
```
6290

6391
### Create a source release
6492

6593
```bash
66-
./dev/release/create_tarball 0.7.0 1
94+
./dev/release/create_tarball 0.8.0 1
6795
```
6896

6997
This will also create the email template to send to the mailing list. Here is an example:
@@ -107,17 +135,12 @@ Create a draft email using this content, but do not send until after completing
107135

108136
### Publish Python Artifacts to testpypi
109137

110-
To securely upload your project, you’ll need a PyPI API token. Create one at
111-
https://test.pypi.org/manage/account/#api-tokens, setting the “Scope” to “Entire account”.
112-
113-
You will also need access to the [datafusion](https://test.pypi.org/project/datafusion/) project on testpypi.
114-
115-
This section assumes some familiary with publishing Python packages to PyPi. For more information, refer to \
138+
This section assumes some familiarity with publishing Python packages to PyPi. For more information, refer to \
116139
[this tutorial](https://packaging.python.org/en/latest/tutorials/packaging-projects/#uploading-the-distribution-archives).
117140

118141
#### Publish Python Wheels to testpypi
119142

120-
Pushing an `rc` tag to main will cause a GitHub Workflow to run that will build the Python wheels.
143+
Pushing an `rc` tag to the release branch will cause a GitHub Workflow to run that will build the Python wheels.
121144

122145
Go to https://github.com/apache/arrow-datafusion-python/actions and look for an action named "Python Release Build"
123146
that has run against the pushed tag.
@@ -195,7 +218,7 @@ Once the vote passes, we can publish the release.
195218
Create the source release tarball:
196219

197220
```bash
198-
./dev/release/release-tarball.sh 0.7.0 1
221+
./dev/release/release-tarball.sh 0.8.0 1
199222
```
200223

201224
### Publishing Python Artifacts
@@ -211,7 +234,7 @@ twine upload --repository pypi dist-release/*
211234
### Push the Release Tag
212235

213236
```bash
214-
git checkout 0.7.0-rc1
215-
git tag 0.7.0
216-
git push apache 0.7.0
237+
git checkout 0.8.0-rc1
238+
git tag 0.8.0
239+
git push apache 0.8.0
217240
```

0 commit comments

Comments
 (0)