Skip to content

Commit 8a3ce59

Browse files
jdye64jdye64
authored
Introduce conda directory containing datafusion-dev.yaml conda enviro… (apache#124)
* Introduce conda directory containing datafusion-dev.yaml conda environment file, conda build files for building and publishing packages to anaconda.org as well as supporting documentation updates to README.md * Added notes for performing conda release
1 parent cb8afac commit 8a3ce59

5 files changed

Lines changed: 155 additions & 1 deletion

File tree

‎README.md‎

Lines changed: 21 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -149,12 +149,18 @@ assert result.column(0) == pyarrow.array([6.0])
149149

150150
## How to install (from pip)
151151

152+
### Pip
152153
```bash
153154
pip install datafusion
154155
# or
155156
python -m pip install datafusion
156157
```
157158

159+
### Conda
160+
```bash
161+
conda install -c conda-forge datafusion
162+
```
163+
158164
You can verify the installation by running:
159165

160166
```python
@@ -163,11 +169,25 @@ You can verify the installation by running:
163169
'0.6.0'
164170
```
165171

172+
166173
## How to develop
167174

168175
This assumes that you have rust and cargo installed. We use the workflow recommended by [pyo3](https://github.com/PyO3/pyo3) and [maturin](https://github.com/PyO3/maturin).
169176

170-
Bootstrap:
177+
The Maturin tools used in this workflow can be installed either via Conda or Pip. Both approaches should offer the same experience. Multiple approaches are only offered to appease developer preference. Bootstrapping for both Conda and Pip are as follows.
178+
179+
Bootstrap (Conda):
180+
181+
```bash
182+
# fetch this repo
183+
git clone git@github.com:apache/arrow-datafusion-python.git
184+
# create the conda environment for dev
185+
conda env create -f ./conda/environments/datafusion-dev.yaml -n datafusion-dev
186+
# activate the conda environment
187+
conda activate datafusion-dev
188+
```
189+
190+
Bootstrap (Pip):
171191

172192
```bash
173193
# fetch this repo

‎conda/environments/datafusion-dev.yaml‎

Lines changed: 41 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,41 @@
1+
# Licensed to the Apache Software Foundation (ASF) under one
2+
# or more contributor license agreements. See the NOTICE file
3+
# distributed with this work for additional information
4+
# regarding copyright ownership. The ASF licenses this file
5+
# to you under the Apache License, Version 2.0 (the
6+
# "License"); you may not use this file except in compliance
7+
# with the License. You may obtain a copy of the License at
8+
#
9+
# http://www.apache.org/licenses/LICENSE-2.0
10+
#
11+
# Unless required by applicable law or agreed to in writing,
12+
# software distributed under the License is distributed on an
13+
# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
14+
# KIND, either express or implied. See the License for the
15+
# specific language governing permissions and limitations
16+
# under the License.
17+
18+
channels:
19+
- conda-forge
20+
dependencies:
21+
- black
22+
- flake8
23+
- isort
24+
- maturin
25+
- mypy
26+
- numpy
27+
- pyarrow
28+
- pytest
29+
- toml
30+
- importlib_metadata
31+
- python>=3.7,<3.11
32+
# Packages useful for building distributions and releasing
33+
- mamba
34+
- conda-build
35+
- anaconda-client
36+
# Packages for documentation building
37+
- sphinx
38+
- pydata-sphinx-theme==0.8.0
39+
- myst-parser
40+
- jinja2
41+
name: datafusion-dev

‎conda/recipes/build.sh‎

Lines changed: 20 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,20 @@
1+
#!/bin/bash
2+
3+
# Licensed to the Apache Software Foundation (ASF) under one
4+
# or more contributor license agreements. See the NOTICE file
5+
# distributed with this work for additional information
6+
# regarding copyright ownership. The ASF licenses this file
7+
# to you under the Apache License, Version 2.0 (the
8+
# "License"); you may not use this file except in compliance
9+
# with the License. You may obtain a copy of the License at
10+
#
11+
# http://www.apache.org/licenses/LICENSE-2.0
12+
#
13+
# Unless required by applicable law or agreed to in writing,
14+
# software distributed under the License is distributed on an
15+
# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
16+
# KIND, either express or implied. See the License for the
17+
# specific language governing permissions and limitations
18+
# under the License.
19+
20+
maturin build -r

‎conda/recipes/meta.yaml‎

Lines changed: 50 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,50 @@
1+
# Licensed to the Apache Software Foundation (ASF) under one
2+
# or more contributor license agreements. See the NOTICE file
3+
# distributed with this work for additional information
4+
# regarding copyright ownership. The ASF licenses this file
5+
# to you under the Apache License, Version 2.0 (the
6+
# "License"); you may not use this file except in compliance
7+
# with the License. You may obtain a copy of the License at
8+
#
9+
# http://www.apache.org/licenses/LICENSE-2.0
10+
#
11+
# Unless required by applicable law or agreed to in writing,
12+
# software distributed under the License is distributed on an
13+
# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
14+
# KIND, either express or implied. See the License for the
15+
# specific language governing permissions and limitations
16+
# under the License.
17+
18+
{% set version = environ.get('GIT_DESCRIBE_TAG', '0.0.0.dev').lstrip('v') + environ.get('VERSION_SUFFIX', '') %}
19+
{% set minor_version = version.split('.')[0] + '.' + version.split('.')[1] %}
20+
21+
package:
22+
name: datafusion
23+
version: {{ version }}
24+
25+
source:
26+
git_url: ../..
27+
28+
build:
29+
noarch: python
30+
number: 0
31+
32+
requirements:
33+
host:
34+
- {{ compiler('rust') }}
35+
- python
36+
- setuptools
37+
- maturin
38+
run:
39+
- python
40+
41+
test:
42+
imports:
43+
- datafusion
44+
45+
about:
46+
home: https://arrow.apache.org/datafusion
47+
license: Apache-2.0
48+
license_family: APACHE
49+
license_file: LICENSE.txt
50+
summary: Apache Arrow DataFusion Python Bindings

‎dev/release/README.md‎

Lines changed: 23 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -148,6 +148,29 @@ This will create a file named `dist/datafusion-0.7.0.tar.gz`. Upload this to tes
148148
python3 -m twine upload --repository testpypi dist/datafusion-0.7.0.tar.gz
149149
```
150150

151+
### Publish Python Artifacts to Anaconda
152+
153+
Publishing artifacts to Anaconda is similar to PyPi. First, Download the source tarball created in the previous step and untar it.
154+
155+
```bash
156+
# Assuming you have an existing conda environment named `datafusion-dev` if not see root README for instructions
157+
conda activate datafusion-dev
158+
conda build .
159+
```
160+
161+
This will setup a virtual conda environment and build the artifacts inside of that virtual env. This step can take a few minutes as the entire build, host, and runtime environments are setup. Once complete a local filesystem path will be emitted for the location of the resulting package. Observe that path and copy to your clipboard.
162+
163+
Ex: `/home/conda/envs/datafusion/conda-bld/linux-64/datafusion-0.7.0.tar.bz2`
164+
165+
Now you are ready to publish this resulting package to anaconda.org. This can be accomplished in a few simple steps.
166+
167+
```bash
168+
# First login to Anaconda with the datafusion credentials
169+
anaconda login
170+
# Upload the package
171+
anaconda upload /home/conda/envs/datafusion/conda-bld/linux-64/datafusion-0.7.0.tar.bz2
172+
```
173+
151174
### Send the Email
152175

153176
Send the email to start the vote.

0 commit comments

Comments
 (0)