Skip to content

Commit 8586a2a

Browse files
jonmmeasejonmmease
authored
[WIP] Use circleci to build conda packages (#1154)
Use circleci to build conda packages Previously the conda package was a noarch Python package that was not specific to Python version of operating system. This works fine, except for one problem. Conda always ensures that all *.py files are byte-compiled to *.pyc when a package is installed, and this can take a long time (~5-10 minutes) for plotly.py since we have so many python files thanks to code generation. To avoid this problem, this PR uses circleci to build conda packages for each Python version, and then converts that build to each operating system using conda convert. This way, each package includes the pre-compiled byte code for the correct version of Python. This results in a much fast installation experience (compared to conda noarch) and a faster initial import (compared to pip wheel).
1 parent d284e70 commit 8586a2a

10 files changed

Lines changed: 206607 additions & 142 deletions

File tree

‎.circleci/config.yml‎

Lines changed: 147 additions & 32 deletions
Original file line numberDiff line numberDiff line change
@@ -2,7 +2,7 @@ version: 2
22

33
jobs:
44
# Core
5-
"python-2.7-core":
5+
python-2.7-core:
66
docker:
77
- image: circleci/python:2.7-stretch-node-browsers
88
environment:
@@ -18,7 +18,7 @@ jobs:
1818
command: 'tox -e py27-core'
1919
no_output_timeout: 20m
2020

21-
"python-3.4-core":
21+
python-3.4-core:
2222
docker:
2323
- image: circleci/python:3.4-stretch-node-browsers
2424
environment:
@@ -34,7 +34,7 @@ jobs:
3434
command: 'tox -e py34-core'
3535
no_output_timeout: 20m
3636

37-
"python-3.5-core":
37+
python-3.5-core:
3838
docker:
3939
- image: circleci/python:3.5-stretch-node-browsers
4040
environment:
@@ -50,7 +50,7 @@ jobs:
5050
command: 'tox -e py35-core'
5151
no_output_timeout: 20m
5252

53-
"python-3.6-core":
53+
python-3.6-core:
5454
docker:
5555
- image: circleci/python:3.6-stretch-node-browsers
5656
environment:
@@ -66,7 +66,7 @@ jobs:
6666
command: 'tox -e py36-core'
6767
no_output_timeout: 20m
6868

69-
"python-3.7-core":
69+
python-3.7-core:
7070
docker:
7171
- image: circleci/python:3.7-stretch-node-browsers
7272
environment:
@@ -83,7 +83,7 @@ jobs:
8383
no_output_timeout: 20m
8484

8585
# Optional
86-
"python-2.7-optional":
86+
python-2.7-optional:
8787
docker:
8888
- image: circleci/python:2.7-stretch-node-browsers
8989
environment:
@@ -103,7 +103,7 @@ jobs:
103103
command: 'tox -e py27-optional'
104104
no_output_timeout: 20m
105105

106-
"python-3.4-optional":
106+
python-3.4-optional:
107107
docker:
108108
- image: circleci/python:3.4-stretch-node-browsers
109109
environment:
@@ -123,7 +123,7 @@ jobs:
123123
command: 'tox -e py34-optional'
124124
no_output_timeout: 20m
125125

126-
"python-3.5-optional":
126+
python-3.5-optional:
127127
docker:
128128
- image: circleci/python:3.5-stretch-node-browsers
129129
environment:
@@ -143,7 +143,7 @@ jobs:
143143
command: 'tox -e py35-optional'
144144
no_output_timeout: 20m
145145

146-
"python-3.6-optional":
146+
python-3.6-optional:
147147
docker:
148148
- image: circleci/python:3.6-stretch-node-browsers
149149
environment:
@@ -163,7 +163,7 @@ jobs:
163163
command: 'tox -e py36-optional'
164164
no_output_timeout: 20m
165165

166-
"python-3.7-optional":
166+
python-3.7-optional:
167167
docker:
168168
- image: circleci/python:3.7-stretch-node-browsers
169169
environment:
@@ -184,7 +184,7 @@ jobs:
184184
no_output_timeout: 20m
185185

186186
# Plot.ly
187-
"python-2.7-plot_ly":
187+
python-2.7-plot_ly:
188188
docker:
189189
- image: circleci/python:2.7-stretch-node-browsers
190190
environment:
@@ -200,7 +200,7 @@ jobs:
200200
command: 'tox -e py27-plot_ly'
201201
no_output_timeout: 20m
202202

203-
"python-3.4-plot_ly":
203+
python-3.4-plot_ly:
204204
docker:
205205
- image: circleci/python:3.4-stretch-node-browsers
206206
environment:
@@ -216,7 +216,7 @@ jobs:
216216
command: 'tox -e py34-plot_ly'
217217
no_output_timeout: 20m
218218

219-
"python-3.7-plot_ly":
219+
python-3.7-plot_ly:
220220
docker:
221221
- image: circleci/python:3.7-stretch-node-browsers
222222
environment:
@@ -232,7 +232,7 @@ jobs:
232232
command: 'tox -e py37-plot_ly'
233233
no_output_timeout: 20m
234234

235-
"python-2-7-orca":
235+
python-2-7-orca:
236236
docker:
237237
- image: circleci/node:10.9-stretch-browsers
238238
environment:
@@ -262,7 +262,7 @@ jobs:
262262
- store_artifacts:
263263
path: plotly/tests/test_orca/images/linux/failed
264264

265-
"python-3-5-orca":
265+
python-3-5-orca:
266266
docker:
267267
- image: circleci/node:10.9-stretch-browsers
268268
environment:
@@ -292,7 +292,7 @@ jobs:
292292
- store_artifacts:
293293
path: plotly/tests/test_orca/images/linux/failed
294294

295-
"python-3-7-orca":
295+
python-3-7-orca:
296296
docker:
297297
- image: circleci/node:10.9-stretch-browsers
298298
environment:
@@ -322,24 +322,139 @@ jobs:
322322
- store_artifacts:
323323
path: plotly/tests/test_orca/images/linux/failed
324324

325+
326+
python-2-7-conda:
327+
docker:
328+
- image: continuumio/miniconda3:4.5.4
329+
environment:
330+
PYTHON_VERSION: 2.7
331+
332+
steps:
333+
- checkout
334+
- run:
335+
name: Create packages
336+
command: .circleci/create_conda_packages.sh
337+
- store_artifacts:
338+
path: artifacts
339+
340+
python-3-5-conda:
341+
docker:
342+
- image: continuumio/miniconda3:4.5.4
343+
environment:
344+
PYTHON_VERSION: 3.5
345+
346+
steps:
347+
- checkout
348+
- run:
349+
name: Create packages
350+
command: .circleci/create_conda_packages.sh
351+
- store_artifacts:
352+
path: artifacts
353+
354+
python-3-6-conda:
355+
docker:
356+
- image: continuumio/miniconda3:4.5.4
357+
environment:
358+
PYTHON_VERSION: 3.6
359+
360+
steps:
361+
- checkout
362+
- run:
363+
name: Create packages
364+
command: .circleci/create_conda_packages.sh
365+
- store_artifacts:
366+
path: artifacts
367+
368+
python-3-7-conda:
369+
docker:
370+
- image: continuumio/miniconda3:4.5.4
371+
environment:
372+
PYTHON_VERSION: 3.7
373+
374+
steps:
375+
- checkout
376+
- run:
377+
name: Create packages
378+
command: .circleci/create_conda_packages.sh
379+
- store_artifacts:
380+
path: artifacts
381+
382+
325383
workflows:
326384
version: 2
327385
build:
328386
jobs:
329-
- "python-2.7-core"
330-
- "python-3.4-core"
331-
- "python-3.5-core"
332-
- "python-3.6-core"
333-
- "python-3.7-core"
334-
- "python-2.7-optional"
335-
- "python-3.4-optional"
336-
- "python-3.5-optional"
337-
- "python-3.6-optional"
387+
388+
- python-2.7-core
389+
- python-3.4-core
390+
- python-3.5-core
391+
- python-3.6-core
392+
- python-3.7-core
393+
- python-2.7-optional
394+
- python-3.4-optional
395+
- python-3.5-optional
396+
- python-3.6-optional
338397
# 3.7 optional disabled due to current shapely incompatibility
339-
# - "python-3.7-optional"
340-
- "python-2.7-plot_ly"
341-
- "python-3.4-plot_ly"
342-
- "python-3.7-plot_ly"
343-
- "python-2-7-orca"
344-
- "python-3-5-orca"
345-
- "python-3-7-orca"
398+
# - python-3.7-optional
399+
- python-2.7-plot_ly
400+
- python-3.4-plot_ly
401+
- python-3.7-plot_ly
402+
- python-2-7-orca
403+
- python-3-5-orca
404+
- python-3-7-orca
405+
406+
- python-2-7-conda:
407+
requires:
408+
- python-2.7-core
409+
- python-2.7-optional
410+
- python-2.7-plot_ly
411+
- python-2-7-orca
412+
413+
filters:
414+
branches:
415+
only:
416+
- master
417+
- /release_.*/
418+
- /conda_build_.*/
419+
420+
- python-3-5-conda:
421+
requires:
422+
- python-3.5-core
423+
- python-3.5-optional
424+
# - python-3.5-plot_ly
425+
- python-3-5-orca
426+
427+
filters:
428+
branches:
429+
only:
430+
- master
431+
- /release_.*/
432+
- /conda_build_.*/
433+
434+
- python-3-6-conda:
435+
requires:
436+
- python-3.6-core
437+
- python-3.6-optional
438+
# - python-3.6-plot_ly
439+
# - python-3-6-orca
440+
441+
filters:
442+
branches:
443+
only:
444+
- master
445+
- /release_.*/
446+
- /conda_build_.*/
447+
448+
- python-3-7-conda:
449+
requires:
450+
- python-3.7-core
451+
# - python-3.7-optional
452+
- python-3.7-plot_ly
453+
- python-3-7-orca
454+
455+
filters:
456+
branches:
457+
only:
458+
- master
459+
- /release_.*/
460+
- /conda_build_.*/

‎.circleci/create_conda_packages.sh‎

Lines changed: 17 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,17 @@
1+
# Install conda dependencies
2+
conda install -y conda-build conda-verify zip
3+
4+
# Perform build
5+
conda build --python $PYTHON_VERSION recipe/
6+
7+
# Convert to other architectures
8+
mkdir -p ./conda_packages_${PYTHON_VERSION}/linux-64/
9+
cp /opt/conda/conda-bld/linux-64/plotly-*.tar.bz2 ./conda_packages_${PYTHON_VERSION}/linux-64/
10+
conda convert -p linux-32 ./conda_packages_${PYTHON_VERSION}/linux-64/plotly-*.tar.bz2 -o ./conda_packages_${PYTHON_VERSION}/
11+
conda convert -p osx-64 ./conda_packages_${PYTHON_VERSION}/linux-64/plotly-*.tar.bz2 -o ./conda_packages_${PYTHON_VERSION}/
12+
conda convert -p win-64 ./conda_packages_${PYTHON_VERSION}/linux-64/plotly-*.tar.bz2 -o ./conda_packages_${PYTHON_VERSION}/
13+
conda convert -p win-32 ./conda_packages_${PYTHON_VERSION}/linux-64/plotly-*.tar.bz2 -o ./conda_packages_${PYTHON_VERSION}/
14+
15+
# zip up packages into artifacts directory
16+
mkdir artifacts
17+
zip -r artifacts/conda_packages_${PYTHON_VERSION}.zip ./conda_packages_${PYTHON_VERSION}/

‎.circleci/install_miniconda_and_build_.sh‎

Lines changed: 15 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,15 @@
1+
#!/usr/bin/env bash
2+
3+
if [ ! -d $HOME/miniconda/ ]; then
4+
# Download miniconda
5+
wget http://repo.continuum.io/miniconda/Miniconda3-latest-Linux-x86_64.sh -O miniconda.sh
6+
7+
chmod +x miniconda.sh
8+
9+
# Install miniconda
10+
./miniconda.sh -b -p $HOME/miniconda
11+
12+
# Create environment
13+
# PYTHON_VERSION=3.6
14+
$HOME/miniconda/bin/conda install conda-build conda-verify
15+
fi

‎.gitignore‎

Lines changed: 0 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -22,9 +22,6 @@ plotly/presentation_objs/
2222

2323
js/node_modules/
2424

25-
# Compiled javascript
26-
plotlywidget/static/
27-
2825
.pytest_cache
2926

3027
# virtual envs

0 commit comments

Comments
 (0)