Skip to content

Commit 7e69d04

Browse files
PGijsbersPGijsbers
authored
Docker enhancement #1277 (#1278)
* Add multiple build platforms * Automatically update Dockerhub description * Launch Python instead of Bash by default * Change `omlp` directory name to less cryptic `openml` * Change directory to `openml` for running purpose of running script For mounted scripts, instructions say to mount them to `/openml`, so we have to `cd` before invoking `python`. * Update readme to reflect updates (python by default, rename dirs) * Add branch/code for doc and test examples as they are required * Ship docker images with readme * Only update readme on release, also try build docker on PR * Update the toc descriptions
1 parent 1d7fbda commit 7e69d04

File tree

4 files changed

+135
-68
lines changed

4 files changed

+135
-68
lines changed

‎.github/workflows/release_docker.yaml‎

Lines changed: 18 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -3,9 +3,13 @@ name: release-docker
33
on:
44
push:
55
branches:
6-
- 'main'
76
- 'develop'
87
- 'docker'
8+
tags:
9+
- 'v*'
10+
pull_request:
11+
branches:
12+
- 'develop'
913

1014
jobs:
1115

@@ -21,6 +25,7 @@ jobs:
2125
uses: docker/setup-buildx-action@v2
2226

2327
- name: Login to DockerHub
28+
if: github.event_name != 'pull_request'
2429
uses: docker/login-action@v2
2530
with:
2631
username: ${{ secrets.DOCKERHUB_USERNAME }}
@@ -40,9 +45,20 @@ jobs:
4045
uses: docker/build-push-action@v4
4146
with:
4247
context: ./docker/
43-
push: true
4448
tags: ${{ steps.meta_dockerhub.outputs.tags }}
4549
labels: ${{ steps.meta_dockerhub.outputs.labels }}
50+
platforms: linux/amd64,linux/arm64
51+
push: ${{ github.event_name == 'push' }}
52+
53+
- name: Update repo description
54+
if: ${{ startsWith(github.ref, 'refs/tags/v') }}
55+
uses: peter-evans/dockerhub-description@v3
56+
with:
57+
username: ${{ secrets.DOCKERHUB_USERNAME }}
58+
password: ${{ secrets.DOCKERHUB_TOKEN }}
59+
repository: openml/openml-python
60+
short-description: "pre-installed openml-python environment"
61+
readme-filepath: ./docker/readme.md
4662

4763
- name: Image digest
4864
run: echo ${{ steps.docker_build.outputs.digest }}

‎docker/Dockerfile‎

Lines changed: 4 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -2,15 +2,17 @@
22
# Useful building docs or running unix tests from a Windows host.
33
FROM python:3.10
44

5-
RUN git clone https://github.com/openml/openml-python.git omlp
6-
WORKDIR omlp
5+
RUN git clone https://github.com/openml/openml-python.git openml
6+
WORKDIR openml
77
RUN python -m venv venv
88
RUN venv/bin/pip install wheel setuptools
99
RUN venv/bin/pip install -e .[test,examples,docs,examples_unix]
1010

1111
WORKDIR /
1212
RUN mkdir scripts
1313
ADD startup.sh scripts/
14+
ADD readme.md /
15+
1416
# Due to the nature of the Docker container it might often be built from Windows.
1517
# It is typical to have the files with \r\n line-ending, we want to remove it for the unix image.
1618
RUN sed -i 's/\r//g' scripts/startup.sh

‎docker/readme.md‎

Lines changed: 99 additions & 54 deletions
Original file line numberDiff line numberDiff line change
@@ -1,86 +1,131 @@
11
# OpenML Python Container
22

3-
This docker container has the latest development version of openml-python downloaded and pre-installed.
4-
It can be used to run the unit tests or build the docs in a fresh and/or isolated unix environment.
5-
Instructions only tested on a Windows host machine.
3+
This docker container has the latest version of openml-python downloaded and pre-installed.
4+
It can also be used by developers to run unit tests or build the docs in
5+
a fresh and/or isolated unix environment.
6+
This document contains information about:
67

7-
First pull the docker image:
8+
1. [Usage](#usage): how to use the image and its main modes.
9+
2. [Using local or remote code](#using-local-or-remote-code): useful when testing your own latest changes.
10+
3. [Versions](#versions): identify which image to use.
11+
4. [Development](#for-developers): information about the Docker image for developers.
812

9-
docker pull openml/openml-python
13+
*note:* each docker image is shipped with a readme, which you can read with:
14+
`docker run --entrypoint=/bin/cat openml/openml-python:TAG readme.md`
1015

1116
## Usage
1217

18+
There are three main ways to use the image: running a pre-installed Python environment,
19+
running tests, and building documentation.
1320

14-
docker run -it openml/openml-python [DOC,TEST] [BRANCH]
21+
### Running `Python` with pre-installed `OpenML-Python` (default):
1522

16-
The image is designed to work with two specified directories which may be mounted ([`docker --mount documentation`](https://docs.docker.com/storage/bind-mounts/#start-a-container-with-a-bind-mount)).
17-
You can mount your openml-python folder to the `/code` directory to run tests or build docs on your local files.
18-
You can mount an `/output` directory to which the container will write output (currently only used for docs).
19-
Each can be mounted by adding a `--mount type=bind,source=SOURCE,destination=/DESTINATION` where `SOURCE` is the absolute path to your code or output directory, and `DESTINATION` is either `code` or `output`.
20-
21-
E.g. mounting a code directory:
23+
To run `Python` with a pre-installed `OpenML-Python` environment run:
2224

23-
docker run -i --mount type=bind,source="E:\\repositories/openml-python",destination="/code" -t openml/openml-python
25+
```text
26+
docker run -it openml/openml-python
27+
```
2428

25-
E.g. mounting an output directory:
29+
this accepts the normal `Python` arguments, e.g.:
2630

27-
docker run -i --mount type=bind,source="E:\\files/output",destination="/output" -t openml/openml-python
31+
```text
32+
docker run openml/openml-python -c "import openml; print(openml.__version__)"
33+
```
2834

29-
You can mount both at the same time.
35+
if you want to run a local script, it needs to be mounted first. Mount it into the
36+
`openml` folder:
3037

31-
### Bash (default)
32-
By default bash is invoked, you should also use the `-i` flag when starting the container so it processes input:
38+
```
39+
docker run -v PATH/TO/FILE:/openml/MY_SCRIPT.py openml/openml-python MY_SCRIPT.py
40+
```
3341

34-
docker run -it openml/openml-python
42+
### Running unit tests
3543

36-
### Building Documentation
37-
There are two ways to build documentation, either directly from the `HEAD` of a branch on Github or from your local directory.
44+
You can run the unit tests by passing `test` as the first argument.
45+
It also requires a local or remote repository to be specified, which is explained
46+
[below]((#using-local-or-remote-code). For this example, we specify to test the
47+
`develop` branch:
3848

39-
#### Building from a local repository
40-
Building from a local directory requires you to mount it to the ``/code`` directory:
49+
```text
50+
docker run openml/openml-python test develop
51+
```
4152

42-
docker run --mount type=bind,source=PATH_TO_REPOSITORY,destination=/code -t openml/openml-python doc
53+
### Building documentation
4354

44-
The produced documentation will be in your repository's ``doc/build`` folder.
45-
If an `/output` folder is mounted, the documentation will *also* be copied there.
55+
You can build the documentation by passing `doc` as the first argument,
56+
you should [mount]((https://docs.docker.com/storage/bind-mounts/#start-a-container-with-a-bind-mount))
57+
an output directory in which the docs will be stored. You also need to provide a remote
58+
or local repository as explained in [the section below]((#using-local-or-remote-code).
59+
In this example, we build documentation for the `develop` branch.
60+
On Windows:
4661

47-
#### Building from an online repository
48-
Building from a remote repository requires you to specify a branch.
49-
The branch may be specified by name directly if it exists on the original repository (https://github.com/openml/openml-python/):
62+
```text
63+
docker run --mount type=bind,source="E:\\files/output",destination="/output" openml/openml-python doc develop
64+
```
5065

51-
docker run --mount type=bind,source=PATH_TO_OUTPUT,destination=/output -t openml/openml-python doc BRANCH
66+
on Linux:
67+
```text
68+
docker run --mount type=bind,source="./output",destination="/output" openml/openml-python doc develop
69+
```
70+
71+
see [the section below]((#using-local-or-remote-code) for running against local changes
72+
or a remote branch.
5273

53-
Where `BRANCH` is the name of the branch for which to generate the documentation.
54-
It is also possible to build the documentation from the branch on a fork, in this case the `BRANCH` should be specified as `GITHUB_NAME#BRANCH` (e.g. `PGijsbers#my_feature`) and the name of the forked repository should be `openml-python`.
74+
*Note: you can forgo mounting an output directory to test if the docs build successfully,
75+
but the result will only be available within the docker container under `/openml/docs/build`.*
5576

56-
### Running tests
57-
There are two ways to run tests, either directly from the `HEAD` of a branch on Github or from your local directory.
58-
It works similar to building docs, but should specify `test` as mode.
59-
For example, to run tests on your local repository:
77+
## Using local or remote code
6078

61-
docker run --mount type=bind,source=PATH_TO_REPOSITORY,destination=/code -t openml/openml-python test
62-
63-
Running tests from the state of an online repository is supported similar to building documentation (i.e. specify `BRANCH` instead of mounting `/code`).
64-
65-
## Troubleshooting
79+
You can build docs or run tests against your local repository or a Github repository.
80+
In the examples below, change the `source` to match the location of your local repository.
81+
82+
### Using a local repository
83+
84+
To use a local directory, mount it in the `/code` directory, on Windows:
85+
86+
```text
87+
docker run --mount type=bind,source="E:\\repositories/openml-python",destination="/code" openml/openml-python test
88+
```
6689

67-
When you are mounting a directory you can check that it is mounted correctly by running the image in bash mode.
68-
Navigate to the `/code` and `/output` directories and see if the expected files are there.
69-
If e.g. there is no code in your mounted `/code`, you should double-check the provided path to your host directory.
90+
on Linux:
91+
```text
92+
docker run --mount type=bind,source="/Users/pietergijsbers/repositories/openml-python",destination="/code" openml/openml-python test
93+
```
7094

71-
## Notes for developers
72-
This section contains some notes about the structure of the image, intended for those who want to work on it.
95+
when building docs, you also need to mount an output directory as shown above, so add both:
96+
97+
```text
98+
docker run --mount type=bind,source="./output",destination="/output" --mount type=bind,source="/Users/pietergijsbers/repositories/openml-python",destination="/code" openml/openml-python doc
99+
```
100+
101+
### Using a Github repository
102+
Building from a remote repository requires you to specify a branch.
103+
The branch may be specified by name directly if it exists on the original repository (https://github.com/openml/openml-python/):
104+
105+
docker run --mount type=bind,source=PATH_TO_OUTPUT,destination=/output openml/openml-python [test,doc] BRANCH
106+
107+
Where `BRANCH` is the name of the branch for which to generate the documentation.
108+
It is also possible to build the documentation from the branch on a fork,
109+
in this case the `BRANCH` should be specified as `GITHUB_NAME#BRANCH` (e.g.
110+
`PGijsbers#my_feature_branch`) and the name of the forked repository should be `openml-python`.
111+
112+
## For developers
113+
This section contains some notes about the structure of the image,
114+
intended for those who want to work on it.
73115

74116
### Added Directories
75117
The `openml/openml-python` image is built on a vanilla `python:3` image.
76-
Additionally it contains the following files are directories:
77-
78-
- `/omlp`: contains the openml-python repository in the state with which the image was built by default.
79-
If working with a `BRANCH`, this repository will be set to the `HEAD` of `BRANCH`.
80-
- `/omlp/venv/`: contains the used virtual environment for `doc` and `test`. It has `openml-python` dependencies pre-installed.
81-
When invoked with `doc` or `test`, the dependencies will be updated based on the `setup.py` of the `BRANCH` or mounted `/code`.
118+
Additionally, it contains the following files are directories:
119+
120+
- `/openml`: contains the openml-python repository in the state with which the image
121+
was built by default. If working with a `BRANCH`, this repository will be set to
122+
the `HEAD` of `BRANCH`.
123+
- `/openml/venv/`: contains the used virtual environment for `doc` and `test`. It has
124+
`openml-python` dependencies pre-installed. When invoked with `doc` or `test`, the
125+
dependencies will be updated based on the `setup.py` of the `BRANCH` or mounted `/code`.
82126
- `/scripts/startup.sh`: the entrypoint of the image. Takes care of the automated features (e.g. `doc` and `test`).
83127

84128
## Building the image
85-
To build the image yourself, execute `docker build -f Dockerfile .` from this directory.
86-
It will use the `startup.sh` as is, so any local changes will be present in the image.
129+
To build the image yourself, execute `docker build -f Dockerfile .` from the `docker`
130+
directory of the `openml-python` repository. It will use the `startup.sh` as is, so any
131+
local changes will be present in the image.

‎docker/startup.sh‎

Lines changed: 14 additions & 10 deletions
Original file line numberDiff line numberDiff line change
@@ -1,3 +1,6 @@
1+
# Entry script to switch between the different Docker functionalities.
2+
# By default, execute Python with OpenML pre-installed
3+
#
14
# Entry script to allow docker to be ran for bash, tests and docs.
25
# The script assumes a code repository can be mounted to ``/code`` and an output directory to ``/output``.
36
# Executes ``mode`` on ``branch`` or the provided ``code`` directory.
@@ -10,10 +13,11 @@
1013
# Can be a branch on a Github fork, specified with the USERNAME#BRANCH format.
1114
# The test or doc build is executed on this branch.
1215

13-
if [ -z "$1" ]; then
14-
echo "Executing in BASH mode."
15-
bash
16-
exit
16+
if [[ ! ( $1 = "doc" || $1 = "test" ) ]]; then
17+
cd openml
18+
source venv/bin/activate
19+
python "$@"
20+
exit 0
1721
fi
1822

1923
# doc and test modes require mounted directories and/or specified branches
@@ -32,8 +36,8 @@ if [ "$1" == "doc" ] && [ -n "$2" ] && ! [ -d "/output" ]; then
3236
fi
3337

3438
if [ -n "$2" ]; then
35-
# if a branch is provided, we will pull it into the `omlp` local repository that was created with the image.
36-
cd omlp
39+
# if a branch is provided, we will pull it into the `openml` local repository that was created with the image.
40+
cd openml
3741
if [[ $2 == *#* ]]; then
3842
# If a branch is specified on a fork (with NAME#BRANCH format), we have to construct the url before pulling
3943
# We add a trailing '#' delimiter so the second element doesn't get the trailing newline from <<<
@@ -52,12 +56,12 @@ if [ -n "$2" ]; then
5256
exit 1
5357
fi
5458
git pull
55-
code_dir="/omlp"
59+
code_dir="/openml"
5660
else
5761
code_dir="/code"
5862
fi
5963

60-
source /omlp/venv/bin/activate
64+
source /openml/venv/bin/activate
6165
cd $code_dir
6266
# The most recent ``main`` is already installed, but we want to update any outdated dependencies
6367
pip install -e .[test,examples,docs,examples_unix]
@@ -71,6 +75,6 @@ if [ "$1" == "doc" ]; then
7175
make html
7276
make linkcheck
7377
if [ -d "/output" ]; then
74-
cp -r /omlp/doc/build /output
78+
cp -r /openml/doc/build /output
7579
fi
76-
fi
80+
fi

0 commit comments

Comments
 (0)