Skip to content

Commit a76042b

Browse files
jmbowmanjmbowman
committed
TE-2639 Support for named release images
1 parent c7b0d05 commit a76042b

8 files changed

Lines changed: 86 additions & 27 deletions

‎Makefile‎

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -34,6 +34,9 @@ upgrade: ## Upgrade requirements with pip-tools
3434
requirements/pip-tools.txt \
3535
requirements/base.txt \
3636

37+
dev.checkout: ## Check out "openedx-release/$OPENEDX_RELEASE" in each repo if set, "master" otherwise
38+
./repo.sh checkout
39+
3740
dev.clone: ## Clone service repos to the parent directory
3841
./repo.sh clone
3942

‎README.rst‎

Lines changed: 23 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -351,14 +351,17 @@ How do I build images?
351351

352352
There are `Docker CI Jenkins jobs`_ on tools-edx-jenkins that build and push new
353353
Docker images to DockerHub on code changes to either the configuration repository or the IDA's codebase. These images
354-
are tagged ``latest``. Images that require tags other than ``latest`` are built and pushed by hand (see NOTES below).
354+
are tagged according to the branch from which they were built (see NOTES below).
355355
If you want to build the images on your own, the Dockerfiles are available in the ``edx/configuration`` repo.
356356

357357
NOTES:
358358

359-
1. edxapp and IDAs use the ``latest`` tag since their configuration changes have been merged to master branch of
360-
``edx/configuration``.
361-
2. The elasticsearch used in devstack is built using elasticsearch-devstack/Dockerfile and the ``devstack`` tag.
359+
1. edxapp and IDAs use the ``latest`` tag for configuration changes which have been merged to master branch of
360+
their repository and ``edx/configuration``.
361+
2. Images for a named Open edX release are built from the corresponding branch
362+
of each repository and tagged appropriately, for example ``hawthorn.master``
363+
or ``hawthorn.rc1``.
364+
3. The elasticsearch used in devstack is built using elasticsearch-devstack/Dockerfile and the ``devstack`` tag.
362365

363366
BUILD COMMANDS:
364367

@@ -385,6 +388,22 @@ For example, if you wanted to build tag ``release-2017-03-03`` for the
385388
E-Commerce Service, you would modify ``ECOMMERCE_VERSION`` in
386389
``docker/build/ecommerce/ansible_overrides.yml``.
387390

391+
How do I run the images for a named Open edX release?
392+
-----------------------------------------------------
393+
394+
1. Set the ``OPENEDX_RELEASE`` environment variable to the appropriate image
395+
tag; "hawthorn.master", "zebrawood.rc1", etc. Note that unlike a server
396+
install, ``OPENEDX_RELEASE`` should not have the "open-release/" prefix.
397+
2. Use ``make dev.checkout`` to check out the correct branch in the local
398+
checkout of each service repository once you've set the ``OPENEDX_RELEASE``
399+
environment variable above.
400+
3. ``make pull`` to get the correct images.
401+
402+
All ``make`` target and ``docker-compose`` calls should now use the correct
403+
images until you change or unset ``OPENEDX_RELEASE`` again. To work on the
404+
master branches and ``latest`` images, unset ``OPENEDX_RELEASE`` or set it to
405+
an empty string.
406+
388407
How do I create database dumps?
389408
-------------------------------
390409
We use database dumps to speed up provisioning and generally spend less time running migrations. These dumps should be

‎docker-compose-analytics-pipeline.yml‎

Lines changed: 7 additions & 7 deletions
Original file line numberDiff line numberDiff line change
@@ -2,7 +2,7 @@ version: "2.1"
22

33
services:
44
namenode:
5-
image: edxops/analytics_pipeline_hadoop_namenode:latest
5+
image: edxops/analytics_pipeline_hadoop_namenode:${OPENEDX_RELEASE:-latest}
66
container_name: edx.devstack.analytics_pipeline.namenode
77
hostname: namenode
88
environment:
@@ -14,7 +14,7 @@ services:
1414
- namenode_data:/hadoop/dfs/name
1515

1616
datanode:
17-
image: edxops/analytics_pipeline_hadoop_datanode:latest
17+
image: edxops/analytics_pipeline_hadoop_datanode:${OPENEDX_RELEASE:-latest}
1818
container_name: edx.devstack.analytics_pipeline.datanode
1919
hostname: datanode
2020
environment:
@@ -28,7 +28,7 @@ services:
2828
- datanode_data:/hadoop/dfs/data
2929

3030
resourcemanager:
31-
image: edxops/analytics_pipeline_hadoop_resourcemanager:latest
31+
image: edxops/analytics_pipeline_hadoop_resourcemanager:${OPENEDX_RELEASE:-latest}
3232
container_name: edx.devstack.analytics_pipeline.resourcemanager
3333
hostname: resourcemanager
3434
environment:
@@ -45,7 +45,7 @@ services:
4545
command: ["/run.sh"]
4646

4747
nodemanager:
48-
image: edxops/analytics_pipeline_hadoop_nodemanager:latest
48+
image: edxops/analytics_pipeline_hadoop_nodemanager:${OPENEDX_RELEASE:-latest}
4949
container_name: edx.devstack.analytics_pipeline.nodemanager
5050
hostname: nodemanager
5151
environment:
@@ -66,7 +66,7 @@ services:
6666
command: ["/run.sh"]
6767

6868
sparkmaster:
69-
image: edxops/analytics_pipeline_spark_master:latest
69+
image: edxops/analytics_pipeline_spark_master:${OPENEDX_RELEASE:-latest}
7070
container_name: edx.devstack.analytics_pipeline.sparkmaster
7171
hostname: sparkmaster
7272
ports:
@@ -76,7 +76,7 @@ services:
7676
- 127.0.0.1:18080:18080 # spark history server
7777

7878
sparkworker:
79-
image: edxops/analytics_pipeline_spark_worker:latest
79+
image: edxops/analytics_pipeline_spark_worker:${OPENEDX_RELEASE:-latest}
8080
container_name: edx.devstack.analytics_pipeline.sparkworker
8181
hostname: sparkworker
8282
depends_on:
@@ -93,7 +93,7 @@ services:
9393
- vertica_data:/home/dbadmin/docker
9494

9595
analyticspipeline:
96-
image: edxops/analytics_pipeline:latest
96+
image: edxops/analytics_pipeline:${OPENEDX_RELEASE:-latest}
9797
container_name: edx.devstack.analytics_pipeline
9898
hostname: analyticspipeline
9999
volumes:

‎docker-compose-marketing-site.yml‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -25,6 +25,6 @@ services:
2525
- DRUPAL_EXTRA_SETTINGS=${DRUPAL_EXTRA_SETTINGS:-/var/www/html/sites/default/docker.settings.php}
2626
# IP address of your machine to enable debugging (IP_ADDRESS set in .env file)
2727
- XDEBUG_CONFIG=remote_host=${XDEBUG_IP_ADDRESS:-127.0.0.1}
28-
image: edxops/edx-mktg:latest
28+
image: edxops/edx-mktg:${OPENEDX_RELEASE:-latest}
2929
ports:
3030
- "8080:80"

‎docker-compose-watchers.yml‎

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -7,7 +7,7 @@ services:
77
environment:
88
BOK_CHOY_HOSTNAME: edx.devstack.lms_watcher
99
ASSET_WATCHER_TIMEOUT: 12
10-
image: edxops/edxapp:latest
10+
image: edxops/edxapp:${OPENEDX_RELEASE:-latest}
1111
volumes:
1212
- ${DEVSTACK_WORKSPACE}/edx-platform:/edx/app/edxapp/edx-platform:cached
1313
- edxapp_lms_assets:/edx/var/edxapp/staticfiles/
@@ -19,7 +19,7 @@ services:
1919
environment:
2020
BOK_CHOY_HOSTNAME: edx.devstack.studio_watcher
2121
ASSET_WATCHER_TIMEOUT: 12
22-
image: edxops/edxapp:latest
22+
image: edxops/edxapp:${OPENEDX_RELEASE:-latest}
2323
volumes:
2424
- edxapp_studio_assets:/edx/var/edxapp/staticfiles/
2525
- ${DEVSTACK_WORKSPACE}/edx-platform:/edx/app/edxapp/edx-platform:cached

‎docker-compose-xqueue.yml‎

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -3,7 +3,7 @@ version: "2.1"
33
services:
44
xqueue:
55
container_name: edx.devstack.xqueue
6-
image: edxops/xqueue:latest
6+
image: edxops/xqueue:${OPENEDX_RELEASE:-latest}
77
command: bash -c 'source /edx/app/xqueue/xqueue_env && while true; do python /edx/app/xqueue/xqueue/manage.py runserver 0.0.0.0:18040 ; sleep 2; done'
88
volumes:
99
- ${DEVSTACK_WORKSPACE}/xqueue:/edx/app/xqueue/xqueue:cached
@@ -13,7 +13,7 @@ services:
1313

1414
xqueue_consumer:
1515
container_name: edx.devstack.xqueue_consumer
16-
image: edxops/xqueue:latest
16+
image: edxops/xqueue:${OPENEDX_RELEASE:-latest}
1717
command: bash -c 'source /edx/app/xqueue/xqueue_env && while true; do python /edx/app/xqueue/xqueue/manage.py run_consumer ; sleep 2; done'
1818
volumes:
1919
- ${DEVSTACK_WORKSPACE}/xqueue:/edx/app/xqueue/xqueue:cached

‎docker-compose.yml‎

Lines changed: 10 additions & 10 deletions
Original file line numberDiff line numberDiff line change
@@ -14,7 +14,7 @@ services:
1414
# Third-party services
1515
chrome:
1616
container_name: edx.devstack.chrome
17-
image: edxops/chrome:latest
17+
image: edxops/chrome:${OPENEDX_RELEASE:-latest}
1818
shm_size: 2g
1919
ports:
2020
- "15900:5900"
@@ -35,7 +35,7 @@ services:
3535

3636
firefox:
3737
container_name: edx.devstack.firefox
38-
image: edxops/firefox:latest
38+
image: edxops/firefox:${OPENEDX_RELEASE:-latest}
3939
shm_size: 2g
4040
ports:
4141
- "25900:5900"
@@ -88,7 +88,7 @@ services:
8888
DB_HOST: edx.devstack.mysql
8989
SOCIAL_AUTH_EDX_OIDC_URL_ROOT: http://edx.devstack.lms:18000/oauth2
9090
ENABLE_DJANGO_TOOLBAR: 1
91-
image: edxops/credentials:latest
91+
image: edxops/credentials:${OPENEDX_RELEASE:-latest}
9292
ports:
9393
- "18150:18150"
9494

@@ -105,7 +105,7 @@ services:
105105
environment:
106106
TEST_ELASTICSEARCH_URL: "http://edx.devstack.elasticsearch:9200"
107107
ENABLE_DJANGO_TOOLBAR: 1
108-
image: edxops/discovery:latest
108+
image: edxops/discovery:${OPENEDX_RELEASE:-latest}
109109
ports:
110110
- "18381:18381"
111111
volumes:
@@ -122,7 +122,7 @@ services:
122122
tty: true
123123
environment:
124124
ENABLE_DJANGO_TOOLBAR: 0
125-
image: edxops/ecommerce:latest
125+
image: edxops/ecommerce:${OPENEDX_RELEASE:-latest}
126126
ports:
127127
- "18130:18130"
128128

@@ -143,7 +143,7 @@ services:
143143
BOK_CHOY_CMS_PORT: 18031
144144
EDXAPP_TEST_MONGO_HOST: edx.devstack.mongo
145145
NO_PYTHON_UNINSTALL: 1
146-
image: edxops/edxapp:latest
146+
image: edxops/edxapp:${OPENEDX_RELEASE:-latest}
147147
ports:
148148
- "18000:18000"
149149
- "19876:19876" # JS test debugging
@@ -159,7 +159,7 @@ services:
159159
- devpi
160160
- elasticsearch
161161
- mysql
162-
image: edxops/notes:latest
162+
image: edxops/notes:${OPENEDX_RELEASE:-latest}
163163
ports:
164164
- "18120:18120"
165165
environment:
@@ -189,7 +189,7 @@ services:
189189
BOK_CHOY_CMS_PORT: 18131
190190
EDXAPP_TEST_MONGO_HOST: edx.devstack.mongo
191191
NO_PYTHON_UNINSTALL: 1
192-
image: edxops/edxapp:latest
192+
image: edxops/edxapp:${OPENEDX_RELEASE:-latest}
193193
ports:
194194
- "18010:18010"
195195
- "19877:19877" # JS test debugging
@@ -205,13 +205,13 @@ services:
205205
- mongo
206206
- memcached
207207
- elasticsearch
208-
image: edxops/forum:latest
208+
image: edxops/forum:${OPENEDX_RELEASE:-latest}
209209
ports:
210210
- "44567:4567"
211211

212212
devpi:
213213
container_name: edx.devstack.devpi
214-
image: edxops/devpi:latest
214+
image: edxops/devpi:${OPENEDX_RELEASE:-latest}
215215
ports:
216216
- "3141:3141"
217217
volumes:

‎repo.sh‎

Lines changed: 38 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -36,6 +36,38 @@ private_repos=(
3636

3737
name_pattern=".*edx/(.*).git"
3838

39+
_checkout ()
40+
{
41+
repos_to_checkout=("$@")
42+
43+
if [ -z "$OPENEDX_RELEASE" ]; then
44+
branch="master"
45+
else
46+
branch="open-release/${OPENEDX_RELEASE}"
47+
fi
48+
for repo in "${repos_to_checkout[@]}"
49+
do
50+
# Use Bash's regex match operator to capture the name of the repo.
51+
# Results of the match are saved to an array called $BASH_REMATCH.
52+
[[ $repo =~ $name_pattern ]]
53+
name="${BASH_REMATCH[1]}"
54+
55+
# If a directory exists and it is nonempty, assume the repo has been cloned.
56+
if [ -d "$name" -a -n "$(ls -A "$name" 2>/dev/null)" ]; then
57+
cd $name
58+
echo "Checking out branch $branch of $name"
59+
git pull
60+
git checkout "$branch"
61+
cd ..
62+
fi
63+
done
64+
}
65+
66+
checkout ()
67+
{
68+
_checkout "${repos[@]}"
69+
}
70+
3971
_clone ()
4072
{
4173
# for repo in ${repos[*]}
@@ -57,6 +89,9 @@ _clone ()
5789
else
5890
git clone $repo
5991
fi
92+
if [ -n "${OPENEDX_RELEASE}" ]; then
93+
git checkout open-release/${OPENEDX_RELEASE}
94+
fi
6095
fi
6196
done
6297
cd - &> /dev/null
@@ -107,7 +142,9 @@ status ()
107142
cd - &> /dev/null
108143
}
109144

110-
if [ "$1" == "clone" ]; then
145+
if [ "$1" == "checkout" ]; then
146+
checkout
147+
elif [ "$1" == "clone" ]; then
111148
clone
112149
elif [ "$1" == "whitelabel" ]; then
113150
clone_private

0 commit comments

Comments
 (0)