Skip to content

Commit f5ea443

Browse files
ericlrobertnishihara
ericl
authored and
robertnishihara
committed
EC2 cluster setup scripts and initial version of auto-scaler (ray-project#1311)
1 parent 76b6b4a commit f5ea443

20 files changed

Lines changed: 1665 additions & 16 deletions

‎.travis.yml‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -112,6 +112,7 @@ script:
112112
- python test/runtest.py
113113
- python test/array_test.py
114114
- python test/actor_test.py
115+
- python test/autoscaler_test.py
115116
- python test/tensorflow_test.py
116117
- python test/failure_test.py
117118
- python test/microbenchmarks.py

‎.travis/install-dependencies.sh‎

Lines changed: 4 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -24,15 +24,15 @@ if [[ "$PYTHON" == "2.7" ]] && [[ "$platform" == "linux" ]]; then
2424
wget https://repo.continuum.io/miniconda/Miniconda2-latest-Linux-x86_64.sh -O miniconda.sh -nv
2525
bash miniconda.sh -b -p $HOME/miniconda
2626
export PATH="$HOME/miniconda/bin:$PATH"
27-
pip install -q numpy cloudpickle==0.5.2 cython cmake funcsigs click colorama psutil redis tensorflow gym flatbuffers opencv-python
27+
pip install -q numpy cloudpickle==0.5.2 cython cmake funcsigs click colorama psutil redis tensorflow gym flatbuffers opencv-python pyyaml
2828
elif [[ "$PYTHON" == "3.5" ]] && [[ "$platform" == "linux" ]]; then
2929
sudo apt-get update
3030
sudo apt-get install -y cmake pkg-config python-dev python-numpy build-essential autoconf curl libtool unzip
3131
# Install miniconda.
3232
wget https://repo.continuum.io/miniconda/Miniconda3-latest-Linux-x86_64.sh -O miniconda.sh -nv
3333
bash miniconda.sh -b -p $HOME/miniconda
3434
export PATH="$HOME/miniconda/bin:$PATH"
35-
pip install -q numpy cloudpickle==0.5.2 cython cmake funcsigs click colorama psutil redis tensorflow gym flatbuffers opencv-python
35+
pip install -q numpy cloudpickle==0.5.2 cython cmake funcsigs click colorama psutil redis tensorflow gym flatbuffers opencv-python pyyaml
3636
elif [[ "$PYTHON" == "2.7" ]] && [[ "$platform" == "macosx" ]]; then
3737
# check that brew is installed
3838
which -s brew
@@ -48,7 +48,7 @@ elif [[ "$PYTHON" == "2.7" ]] && [[ "$platform" == "macosx" ]]; then
4848
wget https://repo.continuum.io/miniconda/Miniconda2-latest-MacOSX-x86_64.sh -O miniconda.sh -nv
4949
bash miniconda.sh -b -p $HOME/miniconda
5050
export PATH="$HOME/miniconda/bin:$PATH"
51-
pip install -q numpy cloudpickle==0.5.2 cython cmake funcsigs click colorama psutil redis tensorflow gym flatbuffers opencv-python
51+
pip install -q numpy cloudpickle==0.5.2 cython cmake funcsigs click colorama psutil redis tensorflow gym flatbuffers opencv-python pyyaml
5252
elif [[ "$PYTHON" == "3.5" ]] && [[ "$platform" == "macosx" ]]; then
5353
# check that brew is installed
5454
which -s brew
@@ -64,7 +64,7 @@ elif [[ "$PYTHON" == "3.5" ]] && [[ "$platform" == "macosx" ]]; then
6464
wget https://repo.continuum.io/miniconda/Miniconda3-latest-MacOSX-x86_64.sh -O miniconda.sh -nv
6565
bash miniconda.sh -b -p $HOME/miniconda
6666
export PATH="$HOME/miniconda/bin:$PATH"
67-
pip install -q numpy cloudpickle==0.5.2 cython cmake funcsigs click colorama psutil redis tensorflow gym flatbuffers opencv-python
67+
pip install -q numpy cloudpickle==0.5.2 cython cmake funcsigs click colorama psutil redis tensorflow gym flatbuffers opencv-python pyyaml
6868
elif [[ "$LINT" == "1" ]]; then
6969
sudo apt-get update
7070
sudo apt-get install -y cmake build-essential autoconf curl libtool unzip

‎doc/requirements-doc.txt‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -6,6 +6,7 @@ mock
66
numpy
77
opencv-python
88
pyarrow
9+
pyyaml
910
psutil
1011
recommonmark
1112
redis

‎doc/source/autoscaling.rst‎

Lines changed: 88 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,88 @@
1+
Cluster setup and auto-scaling (Experimental)
2+
=============================================
3+
4+
Quick start
5+
-----------
6+
7+
First, ensure you have configured your AWS credentials in ``~/.aws/credentials``,
8+
as described in `the boto docs <http://boto3.readthedocs.io/en/latest/guide/configuration.html>`__.
9+
10+
Then you're ready to go. The provided `ray/python/ray/autoscaler/aws/example.yaml <https://github.com/ray-project/ray/tree/master/python/ray/autoscaler/aws/example.yaml>`__ cluster config file will create a small cluster with a m4.large
11+
head node (on-demand), and two m4.large `spot workers <https://aws.amazon.com/ec2/spot/>`__.
12+
Try it out with these commands:
13+
14+
.. code-block:: bash
15+
16+
# Create or update the cluster
17+
$ ray create_or_update ray/python/ray/autoscaler/aws/example.yaml
18+
19+
# Resize the cluster without interrupting running jobs
20+
$ ray create_or_update ray/python/ray/autoscaler/aws/example.yaml \
21+
--max-workers=N --sync-only
22+
23+
# Teardown the cluster
24+
$ ray teardown ray/python/ray/autoscaler/aws/example.yaml
25+
26+
Common configurations
27+
---------------------
28+
29+
Note: auto-scaling support is not fully implemented yet (targeted for 0.4.0).
30+
31+
The example configuration above is enough to get started with Ray, but for more
32+
compute intensive workloads you will want to change the instance types to e.g.
33+
use GPU or larger compute instance by editing the yaml file. Here are a few common
34+
configurations:
35+
36+
**GPU single node**: use Ray on a single large GPU instance.
37+
38+
.. code-block:: yaml
39+
40+
max_workers: 0
41+
head_node:
42+
InstanceType: p2.8xlarge
43+
44+
**Mixed GPU and CPU nodes**: for RL applications that require proportionally more
45+
CPU than GPU resources, you can use additional CPU workers with a GPU head node.
46+
47+
.. code-block:: yaml
48+
49+
max_workers: 10
50+
head_node:
51+
InstanceType: p2.8xlarge
52+
worker_nodes:
53+
InstanceType: m4.16xlarge
54+
55+
**Autoscaling CPU cluster**: use a small head node and have Ray auto-scale
56+
workers as needed. This can be a cost-efficient configuration for clusters with
57+
bursty workloads. You can also request spot workers for additional cost savings.
58+
59+
.. code-block:: yaml
60+
61+
min_workers: 0
62+
max_workers: 10
63+
head_node:
64+
InstanceType: m4.large
65+
worker_nodes:
66+
InstanceMarketOptions:
67+
MarketType: spot
68+
InstanceType: m4.16xlarge
69+
70+
**Autoscaling GPU cluster**: similar to the autoscaling CPU cluster, but
71+
with GPU worker nodes instead.
72+
73+
.. code-block:: yaml
74+
75+
min_workers: 0
76+
max_workers: 10
77+
head_node:
78+
InstanceType: m4.large
79+
worker_nodes:
80+
InstanceMarketOptions:
81+
MarketType: spot
82+
InstanceType: p2.8xlarge
83+
84+
Additional Cloud providers
85+
--------------------------
86+
87+
To use Ray autoscaling on other Cloud providers or cluster management systems, you can implement the ``NodeProvider`` interface
88+
(~100 LOC) and register it in `node_provider.py <https://github.com/ray-project/ray/tree/master/python/ray/autoscaler/node_provider.py>`__.

‎doc/source/index.rst‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -84,6 +84,7 @@ Example Program
8484
:maxdepth: 1
8585
:caption: Cluster Usage
8686

87+
autoscaling.rst
8788
using-ray-on-a-cluster.rst
8889
using-ray-on-a-large-cluster.rst
8990
using-ray-and-docker-on-a-cluster.md

‎python/ray/autoscaler/__init__.py‎

Whitespace-only changes.

0 commit comments

Comments
 (0)