Skip to content

Commit 23352e2

Browse files
DevelopedByAnuragDevelopedByAnurag
committed
added metaflow AI tutorial Back
1 parent 230f81a commit 23352e2

5 files changed

Lines changed: 318 additions & 0 deletions

File tree

‎tutorials/ai-core-metaflow-setup/ai-core-metaflow-setup.md‎

Lines changed: 318 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,318 @@
1+
---
2+
parser: v2
3+
auto_validation: true
4+
time: 20
5+
tags: [ tutorial>beginner, topic>artificial-intelligence, topic>machine-learning, software-product>sap-ai-launchpad, software-product>sap-ai-core ]
6+
primary_tag: software-product>sap-ai-core
7+
author_name: Karim Mohraz
8+
author_profile: https://github.com/karimmohraz
9+
---
10+
11+
# Set Up the Metaflow Library for SAP AI Core
12+
<!-- description --> Explore different ways of logging metrics during training. Compare generated models.
13+
14+
## Prerequisites
15+
- You have Docker Desktop installed.
16+
- You have created your first pipeline with SAP AI Core, using [this tutorial](https://developers.sap.com/tutorials/ai-core-code.html/#).
17+
18+
## You will learn
19+
- How to create a sandbox Python or Docker environment.
20+
- How to set up the Metaflow Python package for SAP AI Core
21+
- How to run a local test of your Metaflow pipeline.
22+
23+
## Intro
24+
Discover how Metaflow assists you, with diagrams and visualization from production to deployment. For more information, see the [the Metaflow documentation.](https://docs.metaflow.org/introduction/why-metaflow)
25+
26+
---
27+
28+
### Set up your system, and Python
29+
30+
31+
32+
[OPTION BEGIN [Linux & MacOS]]
33+
34+
> **CAUTION** **For Windows Users**: Please use an alternate option tab to set up. Otherwise the Metaflow library could cause issues in installation.
35+
36+
Download and install Python 3.X from [python.org](https://www.python.org/downloads/).
37+
38+
Create and activate a virtual Python environment using the following snippet. Note, `tutorial_metaflow` is the name of your environment. The Python virtual environment helps you install Python packages inside a sandbox-like environment. You use the environment to maintain required versions of the packages for your project.
39+
40+
```SHELL
41+
python -m venv tutorial_metaflow
42+
source tutorial_metaflow/bin/activate
43+
```
44+
45+
Check which Python is used by your virtual environment. You can see the path of the Python executable in your virtual environment.
46+
47+
```SHELL
48+
which python
49+
```
50+
51+
[OPTION END]
52+
53+
[OPTION BEGIN [Docker environment (All OS)]]
54+
55+
Create a file named `Dockerfile` with following contents. This file stores instructions for Docker to build an image. You will run this Docker image in your local system to create a sandbox environment similar to a Virtual Machine.
56+
57+
```DOCKERFILE
58+
FROM python
59+
# FROM python:3.9.13 # (optional) specific version
60+
61+
RUN apt update
62+
RUN apt install -y docker.io docker
63+
```
64+
65+
Build the Docker image and run the Docker environment.
66+
67+
```SHELL
68+
docker build -t tutorial_metaflow .
69+
docker run -it -v //var/run/docker.sock:/var/run/docker.sock tutorial_metaflow /bin/sh
70+
```
71+
72+
The command option description:
73+
74+
- `-v //var/run/docker.sock:/var/run/docker.sock` connects your system's Docker to the Docker contained within container of your sandbox environment.
75+
- `-it` connects your system's terminal to the Shell terminal running inside your Docker container.
76+
77+
You should now use this Docker container as the sandbox environment to complete the tutorial.
78+
79+
80+
<!-- border -->![Preview of Docker environment](img/env-docker.png)
81+
82+
> **INFORMATION** If you are developing with Visual Studio Code (VS Code), you may connect to your VS Code to this sandbox environment using following VS Code extensions: [Remote Explorer Extension by Microsoft](https://marketplace.visualstudio.com/items?itemName=ms-vscode.remote-explorer) and [Docker Extension by Microsoft](https://marketplace.visualstudio.com/items?itemName=ms-azuretools.vscode-docker)
83+
>
84+
> <!-- border -->![Preview of VS Code](img/env-vscode2.gif)
85+
86+
87+
[OPTION END]
88+
89+
[OPTION BEGIN [Docker (Advanced computer vision)]]
90+
91+
> **INFORMATION** Only for users comfortable at self debugging code. If you are using Linux or MacOS then you may also try executing the lines from the Dockerfile without the `FROM` & `RUN` commands to set up in you local machine rather than in Docker's sandbox container.
92+
93+
Create file `Dockerfile` with following contents. The base Docker image is `pytorch v1.10` which is required for the `dectectron2` package of python.
94+
95+
```DOCKERFILE
96+
FROM pytorch/pytorch:1.10.0-cuda11.3-cudnn8-runtime
97+
98+
RUN apt update
99+
RUN apt install -y docker.io docker
100+
101+
# Libraries required for graphical processing
102+
RUN apt install -y gcc g++
103+
RUN apt install -y libgl1-mesa-glx libglib2.0-0
104+
105+
# Detectron 2 Installation
106+
RUN pip install https://github.com/facebookresearch/detectron2/archive/refs/tags/v0.6.zip
107+
```
108+
109+
Build the Docker image and run the Docker environment.
110+
111+
```SHELL
112+
docker build -t tutorial_metaflow .
113+
docker run -it -v //var/run/docker.sock:/var/run/docker.sock tutorial_metaflow /bin/sh
114+
```
115+
116+
The command option description:
117+
118+
- `-v //var/run/docker.sock:/var/run/docker.sock` connects your system's Docker to the Docker contained within container of your sandbox environment.
119+
- `-it` connects your system's terminal to the Shell terminal running inside your Docker container.
120+
121+
You should now use this Docker container as the sandbox environment to complete the tutorial.
122+
123+
124+
<!-- border -->![Preview of Docker environment](img/env-docker.png)
125+
> **INFORMATION** If you are developing with Visual Studio Code (VS Code), you may connect to your VS Code to this sandbox environment using following VS Code extensions: [Remote Explorer Extension by Microsoft](https://marketplace.visualstudio.com/items?itemName=ms-vscode.remote-explorer) and [Docker Extension by Microsoft](https://marketplace.visualstudio.com/items?itemName=ms-azuretools.vscode-docker)
126+
>
127+
> <!-- border -->![Preview of VS Code](img/env-vscode2.gif)
128+
129+
[OPTION END]
130+
131+
132+
133+
### Set environment variables
134+
135+
136+
Set your environment variables.
137+
138+
```SHELL
139+
# set username if not set
140+
export USERNAME=tutorialuser
141+
142+
# set location to find settings (configuration) for metaflow, customizable
143+
export HOME=/root
144+
```
145+
146+
147+
### Install Metaflow package
148+
149+
150+
Install the Metaflow package.
151+
152+
```SHELL
153+
pip install sap-ai-core-metaflow
154+
```
155+
156+
157+
158+
### Configure Metaflow
159+
160+
161+
Create a configuration file for Metaflow. Metaflow uses settings from configurations to store snippets of pipeline in your AWS S3 storage.
162+
163+
```SHELL
164+
mkdir -p $HOME/.metaflowconfig
165+
touch $HOME/.metaflowconfig/config.json
166+
```
167+
168+
Edit the following snippet and the paste in the `config.json` file, created above. Replace `<YOUR_S3_BUCKET_NAME>` with AWS S3 bucket ID that you want Metaflow to use to store files.
169+
170+
```JSON [2-3]
171+
{
172+
"METAFLOW_DATASTORE_SYSROOT_S3": "s3://<YOUR_S3_BUCKET_NAME>/metaflow",
173+
"METAFLOW_DATATOOLS_SYSROOT_S3": "s3://<YOUR_S3_BUCKET_NAME>/metaflow/data",
174+
"METAFLOW_DEFAULT_DATASTORE": "s3"
175+
}
176+
```
177+
178+
179+
### Configure AWS
180+
181+
182+
Create a credentials file for your AWS S3 Object Store. The file is used by the Metaflow package access your AWS S3 store.
183+
184+
```SHELL
185+
mkdir -p $HOME/.aws
186+
touch $HOME/.aws/credentials
187+
```
188+
189+
Edit the following snippet and paste it in your `credentials` file. Replace `<YOUR_S3_ACCESS...>` with your AWS S3 credentials, **do not** enclose your credentials within quotes(`""`).
190+
191+
```TEXT [2-3]
192+
[default]
193+
aws_access_key_id = <YOUR_S3_ACCESS_ID>
194+
aws_secret_access_key = <YOUR_S3_ACCESS_KEY>
195+
```
196+
197+
> **INFORMATION** the Metaflow library for SAP AI Core uses AWS, however you may skip the installation of AWS CLI.
198+
199+
200+
201+
### Metaflow HelloWorld
202+
203+
204+
Create a file `hellometaflow.py` with following contents.
205+
206+
```PYTHON
207+
# You should divide the contents of the steps of the pipeline file into different python modules.
208+
# However the contents within each step may have snippets imported from separate python packages/modules.
209+
from metaflow import FlowSpec, step
210+
211+
212+
class HelloFlow(FlowSpec):
213+
"""
214+
A flow where Metaflow prints 'Hi'.
215+
216+
Run this flow to validate that Metaflow is installed correctly.
217+
218+
"""
219+
220+
@step
221+
def start(self):
222+
"""
223+
This is the 'start' step. All flows must have a step named 'start' that
224+
is the first step in the flow.
225+
226+
"""
227+
print("HelloFlow is starting.")
228+
self.next(self.hello)
229+
230+
@step
231+
def hello(self):
232+
"""
233+
A step for metaflow to introduce itself.
234+
235+
"""
236+
print("Metaflow says: Hi!")
237+
self.next(self.end)
238+
239+
@step
240+
def end(self):
241+
"""
242+
This is the 'end' step. All flows must have an 'end' step, which is the
243+
last step in the flow.
244+
245+
"""
246+
print("HelloFlow is all done.")
247+
248+
249+
if __name__ == "__main__":
250+
HelloFlow()
251+
```
252+
253+
> **INFORMATION** You may discover more snippets in the [official Metaflow tutorials.](https://docs.metaflow.org/getting-started/tutorials)
254+
255+
256+
257+
### Run Metaflow pipeline locally
258+
259+
260+
Inspect the steps of the Metaflow pipeline `hellometaflow.py` using following snippet.
261+
262+
```SHELL
263+
python hellometaflow.py show
264+
```
265+
266+
Run this snippet locally.
267+
268+
```SHELL
269+
python hellometaflow.py run
270+
```
271+
272+
<!-- border -->![metaflow](img/hello-metaflow.png)
273+
274+
275+
276+
### Save set up environment
277+
278+
279+
280+
[OPTION BEGIN [Python virtual environment]]
281+
282+
To deactivate Python virtual environment, use the following command.
283+
284+
```SHELL
285+
deactivate
286+
```
287+
288+
[OPTION END]
289+
290+
291+
[OPTION BEGIN [Docker environment]]
292+
293+
> **CAUTION**: These steps are only applicable if you used the Docker environment in the step 1.
294+
295+
Docker containers are ephemeral, meaning that the installation you did inside will lost upon closing. To save the changes, first locate your container ID using the following snippet on your local system (not inside the Docker container).
296+
297+
```SHELL
298+
docker ps
299+
```
300+
301+
Use container resulting container ID to save the changes, including files, installations and commands, in the form of Docker images:
302+
303+
```SHELL
304+
docker commit <CONTAINER_ID> tutorial_metaflow:0.1
305+
```
306+
307+
You may now close the Docker Container, by using the `exit` command inside your Docker container.
308+
309+
<!-- border -->![image](img/env-docker-save.png)
310+
311+
To pick up where you left off, use the following snippet. Ensure you are using the tag `:01` with which you saved the changes.
312+
313+
```SELL
314+
docker run -it -v //var/run/docker.sock:/var/run/docker.sock tutorial_metaflow:0.1 /bin/sh
315+
```
316+
317+
[OPTION END]
318+

‎tutorials/ai-core-metaflow-setup/img/env-docker-save.png‎

94.7 KB
Loading

‎tutorials/ai-core-metaflow-setup/img/env-docker.png‎

122 KB
Loading

‎tutorials/ai-core-metaflow-setup/img/env-vscode2.gif‎

974 KB
Loading

‎tutorials/ai-core-metaflow-setup/img/hello-metaflow.png‎

212 KB
Loading

0 commit comments

Comments
 (0)