Skip to content

Commit 1568d32

Browse files
helenaaaaaaaaaahelenaaaaaaaaaa
committed
Publishing
Publishing
1 parent ba485ff commit 1568d32

5 files changed

Lines changed: 323 additions & 0 deletions

File tree

‎tutorials/ai-core-metaflow-setup/ai-core-metaflow-setup.md‎

Lines changed: 323 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,323 @@
1+
---
2+
title: Set Up the Metaflow Library for SAP AI Core
3+
description: Explore different ways of logging metrics during training. Compare generated models.
4+
auto_validation: true
5+
time: 20
6+
tags: [ tutorial>license, tutorial>beginner, topic>artificial-intelligence, topic>machine-learning, software-product>sap-ai-launchpad, software-product>sap-ai-core ]
7+
primary_tag: software-product>sap-ai-core
8+
author_name: Karim Mohraz
9+
author_profile: https://github.com/karimmohraz
10+
---
11+
12+
## Prerequisites
13+
- You have Docker Desktop installed.
14+
- You have created your first pipeline with SAP AI Core, using [this tutorial](https://developers.sap.com/tutorials/ai-core-code.html/#).
15+
16+
## Details
17+
### You will learn
18+
- How to create a sandbox Python or Docker environment.
19+
- How to set up the Metaflow Python package for SAP AI Core
20+
- How to run a local test of your Metaflow pipeline.
21+
22+
Discover how Metaflow assists you, with diagrams and visualisationm from production to deployment. For more information, see the [the Metaflow documentation.](https://docs.metaflow.org/introduction/why-metaflow)
23+
24+
---
25+
26+
[ACCORDION-BEGIN [Step 1: ](Set up your system, and Python)]
27+
28+
29+
[OPTION BEGIN [Linux & MacOS]]
30+
31+
> **CAUTION** **For Windows Users**: Please use an alternate option tab to set up. Otherwise the Metaflow library could cause issues in installation.
32+
33+
Download and install Python 3.X from [python.org](https://www.python.org/downloads/).
34+
35+
Create and activate a virtual Python environment using the following snippet. Note, `tutorial_metaflow` is the name of your environment. The Python virtual environment helps you install Python packages inside a sandbox-like environment. You use the environment to maintain required versions of the packages for your project.
36+
37+
```SHELL
38+
python -m venv tutorial_metaflow
39+
source tutorial_metaflow/bin/activate
40+
```
41+
42+
Check which Python is used by your virtual environment. You can see the path of the Python executable in your virtual environment.
43+
44+
```SHELL
45+
which python
46+
```
47+
48+
[OPTION END]
49+
50+
[OPTION BEGIN [Docker environment (All OS)]]
51+
52+
Create a file named `Dockerfile` with following contents. This file stores instructions for Docker to build an image. You will run this Docker image in your local system to create a sandbox environment similar to a Virtual Machine.
53+
54+
```DOCKERFILE
55+
FROM python
56+
# FROM python:3.9.13 # (optional) specific version
57+
58+
RUN apt update
59+
RUN apt install -y docker.io docker
60+
```
61+
62+
Build the Docker image and run the Docker environment.
63+
64+
```SHELL
65+
docker build -t tutorial_metaflow .
66+
docker run -it -v //var/run/docker.sock:/var/run/docker.sock tutorial_metaflow /bin/sh
67+
```
68+
69+
The command option description:
70+
71+
- `-v //var/run/docker.sock:/var/run/docker.sock` connects your system's Docker to the Docker contained within container of your sandbox environment.
72+
- `-it` connects your system's terminal to the Shell terminal running inside your Docker container.
73+
74+
You should now use this Docker container as the sandbox environment to complete the tutorial.
75+
76+
77+
!![Preview of Docker environment](img/env-docker.png)
78+
79+
> **INFORMATION** If you are developing with Visual Studio Code (VS Code), you may connect to your VS Code to this sandbox environment using following VS Code extensions: [Remote Explorer Extension by Microsoft](https://marketplace.visualstudio.com/items?itemName=ms-vscode.remote-explorer) and [Docker Extension by Microsoft](https://marketplace.visualstudio.com/items?itemName=ms-azuretools.vscode-docker)
80+
>
81+
> !![Preview of VS Code](img/env-vscode2.gif)
82+
83+
84+
[OPTION END]
85+
86+
[OPTION BEGIN [Docker (Advanced computer vision)]]
87+
88+
> **INFORMATION** Only for users comfortable at self debugging code. If you are using Linux or MacOS then you may also try executing the lines from the Dockerfile without the `FROM` & `RUN` commands to set up in you local machine rather than in Docker's sandbox container.
89+
90+
Create file `Dockerfile` with following contents. The base Docker image is `pytorch v1.10` which is required for the `dectectron2` package of python.
91+
92+
```DOCKERFILE
93+
FROM pytorch/pytorch:1.10.0-cuda11.3-cudnn8-runtime
94+
95+
RUN apt update
96+
RUN apt install -y docker.io docker
97+
98+
# Libraries required for graphical processing
99+
RUN apt install -y gcc g++
100+
RUN apt install -y libgl1-mesa-glx libglib2.0-0
101+
102+
# Detectron 2 Installation
103+
RUN pip install https://github.com/facebookresearch/detectron2/archive/refs/tags/v0.6.zip
104+
```
105+
106+
Build the Docker image and run the Docker environment.
107+
108+
```SHELL
109+
docker build -t tutorial_metaflow .
110+
docker run -it -v //var/run/docker.sock:/var/run/docker.sock tutorial_metaflow /bin/sh
111+
```
112+
113+
The command option description:
114+
115+
- `-v //var/run/docker.sock:/var/run/docker.sock` connects your system's Docker to the Docker contained within container of your sandbox environment.
116+
- `-it` connects your system's terminal to the Shell terminal running inside your Docker container.
117+
118+
You should now use this Docker container as the sandbox environment to complete the tutorial.
119+
120+
121+
!![Preview of Docker environment](img/env-docker.png)
122+
> **INFORMATION** If you are developing with Visual Studio Code (VS Code), you may connect to your VS Code to this sandbox environment using following VS Code extensions: [Remote Explorer Extension by Microsoft](https://marketplace.visualstudio.com/items?itemName=ms-vscode.remote-explorer) and [Docker Extension by Microsoft](https://marketplace.visualstudio.com/items?itemName=ms-azuretools.vscode-docker)
123+
>
124+
> !![Preview of VS Code](img/env-vscode2.gif)
125+
126+
[OPTION END]
127+
128+
129+
[DONE]
130+
[ACCORDION-END]
131+
132+
[ACCORDION-BEGIN [Step 2: ](Set environment variables)]
133+
134+
Set your environment variables.
135+
136+
```SHELL
137+
# set username if not set
138+
export USERNAME=tutorialuser
139+
140+
# set location to find settings (configuration) for metaflow, customizable
141+
export HOME=/root
142+
```
143+
144+
[DONE]
145+
[ACCORDION-END]
146+
147+
[ACCORDION-BEGIN [Step 3: ](Install Metaflow package)]
148+
149+
Install the Metaflow package.
150+
151+
```SHELL
152+
pip install sap-ai-core-metaflow
153+
```
154+
155+
[DONE]
156+
[ACCORDION-END]
157+
158+
159+
[ACCORDION-BEGIN [Step 4: ](Configure Metaflow)]
160+
161+
Create a configuration file for Metaflow. Metaflow uses settings from configurations to store snippets of pipeline in your AWS S3 storage.
162+
163+
```SHELL
164+
mkdir -p $HOME/.metaflowconfig
165+
touch $HOME/.metaflowconfig/config.json
166+
```
167+
168+
Edit the following snippet and the paste in the `config.json` file, created above. Replace `<YOUR_S3_BUCKET_NAME>` with AWS S3 bucket ID that you want Metaflow to use to store files.
169+
170+
```JSON [2-3]
171+
{
172+
"METAFLOW_DATASTORE_SYSROOT_S3": "s3://<YOUR_S3_BUCKET_NAME>/metaflow",
173+
"METAFLOW_DATATOOLS_SYSROOT_S3": "s3://<YOUR_S3_BUCKET_NAME>/metaflow/data",
174+
"METAFLOW_DEFAULT_DATASTORE": "s3"
175+
}
176+
```
177+
178+
[DONE]
179+
[ACCORDION-END]
180+
181+
[ACCORDION-BEGIN [Step 5: ](Configure AWS)]
182+
183+
Create a credentials file for your AWS S3 Object Store. The file is used by the Metaflow package access your AWS S3 store.
184+
185+
```SHELL
186+
mkdir -p $HOME/.aws
187+
touch $HOME/.aws/credentials
188+
```
189+
190+
Edit the following snippet and paste it in your `credentials` file. Replace `<YOUR_S3_ACCESS...>` with your AWS S3 credentials, **do not** enclose your credentials within quotes(`""`).
191+
192+
```TEXT [2-3]
193+
[default]
194+
aws_access_key_id = <YOUR_S3_ACCESS_ID>
195+
aws_secret_access_key = <YOUR_S3_ACCESS_KEY>
196+
```
197+
198+
> **INFORMATION** the Metaflow library for SAP AI Core uses AWS, however you may skip the installation of AWS CLI.
199+
200+
[VALIDATE_1]
201+
[ACCORDION-END]
202+
203+
204+
[ACCORDION-BEGIN [Step 6: ](Metaflow HelloWorld)]
205+
206+
Create a file `hellometaflow.py` with following contents.
207+
208+
```PYTHON
209+
# You should divide the contents of the steps of the pipeline file into different python modules.
210+
# However the contents within each step may have snippets imported from separate python packages/modules.
211+
from metaflow import FlowSpec, step
212+
213+
214+
class HelloFlow(FlowSpec):
215+
"""
216+
A flow where Metaflow prints 'Hi'.
217+
218+
Run this flow to validate that Metaflow is installed correctly.
219+
220+
"""
221+
222+
@step
223+
def start(self):
224+
"""
225+
This is the 'start' step. All flows must have a step named 'start' that
226+
is the first step in the flow.
227+
228+
"""
229+
print("HelloFlow is starting.")
230+
self.next(self.hello)
231+
232+
@step
233+
def hello(self):
234+
"""
235+
A step for metaflow to introduce itself.
236+
237+
"""
238+
print("Metaflow says: Hi!")
239+
self.next(self.end)
240+
241+
@step
242+
def end(self):
243+
"""
244+
This is the 'end' step. All flows must have an 'end' step, which is the
245+
last step in the flow.
246+
247+
"""
248+
print("HelloFlow is all done.")
249+
250+
251+
if __name__ == "__main__":
252+
HelloFlow()
253+
```
254+
255+
> **INFORMATION** You may discover more snippets in the [official Metaflow tutorials.](https://docs.metaflow.org/getting-started/tutorials)
256+
257+
[VALIDATE_1]
258+
[ACCORDION-END]
259+
260+
261+
[ACCORDION-BEGIN [Step 7: ](Run Metaflow pipeline locally)]
262+
263+
Inspect the steps of the Metaflow pipeline `hellometaflow.py` using following snippet.
264+
265+
```SHELL
266+
python hellometaflow.py show
267+
```
268+
269+
Run this snippet locally.
270+
271+
```SHELL
272+
python hellometaflow.py run
273+
```
274+
275+
!![metaflow](img/hello-metaflow.png)
276+
277+
[VALIDATE_1]
278+
[ACCORDION-END]
279+
280+
[ACCORDION-BEGIN [Step 7: ](Save set up environment)]
281+
282+
283+
[OPTION BEGIN [Python virtual environment]]
284+
285+
To deactivate Python virtual environment, use the following command.
286+
287+
```SHELL
288+
deactivate
289+
```
290+
291+
[OPTION END]
292+
293+
294+
[OPTION BEGIN [Docker environment]]
295+
296+
> **CAUTION**: These steps are only applicable if you used the Docker environment in the step 1.
297+
298+
Docker containers are ephemeral, meaning that the installation you did inside will lost upon closing. To save the changes, first locate your container ID using the following snippet on your local system (not inside the Docker container).
299+
300+
```SHELL
301+
docker ps
302+
```
303+
304+
Use container resulting container ID to save the changes, including files, installations and commands, in the form of Docker images:
305+
306+
```SHELL
307+
docker commit <CONTAINER_ID> tutorial_metaflow:0.1
308+
```
309+
310+
You may now close the Docker Container, by using the `exit` command inside your Docker container.
311+
312+
!![image](img/env-docker-save.png)
313+
314+
To pick up where you left off, use the following snippet. Ensure you are using the tag `:01` with which you saved the changes.
315+
316+
```SELL
317+
docker run -it -v //var/run/docker.sock:/var/run/docker.sock tutorial_metaflow:0.1 /bin/sh
318+
```
319+
320+
[OPTION END]
321+
322+
[DONE]
323+
[ACCORDION-END]

‎tutorials/ai-core-metaflow-setup/img/env-docker-save.png‎

94.7 KB
Loading

‎tutorials/ai-core-metaflow-setup/img/env-docker.png‎

122 KB
Loading

‎tutorials/ai-core-metaflow-setup/img/env-vscode2.gif‎

974 KB
Loading

‎tutorials/ai-core-metaflow-setup/img/hello-metaflow.png‎

212 KB
Loading

0 commit comments

Comments
 (0)