Skip to content

Commit 02af73d

Browse files
author
Deepak Sahu
committed
ai-core-metrics: screenshot update
changes from ai-core-sdk v1.15.1
1 parent 6c020fc commit 02af73d

6 files changed

Lines changed: 35 additions & 33 deletions

File tree

‎tutorials/ai-core-metrics/ai-core-metrics.md‎

Lines changed: 35 additions & 33 deletions
Original file line numberDiff line numberDiff line change
@@ -18,7 +18,7 @@ author_profile: https://github.com/dhrubpaul
1818
- How to log step information along with metrics
1919
- How to log custom metrics structure
2020

21-
In this tutorial, you will use SAP AI Launchpad to compare two models that have been generated using SAP AI Core. This tutorial builds on the previous tutorials on house price prediction and ingesting data.
21+
In this tutorial, you'll use SAP AI Launchpad to compare two models that have been generated using SAP AI Core. This tutorial builds on the previous tutorials on house price prediction and ingesting data.
2222

2323
> Important: Comparing models is only available using SAP AI Launchpad, and not the API endpoints. The comparison step is optional.
2424
@@ -100,7 +100,7 @@ pickle.dump(clf, open(MODEL_PATH, 'wb'))
100100
# <PASTE CODE HERE>
101101
```
102102

103-
> The snippet includes some placeholders that say `# <PASTE CODE HERE>`. We complete these entries throughout the tutorial. For clarity, the comments in the code also include the relevant step number.
103+
> The snippet includes some placeholders that state `# <PASTE CODE HERE>`. You'll complete these entries throughout the tutorial. For clarity, the comments in the code also include the relevant step number.
104104
105105
This Python script contains all of the modifications needed for logging metrics, meaning that you can leave your previous workflows as they are.
106106

@@ -109,15 +109,15 @@ This Python script contains all of the modifications needed for logging metrics,
109109

110110
[ACCORDION-BEGIN [Step 2: ](Add connection)]
111111

112-
Add the following connection snippet. The initialization value of `base_url=''` - an empty string - is mandatory, as it indicates the code should be connected to the SAP AI Core environment.
112+
Add the following code snippet.
113113

114114
```PYTHON
115115
from ai_core_sdk.tracking import Tracking
116-
aic_connection = Tracking(base_url='') # DO NOT Change, the value is intentionally passed as empty string, When this code will run inside SAP AI Core then the values will be auto-populated
116+
aic_connection = Tracking()
117117
...
118118
```
119119

120-
> **CAUTION**: The above code is very similar to the code used to connect the SAP AI Core SDK to your local system. However, here it is required to establish connection within the SAP AI Core execution environment.
120+
> **CAUTION**: This code snippet is very similar to the code used to connect the SAP AI Core SDK to your local system. However, in this case, it is required to establish a connection within the SAP AI Core execution environment.
121121
122122
[DONE]
123123
[ACCORDION-END]
@@ -127,7 +127,8 @@ aic_connection = Tracking(base_url='') # DO NOT Change, the value is intentional
127127
Add the following snippet to log the number of observations in your dataset.
128128

129129
```PYTHON
130-
aic_connection.metrics.log_metrics(
130+
# from ai_core_sdk.models import Metric
131+
aic_connection.log_metrics(
131132
metrics = [
132133
Metric(
133134
name= "N_observations", value= float(df.shape[0]), timestamp=datetime.utcnow()),
@@ -147,14 +148,14 @@ After execution, this is shown in SAP AI Launchpad. You can zoom in for details.
147148
Add the following snippet to store metrics for step information. This snippet is also useful for tracking the metrics on epochs of the training process.
148149

149150
```PYTHON
150-
aic_connection.metrics.log_metrics(
151+
aic_connection.log_metrics(
151152
metrics = [
152153
Metric(name= "(Val) Fold R2", value= float(val_step_r2), timestamp=datetime.utcnow(), step=i),
153154
]
154155
)
155156
```
156157

157-
The variable `i` in is already present in your code to pass to the parameter `step=i`. This reflects as shown in SAP AI Launchpad, when executed. Please zoom in to view.
158+
The variable `i` in is already present in your code to pass to the parameter `step=i`. This is also shown in SAP AI Launchpad. You can zoom in to view.
158159

159160
!![image](img/ail/step.png)
160161

@@ -166,7 +167,7 @@ The variable `i` in is already present in your code to pass to the parameter `st
166167
Add the following snippet to store metrics for step information.
167168

168169
```PYTHON
169-
aic_connection.metrics.log_metrics(
170+
aic_connection.log_metrics(
170171
metrics = [
171172
Metric(
172173
name= "Test data R2",
@@ -186,7 +187,8 @@ Your code should resemble:
186187

187188
!![image](img/ail/modelA.png)
188189

189-
After execution, this is shown in SAP AI Launchpad.
190+
After execution, this is shown in SAP AI Launchpad.
191+
190192
!![image](img/ail/modelB.png)
191193

192194
[DONE]
@@ -197,7 +199,7 @@ After execution, this is shown in SAP AI Launchpad.
197199
Add the following snippet to store metrics based on a customized structure.
198200

199201
```PYTHON
200-
aic_connection.metrics.set_custom_info(
202+
aic_connection.set_custom_info(
201203
custom_info= [
202204
MetricCustomInfo(name= "Feature Importance (verbose)", value= str(r)),
203205
MetricCustomInfo(name= "Feature Importance (brief)", value= feature_importances )
@@ -209,7 +211,7 @@ The structure must be type-cast to `str` (string). Here, the structure used is [
209211

210212
The variables `r` and `feature_importances` are already created in the starter code.
211213

212-
After execution, this is shown in SAP AI Launchpad.
214+
After execution, you can see this in SAP AI Launchpad.
213215

214216
!![image](img/ail/custom.png)
215217

@@ -219,13 +221,13 @@ After execution, this is shown in SAP AI Launchpad.
219221
>
220222
> What it is?
221223
>
222-
> - Indicates for a given target, model, dataset and task, how much the model depends on a given feature.
224+
> - Shows how much a model depends on a given feature for a given target, model, dataset and task.
223225
> - Gives an empirical estimate of how much loss is attributed to the removal of a given feature.
224226
>
225227
> What it is not:
226228
>
227-
> - A model, dataset or task agnostic indication of the importance of a given feature. While the method is agnostic, the results are applicable only to the specific input combination.
228-
> - A perfectly accurate indication of the importance of a given feature for a specific prediction. While this is the goal of the method, it does not account for weaknesses in the model.
229+
> - A model, dataset or task-agnostic indication of the importance of a given feature. While the method is agnostic, the results are applicable only to the specific input combination.
230+
> - An accurate indication of the importance of a given feature for a specific prediction. Although this is the goal of the method, it does not account for weaknesses in the model.
229231
>
230232
> Advantages:
231233
>
@@ -238,27 +240,27 @@ After execution, this is shown in SAP AI Launchpad.
238240
[DONE]
239241
[ACCORDION-END]
240242

241-
[ACCORDION-BEGIN [Step 6: ](Add tags for execution meta after training)]
243+
[ACCORDION-BEGIN [Step 7: ](Add tags for execution meta after training)]
242244

243-
Add the following snippet to tag you execution. The `tags` are customizable key-values.
245+
Add the following snippet to tag your execution. The `tags` are customizable key-values.
244246

245247
```PYTHON
246-
aic_connection.metrics.set_tags(
248+
aic_connection.set_tags(
247249
tags= [
248250
MetricTag(name="Validation Method Used", value= "K-Fold"), # your custom name and value
249251
MetricTag(name="Metrics", value= "R2"),
250252
]
251253
)
252254
```
253255

254-
After execution, this is shown in SAP AI Launchpad.
256+
After execution, you can see this in SAP AI Launchpad.
255257

256258
!![image](img/ail/tag.png)
257259

258260
[DONE]
259261
[ACCORDION-END]
260262

261-
[ACCORDION-BEGIN [Step 7: ](Complete files)]
263+
[ACCORDION-BEGIN [Step 8: ](Complete files)]
262264

263265
Check your modified `main.py` by comparing it with the following expected `main.py`.
264266

@@ -271,8 +273,8 @@ from datetime import datetime
271273
import pandas as pd
272274
from ai_core_sdk.models import Metric, MetricTag, MetricCustomInfo, MetricLabel
273275
#
274-
from ai_core_sdk.ai_core_v2_client import AICoreV2Client
275-
aic_connection = AICoreV2Client(base_url='') # DO NOT Change, the value is intentionally passed as empty string, When this code will run inside SAP AI Core then the values will be auto-populated
276+
from ai_core_sdk.tracking import Tracking
277+
aic_connection = Tracking()
276278
#
277279
# Variables
278280
DATA_PATH = '/app/data/train.csv'
@@ -285,7 +287,7 @@ X = df.drop('target', axis=1)
285287
y = df['target']
286288
#
287289
# Metric Logging: Basic
288-
aic_connection.metrics.log_metrics(
290+
aic_connection.log_metrics(
289291
metrics = [
290292
Metric(
291293
name= "N_observations", value= float(df.shape[0]), timestamp=datetime.utcnow()),
@@ -320,7 +322,7 @@ clf.fit(train_x, train_y)
320322
# Scoring over test data
321323
test_r2_score = clf.score(test_x, test_y)
322324
# Metric Logging: Attaching to metrics to generated model
323-
aic_connection.metrics.log_metrics(
325+
aic_connection.log_metrics(
324326
metrics = [
325327
Metric(
326328
name= "Test data R2",
@@ -344,7 +346,7 @@ feature_importances = str('')
344346
for i in r.importances_mean.argsort()[::-1]:
345347
feature_importances += f"{df.columns[i]}: {r.importances_mean[i]:.3f} +/- {r.importances_std[i]:.3f} \n"
346348
# Metric Logging: Custom Structure
347-
aic_connection.metrics.set_custom_info(
349+
aic_connection.set_custom_info(
348350
custom_info= [
349351
MetricCustomInfo(name= "Feature Importance (verbose)", value= str(r)),
350352
MetricCustomInfo(name= "Feature Importance (brief)", value= feature_importances )
@@ -356,7 +358,7 @@ import pickle
356358
pickle.dump(clf, open(MODEL_PATH, 'wb'))
357359
#
358360
# Metric Logging: Tagging the execution
359-
aic_connection.metrics.set_tags(
361+
aic_connection.set_tags(
360362
tags= [
361363
MetricTag(name="Validation Method Used", value= "K-Fold"), # your custom name and value
362364
MetricTag(name="Metrics", value= "R2"),
@@ -369,10 +371,10 @@ Check your modified `main.py` by comparing it with the following expected `main.
369371
```TEXT
370372
sklearn==0.0
371373
pandas
372-
ai-core-sdk>=1.12.0
374+
ai-core-sdk>=1.15.1
373375
```
374376

375-
Create a file called `Dockerfile` with following snippet. This file must not have a file extension or alternative name.
377+
Create a file called `Dockerfile` with the following snippet. This file must not have a file extension or alternative name.
376378

377379
```TEXT
378380
# Specify which base layers (default dependencies) to use
@@ -473,7 +475,7 @@ Create a configuration using the following values. The values are taken from the
473475
| Scenario ID | `learning-datalines`
474476
| Executable ID | `house-metrics-train`
475477

476-
The value for `Input Parameters` `DT_MAX_DEPTH` is your choice. Until now, this was set using an environment variable. If no variable is specified, this parameter will continue to be defined by the environment variables.
478+
The value for `Input Parameters` `DT_MAX_DEPTH` is your choice. Until now, this has been set using an environment variable. If a variable is not specified, this parameter continues to be defined by the environment variables.
477479

478480
> Information: This parameter can be defined using an integer to set a maximum depth or as `None`, which means that nodes are expanded until all leaves are single nodes, or contain all contain fewer data points than specified in the `min_samples_split samples`, if specified. For more information, see [the Scikit learn documentation](https://scikit-learn.org/stable/modules/generated/sklearn.tree.DecisionTreeClassifier.html).
479481

@@ -489,7 +491,7 @@ Create an execution from this configuration.
489491

490492
[OPTION BEGIN [SAP AI Launchpad]]
491493

492-
In the `ML Operations` app, choose `Executions`. Navigate to your execution and choose the `Metrics Resource` tab.
494+
In the `ML Operations` app, choose `Executions`. Navigate to your execution and choose the `Metric Resources` tab.
493495

494496
!![image](img/ail/locate.png)
495497

@@ -513,7 +515,7 @@ Navigate through `AI Core` > `lm` > `metrics` > `Get metrics` and double check t
513515
Paste and edit, then execute the following snippet:
514516

515517
```PYTHON
516-
response = ai_core_client.metrics.query(
518+
response = tracking_client.query(
517519
execution_ids = [
518520
'e1f2169db8760b5d' # list of execution IDs for which to query metrics
519521
],
@@ -571,11 +573,11 @@ Unnamed: 0: 0.000 +/- 0.000
571573
572574
[ACCORDION-BEGIN [Step 11: ](Compare metrics (optional))]
573575
574-
Create two configurations: one with `DT_MAX_DEPTH = 3` and another with `DT_MAX_DEPTH = 6`. Then create executions for both of those configurations.
576+
Create two configurations: one with `DT_MAX_DEPTH = 3` and the other with `DT_MAX_DEPTH = 6`. Then create executions for both configurations.
575577
576578
!![image](img/ail/compare-1.png)
577579
578-
You can then compare metrics for the executions using two different configurations.
580+
You can then compare metrics for the executions using the two different configurations.
579581
580582
!![image](img/ail/compare-2.png)
581583

‎tutorials/ai-core-metrics/img/ail/basic.png‎

53.5 KB
Loading

‎tutorials/ai-core-metrics/img/ail/custom.png‎

-786 Bytes
Loading

‎tutorials/ai-core-metrics/img/ail/modelA.png‎

-624 Bytes
Loading

‎tutorials/ai-core-metrics/img/ail/step.png‎

37.6 KB
Loading

‎tutorials/ai-core-metrics/img/ail/tag.png‎

41.6 KB
Loading

0 commit comments

Comments
 (0)