Skip to content

Commit 4a312bf

Browse files
jkoseknv-kkudrynski
jkosek
authored and
nv-kkudrynski
committed
[ResNet50/TF] Updated Model Navigator
1 parent 58b7aa3 commit 4a312bf

22 files changed

Lines changed: 285 additions & 605 deletions

‎TensorFlow/Classification/ConvNets/.dockerignore‎

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,2 @@
1+
workspace/
2+
navigator_workspace/

‎TensorFlow/Classification/ConvNets/.gitignore‎

Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -124,3 +124,7 @@ models/
124124
tests/
125125
pytest.ini
126126
requirements_test.txt
127+
128+
# Inference related
129+
workspace/
130+
navigator_workspace/

‎TensorFlow/Classification/ConvNets/Dockerfile.inference‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -25,6 +25,6 @@ RUN pip uninstall -y typing
2525
ADD requirements.txt .
2626
ADD triton/requirements.txt triton/requirements.txt
2727
RUN pip install -r requirements.txt
28-
RUN pip install -r triton/requirements.txt
28+
RUN pip install --extra-index-url https://pypi.ngc.nvidia.com -r triton/requirements.txt
2929

3030
ADD . .

‎TensorFlow/Classification/ConvNets/triton/README.md‎

Lines changed: 27 additions & 27 deletions
Original file line numberDiff line numberDiff line change
@@ -184,7 +184,6 @@ processing, and training of the model.
184184
python3 triton/calculate_metrics.py \
185185
--dump-dir ${SHARED_DIR}/correctness_dump \
186186
--metrics triton/metrics.py \
187-
--output-used-for-metrics classes \
188187
--csv ${SHARED_DIR}/correctness_metrics.csv
189188
190189
cat ${SHARED_DIR}/correctness_metrics.csv
@@ -194,24 +193,26 @@ processing, and training of the model.
194193
9. Configure the model on Triton Inference Server.
195194

196195
Generate the configuration from your model repository.
197-
198196
```
199-
python3 triton/config_model_on_triton.py \
200-
--model-repository ${MODEL_REPOSITORY_PATH} \
201-
--model-path ${SHARED_DIR}/model \
202-
--model-format ${FORMAT} \
203-
--model-name ${MODEL_NAME} \
204-
--model-version 1 \
205-
--max-batch-size ${MAX_BATCH_SIZE} \
206-
--precision ${PRECISION} \
207-
--number-of-model-instances ${NUMBER_OF_MODEL_INSTANCES} \
208-
--max-queue-delay-us ${TRITON_MAX_QUEUE_DELAY} \
209-
--preferred-batch-sizes ${TRITON_PREFERRED_BATCH_SIZES} \
210-
--capture-cuda-graph 0 \
211-
--backend-accelerator ${BACKEND_ACCELERATOR} \
212-
--load-model ${TRITON_LOAD_MODEL_METHOD}
213-
```
214-
197+
model-navigator triton-config-model \
198+
--model-repository ${MODEL_REPOSITORY_PATH} \
199+
--model-name ${MODEL_NAME} \
200+
--model-version 1 \
201+
--model-path ${SHARED_DIR}/model \
202+
--model-format ${FORMAT} \
203+
--load-model \
204+
--load-model-timeout-s 100 \
205+
--verbose \
206+
\
207+
--batching dynamic \
208+
--max-queue-delay-us ${TRITON_MAX_QUEUE_DELAY} \
209+
--preferred-batch-sizes ${TRITON_PREFERRED_BATCH_SIZES} \
210+
--backend-accelerator ${BACKEND_ACCELERATOR} \
211+
--tensorrt-precision ${PRECISION} \
212+
--max-batch-size ${MAX_BATCH_SIZE} \
213+
--engine-count-per-device ${DEVICE_KIND}=${NUMBER_OF_MODEL_INSTANCES}
214+
```
215+
215216
10. Run the Triton Inference Server accuracy tests.
216217

217218
```
@@ -228,7 +229,6 @@ processing, and training of the model.
228229
python3 triton/calculate_metrics.py \
229230
--dump-dir ${SHARED_DIR}/accuracy_dump \
230231
--metrics triton/metrics.py \
231-
--output-used-for-metrics classes \
232232
--csv ${SHARED_DIR}/accuracy_metrics.csv
233233
234234
cat ${SHARED_DIR}/accuracy_metrics.csv
@@ -307,12 +307,12 @@ Example values of some key variables in one configuration:
307307
PRECISION="fp16"
308308
FORMAT="tf-trt"
309309
BATCH_SIZE="1, 2, 4, 8, 16, 32, 64, 128"
310-
BACKEND_ACCELERATOR="trt"
310+
BACKEND_ACCELERATOR="none"
311311
MAX_BATCH_SIZE="128"
312312
NUMBER_OF_MODEL_INSTANCES="2"
313313
TRITON_MAX_QUEUE_DELAY="1"
314314
TRITON_PREFERRED_BATCH_SIZES="64 128"
315-
315+
DEVICE_KIND="gpu"
316316
```
317317

318318

@@ -352,9 +352,9 @@ The performance measurements in this document were conducted at the time of publ
352352
This table lists the common variable parameters for all performance measurements:
353353
| Parameter Name | Parameter Value |
354354
|:-----------------------------|:------------------|
355-
| Max Batch Size | 128.0 |
356-
| Number of model instances | 2.0 |
357-
| Triton Max Queue Delay | 1.0 |
355+
| Max Batch Size | 128 |
356+
| Number of model instances | 2 |
357+
| Triton Max Queue Delay | 1 |
358358
| Triton Preferred Batch Sizes | 64 128 |
359359

360360

@@ -495,9 +495,9 @@ Full tabular data
495495
This table lists the common variable parameters for all performance measurements:
496496
| Parameter Name | Parameter Value |
497497
|:-----------------------------|:------------------|
498-
| Max Batch Size | 128.0 |
499-
| Number of model instances | 2.0 |
500-
| Triton Max Queue Delay | 1.0 |
498+
| Max Batch Size | 128 |
499+
| Number of model instances | 2 |
500+
| Triton Max Queue Delay | 1 |
501501
| Triton Preferred Batch Sizes | 64 128 |
502502

503503

‎TensorFlow/Classification/ConvNets/triton/calculate_metrics.py‎

Lines changed: 9 additions & 45 deletions
Original file line numberDiff line numberDiff line change
@@ -17,7 +17,7 @@
1717
r"""
1818
Using `calculate_metrics.py` script, you can obtain model accuracy/error metrics using defined `MetricsCalculator` class.
1919
20-
Data provided to `MetricsCalculator` are obtained from npz dump files
20+
Data provided to `MetricsCalculator` are obtained from dump files
2121
stored in directory pointed by `--dump-dir` argument.
2222
Above files are prepared by `run_inference_on_fw.py` and `run_inference_on_triton.py` scripts.
2323
@@ -40,49 +40,24 @@
4040
import string
4141
from pathlib import Path
4242

43-
import numpy as np
44-
4543
# method from PEP-366 to support relative import in executed modules
4644

4745
if __package__ is None:
4846
__package__ = Path(__file__).parent.name
4947

5048
from .deployment_toolkit.args import ArgParserGenerator
5149
from .deployment_toolkit.core import BaseMetricsCalculator, load_from_file
52-
from .deployment_toolkit.dump import pad_except_batch_axis
50+
from .deployment_toolkit.dump import JsonDumpReader
5351

5452
LOGGER = logging.getLogger("calculate_metrics")
5553
TOTAL_COLUMN_NAME = "_total_"
5654

5755

58-
def get_data(dump_dir, prefix):
59-
"""Loads and concatenates dump files for given prefix (ex. inputs, outputs, labels, ids)"""
60-
dump_dir = Path(dump_dir)
61-
npz_files = sorted(dump_dir.glob(f"{prefix}*.npz"))
62-
data = None
63-
if npz_files:
64-
# assume that all npz files with given prefix contain same set of names
65-
names = list(np.load(npz_files[0].as_posix()).keys())
66-
# calculate target shape
67-
target_shape = {
68-
name: tuple(np.max([np.load(npz_file.as_posix())[name].shape for npz_file in npz_files], axis=0))
69-
for name in names
70-
}
71-
# pad and concatenate data
72-
data = {
73-
name: np.concatenate(
74-
[pad_except_batch_axis(np.load(npz_file.as_posix())[name], target_shape[name]) for npz_file in npz_files]
75-
)
76-
for name in names
77-
}
78-
return data
79-
80-
8156
def main():
8257
logging.basicConfig(level=logging.INFO)
8358

8459
parser = argparse.ArgumentParser(description="Run models with given dataloader", allow_abbrev=False)
85-
parser.add_argument("--metrics", help=f"Path to python module containing metrics calculator", required=True)
60+
parser.add_argument("--metrics", help="Path to python module containing metrics calculator", required=True)
8661
parser.add_argument("--csv", help="Path to csv file", required=True)
8762
parser.add_argument("--dump-dir", help="Path to directory with dumped outputs (and labels)", required=True)
8863

@@ -93,29 +68,18 @@ def main():
9368

9469
args = parser.parse_args()
9570

96-
LOGGER.info(f"args:")
71+
LOGGER.info("args:")
9772
for key, value in vars(args).items():
9873
LOGGER.info(f" {key} = {value}")
9974

10075
MetricsCalculator = load_from_file(args.metrics, "metrics", "MetricsCalculator")
10176
metrics_calculator: BaseMetricsCalculator = ArgParserGenerator(MetricsCalculator).from_args(args)
10277

103-
ids = get_data(args.dump_dir, "ids")["ids"]
104-
x = get_data(args.dump_dir, "inputs")
105-
y_true = get_data(args.dump_dir, "labels")
106-
y_pred = get_data(args.dump_dir, "outputs")
107-
108-
common_keys = list({k for k in (y_true or [])} & {k for k in (y_pred or [])})
109-
for key in common_keys:
110-
if y_true[key].shape != y_pred[key].shape:
111-
LOGGER.warning(
112-
f"Model predictions and labels shall have equal shapes. "
113-
f"y_pred[{key}].shape={y_pred[key].shape} != "
114-
f"y_true[{key}].shape={y_true[key].shape}"
115-
)
116-
117-
metrics = metrics_calculator.calc(ids=ids, x=x, y_pred=y_pred, y_real=y_true)
118-
metrics = {TOTAL_COLUMN_NAME: len(ids), **metrics}
78+
reader = JsonDumpReader(args.dump_dir)
79+
for ids, x, y_true, y_pred in reader.iterate_over(["ids", "inputs", "labels", "outputs"]):
80+
ids = list(ids["ids"]) if ids is not None else None
81+
metrics_calculator.update(ids=ids, x=x, y_pred=y_pred, y_real=y_true)
82+
metrics = metrics_calculator.metrics
11983

12084
metric_names_with_space = [name for name in metrics if any([c in string.whitespace for c in name])]
12185
if metric_names_with_space:

0 commit comments

Comments
 (0)