Skip to content

Commit e2dfe0d

Browse files
cli99cli99
authored
Add flops profiler tutorial (deepspeedai#682)
* work on flops profiler tutorial * update flops profiler tutorial * add flops profiler tutorial and fix names * work on flops profiler tutorial * update flops profiler tutorial * add flops profiler tutorial and fix names * fix tailing ws * fix names * remove multistep profiling and update docs * fix cases where functionals and submodules coexist in a parent module, update readme * fix typo * always invoke post hook function * fix module flops sum and update tests * update tutorial
1 parent 6ee3b29 commit e2dfe0d

12 files changed

Lines changed: 1248 additions & 485 deletions

File tree

‎deepspeed/out2‎

Lines changed: 7 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,7 @@
1+
============================= test session starts ==============================
2+
platform linux -- Python 3.6.9, pytest-6.0.1, py-1.9.0, pluggy-0.13.1
3+
rootdir: /home/chengli1/projects/DeepSpeed
4+
plugins: forked-1.3.0, hypothesis-5.41.3, xdist-2.1.0, cov-2.10.1
5+
collected 0 items
6+
7+
============================ no tests ran in 0.01s =============================

‎deepspeed/profiling/config.py‎

Lines changed: 8 additions & 9 deletions
Original file line numberDiff line numberDiff line change
@@ -15,8 +15,7 @@ def __init__(self, param_dict):
1515
super(DeepSpeedFlopsProfilerConfig, self).__init__()
1616

1717
self.enabled = None
18-
self.start_step = None
19-
self.end_step = None
18+
self.profile_step = None
2019
self.module_depth = None
2120
self.top_modules = None
2221

@@ -35,13 +34,9 @@ def _initialize(self, flops_profiler_dict):
3534
FLOPS_PROFILER_ENABLED,
3635
FLOPS_PROFILER_ENABLED_DEFAULT)
3736

38-
self.start_step = get_scalar_param(flops_profiler_dict,
39-
FLOPS_PROFILER_START_STEP,
40-
FLOPS_PROFILER_START_STEP_DEFAULT)
41-
42-
self.end_step = get_scalar_param(flops_profiler_dict,
43-
FLOPS_PROFILER_END_STEP,
44-
FLOPS_PROFILER_END_STEP_DEFAULT)
37+
self.profile_step = get_scalar_param(flops_profiler_dict,
38+
FLOPS_PROFILER_PROFILE_STEP,
39+
FLOPS_PROFILER_PROFILE_STEP_DEFAULT)
4540

4641
self.module_depth = get_scalar_param(flops_profiler_dict,
4742
FLOPS_PROFILER_MODULE_DEPTH,
@@ -50,3 +45,7 @@ def _initialize(self, flops_profiler_dict):
5045
self.top_modules = get_scalar_param(flops_profiler_dict,
5146
FLOPS_PROFILER_TOP_MODULES,
5247
FLOPS_PROFILER_TOP_MODULES_DEFAULT)
48+
49+
self.detailed = get_scalar_param(flops_profiler_dict,
50+
FLOPS_PROFILER_DETAILED,
51+
FLOPS_PROFILER_DETAILED_DEFAULT)

‎deepspeed/profiling/constants.py‎

Lines changed: 8 additions & 8 deletions
Original file line numberDiff line numberDiff line change
@@ -12,11 +12,11 @@
1212
flops profiler should be enabled as:
1313
"session_params": {
1414
"flops_profiler": {
15-
"enalbe": [true|false],
16-
"start_step": 5,
17-
"end_step": 6,
15+
"enabled": true,
16+
"profile_step": 1,
1817
"module_depth": -1,
1918
"top_modules": 3,
19+
"detailed": true,
2020
}
2121
}
2222
'''
@@ -26,14 +26,14 @@
2626
FLOPS_PROFILER_ENABLED = "enabled"
2727
FLOPS_PROFILER_ENABLED_DEFAULT = False
2828

29-
FLOPS_PROFILER_START_STEP = "start_step"
30-
FLOPS_PROFILER_START_STEP_DEFAULT = 5
31-
32-
FLOPS_PROFILER_END_STEP = "end_step"
33-
FLOPS_PROFILER_END_STEP_DEFAULT = FLOPS_PROFILER_START_STEP_DEFAULT + 1
29+
FLOPS_PROFILER_PROFILE_STEP = "profile_step"
30+
FLOPS_PROFILER_PROFILE_STEP_DEFAULT = 1
3431

3532
FLOPS_PROFILER_MODULE_DEPTH = "module_depth"
3633
FLOPS_PROFILER_MODULE_DEPTH_DEFAULT = -1
3734

3835
FLOPS_PROFILER_TOP_MODULES = "top_modules"
3936
FLOPS_PROFILER_TOP_MODULES_DEFAULT = 3
37+
38+
FLOPS_PROFILER_DETAILED = "detailed"
39+
FLOPS_PROFILER_DETAILED_DEFAULT = True

‎deepspeed/profiling/flops_profiler/README.md‎

Lines changed: 377 additions & 182 deletions
Large diffs are not rendered by default.

0 commit comments

Comments
 (0)