-
Notifications
You must be signed in to change notification settings - Fork 0
Expand file tree
/
Copy pathpapers.html
More file actions
196 lines (185 loc) · 11.1 KB
/
Copy pathpapers.html
File metadata and controls
196 lines (185 loc) · 11.1 KB
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
<!DOCTYPE html>
<html lang="en">
<head>
<meta charset="UTF-8" />
<meta name="viewport" content="width=device-width, initial-scale=1.0" />
<title>Tech Papers · feiyang3cat</title>
<link rel="preconnect" href="https://fonts.googleapis.com" />
<link rel="preconnect" href="https://fonts.gstatic.com" crossorigin />
<link href="https://fonts.googleapis.com/css2?family=Baloo+2:wght@600;700;800&family=Nunito:ital,wght@0,600;0,700;0,800;1,600&display=swap" rel="stylesheet" />
<link rel="stylesheet" href="style.css" />
</head>
<body>
<div class="container">
<nav class="nav">
<a class="nav-home" href="index.html">=^· ⩊ ·^=</a>
<div class="nav-links">
<a href="papers.html" class="active">Papers</a>
<a href="learning.html">Learning</a>
<a href="cheetah.html">Cheetahs</a>
<a href="bouldering.html">Bouldering</a>
</div>
</nav>
<header class="hero">
<h1 class="hero-title">Tech Papers</h1>
<p class="hero-sub">~ papers i read, with notes & summaries ~</p>
<p class="hero-stat">
paper count: <span class="stat-pill">0</span>
key: <span class="stat-pill stat-pill-key">0</span>
</p>
</header>
<div class="filter-bar" role="group" aria-label="Filter papers by topic">
<button class="filter-btn active" data-filter="all">all</button>
<button class="filter-btn key" data-filter="key">Key</button>
<button class="filter-btn" data-filter="parallelism">Parallelism</button>
<button class="filter-btn" data-filter="memory">Memory Optimization</button>
<button class="filter-btn" data-filter="inference">Inference Serving</button>
<button class="filter-btn" data-filter="framework">Coding Framework</button>
</div>
<main class="papers">
<!-- ════ To add a paper, copy one <article> block and edit it. ════ -->
<article class="paper" data-topics="key parallelism memory">
<div class="paper-head">
<h2 class="paper-title">ZeRO (SC 2020)</h2>
<div class="paper-meta">
<span class="meta-date">June 2026</span>
<span class="tag tag-key">Key</span>
<span class="tag tag-topic">Parallelism</span>
<span class="tag tag-topic">Memory Optimization</span>
</div>
</div>
<div class="notes">
<p>TL;DR in Chinese</p>
<p>
(1) 首先这篇需要提前理解模型是如何分布式训练的：用 Data Parallelism 分布式训练（horovod 那一篇学过了），和两种 Model Parallelism 的分布式训练思路（横切 -> pipeline pattern or layer parallelism，或者竖切 -> tensor parallelism）需要了解下。之所以有了 DP 和 MP 两种方式的 parallelism，主要还是因为原来光是数据大，现在模型也大，很难用一个 GPU 或者 node 来 support 一个 model。
</p>
<p>
(2-a) 优化 memory 的 method/方法论：Horovod 主要讲网络 overhead 优化，这里主要讲内存怎么优化，整个思考方式还是挺 clean and neat 的。memory footprint 分析分别有什么类型的 state 占比，或者数量级有多大（在 training 整个角度分了 optimizer state、gradient 和 parameters 作为三种核心的 model state，剩下的一些 memory footprint 简单叫做 residual state）。第二步细分每一种 state 是不是有冗余，like cluster 角度多次存储，像 model state 全部模型都 hold 相当于存了 N 份，当然如果是实时系统这个 makes sense for perf，但是离线训练系统这个就可以继续优化，全部 shard 成 1/N 然后通过 all reduce 或者其他方式再聚合。最后需要评估改进，like 内存和网络的全面影响。
</p>
<p>
(2-b) 感觉还需要一点的方法论基础就是 parallel computing，like all-reduce/reduce-scatter/all-gather…… 目前我只有上学的时候这门课的基础知识，似乎作为基础就够了。
</p>
<p>
(3) 具体的领域知识很重要：(a) 由于 optimizer states 是一个巨大的、和 weights 一一对应的，且发生在 forward/backward propagate 之后，就可以 reduce-scatter 1/N。(b) layer 的 backward propagation 并不需要后一层 layer 的 gradients，只需要 input gradients 和 activation gradients，所以每一层计算完可以马上发生 reduce-scatter 来释放内存（注意 zero-dp 里是假设在 dp 的基础上进行的）。但是注意 layer parameter gradients optimizer 计算是需要的，所以 reduce-scatter 的时候要 scatter aligned with optimizer ownership。-> 上课学过的 dependency analysis。(c) parameter partitioning is based on the fact that forward propagation happens layer by layer, the whole model is never used at one time fully。不过这样每层 forward propagation 时候需要 all-gather。这种做法和 tensor parallelism 的区别就是 for batches of data 计算 forward propagation 的时候，每一层还是在独立计算的，需要 all-gather 计算的这一层，因为这里每个 GPU 的数据不一样。
</p>
<p><a href="papers/zero.html">this paper has a full page for notes</a></p>
</div>
<a class="paper-link" href="papers/zero.html">→ go to the paper</a>
</article>
<article class="paper" data-topics="inference">
<div class="paper-head">
<h2 class="paper-title">Orca (OSDI 2022)</h2>
<div class="paper-meta">
<span class="meta-date">June 2026</span>
<span class="tag tag-topic">Inference Serving</span>
<span class="tag tag-todo">todo</span>
</div>
</div>
<div class="notes">
<!-- TODO: write full summary -->
<p>
The foundational paper for understanding LLM serving. PagedAttention,
DistServe, and Splitwise all build on the surfaces Orca introduces.
</p>
</div>
<a class="paper-link" href="https://www.usenix.org/conference/osdi22/presentation/yu" target="_blank" rel="noopener">→ go to the paper</a>
</article>
<article class="paper" data-topics="parallelism">
<div class="paper-head">
<h2 class="paper-title">Horovod</h2>
<div class="paper-meta">
<span class="meta-date">May 2026</span>
<span class="tag tag-topic">Parallelism / Distributed Training for AI</span>
<span class="tag level-simple">simple</span>
</div>
</div>
<div class="notes">
<p>
TL;DR in Chinese
说到底就是先介绍了一种分布式训练的方式就是Data Parallelism是怎么work的每个节点都有相同的模型，数据分片。
之所以这么说是因为向前传递和向后传递不改参数所有节点参数一致，
optimizer最后需要根据graident全局计算weights变更的时候再进行all reduce/gather。
Horovod提出不用参数服务器而是workers ring-based reduce and propagation减少通信冗余from NM+N? to 2(N-1).
N are data, M are nodes.
</p>
<p>
Data-parallel distributed training framework. Three questions:
<strong>how</strong> it distributes training, <strong>why</strong> that
works, and the <strong>arch + algorithm</strong>.
</p>
<p>
<strong>1. How — data parallelism.</strong> Data is split across servers;
each server holds a full copy of the model.
</p>
<p>
<strong>2. Why it works ⭐</strong> Each step starts from identical weights.
The <strong>forward pass</strong> computes a local loss and doesn't change
the weights; the <strong>backward pass</strong> produces <em>local</em>
gradients that differ per server. Those gradients are then
<strong>averaged across servers (all-reduce)</strong>, and every server
applies the <strong>same averaged gradient</strong> to the
<strong>same starting weights</strong> — so all replicas end the step
identical and stay in sync.
</p>
<p>
<strong>3. Arch + algorithm.</strong> Older systems used parameter servers +
workers (now deprecated); Horovod instead uses ring-allreduce with tensor
fusion. Ring-allreduce takes <code>2(N-1)</code> steps — two passes around
the ring (a reduce-scatter then an all-gather) — and each step sends only
<code>1/N</code> of the data, so bandwidth usage stays low and independent of
the number of servers. Tensor fusion batches small tensors before sending,
improving bandwidth efficiency and average latency: it slightly delays the
first tensor (which waits to be batched) but cuts the queueing time of the
tensors that follow.
</p>
</div>
<a class="paper-link" href="https://arxiv.org/abs/1802.05799" target="_blank" rel="noopener">→ go to the paper</a>
</article>
<article class="paper" data-topics="framework">
<div class="paper-head">
<h2 class="paper-title">PyTorch (NeurIPS 2019)</h2>
<div class="paper-meta">
<span class="meta-date">March 2026</span>
<span class="tag tag-topic">Coding Framework</span>
<span class="tag tag-todo">todo</span>
</div>
</div>
<div class="notes">
<!-- TODO: write full summary -->
<p>
PyTorch's imperative style makes it far easier to grasp how deep learning
frameworks work conceptually than TensorFlow's graph-based approach.
</p>
</div>
<a class="paper-link" href="https://arxiv.org/abs/1912.01703" target="_blank" rel="noopener">→ go to the paper</a>
</article>
</main>
<footer class="site-footer">
<div class="footer-cats" aria-hidden="true">=^..^= =^..^= =^..^=</div>
<a href="https://github.com/feiyang3cat/feiyang3cat.github.io" target="_blank" rel="noopener">source on GitHub</a>
<p class="footer-link">
a selection of papers:
<a href="https://yangwang83.github.io/sys4ml.html" target="_blank" rel="noopener">yangwang83.github.io/sys4ml.html</a>
</p>
</footer>
</div>
<script>
const buttons = document.querySelectorAll(".filter-btn");
const papers = document.querySelectorAll(".papers .paper");
const keyPapers = document.querySelectorAll('.papers .paper[data-topics~="key"]');
document.querySelector(".stat-pill:not(.stat-pill-key)").textContent = papers.length;
document.querySelector(".stat-pill-key").textContent = keyPapers.length;
buttons.forEach((btn) => {
btn.addEventListener("click", () => {
const filter = btn.dataset.filter;
buttons.forEach((b) => b.classList.toggle("active", b === btn));
papers.forEach((paper) => {
const topics = (paper.dataset.topics || "").split(" ");
const show = filter === "all" || topics.includes(filter);
paper.style.display = show ? "" : "none";
});
});
});
</script>
</body>
</html>