-
Notifications
You must be signed in to change notification settings - Fork 0
Expand file tree
/
Copy pathatom.xml
More file actions
650 lines (406 loc) · 413 KB
/
Copy pathatom.xml
File metadata and controls
650 lines (406 loc) · 413 KB
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
400
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
424
425
426
427
428
429
430
431
432
433
434
435
436
437
438
439
440
441
442
443
444
445
446
447
448
449
450
451
452
453
454
455
456
457
458
459
460
461
462
463
464
465
466
467
468
469
470
471
472
473
474
475
476
477
478
479
480
481
482
483
484
485
486
487
488
489
490
491
492
493
494
495
496
497
498
499
500
501
502
503
504
505
506
507
508
509
510
511
512
513
514
515
516
517
518
519
520
521
522
523
524
525
526
527
528
529
530
531
532
533
534
535
536
537
538
539
540
541
542
543
544
545
546
547
548
549
550
551
552
553
554
555
556
557
558
559
560
561
562
563
564
565
566
567
568
569
570
571
572
573
574
575
576
577
578
579
580
581
582
583
584
585
586
587
588
589
590
591
592
593
594
595
596
597
598
599
600
601
602
603
604
605
606
607
608
609
610
611
612
613
614
615
616
617
618
619
620
621
622
623
624
625
626
627
628
629
630
631
632
633
634
635
636
637
638
639
640
641
642
643
644
645
646
647
648
649
650
<?xml version="1.0" encoding="utf-8"?>
<feed xmlns="http://www.w3.org/2005/Atom">
<title>L&H SITE</title>
<subtitle>两个背包旅行者的网络自留地。分享旅行日记，Linux技术，机器学习，建站技巧</subtitle>
<link href="/atom.xml" rel="self"/>
<link href="http://www.l2h.site/"/>
<updated>2023-06-01T13:31:31.969Z</updated>
<id>http://www.l2h.site/</id>
<author>
<name>Lambert</name>
</author>
<generator uri="https://hexo.io/">Hexo</generator>
<entry>
<title>Linux内核内存管理 - 缺页异常处理</title>
<link href="http://www.l2h.site/p/db78840c.html"/>
<id>http://www.l2h.site/p/db78840c.html</id>
<published>2021-11-19T10:38:02.000Z</published>
<updated>2023-06-01T13:31:31.969Z</updated>
<content type="html"><![CDATA[<blockquote><p> 这是<Linux内核内存管理>系列的第八篇:</p><blockquote><p><a href="/p/61304bbb.html">第一篇</a>为内核内存管理过程知识点的的简单梳理</p><p><a href="/p/1dc6adc2.html">第二篇</a>介绍了内核的数据结构</p><p><a href="/p/8f5117c5.html">第三篇</a>介绍了从内核第一行代码加载到跳转到C代码前的内存处理。</p><p><a href="/p/94b22c5d.html">第四篇</a>概览了初始化C代码中的内存处理</p><p><a href="/p/20210923linuxmm4.html">第五篇（上）</a>和<a href="/p/5347665a.html">第五篇（下）</a>介绍了Memblock和伙伴系统分配器</p><p><a href="/p/75308136.html">第六篇</a>介绍了内存检测工具KFence工作原理</p><p><a href="/p/e84c7fac.html">第七篇</a>介绍了进程内存分配malloc的原理</p><p><a href="/p/8cc7cf15.html">第八篇</a>介绍了MMAP映射和反向映射原理</p></blockquote></blockquote><a id="more"></a><h2 id="前言"><a href="#前言" class="headerlink" title="前言"></a>前言</h2><p>Page Fault（缺页异常）大概是最为常见的异常，它发生在CPU访问不在内存的页时。本文以Intel IA32体系结构为例，介绍Linux对缺页异常的处理过程。</p><h2 id="流程"><a href="#流程" class="headerlink" title="流程"></a>流程</h2>]]></content>
<summary type="html">
<blockquote>
<p> 这是&lt;Linux内核内存管理&gt;系列的第八篇:</p>
<blockquote>
<p><a href="/p/61304bbb.html">第一篇</a>为内核内存管理过程知识点的的简单梳理</p>
<p><a href="/p/1dc6adc2.html">第二篇</a>介绍了内核的数据结构</p>
<p><a href="/p/8f5117c5.html">第三篇</a>介绍了从内核第一行代码加载到跳转到C代码前的内存处理。</p>
<p><a href="/p/94b22c5d.html">第四篇</a>概览了初始化C代码中的内存处理</p>
<p><a href="/p/20210923linuxmm4.html">第五篇（上）</a>和<a href="/p/5347665a.html">第五篇（下）</a>介绍了Memblock和伙伴系统分配器</p>
<p><a href="/p/75308136.html">第六篇</a>介绍了内存检测工具KFence工作原理</p>
<p><a href="/p/e84c7fac.html">第七篇</a>介绍了进程内存分配malloc的原理</p>
<p><a href="/p/8cc7cf15.html">第八篇</a>介绍了MMAP映射和反向映射原理</p>
</blockquote>
</blockquote>
</summary>
<category term="Linux" scheme="http://www.l2h.site/categories/Linux/"/>
<category term="Linux内存管理" scheme="http://www.l2h.site/categories/Linux/Linux%E5%86%85%E5%AD%98%E7%AE%A1%E7%90%86/"/>
<category term="Linux" scheme="http://www.l2h.site/tags/Linux/"/>
<category term="内存管理" scheme="http://www.l2h.site/tags/%E5%86%85%E5%AD%98%E7%AE%A1%E7%90%86/"/>
<category term="进程内存" scheme="http://www.l2h.site/tags/%E8%BF%9B%E7%A8%8B%E5%86%85%E5%AD%98/"/>
</entry>
<entry>
<title>Linux内核内存管理 - 文件mmap映射和反向映射</title>
<link href="http://www.l2h.site/p/8cc7cf15.html"/>
<id>http://www.l2h.site/p/8cc7cf15.html</id>
<published>2021-11-13T13:45:55.000Z</published>
<updated>2023-06-01T13:14:04.484Z</updated>
<content type="html"><![CDATA[<blockquote><p>这是<Linux内核内存管理>系列的第八篇:</p><blockquote><p><a href="/p/61304bbb.html">第一篇</a>为内核内存管理过程知识点的的简单梳理</p><p><a href="/p/1dc6adc2.html">第二篇</a>介绍了内核的数据结构</p><p><a href="/p/8f5117c5.html">第三篇</a>介绍了从内核第一行代码加载到跳转到C代码前的内存处理。</p><p><a href="/p/94b22c5d.html">第四篇</a>概览了初始化C代码中的内存处理</p><p><a href="/p/20210923linuxmm4.html">第五篇（上）</a>和<a href="/p/5347665a.html">第五篇（下）</a>介绍了Memblock和伙伴系统分配器</p><p><a href="/p/75308136.html">第六篇</a>介绍了内存检测工具KFence工作原理</p><p><a href="/p/e84c7fac.html">第七篇</a>介绍了进程内存分配malloc的原理</p></blockquote></blockquote><a id="more"></a><h2 id="mmap和munmap"><a href="#mmap和munmap" class="headerlink" title="mmap和munmap"></a>mmap和munmap</h2><p><a href="https://man7.org/linux/man-pages/man2/mmap.2.html" target="_blank" rel="noopener"><strong><em>mmap()</em></strong></a>的主要作用是将文件（普通文件或者设备文件）映射到进程的内存地址空间中，让应用程序可以以读写内存的方式来访问文件。与之对应的操作是<a href="https://man7.org/linux/man-pages/man3/munmap.3p.html" target="_blank" rel="noopener"><strong><em>munmap()</em></strong></a>。</p><p>一段示例代码来自<a href="https://en.wikipedia.org/wiki/Mmap" target="_blank" rel="noopener">维基百科</a>如下：</p><figure class="highlight c"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br><span class="line">21</span><br><span class="line">22</span><br><span class="line">23</span><br><span class="line">24</span><br><span class="line">25</span><br><span class="line">26</span><br><span class="line">27</span><br><span class="line">28</span><br><span class="line">29</span><br><span class="line">30</span><br><span class="line">31</span><br><span class="line">32</span><br><span class="line">33</span><br><span class="line">34</span><br><span class="line">35</span><br><span class="line">36</span><br><span class="line">37</span><br><span class="line">38</span><br><span class="line">39</span><br><span class="line">40</span><br><span class="line">41</span><br><span class="line">42</span><br><span class="line">43</span><br><span class="line">44</span><br><span class="line">45</span><br><span class="line">46</span><br><span class="line">47</span><br><span class="line">48</span><br><span class="line">49</span><br><span class="line">50</span><br><span class="line">51</span><br><span class="line">52</span><br><span class="line">53</span><br><span class="line">54</span><br><span class="line">55</span><br><span class="line">56</span><br></pre></td><td class="code"><pre><span class="line"><span class="meta">#<span class="meta-keyword">include</span> <span class="meta-string"><sys/types.h></span></span></span><br><span class="line"><span class="meta">#<span class="meta-keyword">include</span> <span class="meta-string"><sys/mman.h></span></span></span><br><span class="line"><span class="meta">#<span class="meta-keyword">include</span> <span class="meta-string"><err.h></span></span></span><br><span class="line"><span class="meta">#<span class="meta-keyword">include</span> <span class="meta-string"><fcntl.h></span></span></span><br><span class="line"><span class="meta">#<span class="meta-keyword">include</span> <span class="meta-string"><stdio.h></span></span></span><br><span class="line"><span class="meta">#<span class="meta-keyword">include</span> <span class="meta-string"><stdlib.h></span></span></span><br><span class="line"><span class="meta">#<span class="meta-keyword">include</span> <span class="meta-string"><string.h></span></span></span><br><span class="line"><span class="meta">#<span class="meta-keyword">include</span> <span class="meta-string"><unistd.h></span></span></span><br><span class="line"></span><br><span class="line"><span class="function"><span class="keyword">int</span> <span class="title">main</span><span class="params">(<span class="keyword">void</span>)</span></span></span><br><span class="line"><span class="function"></span>{</span><br><span class="line"> <span class="keyword">const</span> <span class="keyword">char</span> str1[] = <span class="string">"string 1"</span>;</span><br><span class="line"> <span class="keyword">const</span> <span class="keyword">char</span> str2[] = <span class="string">"string 2"</span>;</span><br><span class="line"> <span class="keyword">pid_t</span> parpid = getpid(), childpid;</span><br><span class="line"> <span class="keyword">int</span> fd = <span class="number">-1</span>;</span><br><span class="line"> <span class="keyword">char</span> *anon, *zero;</span><br><span class="line"></span><br><span class="line"> <span class="keyword">if</span> ((fd = <span class="built_in">open</span>(<span class="string">"/dev/zero"</span>, O_RDWR, <span class="number">0</span>)) == <span class="number">-1</span>)</span><br><span class="line"> err(<span class="number">1</span>, <span class="string">"open"</span>);</span><br><span class="line"></span><br><span class="line"> anon = (<span class="keyword">char</span>*)mmap(<span class="literal">NULL</span>, <span class="number">4096</span>, PROT_READ|PROT_WRITE, MAP_ANON|MAP_SHARED, <span class="number">-1</span>, <span class="number">0</span>);</span><br><span class="line"> zero = (<span class="keyword">char</span>*)mmap(<span class="literal">NULL</span>, <span class="number">4096</span>, PROT_READ|PROT_WRITE, MAP_SHARED, fd, <span class="number">0</span>);</span><br><span class="line"></span><br><span class="line"> <span class="keyword">if</span> (anon == MAP_FAILED || zero == MAP_FAILED)</span><br><span class="line"> errx(<span class="number">1</span>, <span class="string">"either mmap"</span>);</span><br><span class="line"></span><br><span class="line"> <span class="built_in">strcpy</span>(anon, str1);</span><br><span class="line"> <span class="built_in">strcpy</span>(zero, str1);</span><br><span class="line"></span><br><span class="line"> <span class="built_in">printf</span>(<span class="string">"PID %d:\tanonymous %s, zero-backed %s\n"</span>, parpid, anon, zero);</span><br><span class="line"> <span class="keyword">switch</span> ((childpid = fork())) {</span><br><span class="line"> <span class="keyword">case</span> <span class="number">-1</span>:</span><br><span class="line"> err(<span class="number">1</span>, <span class="string">"fork"</span>);</span><br><span class="line"> <span class="comment">/* NOTREACHED */</span></span><br><span class="line"> <span class="keyword">case</span> <span class="number">0</span>:</span><br><span class="line"> childpid = getpid();</span><br><span class="line"> <span class="built_in">printf</span>(<span class="string">"PID %d:\tanonymous %s, zero-backed %s\n"</span>, childpid, anon, zero);</span><br><span class="line"> sleep(<span class="number">3</span>);</span><br><span class="line"></span><br><span class="line"> <span class="built_in">printf</span>(<span class="string">"PID %d:\tanonymous %s, zero-backed %s\n"</span>, childpid, anon, zero);</span><br><span class="line"> munmap(anon, <span class="number">4096</span>);</span><br><span class="line"> munmap(zero, <span class="number">4096</span>);</span><br><span class="line"> <span class="built_in">close</span>(fd);</span><br><span class="line"> <span class="keyword">return</span> EXIT_SUCCESS;</span><br><span class="line"> }</span><br><span class="line"></span><br><span class="line"> sleep(<span class="number">2</span>);</span><br><span class="line"> <span class="built_in">strcpy</span>(anon, str2);</span><br><span class="line"> <span class="built_in">strcpy</span>(zero, str2);</span><br><span class="line"></span><br><span class="line"> <span class="built_in">printf</span>(<span class="string">"PID %d:\tanonymous %s, zero-backed %s\n"</span>, parpid, anon, zero);</span><br><span class="line"> munmap(anon, <span class="number">4096</span>);</span><br><span class="line"> munmap(zero, <span class="number">4096</span>);</span><br><span class="line"> <span class="built_in">close</span>(fd);</span><br><span class="line"> <span class="keyword">return</span> EXIT_SUCCESS;</span><br><span class="line">}</span><br></pre></td></tr></table></figure><p>上述代码的输出是：</p><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br></pre></td><td class="code"><pre><span class="line">PID 22475: anonymous string 1, zero-backed string 1</span><br><span class="line">PID 22476: anonymous string 1, zero-backed string 1</span><br><span class="line">PID 22475: anonymous string 2, zero-backed string 2</span><br><span class="line">PID 22476: anonymous string 2, zero-backed string 2</span><br></pre></td></tr></table></figure><p>主要作用是创建了两个匿名映射，父进程和子进程可以通过匿名映射来访问共享的内存。</p><p>mmap和munmap的架构如图所示，与<a href="/p/e84c7fac.html">上篇文章</a>中介绍的<strong>malloc()</strong>工作原理类似：</p><ul><li>应用程序通过libc的API <strong><em>mmap()</em></strong>和<strong><em>munmap()</em></strong>来创建和销毁映射</li><li>libc执行对应的系统调用SYS_mmap和SYS_munmap等</li><li><strong><em>mmap()</em></strong>，内核根据传入的地址，映射长度，文件信息进行<a href="/p/e84c7fac.html#VMA">VMA映射</a>的创建</li><li><strong><em>munmap()</em></strong>则根据地址信息进行VMA映射的删除</li></ul><p><img src="/images/posts/20211113/mmap.png" alt="mmap and munmap" title="mmap and munmap"></p><p>内核实现mmap的核心函数是<strong><em><a href="https://elixir.bootlin.com/linux/latest/source/mm/mmap.c#L1404" target="_blank" rel="noopener">do_mmap()</a></em></strong>：</p><figure class="highlight c"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br><span class="line">21</span><br><span class="line">22</span><br><span class="line">23</span><br><span class="line">24</span><br><span class="line">25</span><br><span class="line">26</span><br><span class="line">27</span><br><span class="line">28</span><br><span class="line">29</span><br><span class="line">30</span><br><span class="line">31</span><br><span class="line">32</span><br><span class="line">33</span><br><span class="line">34</span><br><span class="line">35</span><br><span class="line">36</span><br><span class="line">37</span><br><span class="line">38</span><br><span class="line">39</span><br><span class="line">40</span><br><span class="line">41</span><br><span class="line">42</span><br><span class="line">43</span><br><span class="line">44</span><br><span class="line">45</span><br><span class="line">46</span><br><span class="line">47</span><br><span class="line">48</span><br></pre></td><td class="code"><pre><span class="line"><span class="function"><span class="keyword">unsigned</span> <span class="keyword">long</span> <span class="title">do_mmap</span><span class="params">(struct file *file, <span class="keyword">unsigned</span> <span class="keyword">long</span> addr,</span></span></span><br><span class="line"><span class="function"><span class="params"><span class="keyword">unsigned</span> <span class="keyword">long</span> len, <span class="keyword">unsigned</span> <span class="keyword">long</span> prot,</span></span></span><br><span class="line"><span class="function"><span class="params"><span class="keyword">unsigned</span> <span class="keyword">long</span> flags, <span class="keyword">unsigned</span> <span class="keyword">long</span> pgoff,</span></span></span><br><span class="line"><span class="function"><span class="params"><span class="keyword">unsigned</span> <span class="keyword">long</span> *populate, struct list_head *uf)</span></span></span><br><span class="line"><span class="function"></span>{</span><br><span class="line"><span class="class"><span class="keyword">struct</span> <span class="title">mm_struct</span> *<span class="title">mm</span> = <span class="title">current</span>-><span class="title">mm</span>;</span></span><br><span class="line">.....................</span><br><span class="line"><span class="keyword">if</span> ((prot & PROT_READ) && (current->personality & READ_IMPLIES_EXEC))</span><br><span class="line"><span class="keyword">if</span> (!(file && path_noexec(&file->f_path)))</span><br><span class="line">prot |= PROT_EXEC;</span><br><span class="line"></span><br><span class="line"><span class="comment">/* force arch specific MAP_FIXED handling in get_unmapped_area */</span></span><br><span class="line"><span class="keyword">if</span> (flags & MAP_FIXED_NOREPLACE)</span><br><span class="line">flags |= MAP_FIXED;</span><br><span class="line"></span><br><span class="line"><span class="keyword">if</span> (!(flags & MAP_FIXED))</span><br><span class="line">addr = round_hint_to_min(addr);</span><br><span class="line"></span><br><span class="line"><span class="comment">/* Careful about overflows.. */</span></span><br><span class="line">len = PAGE_ALIGN(len);</span><br><span class="line"><span class="keyword">if</span> (!len)</span><br><span class="line"><span class="keyword">return</span> -ENOMEM;</span><br><span class="line"></span><br><span class="line"><span class="comment">/* offset overflow? */</span></span><br><span class="line"><span class="keyword">if</span> ((pgoff + (len >> PAGE_SHIFT)) < pgoff)</span><br><span class="line"><span class="keyword">return</span> -EOVERFLOW;</span><br><span class="line"></span><br><span class="line"><span class="comment">/* Too many mappings? */</span></span><br><span class="line"><span class="keyword">if</span> (mm->map_count > sysctl_max_map_count)</span><br><span class="line"><span class="keyword">return</span> -ENOMEM;</span><br><span class="line">.........</span><br><span class="line"><span class="keyword">if</span> (flags & MAP_NORESERVE) {</span><br><span class="line"><span class="comment">/* We honor MAP_NORESERVE if allowed to overcommit */</span></span><br><span class="line"><span class="keyword">if</span> (sysctl_overcommit_memory != OVERCOMMIT_NEVER)</span><br><span class="line">vm_flags |= VM_NORESERVE;</span><br><span class="line"></span><br><span class="line"><span class="comment">/* hugetlb applies strict overcommit unless MAP_NORESERVE */</span></span><br><span class="line"><span class="keyword">if</span> (file && is_file_hugepages(file))</span><br><span class="line">vm_flags |= VM_NORESERVE;</span><br><span class="line">}</span><br><span class="line"></span><br><span class="line">addr = mmap_region(file, addr, len, vm_flags, pgoff, uf);</span><br><span class="line"><span class="keyword">if</span> (!IS_ERR_VALUE(addr) &&</span><br><span class="line"> ((vm_flags & VM_LOCKED) ||</span><br><span class="line"> (flags & (MAP_POPULATE | MAP_NONBLOCK)) == MAP_POPULATE))</span><br><span class="line">*populate = len;</span><br><span class="line"><span class="keyword">return</span> addr;</span><br><span class="line">}</span><br></pre></td></tr></table></figure><p>该函数主要就是根据输入参数做一系列检查，并根据参数配置vm_flags，最终传入<strong><em>mmap_region()</em></strong>函数开始创建映射。</p><figure class="highlight c"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br><span class="line">21</span><br><span class="line">22</span><br><span class="line">23</span><br><span class="line">24</span><br><span class="line">25</span><br><span class="line">26</span><br><span class="line">27</span><br><span class="line">28</span><br><span class="line">29</span><br><span class="line">30</span><br><span class="line">31</span><br><span class="line">32</span><br><span class="line">33</span><br><span class="line">34</span><br><span class="line">35</span><br><span class="line">36</span><br><span class="line">37</span><br><span class="line">38</span><br><span class="line">39</span><br><span class="line">40</span><br><span class="line">41</span><br><span class="line">42</span><br><span class="line">43</span><br><span class="line">44</span><br><span class="line">45</span><br><span class="line">46</span><br><span class="line">47</span><br><span class="line">48</span><br><span class="line">49</span><br><span class="line">50</span><br><span class="line">51</span><br><span class="line">52</span><br><span class="line">53</span><br><span class="line">54</span><br><span class="line">55</span><br><span class="line">56</span><br><span class="line">57</span><br><span class="line">58</span><br><span class="line">59</span><br><span class="line">60</span><br><span class="line">61</span><br><span class="line">62</span><br><span class="line">63</span><br><span class="line">64</span><br><span class="line">65</span><br><span class="line">66</span><br><span class="line">67</span><br><span class="line">68</span><br><span class="line">69</span><br><span class="line">70</span><br><span class="line">71</span><br><span class="line">72</span><br><span class="line">73</span><br><span class="line">74</span><br><span class="line">75</span><br><span class="line">76</span><br><span class="line">77</span><br><span class="line">78</span><br><span class="line">79</span><br><span class="line">80</span><br><span class="line">81</span><br><span class="line">82</span><br><span class="line">83</span><br><span class="line">84</span><br><span class="line">85</span><br><span class="line">86</span><br><span class="line">87</span><br><span class="line">88</span><br><span class="line">89</span><br><span class="line">90</span><br><span class="line">91</span><br><span class="line">92</span><br><span class="line">93</span><br><span class="line">94</span><br><span class="line">95</span><br><span class="line">96</span><br><span class="line">97</span><br><span class="line">98</span><br><span class="line">99</span><br><span class="line">100</span><br><span class="line">101</span><br><span class="line">102</span><br><span class="line">103</span><br><span class="line">104</span><br><span class="line">105</span><br><span class="line">106</span><br><span class="line">107</span><br><span class="line">108</span><br><span class="line">109</span><br><span class="line">110</span><br><span class="line">111</span><br><span class="line">112</span><br><span class="line">113</span><br><span class="line">114</span><br><span class="line">115</span><br><span class="line">116</span><br><span class="line">117</span><br><span class="line">118</span><br><span class="line">119</span><br><span class="line">120</span><br><span class="line">121</span><br><span class="line">122</span><br><span class="line">123</span><br><span class="line">124</span><br><span class="line">125</span><br><span class="line">126</span><br><span class="line">127</span><br><span class="line">128</span><br><span class="line">129</span><br><span class="line">130</span><br><span class="line">131</span><br><span class="line">132</span><br><span class="line">133</span><br><span class="line">134</span><br><span class="line">135</span><br><span class="line">136</span><br><span class="line">137</span><br><span class="line">138</span><br><span class="line">139</span><br></pre></td><td class="code"><pre><span class="line"><span class="function"><span class="keyword">unsigned</span> <span class="keyword">long</span> <span class="title">mmap_region</span><span class="params">(struct file *file, <span class="keyword">unsigned</span> <span class="keyword">long</span> addr,</span></span></span><br><span class="line"><span class="function"><span class="params"><span class="keyword">unsigned</span> <span class="keyword">long</span> len, <span class="keyword">vm_flags_t</span> vm_flags, <span class="keyword">unsigned</span> <span class="keyword">long</span> pgoff,</span></span></span><br><span class="line"><span class="function"><span class="params">struct list_head *uf)</span></span></span><br><span class="line"><span class="function"></span>{</span><br><span class="line"><span class="class"><span class="keyword">struct</span> <span class="title">mm_struct</span> *<span class="title">mm</span> = <span class="title">current</span>-><span class="title">mm</span>;</span></span><br><span class="line"><span class="class"><span class="keyword">struct</span> <span class="title">vm_area_struct</span> *<span class="title">vma</span>, *<span class="title">prev</span>, *<span class="title">merge</span>;</span></span><br><span class="line"><span class="keyword">int</span> error;</span><br><span class="line"><span class="class"><span class="keyword">struct</span> <span class="title">rb_node</span> **<span class="title">rb_link</span>, *<span class="title">rb_parent</span>;</span></span><br><span class="line"><span class="keyword">unsigned</span> <span class="keyword">long</span> charged = <span class="number">0</span>;</span><br><span class="line"></span><br><span class="line"><span class="comment">/* Check against address space limit. */</span></span><br><span class="line"><span class="keyword">if</span> (!may_expand_vm(mm, vm_flags, len >> PAGE_SHIFT)) {</span><br><span class="line"><span class="keyword">unsigned</span> <span class="keyword">long</span> nr_pages;</span><br><span class="line">nr_pages = count_vma_pages_range(mm, addr, addr + len);</span><br><span class="line"></span><br><span class="line"><span class="keyword">if</span> (!may_expand_vm(mm, vm_flags,</span><br><span class="line">(len >> PAGE_SHIFT) - nr_pages))</span><br><span class="line"><span class="keyword">return</span> -ENOMEM;</span><br><span class="line">}</span><br><span class="line"></span><br><span class="line"><span class="comment">/* Clear old maps, set up prev, rb_link, rb_parent, and uf */</span></span><br><span class="line"><span class="keyword">if</span> (munmap_vma_range(mm, addr, len, &prev, &rb_link, &rb_parent, uf))</span><br><span class="line"><span class="keyword">return</span> -ENOMEM;</span><br><span class="line"><span class="comment">/*</span></span><br><span class="line"><span class="comment"> * Private writable mapping: check memory availability</span></span><br><span class="line"><span class="comment"> */</span></span><br><span class="line"><span class="keyword">if</span> (accountable_mapping(file, vm_flags)) {</span><br><span class="line">charged = len >> PAGE_SHIFT;</span><br><span class="line"><span class="keyword">if</span> (security_vm_enough_memory_mm(mm, charged))</span><br><span class="line"><span class="keyword">return</span> -ENOMEM;</span><br><span class="line">vm_flags |= VM_ACCOUNT;</span><br><span class="line">}</span><br><span class="line"></span><br><span class="line">vma = vma_merge(mm, prev, addr, addr + len, vm_flags,</span><br><span class="line"><span class="literal">NULL</span>, file, pgoff, <span class="literal">NULL</span>, NULL_VM_UFFD_CTX);</span><br><span class="line"><span class="keyword">if</span> (vma)</span><br><span class="line"><span class="keyword">goto</span> out;</span><br><span class="line"></span><br><span class="line">vma = vm_area_alloc(mm);</span><br><span class="line"><span class="keyword">if</span> (!vma) {</span><br><span class="line">error = -ENOMEM;</span><br><span class="line"><span class="keyword">goto</span> unacct_error;</span><br><span class="line">}</span><br><span class="line"></span><br><span class="line">vma->vm_start = addr;</span><br><span class="line">vma->vm_end = addr + len;</span><br><span class="line">vma->vm_flags = vm_flags;</span><br><span class="line">vma->vm_page_prot = vm_get_page_prot(vm_flags);</span><br><span class="line">vma->vm_pgoff = pgoff;</span><br><span class="line"></span><br><span class="line"><span class="keyword">if</span> (file) {</span><br><span class="line"><span class="keyword">if</span> (vm_flags & VM_SHARED) {</span><br><span class="line">error = mapping_map_writable(file->f_mapping);</span><br><span class="line"><span class="keyword">if</span> (error)</span><br><span class="line"><span class="keyword">goto</span> free_vma;</span><br><span class="line">}</span><br><span class="line"></span><br><span class="line">vma->vm_file = get_file(file);</span><br><span class="line">error = call_mmap(file, vma);</span><br><span class="line"><span class="keyword">if</span> (error)</span><br><span class="line"><span class="keyword">goto</span> unmap_and_free_vma;</span><br><span class="line"></span><br><span class="line">WARN_ON_ONCE(addr != vma->vm_start);</span><br><span class="line"></span><br><span class="line">addr = vma->vm_start;</span><br><span class="line"></span><br><span class="line"><span class="comment">/* If vm_flags changed after call_mmap(), we should try merge vma again</span></span><br><span class="line"><span class="comment"> * as we may succeed this time.</span></span><br><span class="line"><span class="comment"> */</span></span><br><span class="line"><span class="keyword">if</span> (unlikely(vm_flags != vma->vm_flags && prev)) {</span><br><span class="line">merge = vma_merge(mm, prev, vma->vm_start, vma->vm_end, vma->vm_flags,</span><br><span class="line"><span class="literal">NULL</span>, vma->vm_file, vma->vm_pgoff, <span class="literal">NULL</span>, NULL_VM_UFFD_CTX);</span><br><span class="line"><span class="keyword">if</span> (merge) {</span><br><span class="line"><span class="comment">/* ->mmap() can change vma->vm_file and fput the original file. So</span></span><br><span class="line"><span class="comment"> * fput the vma->vm_file here or we would add an extra fput for file</span></span><br><span class="line"><span class="comment"> * and cause general protection fault ultimately.</span></span><br><span class="line"><span class="comment"> */</span></span><br><span class="line">fput(vma->vm_file);</span><br><span class="line">vm_area_free(vma);</span><br><span class="line">vma = merge;</span><br><span class="line"><span class="comment">/* Update vm_flags to pick up the change. */</span></span><br><span class="line">vm_flags = vma->vm_flags;</span><br><span class="line"><span class="keyword">goto</span> unmap_writable;</span><br><span class="line">}</span><br><span class="line">}</span><br><span class="line"></span><br><span class="line">vm_flags = vma->vm_flags;</span><br><span class="line">} <span class="keyword">else</span> <span class="keyword">if</span> (vm_flags & VM_SHARED) {</span><br><span class="line">error = shmem_zero_setup(vma);</span><br><span class="line"><span class="keyword">if</span> (error)</span><br><span class="line"><span class="keyword">goto</span> free_vma;</span><br><span class="line">} <span class="keyword">else</span> {</span><br><span class="line">vma_set_anonymous(vma);</span><br><span class="line">}</span><br><span class="line"></span><br><span class="line"><span class="comment">/* Allow architectures to sanity-check the vm_flags */</span></span><br><span class="line"><span class="keyword">if</span> (!arch_validate_flags(vma->vm_flags)) {</span><br><span class="line">error = -EINVAL;</span><br><span class="line"><span class="keyword">if</span> (file)</span><br><span class="line"><span class="keyword">goto</span> unmap_and_free_vma;</span><br><span class="line"><span class="keyword">else</span></span><br><span class="line"><span class="keyword">goto</span> free_vma;</span><br><span class="line">}</span><br><span class="line"></span><br><span class="line">vma_link(mm, vma, prev, rb_link, rb_parent);</span><br><span class="line"><span class="comment">/* Once vma denies write, undo our temporary denial count */</span></span><br><span class="line">unmap_writable:</span><br><span class="line"><span class="keyword">if</span> (file && vm_flags & VM_SHARED)</span><br><span class="line">mapping_unmap_writable(file->f_mapping);</span><br><span class="line">file = vma->vm_file;</span><br><span class="line">out:</span><br><span class="line">perf_event_mmap(vma);</span><br><span class="line"></span><br><span class="line">vm_stat_account(mm, vm_flags, len >> PAGE_SHIFT);</span><br><span class="line"><span class="keyword">if</span> (vm_flags & VM_LOCKED) {</span><br><span class="line"><span class="keyword">if</span> ((vm_flags & VM_SPECIAL) || vma_is_dax(vma) ||</span><br><span class="line">is_vm_hugetlb_page(vma) ||</span><br><span class="line">vma == get_gate_vma(current->mm))</span><br><span class="line">vma->vm_flags &= VM_LOCKED_CLEAR_MASK;</span><br><span class="line"><span class="keyword">else</span></span><br><span class="line">mm->locked_vm += (len >> PAGE_SHIFT);</span><br><span class="line">}</span><br><span class="line"></span><br><span class="line"><span class="keyword">if</span> (file)</span><br><span class="line">uprobe_mmap(vma);</span><br><span class="line"></span><br><span class="line"><span class="comment">/*</span></span><br><span class="line"><span class="comment"> * New (or expanded) vma always get soft dirty status.</span></span><br><span class="line"><span class="comment"> * Otherwise user-space soft-dirty page tracker won't</span></span><br><span class="line"><span class="comment"> * be able to distinguish situation when vma area unmapped,</span></span><br><span class="line"><span class="comment"> * then new mapped in-place (which must be aimed as</span></span><br><span class="line"><span class="comment"> * a completely new data area).</span></span><br><span class="line"><span class="comment"> */</span></span><br><span class="line">vma->vm_flags |= VM_SOFTDIRTY;</span><br><span class="line">vma_set_page_prot(vma);</span><br><span class="line"></span><br><span class="line"><span class="keyword">return</span> addr;</span><br><span class="line">........</span><br><span class="line">}</span><br></pre></td></tr></table></figure><p><strong><em>mmap_region()</em></strong>函数的实现也比较简单，这里不做过多解释。值得注意的是：如果传入的文件为空，则表示创建匿名映射。若连共享标记VM_SHARED也未指定，则与使用malloc()分配内存相同，仅为对应虚拟地址创建内存映射。</p><p><strong><em>munmap()</em></strong>的内核实现仅为移除对应VMA映射，本文也不再做分析。</p><h2 id="反向映射"><a href="#反向映射" class="headerlink" title="反向映射"></a>反向映射</h2><p><a href="https://lwn.net/Articles/23732/" target="_blank" rel="noopener">反向映射</a>的作用是给定物理页面，找到与其对应的所有进程的VMA。为什么会有这样的查找呢？这是因为所有进程的虚拟内存总大小往往远大于物理内存，为了支撑Linux系统的有效运作，内核在管理内存时，会将暂时不用的物理内存页换出到磁盘上，在有需要时再换入到内存中。</p><p>这种情况下，如何确定该物理内存有哪些进程正在使用？这便需要反向映射。</p><p>系统中内存页很多，在管理反向映射时，即使引入很小的数据结构，也会带来很大的额外内存开销。同时，因为反向映射使用比较频繁，也需要最优化查找效率，避免成为系统瓶颈。</p><p>回顾一下<strong><em>struct page</em></strong>，为了节省管理开销，其定义了很多联合体。其中与逆向映射有关的储存在mapping，_mapcount，index等成员中。</p><figure class="highlight c"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br><span class="line">21</span><br><span class="line">22</span><br><span class="line">23</span><br><span class="line">24</span><br><span class="line">25</span><br><span class="line">26</span><br><span class="line">27</span><br><span class="line">28</span><br><span class="line">29</span><br><span class="line">30</span><br><span class="line">31</span><br><span class="line">32</span><br><span class="line">33</span><br><span class="line">34</span><br><span class="line">35</span><br><span class="line">36</span><br><span class="line">37</span><br></pre></td><td class="code"><pre><span class="line"><span class="class"><span class="keyword">struct</span> <span class="title">page</span> {</span></span><br><span class="line"><span class="keyword">union</span> {</span><br><span class="line"><span class="class"><span class="keyword">struct</span> {</span><span class="comment">/* Page cache and anonymous pages */</span></span><br><span class="line"><span class="comment">/**</span></span><br><span class="line"><span class="comment"> * @lru: Pageout list, eg. active_list protected by</span></span><br><span class="line"><span class="comment"> * lruvec->lru_lock. Sometimes used as a generic list</span></span><br><span class="line"><span class="comment"> * by the page owner.</span></span><br><span class="line"><span class="comment"> */</span></span><br><span class="line"><span class="class"><span class="keyword">struct</span> <span class="title">list_head</span> <span class="title">lru</span>;</span></span><br><span class="line"><span class="comment">/* See page-flags.h for PAGE_MAPPING_FLAGS */</span></span><br><span class="line"><span class="class"><span class="keyword">struct</span> <span class="title">address_space</span> *<span class="title">mapping</span>;</span></span><br><span class="line"><span class="keyword">pgoff_t</span> index;<span class="comment">/* Our offset within mapping. */</span></span><br><span class="line"><span class="comment">/**</span></span><br><span class="line"><span class="comment"> * @private: Mapping-private opaque data.</span></span><br><span class="line"><span class="comment"> * Usually used for buffer_heads if PagePrivate.</span></span><br><span class="line"><span class="comment"> * Used for swp_entry_t if PageSwapCache.</span></span><br><span class="line"><span class="comment"> * Indicates order in the buddy system if PageBuddy.</span></span><br><span class="line"><span class="comment"> */</span></span><br><span class="line"><span class="keyword">unsigned</span> <span class="keyword">long</span> <span class="keyword">private</span>;</span><br><span class="line">};</span><br><span class="line">.............</span><br><span class="line">};</span><br><span class="line"></span><br><span class="line"><span class="keyword">union</span> {<span class="comment">/* This union is 4 bytes in size. */</span></span><br><span class="line"><span class="comment">/*</span></span><br><span class="line"><span class="comment"> * If the page can be mapped to userspace, encodes the number</span></span><br><span class="line"><span class="comment"> * of times this page is referenced by a page table.</span></span><br><span class="line"><span class="comment"> */</span></span><br><span class="line"><span class="keyword">atomic_t</span> _mapcount;</span><br><span class="line"></span><br><span class="line"><span class="keyword">unsigned</span> <span class="keyword">int</span> page_type;</span><br><span class="line"></span><br><span class="line"><span class="keyword">unsigned</span> <span class="keyword">int</span> active;<span class="comment">/* SLAB */</span></span><br><span class="line"><span class="keyword">int</span> units;<span class="comment">/* SLOB */</span></span><br><span class="line">};</span><br><span class="line">.............</span><br><span class="line">} _struct_page_alignment;</span><br></pre></td></tr></table></figure><p><a href="https://lwn.net/Articles/23732/" target="_blank" rel="noopener">内核文档</a>对这种映射有个直观的描述如下图:</p><p><img src="https://static.lwn.net/images/ns/ormap.png" alt="反向映射图示" title="反向映射图示"></p><p>简单一点讲：物理页结构体<strong><em>struct page</em></strong>使用<strong><em>mapping</em></strong>成员查找所有该页对应的VMA，从而找到所有正在使用该物理页的虚拟页。</p><p>mapping成员查找VMA的方法并非如上图那样容易理解。实际需要考虑很多情况，因此内核设计了如下数据结构：</p><figure class="highlight c"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br><span class="line">21</span><br><span class="line">22</span><br><span class="line">23</span><br><span class="line">24</span><br><span class="line">25</span><br><span class="line">26</span><br><span class="line">27</span><br><span class="line">28</span><br><span class="line">29</span><br><span class="line">30</span><br><span class="line">31</span><br><span class="line">32</span><br><span class="line">33</span><br><span class="line">34</span><br><span class="line">35</span><br><span class="line">36</span><br><span class="line">37</span><br><span class="line">38</span><br><span class="line">39</span><br><span class="line">40</span><br><span class="line">41</span><br><span class="line">42</span><br><span class="line">43</span><br><span class="line">44</span><br><span class="line">45</span><br><span class="line">46</span><br></pre></td><td class="code"><pre><span class="line"><span class="comment">/*</span></span><br><span class="line"><span class="comment"> * The anon_vma heads a list of private "related" vmas, to scan if</span></span><br><span class="line"><span class="comment"> * an anonymous page pointing to this anon_vma needs to be unmapped:</span></span><br><span class="line"><span class="comment"> * the vmas on the list will be related by forking, or by splitting.</span></span><br><span class="line"><span class="comment"> *</span></span><br><span class="line"><span class="comment"> * Since vmas come and go as they are split and merged (particularly</span></span><br><span class="line"><span class="comment"> * in mprotect), the mapping field of an anonymous page cannot point</span></span><br><span class="line"><span class="comment"> * directly to a vma: instead it points to an anon_vma, on whose list</span></span><br><span class="line"><span class="comment"> * the related vmas can be easily linked or unlinked.</span></span><br><span class="line"><span class="comment"> *</span></span><br><span class="line"><span class="comment"> * After unlinking the last vma on the list, we must garbage collect</span></span><br><span class="line"><span class="comment"> * the anon_vma object itself: we're guaranteed no page can be</span></span><br><span class="line"><span class="comment"> * pointing to this anon_vma once its vma list is empty.</span></span><br><span class="line"><span class="comment"> */</span></span><br><span class="line"><span class="class"><span class="keyword">struct</span> <span class="title">anon_vma</span> {</span></span><br><span class="line"><span class="class"><span class="keyword">struct</span> <span class="title">anon_vma</span> *<span class="title">root</span>;</span><span class="comment">/* Root of this anon_vma tree */</span></span><br><span class="line"><span class="class"><span class="keyword">struct</span> <span class="title">rw_semaphore</span> <span class="title">rwsem</span>;</span><span class="comment">/* W: modification, R: walking the list */</span></span><br><span class="line"><span class="keyword">atomic_t</span> refcount;</span><br><span class="line"><span class="keyword">unsigned</span> degree;</span><br><span class="line"><span class="class"><span class="keyword">struct</span> <span class="title">anon_vma</span> *<span class="title">parent</span>;</span><span class="comment">/* Parent of this anon_vma */</span></span><br><span class="line"><span class="class"><span class="keyword">struct</span> <span class="title">rb_root_cached</span> <span class="title">rb_root</span>;</span></span><br><span class="line">};</span><br><span class="line"></span><br><span class="line"><span class="comment">/*</span></span><br><span class="line"><span class="comment"> * The copy-on-write semantics of fork mean that an anon_vma</span></span><br><span class="line"><span class="comment"> * can become associated with multiple processes. Furthermore,</span></span><br><span class="line"><span class="comment"> * each child process will have its own anon_vma, where new</span></span><br><span class="line"><span class="comment"> * pages for that process are instantiated.</span></span><br><span class="line"><span class="comment"> *</span></span><br><span class="line"><span class="comment"> * This structure allows us to find the anon_vmas associated</span></span><br><span class="line"><span class="comment"> * with a VMA, or the VMAs associated with an anon_vma.</span></span><br><span class="line"><span class="comment"> * The "same_vma" list contains the anon_vma_chains linking</span></span><br><span class="line"><span class="comment"> * all the anon_vmas associated with this VMA.</span></span><br><span class="line"><span class="comment"> * The "rb" field indexes on an interval tree the anon_vma_chains</span></span><br><span class="line"><span class="comment"> * which link all the VMAs associated with this anon_vma.</span></span><br><span class="line"><span class="comment"> */</span></span><br><span class="line"><span class="class"><span class="keyword">struct</span> <span class="title">anon_vma_chain</span> {</span></span><br><span class="line"><span class="class"><span class="keyword">struct</span> <span class="title">vm_area_struct</span> *<span class="title">vma</span>;</span></span><br><span class="line"><span class="class"><span class="keyword">struct</span> <span class="title">anon_vma</span> *<span class="title">anon_vma</span>;</span></span><br><span class="line"><span class="class"><span class="keyword">struct</span> <span class="title">list_head</span> <span class="title">same_vma</span>;</span> <span class="comment">/* locked by mmap_lock & page_table_lock */</span></span><br><span class="line"><span class="class"><span class="keyword">struct</span> <span class="title">rb_node</span> <span class="title">rb</span>;</span><span class="comment">/* locked by anon_vma->rwsem */</span></span><br><span class="line"><span class="keyword">unsigned</span> <span class="keyword">long</span> rb_subtree_last;</span><br><span class="line"><span class="meta">#<span class="meta-keyword">ifdef</span> CONFIG_DEBUG_VM_RB</span></span><br><span class="line"><span class="keyword">unsigned</span> <span class="keyword">long</span> cached_vma_start, cached_vma_last;</span><br><span class="line"><span class="meta">#<span class="meta-keyword">endif</span></span></span><br><span class="line">};</span><br></pre></td></tr></table></figure><p>事实上，数据结构定义有描述为什么需要这样的数据结构而不是直接由mapping指向vma_area_struct。即：vm_area_struct可能会被合并、拆分等。</p><p>下图描述了当fork一个新进程时，反向映射相关字段的变化状况。</p><ul><li>Fork新进程会将每个VM area(vm_area_struct)进行复制</li><li>将每一个VM area的anon_vma_chain链表的anon_vma_chain进行复制，并与父进程的anon_vma关联</li><li>对每个VM area，创建新的anon_vma和anon_vma_chain，与父进程的anon_vma关联</li></ul><p>这样的关联建立起来后，通过物理页结构体struct page就可以查找到所有的关联VM area。</p><p><img src="/images/posts/20211113/mmap2.png" alt="反向映射数据结构关系" title="反向映射数据结构关系"></p><h2 id="结语"><a href="#结语" class="headerlink" title="结语"></a>结语</h2><p>本文概要介绍了mmap和反向映射的原理。</p><ul><li>mmap主要用于用户空间态进程映射一段虚拟地址，用以共享、分配内存或者使用访问内存的方式来访问文件节点。</li><li>反向映射用于内核查找一个物理页面对应的所有虚拟地址，以便系统换页时使用。</li></ul><p>内存管理系统内容纷繁，也是内核工作者集体智慧的结晶，笔者在理解时不免有遗失或者偏差之处。如您有问题或者建议，请留言提出讨论。</p>]]></content>
<summary type="html">
<blockquote>
<p>这是&lt;Linux内核内存管理&gt;系列的第八篇:</p>
<blockquote>
<p><a href="/p/61304bbb.html">第一篇</a>为内核内存管理过程知识点的的简单梳理</p>
<p><a href="/p/1dc6adc2.html">第二篇</a>介绍了内核的数据结构</p>
<p><a href="/p/8f5117c5.html">第三篇</a>介绍了从内核第一行代码加载到跳转到C代码前的内存处理。</p>
<p><a href="/p/94b22c5d.html">第四篇</a>概览了初始化C代码中的内存处理</p>
<p><a href="/p/20210923linuxmm4.html">第五篇（上）</a>和<a href="/p/5347665a.html">第五篇（下）</a>介绍了Memblock和伙伴系统分配器</p>
<p><a href="/p/75308136.html">第六篇</a>介绍了内存检测工具KFence工作原理</p>
<p><a href="/p/e84c7fac.html">第七篇</a>介绍了进程内存分配malloc的原理</p>
</blockquote>
</blockquote>
</summary>
<category term="Linux" scheme="http://www.l2h.site/categories/Linux/"/>
<category term="Linux内存管理" scheme="http://www.l2h.site/categories/Linux/Linux%E5%86%85%E5%AD%98%E7%AE%A1%E7%90%86/"/>
<category term="Linux" scheme="http://www.l2h.site/tags/Linux/"/>
<category term="内存管理" scheme="http://www.l2h.site/tags/%E5%86%85%E5%AD%98%E7%AE%A1%E7%90%86/"/>
<category term="进程内存" scheme="http://www.l2h.site/tags/%E8%BF%9B%E7%A8%8B%E5%86%85%E5%AD%98/"/>
</entry>
<entry>
<title>Linux内核内存管理 - 进程内存</title>
<link href="http://www.l2h.site/p/e84c7fac.html"/>
<id>http://www.l2h.site/p/e84c7fac.html</id>
<published>2021-11-03T13:45:55.000Z</published>
<updated>2023-06-01T13:14:04.483Z</updated>
<content type="html"><![CDATA[<blockquote><p>这是<Linux内核内存管理>系列的第七篇:</p><blockquote><p><a href="/p/61304bbb.html">第一篇</a>为内核内存管理过程知识点的的简单梳理</p><p><a href="/p/1dc6adc2.html">第二篇</a>介绍了内核的数据结构</p><p><a href="/p/8f5117c5.html">第三篇</a>介绍了从内核第一行代码加载到跳转到C代码前的内存处理。</p><p><a href="/p/94b22c5d.html">第四篇</a>概览了初始化C代码中的内存处理</p><p><a href="/p/20210923linuxmm4.html">第五篇（上）</a>和<a href="/p/5347665a.html">第五篇（下）</a>介绍了Memblock和伙伴系统分配器</p><p><a href>第六篇</a>介绍了内存检测工具KFence工作原理</p></blockquote></blockquote><a id="more"></a><h2 id="前言"><a href="#前言" class="headerlink" title="前言"></a>前言</h2><p><strong><em><a href="https://man7.org/linux/man-pages/man3/realloc.3.html" target="_blank" rel="noopener">malloc()</a></em></strong> 大概是在Linux平台上用户空间态编程，最常用的内存分配函数。大家可能会想，</p><ul><li><p>这个函数是如何拿到内存的？内核如何为它做的映射？</p></li><li><p>另外，一个可执行程序有自己的代码和静态数据，内核如何将这个可执行程序代码加载到内存中执行？其对应的静态变量，全局变量等所需内存又是如何分配的？</p></li></ul><p>以上问题是开发用户空间态程序时，容易被忽略的、甚至完全不会被注意到的问题。因为这些都是由程序所链接的C库和底层内核实现的，程序开发者往往无需在意这些细节。</p><p>一般情况下，这并不影响大家写出一个像样的程序。但是当面临一些疑难问题时，仅有如何使用C函数的知识，是无法胜任和处理的。</p><p>本文意在从以下几点剖析内核处理进程内存有关的过程:</p><ul><li>进程创建过程的段映射</li><li>进程内存分配过程的堆映射</li></ul><h2 id="架构"><a href="#架构" class="headerlink" title="架构"></a>架构</h2><p>下图简要描述了Linux内存管理架构：</p><ul><li>用户空间态程序使用GLIBC来创建进程或管理内存</li></ul><blockquote><p>这里GLIBC并不是唯一选项，也有许多其替代。例如：<a href="http://musl.libc.org/" target="_blank" rel="noopener">musl Libc</a>, 嵌入式设备常用的<a href="https://www.uclibc.org/about.html" target="_blank" rel="noopener">uClibc</a>等。</p></blockquote><ul><li>内核态和用户空间态处理内存分配和进程管理的接口是系统调用。</li></ul><blockquote><p>当然除了系统调用之外，内核和用户空间态通信方式还有<a href="https://en.wikipedia.org/wiki/Netlink" target="_blank" rel="noopener">Netlink</a>等。</p></blockquote><ul><li><strong><em>malloc()</em></strong>、<strong><em>free()</em></strong>等函数并不会直接跟系统要内存，反而会在必要时才会使用系统调用向内核申请内存。</li><li>启动程序时，使用系统调用<strong><em>fork</em></strong>或者<strong><em>clone</em></strong>创建进程，调用<strong><em>execv</em></strong>加载ELF，为进程创建必要的数据结构、分配必要的内存和页表。</li><li>内核内存管理如系列文章所述，有按页分配的伙伴系统、更小级别的分配器SLAB。</li><li>系统调用通过虚拟地址映射管理VMA来为进程分配和管理内存。</li><li>内核根据需要，会将暂时不需要使用的进程内存换出。而当有使用需求时，再将对应内存换入。这就需要内核的缺页中断处理程序及换页机制来保障。</li></ul><p><img src="/images/posts/20211106/arch.png" alt="Linux内存管理架构" title="Linux内存管理架构"></p><h2 id="数据结构"><a href="#数据结构" class="headerlink" title="数据结构"></a>数据结构</h2><p>内核为每个进程分配了一个数据结构<strong><em>task_struct</em></strong>，而其中管理内存的部分是<strong><em>mm_struct</em></strong>:</p><figure class="highlight c"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br><span class="line">21</span><br><span class="line">22</span><br><span class="line">23</span><br><span class="line">24</span><br><span class="line">25</span><br></pre></td><td class="code"><pre><span class="line"><span class="class"><span class="keyword">struct</span> <span class="title">mm_struct</span> {</span></span><br><span class="line"><span class="class"><span class="keyword">struct</span> <span class="title">vm_area_struct</span> *<span class="title">mmap</span>;</span><span class="comment">/* list of VMAs */</span></span><br><span class="line"><span class="class"><span class="keyword">struct</span> <span class="title">rb_root</span> <span class="title">mm_rb</span>;</span></span><br><span class="line">....</span><br><span class="line"><span class="keyword">unsigned</span> <span class="keyword">long</span> mmap_base;<span class="comment">/* base of mmap area */</span></span><br><span class="line">...</span><br><span class="line"><span class="keyword">unsigned</span> <span class="keyword">long</span> task_size;<span class="comment">/* size of task vm space */</span></span><br><span class="line"><span class="keyword">unsigned</span> <span class="keyword">long</span> highest_vm_end;<span class="comment">/* highest vma end address */</span></span><br><span class="line"><span class="keyword">pgd_t</span> * pgd;</span><br><span class="line">...</span><br><span class="line"><span class="class"><span class="keyword">struct</span> <span class="title">list_head</span> <span class="title">mmlist</span>;</span></span><br><span class="line"><span class="keyword">unsigned</span> <span class="keyword">long</span> total_vm; <span class="comment">/* Total pages mapped */</span></span><br><span class="line"><span class="keyword">unsigned</span> <span class="keyword">long</span> locked_vm; <span class="comment">/* Pages that have PG_mlocked set */</span></span><br><span class="line"><span class="keyword">atomic64_t</span> pinned_vm; <span class="comment">/* Refcount permanently increased */</span></span><br><span class="line"><span class="keyword">unsigned</span> <span class="keyword">long</span> data_vm; <span class="comment">/* VM_WRITE & ~VM_SHARED & ~VM_STACK */</span></span><br><span class="line"><span class="keyword">unsigned</span> <span class="keyword">long</span> exec_vm; <span class="comment">/* VM_EXEC & ~VM_WRITE & ~VM_STACK */</span></span><br><span class="line"><span class="keyword">unsigned</span> <span class="keyword">long</span> stack_vm; <span class="comment">/* VM_STACK */</span></span><br><span class="line">...</span><br><span class="line"><span class="keyword">unsigned</span> <span class="keyword">long</span> start_code, end_code, start_data, end_data;</span><br><span class="line"><span class="keyword">unsigned</span> <span class="keyword">long</span> start_brk, brk, start_stack;</span><br><span class="line"><span class="keyword">unsigned</span> <span class="keyword">long</span> arg_start, arg_end, env_start, env_end;</span><br><span class="line">...</span><br><span class="line"><span class="class"><span class="keyword">struct</span> <span class="title">linux_binfmt</span> *<span class="title">binfmt</span>;</span></span><br><span class="line">...</span><br><span class="line">};</span><br></pre></td></tr></table></figure><p>其中所结构体栏位的意义标注如下图：</p><ul><li>mmap_base指向进程的MMAP空间</li><li>brk和start_brk分别指向进程堆的当前位置（若进程需要申请更大的堆，则从该位置开始分配）和起始地址。</li><li>start_code和end_code分别指向代码段的起始地址和结束地址。</li><li>start_data和end_data分别指向数据段的起始地址和结束地址。</li><li>start_stack指向栈的初始地址。</li><li>除此之外，还有为该进程参数、环境变量所分配的内存(arg_start/arg_end/env_start/env_end)。</li></ul><blockquote><p>以上地址皆为虚拟地址，是内核进程启动的过程中，由内核所初始化。</p></blockquote><ul><li>mmap和mm_rb下管理内核所以为该进程分配的虚拟内存，分别使用红黑树和链表管理。</li><li>pgd指向该进程的页目录</li></ul><p><img src="/images/posts/20211106/process_vma.png" alt="Linux进程内存管理" title="Linux进程内存管理"></p><h2 id="进程创建时内存管理"><a href="#进程创建时内存管理" class="headerlink" title="进程创建时内存管理"></a>进程创建时内存管理</h2><p>您可能会想知道，前一节所提及的那些段地址，内核是如何确定的？其实这跟<a href="https://en.wikipedia.org/wiki/Executable_and_Linkable_Format" target="_blank" rel="noopener">ELF格式</a>有关。</p><ul><li>所有的Linux进程都遵循ELF格式，在链接的过程中，链接器按照<a href="https://sourceware.org/binutils/docs/ld/Scripts.html" target="_blank" rel="noopener">Linker Script</a>的指定将程序打包成ELF。</li></ul><blockquote><p>如果没有明确指定，GCC会指定一个默认的Linker Script</p></blockquote><ul><li>内核创建进程时，会初始化<strong><em>mm_struct</em></strong>。</li><li>内核加载进程的过程中，按照ELF头部信息为该进程分配内存。</li></ul><p>一张图描述上述过程。</p><p><img src="/images/posts/20211106/Load_ELF.png" alt="Linux进程内存初始化" title="Linux进程内存初始化"></p><p>内核加载进程执行，也遵守ELF规范，在此期间为进程分配虚拟内存VMA。</p><h2 id="内存分配"><a href="#内存分配" class="headerlink" title="内存分配"></a>内存分配</h2><h3 id="用户空间态"><a href="#用户空间态" class="headerlink" title="用户空间态"></a>用户空间态</h3><p>使用musl Libc来对<strong><em>malloc()</em></strong>进行介绍。</p><blockquote><p>没有选择Glibc分析的原因，是因为没有搞懂<strong><em><a href="https://sourceware.org/systemtap/wiki/UserSpaceProbeImplementation" target="_blank" rel="noopener">Systemtap</a></em></strong>的原理。使用musl Libc分析不会影响理解。</p></blockquote><figure class="highlight c"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br><span class="line">21</span><br><span class="line">22</span><br><span class="line">23</span><br><span class="line">24</span><br><span class="line">25</span><br><span class="line">26</span><br><span class="line">27</span><br><span class="line">28</span><br><span class="line">29</span><br><span class="line">30</span><br><span class="line">31</span><br><span class="line">32</span><br><span class="line">33</span><br><span class="line">34</span><br><span class="line">35</span><br><span class="line">36</span><br><span class="line">37</span><br><span class="line">38</span><br><span class="line">39</span><br><span class="line">40</span><br><span class="line">41</span><br><span class="line">42</span><br><span class="line">43</span><br><span class="line">44</span><br><span class="line">45</span><br><span class="line">46</span><br><span class="line">47</span><br><span class="line">48</span><br><span class="line">49</span><br><span class="line">50</span><br><span class="line">51</span><br><span class="line">52</span><br><span class="line">53</span><br><span class="line">54</span><br><span class="line">55</span><br><span class="line">56</span><br><span class="line">57</span><br><span class="line">58</span><br><span class="line">59</span><br><span class="line">60</span><br><span class="line">61</span><br><span class="line">62</span><br><span class="line">63</span><br><span class="line">64</span><br><span class="line">65</span><br><span class="line">66</span><br><span class="line">67</span><br><span class="line">68</span><br><span class="line">69</span><br><span class="line">70</span><br><span class="line">71</span><br><span class="line">72</span><br><span class="line">73</span><br></pre></td><td class="code"><pre><span class="line"><span class="keyword">static</span> <span class="keyword">void</span> *__simple_malloc(<span class="keyword">size_t</span> n)</span><br><span class="line">{</span><br><span class="line"><span class="keyword">static</span> <span class="keyword">uintptr_t</span> brk, cur, <span class="built_in">end</span>;</span><br><span class="line"><span class="keyword">static</span> <span class="keyword">unsigned</span> mmap_step;</span><br><span class="line"><span class="keyword">size_t</span> align=<span class="number">1</span>;</span><br><span class="line"><span class="keyword">void</span> *p;</span><br><span class="line"></span><br><span class="line"><span class="keyword">if</span> (n > SIZE_MAX/<span class="number">2</span>) {</span><br><span class="line">errno = ENOMEM;</span><br><span class="line"><span class="keyword">return</span> <span class="number">0</span>;</span><br><span class="line">}</span><br><span class="line"></span><br><span class="line"><span class="keyword">if</span> (!n) n++;</span><br><span class="line"><span class="keyword">while</span> (align<n && align<ALIGN)</span><br><span class="line">align += align;</span><br><span class="line"></span><br><span class="line">LOCK(lock);</span><br><span class="line"></span><br><span class="line">cur += -cur & align<span class="number">-1</span>;</span><br><span class="line"></span><br><span class="line"><span class="keyword">if</span> (n > <span class="built_in">end</span>-cur) {</span><br><span class="line"><span class="keyword">size_t</span> req = n - (<span class="built_in">end</span>-cur) + PAGE_SIZE<span class="number">-1</span> & -PAGE_SIZE;</span><br><span class="line"></span><br><span class="line"><span class="keyword">if</span> (!cur) {</span><br><span class="line">brk = __syscall(SYS_brk, <span class="number">0</span>);</span><br><span class="line">brk += -brk & PAGE_SIZE<span class="number">-1</span>;</span><br><span class="line">cur = <span class="built_in">end</span> = brk;</span><br><span class="line">}</span><br><span class="line"></span><br><span class="line"><span class="keyword">if</span> (brk == <span class="built_in">end</span> && req < SIZE_MAX-brk</span><br><span class="line"> && !traverses_stack_p(brk, brk+req)</span><br><span class="line"> && __syscall(SYS_brk, brk+req)==brk+req) {</span><br><span class="line">brk = <span class="built_in">end</span> += req;</span><br><span class="line">} <span class="keyword">else</span> {</span><br><span class="line"><span class="keyword">int</span> new_area = <span class="number">0</span>;</span><br><span class="line">req = n + PAGE_SIZE<span class="number">-1</span> & -PAGE_SIZE;</span><br><span class="line"><span class="comment">/* Only make a new area rather than individual mmap</span></span><br><span class="line"><span class="comment"> * if wasted space would be over 1/8 of the map. */</span></span><br><span class="line"><span class="keyword">if</span> (req-n > req/<span class="number">8</span>) {</span><br><span class="line"><span class="comment">/* Geometric area size growth up to 64 pages,</span></span><br><span class="line"><span class="comment"> * bounding waste by 1/8 of the area. */</span></span><br><span class="line"><span class="keyword">size_t</span> <span class="built_in">min</span> = PAGE_SIZE<<(mmap_step/<span class="number">2</span>);</span><br><span class="line"><span class="keyword">if</span> (<span class="built_in">min</span>-n > <span class="built_in">end</span>-cur) {</span><br><span class="line"><span class="keyword">if</span> (req < <span class="built_in">min</span>) {</span><br><span class="line">req = <span class="built_in">min</span>;</span><br><span class="line"><span class="keyword">if</span> (mmap_step < <span class="number">12</span>)</span><br><span class="line">mmap_step++;</span><br><span class="line">}</span><br><span class="line">new_area = <span class="number">1</span>;</span><br><span class="line">}</span><br><span class="line">}</span><br><span class="line"><span class="keyword">void</span> *mem = __mmap(<span class="number">0</span>, req, PROT_READ|PROT_WRITE,</span><br><span class="line">MAP_PRIVATE|MAP_ANONYMOUS, <span class="number">-1</span>, <span class="number">0</span>);</span><br><span class="line"><span class="keyword">if</span> (mem == MAP_FAILED || !new_area) {</span><br><span class="line">UNLOCK(lock);</span><br><span class="line"><span class="keyword">return</span> mem==MAP_FAILED ? <span class="number">0</span> : mem;</span><br><span class="line">}</span><br><span class="line">cur = (<span class="keyword">uintptr_t</span>)mem;</span><br><span class="line"><span class="built_in">end</span> = cur + req;</span><br><span class="line">}</span><br><span class="line">}</span><br><span class="line"></span><br><span class="line">p = (<span class="keyword">void</span> *)cur;</span><br><span class="line">cur += n;</span><br><span class="line">UNLOCK(lock);</span><br><span class="line"><span class="keyword">return</span> p;</span><br><span class="line">}</span><br><span class="line">weak_alias(__simple_malloc, __libc_malloc_impl);</span><br><span class="line"></span><br><span class="line"><span class="keyword">void</span> *__libc_malloc(<span class="keyword">size_t</span> n)</span><br><span class="line">{</span><br><span class="line"><span class="keyword">return</span> __libc_malloc_impl(n);</span><br><span class="line">}</span><br></pre></td></tr></table></figure><p>这段代码比较容易理解,我们只关注其中<strong><em>__syscall(SYS_brk, ….)</em></strong>。它的作用就是使用brk这个系统调用向内核要内存。</p><blockquote><p> <strong><em>weak_alias</em></strong>的定义如下：</p></blockquote><figure class="highlight c"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><span class="line">#<span class="meta">#<span class="meta-keyword">define</span> weak_alias(old,new) __attribute__((__weak__, __alias__(old)))</span></span><br></pre></td></tr></table></figure><blockquote><p>其中<a href="https://gcc.gnu.org/onlinedocs/gcc-4.3.5/gcc/Function-Attributes.html" target="_blank" rel="noopener">Weak Alias</a>的意义即给<strong><em>old</em></strong> symbol设置一个别名<strong><em>new</em></strong>。</p></blockquote><h3 id="内核空间态"><a href="#内核空间态" class="headerlink" title="内核空间态"></a>内核空间态</h3><p>内核空间态处理brk系统调用的代码如下：</p><ul><li>首先做一些必要的检查，如检查所申请的堆大小是否超过系统的rlimit，出错则退出返回错误。解释如下：</li></ul><blockquote><p><strong>RLIMIT_DATA</strong></p><p>The maximum size of the process’s data segment (initialized data, uninitialized data, and heap). This limit affects calls to <strong><a href="https://linux.die.net/man/2/brk" target="_blank" rel="noopener">brk</a></strong>(2) and <strong><a href="https://linux.die.net/man/2/sbrk" target="_blank" rel="noopener">sbrk</a></strong>(2), which fail with the error <strong>ENOMEM</strong> upon encountering the soft limit of this resource.</p></blockquote><ul><li>如果当前brk大于所需brk，则将多出的部分从该进程的堆VM映射中移除并返回。</li><li>如果当前brk小于所需brk，则为其扩展堆的VM映射并返回。</li></ul><figure class="highlight c"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br><span class="line">21</span><br><span class="line">22</span><br><span class="line">23</span><br><span class="line">24</span><br><span class="line">25</span><br><span class="line">26</span><br><span class="line">27</span><br><span class="line">28</span><br><span class="line">29</span><br><span class="line">30</span><br><span class="line">31</span><br><span class="line">32</span><br><span class="line">33</span><br><span class="line">34</span><br><span class="line">35</span><br><span class="line">36</span><br><span class="line">37</span><br><span class="line">38</span><br><span class="line">39</span><br><span class="line">40</span><br><span class="line">41</span><br><span class="line">42</span><br><span class="line">43</span><br><span class="line">44</span><br><span class="line">45</span><br><span class="line">46</span><br><span class="line">47</span><br><span class="line">48</span><br><span class="line">49</span><br><span class="line">50</span><br><span class="line">51</span><br><span class="line">52</span><br><span class="line">53</span><br><span class="line">54</span><br><span class="line">55</span><br><span class="line">56</span><br><span class="line">57</span><br><span class="line">58</span><br><span class="line">59</span><br><span class="line">60</span><br><span class="line">61</span><br><span class="line">62</span><br><span class="line">63</span><br><span class="line">64</span><br><span class="line">65</span><br><span class="line">66</span><br><span class="line">67</span><br><span class="line">68</span><br><span class="line">69</span><br><span class="line">70</span><br><span class="line">71</span><br><span class="line">72</span><br><span class="line">73</span><br><span class="line">74</span><br><span class="line">75</span><br><span class="line">76</span><br><span class="line">77</span><br><span class="line">78</span><br><span class="line">79</span><br><span class="line">80</span><br><span class="line">81</span><br><span class="line">82</span><br><span class="line">83</span><br><span class="line">84</span><br><span class="line">85</span><br><span class="line">86</span><br><span class="line">87</span><br><span class="line">88</span><br><span class="line">89</span><br><span class="line">90</span><br><span class="line">91</span><br><span class="line">92</span><br><span class="line">93</span><br><span class="line">94</span><br><span class="line">95</span><br><span class="line">96</span><br></pre></td><td class="code"><pre><span class="line">SYSCALL_DEFINE1(brk, <span class="keyword">unsigned</span> <span class="keyword">long</span>, brk)</span><br><span class="line">{</span><br><span class="line"><span class="keyword">unsigned</span> <span class="keyword">long</span> newbrk, oldbrk, origbrk;</span><br><span class="line"><span class="class"><span class="keyword">struct</span> <span class="title">mm_struct</span> *<span class="title">mm</span> = <span class="title">current</span>-><span class="title">mm</span>;</span></span><br><span class="line"><span class="class"><span class="keyword">struct</span> <span class="title">vm_area_struct</span> *<span class="title">next</span>;</span></span><br><span class="line"><span class="keyword">unsigned</span> <span class="keyword">long</span> min_brk;</span><br><span class="line"><span class="keyword">bool</span> populate;</span><br><span class="line"><span class="keyword">bool</span> downgraded = <span class="literal">false</span>;</span><br><span class="line">LIST_HEAD(uf);</span><br><span class="line"></span><br><span class="line"><span class="keyword">if</span> (mmap_write_lock_killable(mm))</span><br><span class="line"><span class="keyword">return</span> -EINTR;</span><br><span class="line"></span><br><span class="line">origbrk = mm->brk;</span><br><span class="line"></span><br><span class="line"><span class="meta">#<span class="meta-keyword">ifdef</span> CONFIG_COMPAT_BRK</span></span><br><span class="line"><span class="comment">/*</span></span><br><span class="line"><span class="comment"> * CONFIG_COMPAT_BRK can still be overridden by setting</span></span><br><span class="line"><span class="comment"> * randomize_va_space to 2, which will still cause mm->start_brk</span></span><br><span class="line"><span class="comment"> * to be arbitrarily shifted</span></span><br><span class="line"><span class="comment"> */</span></span><br><span class="line"><span class="keyword">if</span> (current->brk_randomized)</span><br><span class="line">min_brk = mm->start_brk;</span><br><span class="line"><span class="keyword">else</span></span><br><span class="line">min_brk = mm->end_data;</span><br><span class="line"><span class="meta">#<span class="meta-keyword">else</span></span></span><br><span class="line">min_brk = mm->start_brk;</span><br><span class="line"><span class="meta">#<span class="meta-keyword">endif</span></span></span><br><span class="line"><span class="keyword">if</span> (brk < min_brk)</span><br><span class="line"><span class="keyword">goto</span> out;</span><br><span class="line"></span><br><span class="line"><span class="comment">/*</span></span><br><span class="line"><span class="comment"> * Check against rlimit here. If this check is done later after the test</span></span><br><span class="line"><span class="comment"> * of oldbrk with newbrk then it can escape the test and let the data</span></span><br><span class="line"><span class="comment"> * segment grow beyond its set limit the in case where the limit is</span></span><br><span class="line"><span class="comment"> * not page aligned -Ram Gupta</span></span><br><span class="line"><span class="comment"> */</span></span><br><span class="line"><span class="keyword">if</span> (check_data_rlimit(rlimit(RLIMIT_DATA), brk, mm->start_brk,</span><br><span class="line"> mm->end_data, mm->start_data))</span><br><span class="line"><span class="keyword">goto</span> out;</span><br><span class="line"></span><br><span class="line">newbrk = PAGE_ALIGN(brk);</span><br><span class="line">oldbrk = PAGE_ALIGN(mm->brk);</span><br><span class="line"><span class="keyword">if</span> (oldbrk == newbrk) {</span><br><span class="line">mm->brk = brk;</span><br><span class="line"><span class="keyword">goto</span> success;</span><br><span class="line">}</span><br><span class="line"></span><br><span class="line"><span class="comment">/*</span></span><br><span class="line"><span class="comment"> * Always allow shrinking brk.</span></span><br><span class="line"><span class="comment"> * __do_munmap() may downgrade mmap_lock to read.</span></span><br><span class="line"><span class="comment"> */</span></span><br><span class="line"><span class="keyword">if</span> (brk <= mm->brk) {</span><br><span class="line"><span class="keyword">int</span> ret;</span><br><span class="line"></span><br><span class="line"><span class="comment">/*</span></span><br><span class="line"><span class="comment"> * mm->brk must to be protected by write mmap_lock so update it</span></span><br><span class="line"><span class="comment"> * before downgrading mmap_lock. When __do_munmap() fails,</span></span><br><span class="line"><span class="comment"> * mm->brk will be restored from origbrk.</span></span><br><span class="line"><span class="comment"> */</span></span><br><span class="line">mm->brk = brk;</span><br><span class="line">ret = __do_munmap(mm, newbrk, oldbrk-newbrk, &uf, <span class="literal">true</span>);</span><br><span class="line"><span class="keyword">if</span> (ret < <span class="number">0</span>) {</span><br><span class="line">mm->brk = origbrk;</span><br><span class="line"><span class="keyword">goto</span> out;</span><br><span class="line">} <span class="keyword">else</span> <span class="keyword">if</span> (ret == <span class="number">1</span>) {</span><br><span class="line">downgraded = <span class="literal">true</span>;</span><br><span class="line">}</span><br><span class="line"><span class="keyword">goto</span> success;</span><br><span class="line">}</span><br><span class="line"></span><br><span class="line"><span class="comment">/* Check against existing mmap mappings. */</span></span><br><span class="line">next = find_vma(mm, oldbrk);</span><br><span class="line"><span class="keyword">if</span> (next && newbrk + PAGE_SIZE > vm_start_gap(next))</span><br><span class="line"><span class="keyword">goto</span> out;</span><br><span class="line"></span><br><span class="line"><span class="comment">/* Ok, looks good - let it rip. */</span></span><br><span class="line"><span class="keyword">if</span> (do_brk_flags(oldbrk, newbrk-oldbrk, <span class="number">0</span>, &uf) < <span class="number">0</span>)</span><br><span class="line"><span class="keyword">goto</span> out;</span><br><span class="line">mm->brk = brk;</span><br><span class="line"></span><br><span class="line">success:</span><br><span class="line">populate = newbrk > oldbrk && (mm->def_flags & VM_LOCKED) != <span class="number">0</span>;</span><br><span class="line"><span class="keyword">if</span> (downgraded)</span><br><span class="line">mmap_read_unlock(mm);</span><br><span class="line"><span class="keyword">else</span></span><br><span class="line">mmap_write_unlock(mm);</span><br><span class="line">userfaultfd_unmap_complete(mm, &uf);</span><br><span class="line"><span class="keyword">if</span> (populate)</span><br><span class="line">mm_populate(oldbrk, newbrk - oldbrk);</span><br><span class="line"><span class="keyword">return</span> brk;</span><br><span class="line"></span><br><span class="line">out:</span><br><span class="line">mmap_write_unlock(mm);</span><br><span class="line"><span class="keyword">return</span> origbrk;</span><br><span class="line">}</span><br></pre></td></tr></table></figure><h2 id="VMA"><a href="#VMA" class="headerlink" title="VMA"></a>VMA</h2><p>要更进一步理解以上过程，皆需理解VMA的管理方式。引用<深入理解Linux内核架构>一书的介绍：</p><p><img src="/images/posts/20211106/vma.png" alt="VMA管理" title="VMA管理"></p><ul><li><p>如果一个新区域紧接着现存区域前后直接添加(因此也包括在两个现存区域之间的情况)，内核将涉及的数据结构合并为一个。当然，前提是涉及的所有区域的访问权限相同，而且是从同一后备存储器映射的连续数据。</p></li><li><p>如果在区域的开始或结束处进行删除，则必须据此截断现存的数据结构。</p></li><li><p>如果删除两个区域之间的一个区域，那么一方面需要减小现存数据结构的长度，另一方面需</p><p>要为形成的新区域创建一个新的数据结构。</p></li></ul><p>代码部分不做进一步分析，大家可以直接看<a href="https://elixir.bootlin.com/linux/latest/source/include/linux/mm.h#L2713" target="_blank" rel="noopener">内核源码</a>或者找相关资料学习。</p><h2 id="结语"><a href="#结语" class="headerlink" title="结语"></a>结语</h2><p>本文概要介绍了Linux内核对进程内存的管理方式。主要有：</p><ul><li>进程内存管理架构</li><li>进程加载执行时内存分配</li><li>堆管理</li><li>VMA的管理方式</li></ul><p>进程内存管理还涉及到以下知识，将会在之后的文章中介绍：</p><ul><li>内存映射mmap</li><li>反向映射</li><li>缺页管理</li></ul>]]></content>
<summary type="html">
<blockquote>
<p>这是&lt;Linux内核内存管理&gt;系列的第七篇:</p>
<blockquote>
<p><a href="/p/61304bbb.html">第一篇</a>为内核内存管理过程知识点的的简单梳理</p>
<p><a href="/p/1dc6adc2.html">第二篇</a>介绍了内核的数据结构</p>
<p><a href="/p/8f5117c5.html">第三篇</a>介绍了从内核第一行代码加载到跳转到C代码前的内存处理。</p>
<p><a href="/p/94b22c5d.html">第四篇</a>概览了初始化C代码中的内存处理</p>
<p><a href="/p/20210923linuxmm4.html">第五篇（上）</a>和<a href="/p/5347665a.html">第五篇（下）</a>介绍了Memblock和伙伴系统分配器</p>
<p><a href>第六篇</a>介绍了内存检测工具KFence工作原理</p>
</blockquote>
</blockquote>
</summary>
<category term="Linux" scheme="http://www.l2h.site/categories/Linux/"/>
<category term="Linux内存管理" scheme="http://www.l2h.site/categories/Linux/Linux%E5%86%85%E5%AD%98%E7%AE%A1%E7%90%86/"/>
<category term="Linux" scheme="http://www.l2h.site/tags/Linux/"/>
<category term="内存管理" scheme="http://www.l2h.site/tags/%E5%86%85%E5%AD%98%E7%AE%A1%E7%90%86/"/>
<category term="进程内存" scheme="http://www.l2h.site/tags/%E8%BF%9B%E7%A8%8B%E5%86%85%E5%AD%98/"/>
</entry>
<entry>
<title>MAC运行Virtual Box时"The support driver is not installed"解决</title>
<link href="http://www.l2h.site/p/7ccf1cb6.html"/>
<id>http://www.l2h.site/p/7ccf1cb6.html</id>
<published>2021-10-17T00:52:17.000Z</published>
<updated>2023-06-01T13:14:04.483Z</updated>
<content type="html"><![CDATA[<blockquote><p>注意：这只是一个Work Around方法，并非完全解决方案</p></blockquote><h2 id="背景"><a href="#背景" class="headerlink" title="背景"></a>背景</h2><p>MAC升级Big Sur后，每次运行Virtual Box都会有如下报错：</p><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br></pre></td><td class="code"><pre><span class="line">Kernel driver not installed (rc=-1908)</span><br><span class="line"></span><br><span class="line">Make sure the kernel module has been loaded successfully.</span><br><span class="line"></span><br><span class="line">where: suplibOsInit what: 3 VERR_VM_DRIVER_NOT_INSTALLED (-1908) - The support driver is not installed. On linux, open returned ENOENT.</span><br></pre></td></tr></table></figure><a id="more"></a><p><a href="https://blog.csdn.net/u011700186/article/details/109741194" target="_blank" rel="noopener">CSDN这篇文章</a>介绍得很详细，但不够完整，这里做一下补充</p><h2 id="下载安装Extension"><a href="#下载安装Extension" class="headerlink" title="下载安装Extension"></a>下载安装Extension</h2><p>到<a href="https://www.virtualbox.org/wiki/Downloads" target="_blank" rel="noopener">Virtual Box官网</a>下载Extension Pack并安装，如下图：</p><p><img src="/images/posts/20211017/VBox_MAC_Extension.png" alt="Extension Pack" title="Virtual Box Extension Pack"></p><p>下载安装时，如果被系统安全阻拦，可以在“安全与隐私”设置里信任相关程序的执行。</p><h2 id="执行kext"><a href="#执行kext" class="headerlink" title="执行kext"></a>执行kext</h2><p>这里很奇怪，无论怎么安装重启，在我的电脑上，对应的Driver总不能开机自动运行(<u>这也是前文说这是Workaround的原因</u>)，因此每次重新开机后还必须执行一次如下命令：</p><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><span class="line">sudo kextload -b org.virtualbox.kext.VBoxDrv</span><br></pre></td></tr></table></figure><p>接着再运行Virtual Box就不会报错了。</p><blockquote><p>后续若找到自动加载的方法，可以再交流讨论更新</p></blockquote><h2 id="简单说明"><a href="#简单说明" class="headerlink" title="简单说明"></a>简单说明</h2><p>这里说明一下，会有以上报错的主要原因就是MAC High Sierra，开始逐渐淘汰运行在内核态的驱动，改推荐（甚至强制推行）厂商改用<a href="https://developer.apple.com/system-extensions/" target="_blank" rel="noopener">System Extension</a>来开发驱动，于是乎，各种兼容性问题就来了。</p><p>这也是Apple强硬的一点。当然反之使用System Extension是有好处的，希望Oracle后续看如何更好地兼容新的MAC OS。</p>]]></content>
<summary type="html">
<blockquote>
<p>注意：这只是一个Work Around方法，并非完全解决方案</p>
</blockquote><h2 id="背景"><a href="#背景" class="headerlink" title="背景"></a>背景</h2><p>MAC升级Big Sur后，每次运行Virtual Box都会有如下报错：</p><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br></pre></td><td class="code"><pre><span class="line">Kernel driver not installed (rc=-1908)</span><br><span class="line"></span><br><span class="line">Make sure the kernel module has been loaded successfully.</span><br><span class="line"></span><br><span class="line">where: suplibOsInit what: 3 VERR_VM_DRIVER_NOT_INSTALLED (-1908) - The support driver is not installed. On linux, open returned ENOENT.</span><br></pre></td></tr></table></figure>
</summary>
<category term="软件技巧" scheme="http://www.l2h.site/categories/%E8%BD%AF%E4%BB%B6%E6%8A%80%E5%B7%A7/"/>
<category term="Virtual Box" scheme="http://www.l2h.site/tags/Virtual-Box/"/>
<category term="MAC OS" scheme="http://www.l2h.site/tags/MAC-OS/"/>
</entry>
<entry>
<title>Linux内核内存管理 - KFENCE</title>
<link href="http://www.l2h.site/p/75308136.html"/>
<id>http://www.l2h.site/p/75308136.html</id>
<published>2021-10-16T12:16:55.000Z</published>
<updated>2023-06-01T13:14:04.482Z</updated>
<content type="html"><![CDATA[<blockquote><p>这是<Linux内核内存管理>系列的第六篇</p><blockquote><p><a href="/p/61304bbb.html">第一篇</a>为内核内存管理过程知识点的的简单梳理</p><p><a href="/p/1dc6adc2.html">第二篇</a>介绍了内核的数据结构</p><p><a href="/p/8f5117c5.html">第三篇</a>介绍了从内核第一行代码加载到跳转到C代码前的内存处理。</p><p><a href="/p/94b22c5d.html">第四篇</a>概览了初始化C代码中的内存处理</p><p><a href="/p/20210923linuxmm4.html">第五篇（上）</a>和<a href="/p/5347665a.html">第五篇（下）</a>介绍了Memblock和伙伴系统分配器</p></blockquote></blockquote><h2 id="前言"><a href="#前言" class="headerlink" title="前言"></a>前言</h2><a id="more"></a><p><a href="https://www.kernel.org/doc/html/latest/dev-tools/kfence.html" target="_blank" rel="noopener">Kernel Electric-Fence (KFENCE)</a>是5.12版本内核新引入的内存使用错误检测机制。它可以检查的错误有：</p><ul><li>内存访问越界</li><li>释放后使用</li><li>无效释放</li></ul><p>显然，它可以检测的内存错误类型不如<a href="https://www.kernel.org/doc/html/latest/dev-tools/kasan.html" target="_blank" rel="noopener">KASAN</a>多。但与KASAN相比，它最大的优势是运行时小Overhead，可以直接用在生产环境中。因此在X86，ARM64，RISCV等平台上均默认开启。</p><blockquote><p>在Arch对应的defconfig中使用CONFIG_HAVE_ARCH_KFENCE开启。</p></blockquote><h2 id="架构及原理"><a href="#架构及原理" class="headerlink" title="架构及原理"></a>架构及原理</h2><p>Kfence的原理比较简单，如下图：</p><p><img src="/images/posts/20211016/Kfence.png" alt="KFENCE" title="KFENCE"></p><h3 id="初始化"><a href="#初始化" class="headerlink" title="初始化"></a>初始化</h3><ul><li>初始化过程中，KFENCE向<a href="https://www.l2h.site/p/20210923linuxmm4.html" target="_blank" rel="noopener">Memblock</a>申请一段内存，作为KFENCE内存池。<ul><li>这个内存池的大小配置为CONFIG_KFENCE_NUM_OBJECTS</li><li>即，预留两个页面作为保护页（Guard Page），接着为每一个用于分配的内存页分配一个Guard Page。因此总大小为：</li></ul></li></ul><figure class="highlight c"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><span class="line"><span class="meta">#<span class="meta-keyword">define</span> KFENCE_POOL_SIZE ((CONFIG_KFENCE_NUM_OBJECTS + 1) * 2 * PAGE_SIZE)</span></span><br></pre></td></tr></table></figure><ul><li>初始化一个Delayed Worker，定期（<strong>CONFIG_KFENCE_SAMPLE_INTEVAL</strong>）重置kfence_alloc_gate值为0。</li></ul><blockquote><p>这个值可以通过sysfs修改</p></blockquote><h3 id="分配"><a href="#分配" class="headerlink" title="分配"></a>分配</h3><ul><li>kfence_alloc_gate值为0时，使用kmem_cache_alloc所作的内存分配从KFENCE内存池中分配，并增加kfence_alloc_gate的值。kfence_alloc_gate值大于等于1时，直接从SLUB中分配。由此可以看出，kfence是基于采样的内存检测。</li></ul><blockquote><p>大于一个Page(4K)的分配不会从KFENCE Pool中分配</p></blockquote><ul><li>每次通过KFENCE进行内存分配时，都会从KFENCE内存池分配一个内存页和一个Guard Page，并在实际使用内存的两端内存填充Canary数据。</li></ul><blockquote><p>解释一下为什么保护数据叫Canary。这是因为在19世纪，金丝雀在采矿业中常用的毒气检测方法，因为它们比人类对毒气更为敏感反应也更快。</p></blockquote><ul><li>如果KFENCE内存池中没有可用内存，则直接从SLAB中分配。</li></ul><h3 id="释放"><a href="#释放" class="headerlink" title="释放"></a>释放</h3><ul><li>释放时，检查Canary数据，将所用内存放回KFENCE内存池。</li></ul><h3 id="检测报错"><a href="#检测报错" class="headerlink" title="检测报错"></a>检测报错</h3><p>在以下情况，会检测报错：</p><ul><li>释放时发现Canary数据不对。</li><li>当KFENCE内存池的内存区域发生Page Fault时，它或者是因为越界访问、或者是释放后使用。</li><li>无效释放：当一段KFENCE内存没有被标记分配，但对齐释放时，会有相应报错提示。</li></ul><h2 id="总结"><a href="#总结" class="headerlink" title="总结"></a>总结</h2><p>开源社区总能带来新的idea。KFENCE，克服了KASAN等工具需要占用大量内存且影响运行时性能的缺点，是一个有效地运行时内存访问错误检测工具。</p><p>当然，因为它所针对的内存区域仅仅是KFENCE内存池，且其是周期性进行采样，检测效果还不得而知。其又有可以动态开关、参数可调节等优点，这些劣势或许也不是问题。后续若有时间可以研究分析对比其和KASAN的检测效果。</p>]]></content>
<summary type="html">
<blockquote>
<p>这是&lt;Linux内核内存管理&gt;系列的第六篇</p>
<blockquote>
<p><a href="/p/61304bbb.html">第一篇</a>为内核内存管理过程知识点的的简单梳理</p>
<p><a href="/p/1dc6adc2.html">第二篇</a>介绍了内核的数据结构</p>
<p><a href="/p/8f5117c5.html">第三篇</a>介绍了从内核第一行代码加载到跳转到C代码前的内存处理。</p>
<p><a href="/p/94b22c5d.html">第四篇</a>概览了初始化C代码中的内存处理</p>
<p><a href="/p/20210923linuxmm4.html">第五篇（上）</a>和<a href="/p/5347665a.html">第五篇（下）</a>介绍了Memblock和伙伴系统分配器</p>
</blockquote>
</blockquote><h2 id="前言"><a href="#前言" class="headerlink" title="前言"></a>前言</h2>
</summary>
<category term="Linux" scheme="http://www.l2h.site/categories/Linux/"/>
<category term="Linux内存管理" scheme="http://www.l2h.site/categories/Linux/Linux%E5%86%85%E5%AD%98%E7%AE%A1%E7%90%86/"/>
<category term="Linux" scheme="http://www.l2h.site/tags/Linux/"/>
<category term="内存管理" scheme="http://www.l2h.site/tags/%E5%86%85%E5%AD%98%E7%AE%A1%E7%90%86/"/>
<category term="KFENCE" scheme="http://www.l2h.site/tags/KFENCE/"/>
</entry>
<entry>
<title>PlantUML主题选项</title>
<link href="http://www.l2h.site/p/c5b29752.html"/>
<id>http://www.l2h.site/p/c5b29752.html</id>
<published>2021-10-10T00:10:24.000Z</published>
<updated>2023-06-01T13:14:04.482Z</updated>
<content type="html"><![CDATA[<h2 id="前言"><a href="#前言" class="headerlink" title="前言"></a>前言</h2><p><a href="http://www.plantuml.com/" target="_blank" rel="noopener">PlantUML</a>可以算是最成功最知名的开源绘图工具了，它可以方便地将您输入文字型描述，转化成您想要的各种图（当然，要遵循其语法）。近日，在做流程图的时候，到网站上去翻语法，赫然发现其增加了一个新的<a href="https://plantuml.com/zh/theme" target="_blank" rel="noopener"><strong>主题</strong></a>功能。</p><h2 id="用法"><a href="#用法" class="headerlink" title="用法"></a>用法</h2><p>用法也很简单, 在文件头部增加如下配置即可：</p><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><span class="line">!theme 主题名</span><br></pre></td></tr></table></figure><p>例如，如下图采用内置的spacelab主题：</p><a id="more"></a><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br></pre></td><td class="code"><pre><span class="line">@startuml</span><br><span class="line">!theme spacelab</span><br><span class="line">Bob -> Alice : hello</span><br><span class="line">Bob <- Alice : $success("success: hello B.")</span><br><span class="line">Bob -x Alice : $failure("failure")</span><br><span class="line">Bob ->> Alice : $warning("warning")</span><br><span class="line">@enduml</span><br></pre></td></tr></table></figure><p>效果如下：</p><p><img src="/images/posts/20211009/sample.png" alt="spacelab主题示例"></p><p>除了内置主题，也支持本地主题：</p><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><span class="line">!theme 主题名 from /本地/摆放/主题的路径</span><br></pre></td></tr></table></figure><p>互联网主题也支持，例如：</p><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><span class="line">!theme 主题名 from https://raw.githubusercontent.com/plantuml/plantuml/master/themes</span><br></pre></td></tr></table></figure><h2 id="内置主题"><a href="#内置主题" class="headerlink" title="内置主题"></a>内置主题</h2><p>您可以使用如下代码查看PLANTUML支持哪些内置主题：</p><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br></pre></td><td class="code"><pre><span class="line">@startuml</span><br><span class="line">help themes</span><br><span class="line">@enduml</span><br></pre></td></tr></table></figure><p>目前plantuml支持如下内置主题，为了方便大家选用，我将所有的示例图显示出来（P.S. 个人比较喜欢sketchy）：</p><p><img src="/images/posts/20211009/vibrant.png" alt="vibrant"></p><p><img src="/images/posts/20211009/amiga.png" alt="amiga"></p><p><img src="/images/posts/20211009/aws-orange.png" alt="aws-orange"></p><p><img src="/images/posts/20211009/black-knight.png" alt="black-knight"></p><p><img src="/images/posts/20211009/bluegray.png" alt="bluegray"></p><p><img src="/images/posts/20211009/blueprint.png" alt="blueprint"></p><p><img src="/images/posts/20211009/cerulean-outline.png" alt="cerulean-outline"></p><p><img src="/images/posts/20211009/cerulean.png" alt="cerulean"></p><p><img src="/images/posts/20211009/crt-amber.png" alt="crt-amber"></p><p><img src="/images/posts/20211009/crt-green.png" alt="crt-green"></p><p><img src="/images/posts/20211009/cyborg-outline.png" alt="cyborg-outline"></p><p><img src="/images/posts/20211009/Cyborg.png" alt="Cyborg"></p><p><img src="/images/posts/20211009/hacker.png" alt="hacker"></p><p><img src="/images/posts/20211009/lightgray.png" alt="lightgray"></p><p><img src="/images/posts/20211009/materia-outline.png" alt="materia-outline"></p><p><img src="/images/posts/20211009/materia.png" alt="materia"></p><p><img src="/images/posts/20211009/metal.png" alt="metal"></p><p><img src="/images/posts/20211009/mimeograph.png" alt="mimeograph"></p><p><img src="/images/posts/20211009/minty.png" alt="minty"></p><p><img src="/images/posts/20211009/plain.png" alt="plain"></p><p><img src="/images/posts/20211009/reddress-darkblue.png" alt="reddress-darkblue"></p><p><img src="/images/posts/20211009/reddress-darkgreen.png" alt="reddress-darkgreen"></p><p><img src="/images/posts/20211009/reddress-darkorange.png" alt="reddress-darkorange"></p><p><img src="/images/posts/20211009/reddress-darkred.png" alt="reddress-darkred"></p><p><img src="/images/posts/20211009/reddress-lightblue.png" alt="reddress-lightblue"></p><p><img src="/images/posts/20211009/reddress-lightgreen.png" alt="reddress-lightgreen"></p><p><img src="/images/posts/20211009/reddress-lightorange.png" alt="reddress-lightorange"></p><p><img src="/images/posts/20211009/reddress-lightred.png" alt="reddress-lightred"></p><p><img src="/images/posts/20211009/sample.png" alt="sample"></p><p><img src="/images/posts/20211009/sandstone.png" alt="sandstone"></p><p><img src="/images/posts/20211009/silver.png" alt="silver"></p><p><img src="/images/posts/20211009/sketchy-outline.png" alt="sketchy-outline"></p><p><img src="/images/posts/20211009/sketchy.png" alt="sketchy"></p><p><img src="/images/posts/20211009/spacelab.png" alt="spacelab"></p><p><img src="/images/posts/20211009/superhero-outline.png" alt="superhero-outline"></p><p><img src="/images/posts/20211009/superhero.png" alt="superhero"></p><p><img src="/images/posts/20211009/toy.png" alt="toy"></p><p><img src="/images/posts/20211009/united.png" alt="united"></p><h2 id="总结"><a href="#总结" class="headerlink" title="总结"></a>总结</h2><p>本文介绍了PlantUML的主题，将内置主题的样式全部呈现出来供参考。这类开发实用工具其实很多，本人后续也会推荐一些优秀的工具给大家。本站的链接栏也有一些链接可以参考。</p>]]></content>
<summary type="html">
<h2 id="前言"><a href="#前言" class="headerlink" title="前言"></a>前言</h2><p><a href="http://www.plantuml.com/" target="_blank" rel="noopener">PlantUML</a>可以算是最成功最知名的开源绘图工具了，它可以方便地将您输入文字型描述，转化成您想要的各种图（当然，要遵循其语法）。近日，在做流程图的时候，到网站上去翻语法，赫然发现其增加了一个新的<a href="https://plantuml.com/zh/theme" target="_blank" rel="noopener"><strong>主题</strong></a>功能。</p><h2 id="用法"><a href="#用法" class="headerlink" title="用法"></a>用法</h2><p>用法也很简单, 在文件头部增加如下配置即可：</p><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><span class="line">!theme 主题名</span><br></pre></td></tr></table></figure><p>例如，如下图采用内置的spacelab主题：</p>
</summary>
<category term="Linux" scheme="http://www.l2h.site/categories/Linux/"/>
<category term="PlantUML" scheme="http://www.l2h.site/tags/PlantUML/"/>
<category term="架构图" scheme="http://www.l2h.site/tags/%E6%9E%B6%E6%9E%84%E5%9B%BE/"/>
<category term="主题" scheme="http://www.l2h.site/tags/%E4%B8%BB%E9%A2%98/"/>
<category term="Theme" scheme="http://www.l2h.site/tags/Theme/"/>
</entry>
<entry>
<title>Linux内核内存管理 - SLUB</title>
<link href="http://www.l2h.site/p/37327e44.html"/>
<id>http://www.l2h.site/p/37327e44.html</id>
<published>2021-10-07T02:48:24.000Z</published>
<updated>2023-06-01T13:14:04.482Z</updated>
<content type="html"><![CDATA[<blockquote><p>这是<Linux内核内存管理>系列的第六篇</p><blockquote><p><a href="/p/61304bbb.html">第一篇</a>为内核内存管理过程知识点的的简单梳理</p><p><a href="/p/1dc6adc2.html">第二篇</a>介绍了内核的数据结构</p><p><a href="/p/8f5117c5.html">第三篇</a>介绍了从内核第一行代码加载到跳转到C代码前的内存处理。</p><p><a href="/p/94b22c5d.html">第四篇</a>概览了初始化C代码中的内存处理</p><p><a href="/p/20210923linuxmm4.html">第五篇（上）</a>和<a href="/p/5347665a.html">第五篇（下）</a>介绍了Memblock和伙伴系统分配器</p></blockquote><p>为了避免晦涩难懂，本文及之后均主要使用图表+文字描述，尽量避免涉及过多代码。专注点会在：</p><ol><li>背景</li><li>架构及其思想</li><li>流程</li><li>特殊处理及其原因</li></ol></blockquote><a id="more"></a><h2 id="前言"><a href="#前言" class="headerlink" title="前言"></a>前言</h2><p>网络上介绍SLAB/SLUB的文章很多，也都很详细，本文以当前内核版本(<strong>5.14.X</strong>)来介绍被广泛采用的SLAB内存管理，希望尽可能地做到详尽易理解。一些更多的参考资料见，文中不再另外标注引用：</p><ul><li><a href="http://www.wowotech.net/memory_management/426.html" target="_blank" rel="noopener">图解Slub</a></li><li><a href="https://www.kernel.org/doc/html/latest/vm/slub.html" target="_blank" rel="noopener">Short users guide for SLUB</a></li></ul><p><strong>kmalloc/kfree</strong> 大概是内核最常用的内存分配和释放函数，其背后的实现就是SLAB分配器。而SLUB是SLAB分配器的一种实现，另外的两种实现分别是SLAB和SLOB。从命名也可以看出SLAB是鼻祖，随着内核的发展，演进出了SLOB和SLUB分配器。</p><ul><li>SLOB分配器是为了应对嵌入式设备内存管理的特殊需求而生</li><li>SLUB则是在SLAB的基础上演进而来，可以很好地适应各种平台的需求，更为有效地使用内存，同时增强了调试的容易度。</li></ul><p><strong>SLAB分配器解决的是什么问题？</strong>这个问题可以换种方式来问，为什么有了Buddy System，还要用SLAB分配器？ 解释如下：</p><ul><li>伙伴系统是以页为单位进行管理的，每页的大小一般为4096字节。内核程序在申请内存时，往往不会刚好申请页大小的倍数，如果我们按页进行分配，没过多久系统内存就会被耗尽。</li><li>因为上述原因，必须采用更小的单位对内存进行管理。这需要考虑频繁内存分配释放造成的内存碎片问题。与此同时，需要考虑如何更有效地利用CPU缓存，以及尽量避免访问同一块内存区域造成的静态，等等。</li></ul><p><strong>为什么这三种分配器又都是SLAB分配器的实现</strong>？这是因为这三种分配器采用一样的数据结构名称和内存分配/释放API（注意，仅仅是“名称”一样）。例，其管理结构体，都叫<strong><em>struct kmem_cache</em></strong>。</p><h2 id="内核配置"><a href="#内核配置" class="headerlink" title="内核配置"></a>内核配置</h2><p><a href="#前言">前言</a>讲到，SLAB/SLOB/SLUB采用相同的API，相同的结构体，那么他们一定是相互排他的，这从内核定义KConfig也可以看出：</p><figure class="highlight c"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br><span class="line">21</span><br><span class="line">22</span><br><span class="line">23</span><br><span class="line">24</span><br><span class="line">25</span><br><span class="line">26</span><br><span class="line">27</span><br><span class="line">28</span><br><span class="line">29</span><br><span class="line">30</span><br><span class="line">31</span><br><span class="line">32</span><br><span class="line">33</span><br><span class="line">34</span><br></pre></td><td class="code"><pre><span class="line">choice</span><br><span class="line">prompt <span class="string">"Choose SLAB allocator"</span></span><br><span class="line"><span class="keyword">default</span> SLUB</span><br><span class="line">help</span><br><span class="line"> This option allows to select a slab allocator.</span><br><span class="line"></span><br><span class="line"><span class="built_in">config</span> SLAB</span><br><span class="line"><span class="keyword">bool</span> <span class="string">"SLAB"</span></span><br><span class="line">select HAVE_HARDENED_USERCOPY_ALLOCATOR</span><br><span class="line">help</span><br><span class="line"> The regular slab allocator that is established <span class="keyword">and</span> known to work</span><br><span class="line"> well in all environments. It organizes cache hot objects in</span><br><span class="line"> per cpu <span class="keyword">and</span> per node queues.</span><br><span class="line"></span><br><span class="line"><span class="built_in">config</span> SLUB</span><br><span class="line"><span class="keyword">bool</span> <span class="string">"SLUB (Unqueued Allocator)"</span></span><br><span class="line">select HAVE_HARDENED_USERCOPY_ALLOCATOR</span><br><span class="line">help</span><br><span class="line"> SLUB is a slab allocator that minimizes cache <span class="built_in">line</span> usage</span><br><span class="line"> instead of managing queues of cached objects (SLAB approach).</span><br><span class="line"> Per cpu caching is realized <span class="keyword">using</span> slabs of objects instead</span><br><span class="line"> of queues of objects. SLUB can use memory efficiently</span><br><span class="line"> <span class="keyword">and</span> has enhanced diagnostics. SLUB is the <span class="keyword">default</span> choice <span class="keyword">for</span></span><br><span class="line"> a slab allocator.</span><br><span class="line"></span><br><span class="line"><span class="built_in">config</span> SLOB</span><br><span class="line">depends on EXPERT</span><br><span class="line"><span class="keyword">bool</span> <span class="string">"SLOB (Simple Allocator)"</span></span><br><span class="line">help</span><br><span class="line"> SLOB replaces the stock allocator with a drastically simpler</span><br><span class="line"> allocator. SLOB is generally more space efficient but</span><br><span class="line"> does <span class="keyword">not</span> perform as well on large systems.</span><br><span class="line"></span><br><span class="line">endchoice</span><br></pre></td></tr></table></figure><p>从中默认选项就是SLUB。</p><blockquote><p>KConfig相关知识可以参考<a href="https://www.kernel.org/doc/html/latest/kbuild/kconfig-language.html" target="_blank" rel="noopener">KConfig Language</a></p></blockquote><h2 id="架构"><a href="#架构" class="headerlink" title="架构"></a>架构</h2><p>SLAB（下文中SLAB也统一代表SLUB）在系统中的位置如<strong>Figure 1</strong>所示.</p><p><img src="/images/posts/20211007/Architecture-in-the-system.png" alt="SLUB在内存管理系统中位置" title="Figure 1.SLUB在内存管理系统中位置"></p><p>简单说明如下:</p><ul><li>内存管理系统的最大管理单位是Node，被划分为多个Memory Zone（<font color="red">ToDo: 上一章Buddy System和第二章补充图片示意</font>）。</li><li>在进行Page分配时（Page分配还记得吗？可以参考<a href="/p/20210923linuxmm4.html">第五篇（上）</a>和<a href="/p/5347665a.html">第五篇（下）</a>对<strong>Buddy System</strong>的介绍），是依据分配传入的参数，选择从哪个Memory Zone分配内存。</li><li>SLAB的分配需要kmem_cache管理结构，而这些管理结构所需的内存也是从的kmem_cache来的。这里内核<u><strong>做了很巧妙的设计</strong></u>: <ul><li>建立kmem_cache的slab所需的slab初始管理对象是boot_kmem_cache和boot_kmem_cache_node。它们被__init属性修饰，表示将会被放到.init.data段，并在内核初始化的后半段释放。</li><li>在整个SLAB系统初始化过程中，会从boot_kmem_cache和boot_kmem_cache_node指向的SLAB，分配出全局SLAB对象kmem_cache和kmem_cache_node。</li><li>之后将boot_kmem_cache和boot_kmem_cache_node内容拷贝到kmem_cache和kmem_cache_node。</li><li>至此，使用全局SLAB对象kmem_cache和kmem_cache_node来进行SLUB管理。</li></ul></li><li>kmalloc的内存也是内核初始化早期分配的。其实质上是建立了大小分别为2, 4, 8, ….的SLAB对象。</li></ul><h2 id="源文件"><a href="#源文件" class="headerlink" title="源文件"></a>源文件</h2><p>下表介绍SLAB及SLUB相关内核源文件：</p><table><thead><tr><th>文件</th><th>描述</th></tr></thead><tbody><tr><td>slab.c</td><td>SLAB分配器（三个分配器之一）的实现</td></tr><tr><td>slab.h</td><td>所有SLAB分配器的头文件定义</td></tr><tr><td>slob.c</td><td>SLOB分配器的实现</td></tr><tr><td>slub.c</td><td>SLUB分配器的实现</td></tr><tr><td>slab_common.c</td><td>所有SLAB分配器公用的，与实现无关的函数。大部分都会调用到具体的某个分配器。</td></tr></tbody></table><h2 id="数据结构"><a href="#数据结构" class="headerlink" title="数据结构"></a>数据结构</h2><p>SLAB的重要的数据结构有三个，其内容和相互关系如下图：</p><p><img src="/images/posts/20211007/SLUB.png" alt="SLUB数据结构" title="Figure 2.SLUB数据结构"></p><p>其中：</p><ul><li>kmem_cache代表一个SLAB对象</li><li>kmem_cache_cpu里存储了该SLAB对象在CPU本地的资源，这里<strong>__percpu</strong>修饰表示这是一个Per CPU对象（每个CPU有一份拷贝）</li><li>kmem_cache_node是一个数组，每个数组成员代表该SLAB对象在每个<a href="/p/1dc6adc2.html#内存节点">内存结点</a>的内存资源。</li></ul><h2 id="管理方式"><a href="#管理方式" class="headerlink" title="管理方式"></a>管理方式</h2><p>SLUB的管理方式如下图：</p><p><img src="/images/posts/20211007/slub_Detail.png" alt="SLUB管理" title="Figure3. SLUB管理方式"></p><p>简单描述如下：</p><ul><li>每个SLUB管理结构分别有多个cpu本地slab和node slab。</li><li>SLUB刚建立时，只有对应的管理结构。</li><li>SLUB分配内存时<ul><li>若此时该对象中没有页面可用，则从伙伴系统中分配页面、挂到cpu本地slab上，从中返回一个所需内存。</li><li>若此时该对象中有可用页面，则从中分配内存。</li><li>若当前kmem_cache_cpu已经没有可用页面(kmem_cache_cpu的freelist和partial所指向页都满)，则从kmem_cache_node的partial处分配内存</li></ul></li></ul><blockquote><p>这样处理可以保证总是优先从该cpu的cache区域分配，提升资源的访问速度。</p></blockquote><ul><li>释放内存：会先将内存释放到该内存所在页。 释放有如下情况：</li></ul><table><thead><tr><th>场景</th><th>释放方式</th></tr></thead><tbody><tr><td><u>释放前</u>该页上内存已经全部使用，per cpu partial链表上的空闲可用内存总数 <strong><font color="red">></font></strong> kmem_cache.cpu_partial</td><td>1. 将kmem_cache_cpu的partial链表上的页挂到per node partial。 2.将该内存所在页放回kmem_cache_cpu的partial链表</td></tr><tr><td><u>释放前</u>该页上内存已经全部使用，per cpu partial链表上的空闲可用内存总数 <strong><font color="red"><=</font></strong> kmem_cache.cpu_partial</td><td>将该内存所在页放回kmem_cache_cpu的partial链表</td></tr><tr><td>1. 该页在per node partial 2.<u>释放后</u>，该页呈未分配状态 3.kmem_cache_node.nr_partial <strong><font color="red">></font></strong> kmem_cache.min_partial</td><td><font color="blue">将该页归还给伙伴系统</font></td></tr><tr><td>其他</td><td>/</td></tr></tbody></table><blockquote><p>设置阈值的主要目的是为了避免SLAB占用过多的内存页，导致系统中其他对象想要分配内存时拿不到内存。</p></blockquote><h2 id="总结"><a href="#总结" class="headerlink" title="总结"></a>总结</h2><p>本文介绍了SLAB内存分配器，其是整个系统运行中，起重要且主要作用的内存分配器。介绍了：</p><ul><li>SLAB分配器的分类</li><li>SLUB分配器的架构</li><li>SLUB分配器的逻辑原理</li></ul><p>希望对您分析内核代码有所帮助。</p>]]></content>
<summary type="html">
<blockquote>
<p>这是&lt;Linux内核内存管理&gt;系列的第六篇</p>
<blockquote>
<p><a href="/p/61304bbb.html">第一篇</a>为内核内存管理过程知识点的的简单梳理</p>
<p><a href="/p/1dc6adc2.html">第二篇</a>介绍了内核的数据结构</p>
<p><a href="/p/8f5117c5.html">第三篇</a>介绍了从内核第一行代码加载到跳转到C代码前的内存处理。</p>
<p><a href="/p/94b22c5d.html">第四篇</a>概览了初始化C代码中的内存处理</p>
<p><a href="/p/20210923linuxmm4.html">第五篇（上）</a>和<a href="/p/5347665a.html">第五篇（下）</a>介绍了Memblock和伙伴系统分配器</p>
</blockquote>
<p>为了避免晦涩难懂，本文及之后均主要使用图表+文字描述，尽量避免涉及过多代码。专注点会在：</p>
<ol>
<li>背景</li>
<li>架构及其思想</li>
<li>流程</li>
<li>特殊处理及其原因</li>
</ol>
</blockquote>
</summary>
<category term="Linux" scheme="http://www.l2h.site/categories/Linux/"/>
<category term="Linux内存管理" scheme="http://www.l2h.site/categories/Linux/Linux%E5%86%85%E5%AD%98%E7%AE%A1%E7%90%86/"/>
<category term="Linux" scheme="http://www.l2h.site/tags/Linux/"/>
<category term="内存管理" scheme="http://www.l2h.site/tags/%E5%86%85%E5%AD%98%E7%AE%A1%E7%90%86/"/>
<category term="SLAB" scheme="http://www.l2h.site/tags/SLAB/"/>
<category term="SLUB" scheme="http://www.l2h.site/tags/SLUB/"/>
</entry>
<entry>
<title>Linux内核内存管理 - Memblock和Buddy System(2)</title>
<link href="http://www.l2h.site/p/5347665a.html"/>
<id>http://www.l2h.site/p/5347665a.html</id>
<published>2021-10-03T00:59:16.000Z</published>
<updated>2023-06-01T13:14:04.481Z</updated>
<content type="html"><![CDATA[<blockquote><p>书接前文，本文介绍《Memblock和Buddy System》的第二篇，第一篇见<a href="/p/20210923linuxmm4.html">前文</a></p></blockquote><h2 id="伙伴系统"><a href="#伙伴系统" class="headerlink" title="伙伴系统"></a>伙伴系统</h2><h3 id="Mem-Block向Buddy-System过渡"><a href="#Mem-Block向Buddy-System过渡" class="headerlink" title="Mem Block向Buddy System过渡"></a>Mem Block向Buddy System过渡</h3><p>伙伴系统便是使用页为单位对内存进行管理的方法。伙伴系统接管前，处理建立mem_section结构，也必须先从Mem Block中释放出不再使用的内存交给伙伴系统管理。本文<a href="#%E9%87%8A%E6%94%BE">Figure 2</a>中略有体现，实现这个过渡的函数是memblock_free_all:</p><a id="more"></a><figure class="highlight c"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br></pre></td><td class="code"><pre><span class="line"><span class="function"><span class="keyword">void</span> __init <span class="title">memblock_free_all</span><span class="params">(<span class="keyword">void</span>)</span></span></span><br><span class="line"><span class="function"></span>{</span><br><span class="line"><span class="keyword">unsigned</span> <span class="keyword">long</span> pages;</span><br><span class="line"></span><br><span class="line">free_unused_memmap();</span><br><span class="line">reset_all_zones_managed_pages();</span><br><span class="line">pages = free_low_memory_core_early();</span><br><span class="line">totalram_pages_add(pages);</span><br><span class="line">}</span><br></pre></td></tr></table></figure><ul><li><p><em>free_unused_memmap</em> 释放未使用mem_map内存。</p></li><li><p><em>reset_all_zones_managed_pages</em> 作用是将所有节点所有区域的managed_pages自动设置为0（<strong>managed_pages</strong>表示被伙伴系统管理的页的数量）。</p></li><li><p>_free_low_memory_core_early_主要做两个动作：</p><ul><li><p>将reserve类型的memblock和明确标记为Memory None的内存对应的页做标记为reserved(PG_reserved)</p></li><li><p>将Mem block类型为memory的区域free掉，并标记为Free页面</p></li></ul></li><li><p>_totalram_pages_add_增加 <em>_totalram_pages</em> ，用于标记系统中可用总页数。</p></li></ul><h3 id="管理方式"><a href="#管理方式" class="headerlink" title="管理方式"></a>管理方式</h3><p>伙伴系统的管理方式可以参考<<a href="http://ptgmedia.pearsoncmg.com/images/0131453483/downloads/gorman_book.pdf" target="_blank" rel="noopener">Understanding the Linux® Virtual Memory Manager</a>>的图：</p><p><img src="https://pic.l2h.site/20211004_buddy_system.png" alt="伙伴系统管理图示" title="Figure 1. 伙伴系统管理图示"></p><p>每个内存区域(zone)，都有一个链表数组，数组元素用来存放 $2^{Order}$个页的链表。内存的分配和释放便围绕着这个表来管理。</p><h3 id="数据结构"><a href="#数据结构" class="headerlink" title="数据结构"></a>数据结构</h3><p><a href="/p/1dc6adc2.html">数据结构</a>一文，我们已经介绍的struct page/struct zone/struct pglist_data等数据结构。我们回顾其中部分字段：</p><figure class="highlight c"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br></pre></td><td class="code"><pre><span class="line"><span class="comment">//include/linux/mmzone.h</span></span><br><span class="line"><span class="class"><span class="keyword">struct</span> <span class="title">zone</span> {</span></span><br><span class="line"> ...</span><br><span class="line"><span class="class"><span class="keyword">struct</span> <span class="title">pglist_data</span>*<span class="title">zone_pgdat</span>;</span></span><br><span class="line"> ...</span><br><span class="line"><span class="comment">/* zone_start_pfn == zone_start_paddr >> PAGE_SHIFT */</span></span><br><span class="line"><span class="keyword">unsigned</span> <span class="keyword">long</span>zone_start_pfn;</span><br><span class="line"> ...</span><br><span class="line"><span class="keyword">atomic_long_t</span>managed_pages; <span class="comment">//managed_pages = present_pages - reserved_pages;</span></span><br><span class="line"><span class="keyword">unsigned</span> <span class="keyword">long</span>spanned_pages; <span class="comment">//spanned_pages = zone_end_pfn - zone_start_pfn;</span></span><br><span class="line"><span class="keyword">unsigned</span> <span class="keyword">long</span>present_pages; <span class="comment">//present_pages = spanned_pages - absent_pages(pages in holes);</span></span><br><span class="line"><span class="meta">#<span class="meta-keyword">ifdef</span> CONFIG_CMA</span></span><br><span class="line"><span class="keyword">unsigned</span> <span class="keyword">long</span>cma_pages;</span><br><span class="line"><span class="meta">#<span class="meta-keyword">endif</span></span></span><br><span class="line"> ...</span><br><span class="line"><span class="comment">/* free areas of different sizes */</span></span><br><span class="line"><span class="class"><span class="keyword">struct</span> <span class="title">free_area</span><span class="title">free_area</span>[<span class="title">MAX_ORDER</span>];</span></span><br><span class="line"> ...</span><br><span class="line">} ____cacheline_internodealigned_in_smp;</span><br></pre></td></tr></table></figure><p><strong>zone</strong></p><ul><li>zone_pgdat: 表示该内存区域所在的内存结点</li><li>zone_start_pfn: 表示该内存区域的起始页帧号</li><li>managed_pages: 表示该区域内由伙伴系统管理的页数</li><li>spanned_pages: 表示该区域跨越的总页数</li><li>present_pages：表示该区域内去掉内存空洞的总页数 （含系统保留页）</li><li>free_area: 如前文，存放伙伴系统有关的可用区域。</li></ul><figure class="highlight c"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br><span class="line">21</span><br><span class="line">22</span><br><span class="line">23</span><br><span class="line">24</span><br></pre></td><td class="code"><pre><span class="line"><span class="keyword">typedef</span> <span class="class"><span class="keyword">struct</span> <span class="title">pglist_data</span> {</span></span><br><span class="line"><span class="comment">/*</span></span><br><span class="line"><span class="comment"> * node_zones contains just the zones for THIS node. Not all of the</span></span><br><span class="line"><span class="comment"> * zones may be populated, but it is the full list. It is referenced by</span></span><br><span class="line"><span class="comment"> * this node's node_zonelists as well as other node's node_zonelists.</span></span><br><span class="line"><span class="comment"> */</span></span><br><span class="line"><span class="class"><span class="keyword">struct</span> <span class="title">zone</span> <span class="title">node_zones</span>[<span class="title">MAX_NR_ZONES</span>];</span></span><br><span class="line"><span class="comment">/*</span></span><br><span class="line"><span class="comment"> * node_zonelists contains references to all zones in all nodes.</span></span><br><span class="line"><span class="comment"> * Generally the first zones will be references to this node's</span></span><br><span class="line"><span class="comment"> * node_zones.</span></span><br><span class="line"><span class="comment"> */</span></span><br><span class="line"><span class="class"><span class="keyword">struct</span> <span class="title">zonelist</span> <span class="title">node_zonelists</span>[<span class="title">MAX_ZONELISTS</span>];</span></span><br><span class="line"> ...</span><br><span class="line"><span class="keyword">wait_queue_head_t</span> kswapd_wait;</span><br><span class="line">...</span><br><span class="line"><span class="class"><span class="keyword">struct</span> <span class="title">task_struct</span> *<span class="title">kswapd</span>;</span><span class="comment">/* Protected by</span></span><br><span class="line"><span class="comment"> mem_hotplug_begin/end() */</span></span><br><span class="line"><span class="keyword">int</span> kswapd_order;</span><br><span class="line"><span class="keyword">enum</span> zone_type kswapd_highest_zoneidx;</span><br><span class="line"><span class="keyword">int</span> kswapd_failures;<span class="comment">/* Number of 'reclaimed == 0' runs */</span></span><br><span class="line"><span class="keyword">unsigned</span> <span class="keyword">long</span>totalreserve_pages;</span><br><span class="line"> ...</span><br><span class="line">} <span class="keyword">pg_data_t</span>;</span><br></pre></td></tr></table></figure><p><strong>pglist_data</strong></p><ul><li>node_zones: 该数组存储该内存节点内所有的内存区域</li><li>node_zonelists：保护所有内存节点中所有内存区域的应用</li><li>kswapd_wait、kswapd、kswapd_order等: kswpad线程运行所需字段。</li></ul><h3 id="内存分配"><a href="#内存分配" class="headerlink" title="内存分配"></a>内存分配</h3><p>内存分配使用<strong>alloc_pages*</strong>系列函数，其核心代码__alloc_pages代码如下：</p><figure class="highlight c"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br><span class="line">21</span><br><span class="line">22</span><br><span class="line">23</span><br><span class="line">24</span><br><span class="line">25</span><br><span class="line">26</span><br><span class="line">27</span><br></pre></td><td class="code"><pre><span class="line"><span class="class"><span class="keyword">struct</span> <span class="title">page</span> *__<span class="title">alloc_pages</span>(<span class="title">gfp_t</span> <span class="title">gfp</span>, <span class="title">unsigned</span> <span class="title">int</span> <span class="title">order</span>, <span class="title">int</span> <span class="title">preferred_nid</span>,</span></span><br><span class="line"><span class="class"><span class="title">nodemask_t</span> *<span class="title">nodemask</span>)</span></span><br><span class="line"><span class="class">{</span></span><br><span class="line"> ....</span><br><span class="line"> ....</span><br><span class="line"><span class="keyword">if</span> (!prepare_alloc_pages(gfp, order, preferred_nid, nodemask, &ac,</span><br><span class="line">&alloc_gfp, &alloc_flags))</span><br><span class="line"><span class="keyword">return</span> <span class="literal">NULL</span>;</span><br><span class="line"> ....</span><br><span class="line"><span class="comment">/* First allocation attempt */</span></span><br><span class="line">page = get_page_from_freelist(alloc_gfp, order, alloc_flags, &ac);</span><br><span class="line"><span class="keyword">if</span> (likely(page))</span><br><span class="line"><span class="keyword">goto</span> out;</span><br><span class="line"> ...</span><br><span class="line">page = __alloc_pages_slowpath(alloc_gfp, order, &ac);</span><br><span class="line"></span><br><span class="line">out:</span><br><span class="line"><span class="keyword">if</span> (memcg_kmem_enabled() && (gfp & __GFP_ACCOUNT) && page &&</span><br><span class="line"> unlikely(__memcg_kmem_charge_page(page, gfp, order) != <span class="number">0</span>)) {</span><br><span class="line">__free_pages(page, order);</span><br><span class="line">page = <span class="literal">NULL</span>;</span><br><span class="line">}</span><br><span class="line"></span><br><span class="line">trace_mm_page_alloc(page, order, alloc_gfp, ac.migratetype);</span><br><span class="line"></span><br><span class="line"><span class="keyword">return</span> page;</span><br><span class="line">}</span><br></pre></td></tr></table></figure><p>代码很多，但是核心部分就是下面三个函数：</p><ul><li><strong>prepare_alloc_pages</strong>: 主要作用是，分配前准备页面分配的上下文，特别是选取合适的内存节点的内存区域（Zone）</li></ul><figure class="highlight c"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br><span class="line">21</span><br></pre></td><td class="code"><pre><span class="line"><span class="function"><span class="keyword">static</span> <span class="keyword">inline</span> <span class="keyword">bool</span> <span class="title">prepare_alloc_pages</span><span class="params">(<span class="keyword">gfp_t</span> gfp_mask, <span class="keyword">unsigned</span> <span class="keyword">int</span> order,</span></span></span><br><span class="line"><span class="function"><span class="params"><span class="keyword">int</span> preferred_nid, <span class="keyword">nodemask_t</span> *nodemask,</span></span></span><br><span class="line"><span class="function"><span class="params">struct alloc_context *ac, <span class="keyword">gfp_t</span> *alloc_gfp,</span></span></span><br><span class="line"><span class="function"><span class="params"><span class="keyword">unsigned</span> <span class="keyword">int</span> *alloc_flags)</span></span></span><br><span class="line"><span class="function"></span>{</span><br><span class="line">ac->highest_zoneidx = gfp_zone(gfp_mask);</span><br><span class="line">ac->zonelist = node_zonelist(preferred_nid, gfp_mask);</span><br><span class="line">ac->nodemask = nodemask;</span><br><span class="line">ac->migratetype = gfp_migratetype(gfp_mask);</span><br><span class="line"> </span><br><span class="line"> ........</span><br><span class="line"><span class="comment">/*</span></span><br><span class="line"><span class="comment"> * The preferred zone is used for statistics but crucially it is</span></span><br><span class="line"><span class="comment"> * also used as the starting point for the zonelist iterator. It</span></span><br><span class="line"><span class="comment"> * may get reset for allocations that ignore memory policies.</span></span><br><span class="line"><span class="comment"> */</span></span><br><span class="line">ac->preferred_zoneref = first_zones_zonelist(ac->zonelist,</span><br><span class="line">ac->highest_zoneidx, ac->nodemask);</span><br><span class="line"></span><br><span class="line"><span class="keyword">return</span> <span class="literal">true</span>;</span><br><span class="line">}</span><br></pre></td></tr></table></figure><ul><li><strong>get_page_from_freelist</strong>: 作用是根据传入的分配参数不断尝试来分配内存。同样引用<<a href="http://ptgmedia.pearsoncmg.com/images/0131453483/downloads/gorman_book.pdf" target="_blank" rel="noopener">Understanding the Linux® Virtual Memory Manager</a>>的图，当所需order的页数不足时，会将更大order的free_area拆分来满足返回对应的page指针。当无法分配成功时，则返回NULL。</li></ul><p><img src="https://pic.l2h.site/2021-10-04Buddy_Alloc.png" alt="伙伴系统分配" title="伙伴系统分配"></p><figure class="highlight c"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br><span class="line">21</span><br><span class="line">22</span><br><span class="line">23</span><br><span class="line">24</span><br><span class="line">25</span><br><span class="line">26</span><br><span class="line">27</span><br><span class="line">28</span><br><span class="line">29</span><br><span class="line">30</span><br><span class="line">31</span><br><span class="line">32</span><br><span class="line">33</span><br><span class="line">34</span><br><span class="line">35</span><br><span class="line">36</span><br><span class="line">37</span><br><span class="line">38</span><br><span class="line">39</span><br><span class="line">40</span><br><span class="line">41</span><br><span class="line">42</span><br><span class="line">43</span><br><span class="line">44</span><br><span class="line">45</span><br><span class="line">46</span><br><span class="line">47</span><br><span class="line">48</span><br><span class="line">49</span><br><span class="line">50</span><br><span class="line">51</span><br><span class="line">52</span><br><span class="line">53</span><br><span class="line">54</span><br><span class="line">55</span><br><span class="line">56</span><br><span class="line">57</span><br><span class="line">58</span><br><span class="line">59</span><br><span class="line">60</span><br><span class="line">61</span><br><span class="line">62</span><br><span class="line">63</span><br><span class="line">64</span><br><span class="line">65</span><br><span class="line">66</span><br><span class="line">67</span><br><span class="line">68</span><br><span class="line">69</span><br><span class="line">70</span><br><span class="line">71</span><br><span class="line">72</span><br><span class="line">73</span><br><span class="line">74</span><br><span class="line">75</span><br><span class="line">76</span><br><span class="line">77</span><br><span class="line">78</span><br><span class="line">79</span><br><span class="line">80</span><br><span class="line">81</span><br><span class="line">82</span><br><span class="line">83</span><br><span class="line">84</span><br><span class="line">85</span><br><span class="line">86</span><br><span class="line">87</span><br><span class="line">88</span><br><span class="line">89</span><br><span class="line">90</span><br><span class="line">91</span><br><span class="line">92</span><br><span class="line">93</span><br><span class="line">94</span><br><span class="line">95</span><br><span class="line">96</span><br><span class="line">97</span><br><span class="line">98</span><br><span class="line">99</span><br><span class="line">100</span><br><span class="line">101</span><br><span class="line">102</span><br><span class="line">103</span><br><span class="line">104</span><br><span class="line">105</span><br></pre></td><td class="code"><pre><span class="line"><span class="keyword">static</span> <span class="class"><span class="keyword">struct</span> <span class="title">page</span> *</span></span><br><span class="line"><span class="class"><span class="title">get_page_from_freelist</span>(<span class="title">gfp_t</span> <span class="title">gfp_mask</span>, <span class="title">unsigned</span> <span class="title">int</span> <span class="title">order</span>, <span class="title">int</span> <span class="title">alloc_flags</span>,</span></span><br><span class="line"><span class="class"><span class="title">const</span> <span class="title">struct</span> <span class="title">alloc_context</span> *<span class="title">ac</span>)</span></span><br><span class="line"><span class="class">{</span></span><br><span class="line"><span class="class"><span class="keyword">struct</span> <span class="title">zoneref</span> *<span class="title">z</span>;</span></span><br><span class="line"><span class="class"><span class="keyword">struct</span> <span class="title">zone</span> *<span class="title">zone</span>;</span></span><br><span class="line"><span class="class"><span class="keyword">struct</span> <span class="title">pglist_data</span> *<span class="title">last_pgdat_dirty_limit</span> = <span class="title">NULL</span>;</span></span><br><span class="line"><span class="keyword">bool</span> no_fallback;</span><br><span class="line"></span><br><span class="line">retry:</span><br><span class="line"><span class="comment">/*</span></span><br><span class="line"><span class="comment"> * Scan zonelist, looking for a zone with enough free.</span></span><br><span class="line"><span class="comment"> * See also __cpuset_node_allowed() comment in kernel/cpuset.c.</span></span><br><span class="line"><span class="comment"> */</span></span><br><span class="line">no_fallback = alloc_flags & ALLOC_NOFRAGMENT;</span><br><span class="line">z = ac->preferred_zoneref;</span><br><span class="line">for_next_zone_zonelist_nodemask(zone, z, ac->highest_zoneidx,</span><br><span class="line">ac->nodemask) {</span><br><span class="line"><span class="class"><span class="keyword">struct</span> <span class="title">page</span> *<span class="title">page</span>;</span></span><br><span class="line"><span class="keyword">unsigned</span> <span class="keyword">long</span> mark;</span><br><span class="line"></span><br><span class="line"><span class="keyword">if</span> (cpusets_enabled() &&</span><br><span class="line">(alloc_flags & ALLOC_CPUSET) &&</span><br><span class="line">!__cpuset_zone_allowed(zone, gfp_mask))</span><br><span class="line"><span class="keyword">continue</span>;</span><br><span class="line"><span class="keyword">if</span> (ac->spread_dirty_pages) {</span><br><span class="line"><span class="keyword">if</span> (last_pgdat_dirty_limit == zone->zone_pgdat)</span><br><span class="line"><span class="keyword">continue</span>;</span><br><span class="line"></span><br><span class="line"><span class="keyword">if</span> (!node_dirty_ok(zone->zone_pgdat)) {</span><br><span class="line">last_pgdat_dirty_limit = zone->zone_pgdat;</span><br><span class="line"><span class="keyword">continue</span>;</span><br><span class="line">}</span><br><span class="line">}</span><br><span class="line"></span><br><span class="line"><span class="keyword">if</span> (no_fallback && nr_online_nodes > <span class="number">1</span> &&</span><br><span class="line"> zone != ac->preferred_zoneref->zone) {</span><br><span class="line"><span class="keyword">int</span> local_nid;</span><br><span class="line"></span><br><span class="line"><span class="comment">/*</span></span><br><span class="line"><span class="comment"> * If moving to a remote node, retry but allow</span></span><br><span class="line"><span class="comment"> * fragmenting fallbacks. Locality is more important</span></span><br><span class="line"><span class="comment"> * than fragmentation avoidance.</span></span><br><span class="line"><span class="comment"> */</span></span><br><span class="line">local_nid = zone_to_nid(ac->preferred_zoneref->zone);</span><br><span class="line"><span class="keyword">if</span> (zone_to_nid(zone) != local_nid) {</span><br><span class="line">alloc_flags &= ~ALLOC_NOFRAGMENT;</span><br><span class="line"><span class="keyword">goto</span> retry;</span><br><span class="line">}</span><br><span class="line">}</span><br><span class="line"></span><br><span class="line">mark = wmark_pages(zone, alloc_flags & ALLOC_WMARK_MASK);</span><br><span class="line"><span class="keyword">if</span> (!zone_watermark_fast(zone, order, mark,</span><br><span class="line"> ac->highest_zoneidx, alloc_flags,</span><br><span class="line"> gfp_mask)) {</span><br><span class="line"><span class="keyword">int</span> ret;</span><br><span class="line"></span><br><span class="line"> ...</span><br><span class="line"><span class="comment">/* Checked here to keep the fast path fast */</span></span><br><span class="line">BUILD_BUG_ON(ALLOC_NO_WATERMARKS < NR_WMARK);</span><br><span class="line"><span class="keyword">if</span> (alloc_flags & ALLOC_NO_WATERMARKS)</span><br><span class="line"><span class="keyword">goto</span> try_this_zone;</span><br><span class="line"></span><br><span class="line"><span class="keyword">if</span> (!node_reclaim_enabled() ||</span><br><span class="line"> !zone_allows_reclaim(ac->preferred_zoneref->zone, zone))</span><br><span class="line"><span class="keyword">continue</span>;</span><br><span class="line"></span><br><span class="line">ret = node_reclaim(zone->zone_pgdat, gfp_mask, order);</span><br><span class="line"><span class="keyword">switch</span> (ret) {</span><br><span class="line"><span class="keyword">case</span> NODE_RECLAIM_NOSCAN:</span><br><span class="line"><span class="comment">/* did not scan */</span></span><br><span class="line"><span class="keyword">continue</span>;</span><br><span class="line"><span class="keyword">case</span> NODE_RECLAIM_FULL:</span><br><span class="line"><span class="comment">/* scanned but unreclaimable */</span></span><br><span class="line"><span class="keyword">continue</span>;</span><br><span class="line"><span class="keyword">default</span>:</span><br><span class="line"><span class="comment">/* did we reclaim enough */</span></span><br><span class="line"><span class="keyword">if</span> (zone_watermark_ok(zone, order, mark,</span><br><span class="line">ac->highest_zoneidx, alloc_flags))</span><br><span class="line"><span class="keyword">goto</span> try_this_zone;</span><br><span class="line"></span><br><span class="line"><span class="keyword">continue</span>;</span><br><span class="line">}</span><br><span class="line">}</span><br><span class="line"></span><br><span class="line">try_this_zone:</span><br><span class="line">page = rmqueue(ac->preferred_zoneref->zone, zone, order,</span><br><span class="line">gfp_mask, alloc_flags, ac->migratetype);</span><br><span class="line"><span class="keyword">if</span> (page) {</span><br><span class="line">prep_new_page(page, order, gfp_mask, alloc_flags);</span><br><span class="line"></span><br><span class="line"><span class="keyword">if</span> (unlikely(order && (alloc_flags & ALLOC_HARDER)))</span><br><span class="line">reserve_highatomic_pageblock(page, zone, order);</span><br><span class="line"><span class="keyword">return</span> page;</span><br><span class="line">} <span class="keyword">else</span> {</span><br><span class="line"> ...</span><br><span class="line">}</span><br><span class="line">}</span><br><span class="line"><span class="keyword">if</span> (no_fallback) {</span><br><span class="line">alloc_flags &= ~ALLOC_NOFRAGMENT;</span><br><span class="line"><span class="keyword">goto</span> retry;</span><br><span class="line">}</span><br><span class="line"></span><br><span class="line"><span class="keyword">return</span> <span class="literal">NULL</span>;</span><br><span class="line">}</span><br></pre></td></tr></table></figure><ul><li><strong>__alloc_pages_slowpath</strong>:如果get_page_from_freelist分配页面失败，则进行慢速分配。这个函数会尝试回收内存，采用以下顺序：<ul><li>触发kswapd尝试回收内存。</li><li>如果回收失败，则尝试杀掉进程回收内存。</li></ul></li></ul><h3 id="内存释放"><a href="#内存释放" class="headerlink" title="内存释放"></a>内存释放</h3><p>内存释放最后会调用到*<em>__free_one_page *</em>：释放过程比较容易来讲，找到可以合并的Buddy页帧号向上一级Order合并直到不能合并，将合并好的页加入到对应Order的free_area。</p><figure class="highlight c"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br><span class="line">21</span><br><span class="line">22</span><br><span class="line">23</span><br><span class="line">24</span><br><span class="line">25</span><br><span class="line">26</span><br><span class="line">27</span><br><span class="line">28</span><br><span class="line">29</span><br><span class="line">30</span><br><span class="line">31</span><br><span class="line">32</span><br><span class="line">33</span><br><span class="line">34</span><br><span class="line">35</span><br><span class="line">36</span><br><span class="line">37</span><br><span class="line">38</span><br><span class="line">39</span><br><span class="line">40</span><br><span class="line">41</span><br><span class="line">42</span><br><span class="line">43</span><br><span class="line">44</span><br><span class="line">45</span><br><span class="line">46</span><br><span class="line">47</span><br><span class="line">48</span><br><span class="line">49</span><br><span class="line">50</span><br><span class="line">51</span><br><span class="line">52</span><br><span class="line">53</span><br><span class="line">54</span><br><span class="line">55</span><br><span class="line">56</span><br><span class="line">57</span><br><span class="line">58</span><br><span class="line">59</span><br><span class="line">60</span><br><span class="line">61</span><br><span class="line">62</span><br><span class="line">63</span><br><span class="line">64</span><br><span class="line">65</span><br><span class="line">66</span><br><span class="line">67</span><br><span class="line">68</span><br><span class="line">69</span><br><span class="line">70</span><br><span class="line">71</span><br><span class="line">72</span><br><span class="line">73</span><br><span class="line">74</span><br><span class="line">75</span><br><span class="line">76</span><br><span class="line">77</span><br><span class="line">78</span><br></pre></td><td class="code"><pre><span class="line"><span class="keyword">static</span> <span class="keyword">inline</span> <span class="keyword">void</span> __free_one_page(struct page *page,</span><br><span class="line"><span class="keyword">unsigned</span> <span class="keyword">long</span> pfn,</span><br><span class="line">struct zone *zone, <span class="keyword">unsigned</span> <span class="keyword">int</span> order,</span><br><span class="line"><span class="keyword">int</span> migratetype, <span class="keyword">fpi_t</span> fpi_flags)</span><br><span class="line">{ .....</span><br><span class="line">max_order = <span class="keyword">min_t</span>(<span class="keyword">unsigned</span> <span class="keyword">int</span>, MAX_ORDER - <span class="number">1</span>, pageblock_order);</span><br><span class="line"> .....</span><br><span class="line">continue_merging:</span><br><span class="line"><span class="keyword">while</span> (order < max_order) {</span><br><span class="line"><span class="keyword">if</span> (compaction_capture(capc, page, order, migratetype)) {</span><br><span class="line">__mod_zone_freepage_state(zone, -(<span class="number">1</span> << order),</span><br><span class="line">migratetype);</span><br><span class="line"><span class="keyword">return</span>;</span><br><span class="line">}</span><br><span class="line">buddy_pfn = __find_buddy_pfn(pfn, order);</span><br><span class="line">buddy = page + (buddy_pfn - pfn);</span><br><span class="line"></span><br><span class="line"><span class="keyword">if</span> (!pfn_valid_within(buddy_pfn))</span><br><span class="line"><span class="keyword">goto</span> done_merging;</span><br><span class="line"><span class="keyword">if</span> (!page_is_buddy(page, buddy, order))</span><br><span class="line"><span class="keyword">goto</span> done_merging;</span><br><span class="line"><span class="comment">/*</span></span><br><span class="line"><span class="comment"> * Our buddy is free or it is CONFIG_DEBUG_PAGEALLOC guard page,</span></span><br><span class="line"><span class="comment"> * merge with it and move up one order.</span></span><br><span class="line"><span class="comment"> */</span></span><br><span class="line"><span class="keyword">if</span> (page_is_guard(buddy))</span><br><span class="line">clear_page_guard(zone, buddy, order, migratetype);</span><br><span class="line"><span class="keyword">else</span></span><br><span class="line">del_page_from_free_list(buddy, zone, order);</span><br><span class="line">combined_pfn = buddy_pfn & pfn;</span><br><span class="line">page = page + (combined_pfn - pfn);</span><br><span class="line">pfn = combined_pfn;</span><br><span class="line">order++;</span><br><span class="line">}</span><br><span class="line"><span class="keyword">if</span> (order < MAX_ORDER - <span class="number">1</span>) {</span><br><span class="line"><span class="comment">/* If we are here, it means order is >= pageblock_order.</span></span><br><span class="line"><span class="comment"> * We want to prevent merge between freepages on isolate</span></span><br><span class="line"><span class="comment"> * pageblock and normal pageblock. Without this, pageblock</span></span><br><span class="line"><span class="comment"> * isolation could cause incorrect freepage or CMA accounting.</span></span><br><span class="line"><span class="comment"> *</span></span><br><span class="line"><span class="comment"> * We don't want to hit this code for the more frequent</span></span><br><span class="line"><span class="comment"> * low-order merging.</span></span><br><span class="line"><span class="comment"> */</span></span><br><span class="line"><span class="keyword">if</span> (unlikely(has_isolate_pageblock(zone))) {</span><br><span class="line"><span class="keyword">int</span> buddy_mt;</span><br><span class="line"></span><br><span class="line">buddy_pfn = __find_buddy_pfn(pfn, order);</span><br><span class="line">buddy = page + (buddy_pfn - pfn);</span><br><span class="line">buddy_mt = get_pageblock_migratetype(buddy);</span><br><span class="line"></span><br><span class="line"><span class="keyword">if</span> (migratetype != buddy_mt</span><br><span class="line">&& (is_migrate_isolate(migratetype) ||</span><br><span class="line">is_migrate_isolate(buddy_mt)))</span><br><span class="line"><span class="keyword">goto</span> done_merging;</span><br><span class="line">}</span><br><span class="line">max_order = order + <span class="number">1</span>;</span><br><span class="line"><span class="keyword">goto</span> continue_merging;</span><br><span class="line">}</span><br><span class="line"></span><br><span class="line">done_merging:</span><br><span class="line">set_buddy_order(page, order);</span><br><span class="line"></span><br><span class="line"><span class="keyword">if</span> (fpi_flags & FPI_TO_TAIL)</span><br><span class="line">to_tail = <span class="literal">true</span>;</span><br><span class="line"><span class="keyword">else</span> <span class="keyword">if</span> (is_shuffle_order(order))</span><br><span class="line">to_tail = shuffle_pick_tail();</span><br><span class="line"><span class="keyword">else</span></span><br><span class="line">to_tail = buddy_merge_likely(pfn, buddy_pfn, page, order);</span><br><span class="line"></span><br><span class="line"><span class="keyword">if</span> (to_tail)</span><br><span class="line">add_to_free_list_tail(page, zone, order, migratetype);</span><br><span class="line"><span class="keyword">else</span></span><br><span class="line">add_to_free_list(page, zone, order, migratetype);</span><br><span class="line"></span><br><span class="line"><span class="comment">/* Notify page reporting subsystem of freed page */</span></span><br><span class="line"><span class="keyword">if</span> (!(fpi_flags & FPI_SKIP_REPORT_NOTIFY))</span><br><span class="line">page_reporting_notify_free(order);</span><br><span class="line">}</span><br></pre></td></tr></table></figure><h3 id="kswapd"><a href="#kswapd" class="headerlink" title="kswapd"></a>kswapd</h3><p>见<a href="#内存分配">内存分配</a>一节，_alloc_pages_slowpath会触发kswapd来回收内存。kswapd在每个内存节点都有一个，其定义和代码如下。其实这里就是调用了balance_pgdat进行内存回收。</p><figure class="highlight c"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br><span class="line">21</span><br><span class="line">22</span><br><span class="line">23</span><br><span class="line">24</span><br><span class="line">25</span><br><span class="line">26</span><br><span class="line">27</span><br><span class="line">28</span><br><span class="line">29</span><br><span class="line">30</span><br><span class="line">31</span><br><span class="line">32</span><br><span class="line">33</span><br><span class="line">34</span><br><span class="line">35</span><br><span class="line">36</span><br><span class="line">37</span><br></pre></td><td class="code"><pre><span class="line"><span class="function"><span class="keyword">int</span> <span class="title">kswapd_run</span><span class="params">(<span class="keyword">int</span> nid)</span></span></span><br><span class="line"><span class="function"></span>{</span><br><span class="line"><span class="keyword">pg_data_t</span> *pgdat = NODE_DATA(nid);</span><br><span class="line"> ...</span><br><span class="line">pgdat->kswapd = kthread_run(kswapd, pgdat, <span class="string">"kswapd%d"</span>, nid);</span><br><span class="line"> ...</span><br><span class="line">}</span><br><span class="line"><span class="function"><span class="keyword">static</span> <span class="keyword">int</span> <span class="title">kswapd</span><span class="params">(<span class="keyword">void</span> *p)</span></span></span><br><span class="line"><span class="function"></span>{</span><br><span class="line"> ...</span><br><span class="line"><span class="keyword">for</span> ( ; ; ) {</span><br><span class="line"><span class="keyword">bool</span> ret;</span><br><span class="line"></span><br><span class="line">alloc_order = reclaim_order = READ_ONCE(pgdat->kswapd_order);</span><br><span class="line">highest_zoneidx = kswapd_highest_zoneidx(pgdat,</span><br><span class="line">highest_zoneidx);</span><br><span class="line"></span><br><span class="line">kswapd_try_sleep:</span><br><span class="line">kswapd_try_to_sleep(pgdat, alloc_order, reclaim_order,</span><br><span class="line">highest_zoneidx);</span><br><span class="line"> ......</span><br><span class="line">ret = try_to_freeze();</span><br><span class="line"><span class="keyword">if</span> (kthread_should_stop())</span><br><span class="line"><span class="keyword">break</span>;</span><br><span class="line"> ......</span><br><span class="line">trace_mm_vmscan_kswapd_wake(pgdat->node_id, highest_zoneidx,</span><br><span class="line">alloc_order);</span><br><span class="line">reclaim_order = balance_pgdat(pgdat, alloc_order,</span><br><span class="line">highest_zoneidx);</span><br><span class="line"><span class="keyword">if</span> (reclaim_order < alloc_order)</span><br><span class="line"><span class="keyword">goto</span> kswapd_try_sleep;</span><br><span class="line">}</span><br><span class="line"></span><br><span class="line">tsk->flags &= ~(PF_MEMALLOC | PF_SWAPWRITE | PF_KSWAPD);</span><br><span class="line"></span><br><span class="line"><span class="keyword">return</span> <span class="number">0</span>;</span><br><span class="line">}</span><br></pre></td></tr></table></figure><h2 id="总结"><a href="#总结" class="headerlink" title="总结"></a>总结</h2><p>本文是自己学习Linux内存管理的简单梳理，介绍了：</p><ul><li>Linux初始化早期的Memblock内存管理</li><li>Linux物理内存模型</li><li>Linux伙伴系统（Buddy Allocator）</li></ul><p>希望也对您理解Linux的内存管理有一些帮助。</p><h2 id="参考资料"><a href="#参考资料" class="headerlink" title="参考资料"></a>参考资料</h2><ol><li>内核代码. <a href="https://elixir.bootlin.com/linux/latest/source/" target="_blank" rel="noopener">https://elixir.bootlin.com/linux/latest/source/</a>.</li><li>内核手册. <a href="https://www.kernel.org/doc/html/latest/vm/index.html" target="_blank" rel="noopener">https://www.kernel.org/doc/html/latest/vm/index.html</a>.</li><li>Understanding the Linux® Virtual Memory Manager. Mel Gorman. <a href="http://ptgmedia.pearsoncmg.com/images/0131453483/downloads/gorman_book.pdf" target="_blank" rel="noopener">http://ptgmedia.pearsoncmg.com/images/0131453483/downloads/gorman_book.pdf</a>.</li></ol>]]></content>
<summary type="html">
<blockquote>
<p>书接前文，本文介绍《Memblock和Buddy System》的第二篇，第一篇见<a href="/p/20210923linuxmm4.html">前文</a></p>
</blockquote><h2 id="伙伴系统"><a href="#伙伴系统" class="headerlink" title="伙伴系统"></a>伙伴系统</h2><h3 id="Mem-Block向Buddy-System过渡"><a href="#Mem-Block向Buddy-System过渡" class="headerlink" title="Mem Block向Buddy System过渡"></a>Mem Block向Buddy System过渡</h3><p>伙伴系统便是使用页为单位对内存进行管理的方法。伙伴系统接管前，处理建立mem_section结构，也必须先从Mem Block中释放出不再使用的内存交给伙伴系统管理。本文<a href="#%E9%87%8A%E6%94%BE">Figure 2</a>中略有体现，实现这个过渡的函数是memblock_free_all:</p>
</summary>
<category term="Linux" scheme="http://www.l2h.site/categories/Linux/"/>
<category term="Linux内存管理" scheme="http://www.l2h.site/categories/Linux/Linux%E5%86%85%E5%AD%98%E7%AE%A1%E7%90%86/"/>
<category term="Linux" scheme="http://www.l2h.site/tags/Linux/"/>
<category term="内存管理" scheme="http://www.l2h.site/tags/%E5%86%85%E5%AD%98%E7%AE%A1%E7%90%86/"/>
<category term="Buddy System" scheme="http://www.l2h.site/tags/Buddy-System/"/>
<category term="伙伴系统" scheme="http://www.l2h.site/tags/%E4%BC%99%E4%BC%B4%E7%B3%BB%E7%BB%9F/"/>
<category term="Memblock" scheme="http://www.l2h.site/tags/Memblock/"/>
</entry>
<entry>
<title>Linux内核内存管理 - Memblock和Buddy System(1)</title>
<link href="http://www.l2h.site/p/20210923linuxmm4.html"/>
<id>http://www.l2h.site/p/20210923linuxmm4.html</id>
<published>2021-09-23T11:27:15.000Z</published>
<updated>2023-06-01T13:14:04.481Z</updated>
<content type="html"><![CDATA[<blockquote><p>这是<Linux内核内存管理>系列的第四篇</p><blockquote><p><a href="/p/61304bbb.html">第一篇</a>为内核内存管理过程知识点的的简单梳理</p><p><a href="/p/1dc6adc2.html">第二篇</a>介绍了内核的数据结构</p><p><a href="/p/8f5117c5.html">第三篇</a>介绍了从内核第一行代码加载到跳转到C代码前的内存处理。</p><p><a href="/p/94b22c5d.html">第四篇</a>概览了初始化C代码中的内存处理</p></blockquote></blockquote><h2 id="概述"><a href="#概述" class="headerlink" title="概述"></a>概述</h2><p>参考<a href="https://www.kernel.org/doc/html/latest/core-api/boot-time-mm.html" target="_blank" rel="noopener">内核文档</a>，系统初始化早期是不能使用我们常用的 <em>kmalloc</em> ，<em>vmalloc</em> 等函数，这是因为此时对应的功能还没初始化好。</p><a id="more"></a><p>尽管如此，早期初始化仍然需要分配内存。因此早期内核提供了基于bitmap的Bootmem分配器，后续逐渐演进成了现在的Memblock。<br>Memblock或者早期的Bootmem并不能适应系统运行时的各种复杂场景（多线程、碎片等）。因此在内核启动到一定阶段后，内存管理的工作会交由伙伴系统(Buddy System)接管。 </p><blockquote><p>当然也并非是仅由伙伴系统管理。除了伙伴系统来以页为单位进行内存分配，还会有SLAB系统的某种（一般是SLUB）来实现对小内存分配的管理。</p></blockquote><h2 id="Memblock"><a href="#Memblock" class="headerlink" title="Memblock"></a>Memblock</h2><p>Memblock将系统中的内存分为一系列不同类型的连续区域。主要有以下几个类型：</p><ul><li><strong>memory</strong>：用于描述当前内核可用的物理内存。</li><li><strong>reserved</strong>: 用于描述不可用内存（已分配）。</li><li><strong>physmem</strong>：特殊体系结构才有，此处不表。</li></ul><p>Memblock使用以上概念对内核启动早期内存分配进行管理。</p><h3 id="数据结构"><a href="#数据结构" class="headerlink" title="数据结构"></a>数据结构</h3><p>Memblock的内存主要数据结构如下：</p><figure class="highlight c"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br><span class="line">21</span><br></pre></td><td class="code"><pre><span class="line"><span class="class"><span class="keyword">struct</span> <span class="title">memblock_region</span> {</span></span><br><span class="line"><span class="keyword">phys_addr_t</span> base;</span><br><span class="line"><span class="keyword">phys_addr_t</span> <span class="built_in">size</span>;</span><br><span class="line"><span class="keyword">enum</span> memblock_flags flags;</span><br><span class="line"><span class="meta">#<span class="meta-keyword">ifdef</span> CONFIG_NUMA</span></span><br><span class="line"><span class="keyword">int</span> nid;</span><br><span class="line"><span class="meta">#<span class="meta-keyword">endif</span></span></span><br><span class="line">};</span><br><span class="line"><span class="class"><span class="keyword">struct</span> <span class="title">memblock_type</span> {</span></span><br><span class="line"><span class="keyword">unsigned</span> <span class="keyword">long</span> cnt;</span><br><span class="line"><span class="keyword">unsigned</span> <span class="keyword">long</span> <span class="built_in">max</span>;</span><br><span class="line"><span class="keyword">phys_addr_t</span> total_size;</span><br><span class="line"><span class="class"><span class="keyword">struct</span> <span class="title">memblock_region</span> *<span class="title">regions</span>;</span></span><br><span class="line"><span class="keyword">char</span> *name;</span><br><span class="line">};</span><br><span class="line"><span class="class"><span class="keyword">struct</span> <span class="title">memblock</span> {</span></span><br><span class="line"><span class="keyword">bool</span> bottom_up; <span class="comment">/* is bottom up direction? */</span></span><br><span class="line"><span class="keyword">phys_addr_t</span> current_limit;</span><br><span class="line"><span class="class"><span class="keyword">struct</span> <span class="title">memblock_type</span> <span class="title">memory</span>;</span></span><br><span class="line"><span class="class"><span class="keyword">struct</span> <span class="title">memblock_type</span> <span class="title">reserved</span>;</span></span><br><span class="line">};</span><br></pre></td></tr></table></figure><ul><li><strong>memblock_region</strong>: 代表一段物理内存区域。<ul><li><strong>base</strong>：表示区域物理起始物理地址</li><li><strong>size</strong>：表示区域大小</li><li><strong>flags</strong>：区域标记，主要有（HOTPLUG, MIRROR, NOMAP），分别表示区域是否是热插拔，镜像区域和是否加入到内核直接映射区。</li><li><strong>nid</strong>: 如果开启了NUMA（Non Unified Memory Access）</li></ul></li><li><strong>memblock_type</strong>: 表示某种内存类型的集合。如<a href="#Memblock">前文</a>所述，目前主要有Memory和Reserved。<ul><li><strong>cnt</strong>：表示该memblock_type的个数</li><li><strong>max</strong>: 此type内区域的数量（即<strong>regions</strong>链表的元素个数）</li><li><strong>total_size</strong>: 此类内存区域内存的总大小</li><li><strong>regions</strong>：所有此类区域内存的链表</li><li><strong>name</strong>: 该类型的符号名</li></ul></li><li><strong>memblock</strong>: 管理整个Memblock的数据结构<ul><li><strong>bottom_up</strong>: 内存分配的方向，是否从底向上</li><li><strong>current_limit</strong>：Memblock分配器管理的物理内存地址的上限</li><li><strong>memory</strong>: memory类型</li><li><strong>reserved</strong>: memory类型</li></ul></li></ul><p>一张图说明以上结构的关系：</p><h3 id="分配和释放"><a href="#分配和释放" class="headerlink" title="分配和释放"></a>分配和释放</h3><p>Memblock分配和释放函数主要有以下：</p><ul><li><strong>memblock_alloc</strong>: 分配内存，主要有(memblock_alloc_range_nid, memblock_alloc_raw, memblock_alloc_from等)</li><li><strong>memblock_add</strong>: 分配内存区域</li><li><strong>memblock_add_node</strong>: 在指定NUMA上分配内存区域</li><li><strong>memblock_add_range</strong>: 在指定NUMA区域内分配指定类型和FLAG标记的内存区域</li><li><strong>memblock_remove</strong>: 删除一个内存区域</li><li><strong>memblock_remove_range</strong>: 删除一个指定类型的内存区域</li><li><strong>memblock_remove_region</strong>: 删除指定内存区域</li><li><strong>memblock_free</strong>: 删除内存区域， 主要有(memblock_free_early, memblock_free_early_nid等)</li><li><strong>memblock_reserve</strong>: 将指定区域设定为保留区域</li></ul><p>以上API名称很像，但其实最终只是对其中某几个API的封装。</p><h4 id="分配"><a href="#分配" class="headerlink" title="分配"></a>分配</h4><p>从如下alloc系函数调用关系可以看到，最终调用到memblock_add_range。</p><p><img src="/images/posts/20210923/Memblock_Alloc.png" alt="Memblock分配函数" title="Figure 1. Memblock分配函数调用"></p><p>从不同分支进到分配函数memblock_add_range，差异仅在与分配时选取的NUMA Node ID、标记（Flag）、类型（Memory还是Reserved）等参数的不同。其具体代码如下：</p><figure class="highlight c"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br><span class="line">21</span><br><span class="line">22</span><br><span class="line">23</span><br><span class="line">24</span><br><span class="line">25</span><br><span class="line">26</span><br><span class="line">27</span><br><span class="line">28</span><br><span class="line">29</span><br><span class="line">30</span><br><span class="line">31</span><br><span class="line">32</span><br><span class="line">33</span><br><span class="line">34</span><br><span class="line">35</span><br><span class="line">36</span><br><span class="line">37</span><br><span class="line">38</span><br><span class="line">39</span><br><span class="line">40</span><br><span class="line">41</span><br><span class="line">42</span><br><span class="line">43</span><br><span class="line">44</span><br><span class="line">45</span><br><span class="line">46</span><br><span class="line">47</span><br><span class="line">48</span><br><span class="line">49</span><br><span class="line">50</span><br><span class="line">51</span><br><span class="line">52</span><br><span class="line">53</span><br><span class="line">54</span><br><span class="line">55</span><br><span class="line">56</span><br><span class="line">57</span><br><span class="line">58</span><br><span class="line">59</span><br><span class="line">60</span><br><span class="line">61</span><br><span class="line">62</span><br><span class="line">63</span><br><span class="line">64</span><br><span class="line">65</span><br><span class="line">66</span><br><span class="line">67</span><br><span class="line">68</span><br><span class="line">69</span><br><span class="line">70</span><br><span class="line">71</span><br><span class="line">72</span><br><span class="line">73</span><br><span class="line">74</span><br><span class="line">75</span><br><span class="line">76</span><br><span class="line">77</span><br><span class="line">78</span><br><span class="line">79</span><br><span class="line">80</span><br><span class="line">81</span><br><span class="line">82</span><br><span class="line">83</span><br><span class="line">84</span><br><span class="line">85</span><br></pre></td><td class="code"><pre><span class="line"><span class="function"><span class="keyword">static</span> <span class="keyword">int</span> __init_memblock <span class="title">memblock_add_range</span><span class="params">(struct memblock_type *type,</span></span></span><br><span class="line"><span class="function"><span class="params"><span class="keyword">phys_addr_t</span> base, <span class="keyword">phys_addr_t</span> <span class="built_in">size</span>,</span></span></span><br><span class="line"><span class="function"><span class="params"><span class="keyword">int</span> nid, <span class="keyword">enum</span> memblock_flags flags)</span></span></span><br><span class="line"><span class="function"></span>{</span><br><span class="line"><span class="keyword">bool</span> insert = <span class="literal">false</span>;</span><br><span class="line"><span class="keyword">phys_addr_t</span> obase = base;</span><br><span class="line"><span class="keyword">phys_addr_t</span> <span class="built_in">end</span> = base + memblock_cap_size(base, &<span class="built_in">size</span>);</span><br><span class="line"><span class="keyword">int</span> idx, nr_new;</span><br><span class="line"><span class="class"><span class="keyword">struct</span> <span class="title">memblock_region</span> *<span class="title">rgn</span>;</span></span><br><span class="line"></span><br><span class="line"><span class="keyword">if</span> (!<span class="built_in">size</span>)</span><br><span class="line"><span class="keyword">return</span> <span class="number">0</span>;</span><br><span class="line"></span><br><span class="line"><span class="comment">/* special case for empty array */</span></span><br><span class="line"><span class="keyword">if</span> (type->regions[<span class="number">0</span>].<span class="built_in">size</span> == <span class="number">0</span>) {</span><br><span class="line">WARN_ON(type->cnt != <span class="number">1</span> || type->total_size);</span><br><span class="line">type->regions[<span class="number">0</span>].base = base;</span><br><span class="line">type->regions[<span class="number">0</span>].<span class="built_in">size</span> = <span class="built_in">size</span>;</span><br><span class="line">type->regions[<span class="number">0</span>].flags = flags;</span><br><span class="line">memblock_set_region_node(&type->regions[<span class="number">0</span>], nid);</span><br><span class="line">type->total_size = <span class="built_in">size</span>;</span><br><span class="line"><span class="keyword">return</span> <span class="number">0</span>;</span><br><span class="line">}</span><br><span class="line">repeat:</span><br><span class="line"><span class="comment">/*</span></span><br><span class="line"><span class="comment"> * The following is executed twice. Once with %false @insert and</span></span><br><span class="line"><span class="comment"> * then with %true. The first counts the number of regions needed</span></span><br><span class="line"><span class="comment"> * to accommodate the new area. The second actually inserts them.</span></span><br><span class="line"><span class="comment"> */</span></span><br><span class="line">base = obase;</span><br><span class="line">nr_new = <span class="number">0</span>;</span><br><span class="line"></span><br><span class="line">for_each_memblock_type(idx, type, rgn) {</span><br><span class="line"><span class="keyword">phys_addr_t</span> rbase = rgn->base;</span><br><span class="line"><span class="keyword">phys_addr_t</span> rend = rbase + rgn-><span class="built_in">size</span>;</span><br><span class="line"></span><br><span class="line"><span class="keyword">if</span> (rbase >= <span class="built_in">end</span>)</span><br><span class="line"><span class="keyword">break</span>;</span><br><span class="line"><span class="keyword">if</span> (rend <= base)</span><br><span class="line"><span class="keyword">continue</span>;</span><br><span class="line"><span class="comment">/*</span></span><br><span class="line"><span class="comment"> * @rgn overlaps. If it separates the lower part of new</span></span><br><span class="line"><span class="comment"> * area, insert that portion.</span></span><br><span class="line"><span class="comment"> */</span></span><br><span class="line"><span class="keyword">if</span> (rbase > base) {</span><br><span class="line"><span class="meta">#<span class="meta-keyword">ifdef</span> CONFIG_NUMA</span></span><br><span class="line">WARN_ON(nid != memblock_get_region_node(rgn));</span><br><span class="line"><span class="meta">#<span class="meta-keyword">endif</span></span></span><br><span class="line">WARN_ON(flags != rgn->flags);</span><br><span class="line">nr_new++;</span><br><span class="line"><span class="keyword">if</span> (insert)</span><br><span class="line">memblock_insert_region(type, idx++, base,</span><br><span class="line"> rbase - base, nid,</span><br><span class="line"> flags);</span><br><span class="line">}</span><br><span class="line"><span class="comment">/* area below @rend is dealt with, forget about it */</span></span><br><span class="line">base = <span class="built_in">min</span>(rend, <span class="built_in">end</span>);</span><br><span class="line">}</span><br><span class="line"></span><br><span class="line"><span class="comment">/* insert the remaining portion */</span></span><br><span class="line"><span class="keyword">if</span> (base < <span class="built_in">end</span>) {</span><br><span class="line">nr_new++;</span><br><span class="line"><span class="keyword">if</span> (insert)</span><br><span class="line">memblock_insert_region(type, idx, base, <span class="built_in">end</span> - base,</span><br><span class="line"> nid, flags);</span><br><span class="line">}</span><br><span class="line"></span><br><span class="line"><span class="keyword">if</span> (!nr_new)</span><br><span class="line"><span class="keyword">return</span> <span class="number">0</span>;</span><br><span class="line"></span><br><span class="line"><span class="comment">/*</span></span><br><span class="line"><span class="comment"> * If this was the first round, resize array and repeat for actual</span></span><br><span class="line"><span class="comment"> * insertions; otherwise, merge and return.</span></span><br><span class="line"><span class="comment"> */</span></span><br><span class="line"><span class="keyword">if</span> (!insert) {</span><br><span class="line"><span class="keyword">while</span> (type->cnt + nr_new > type-><span class="built_in">max</span>)</span><br><span class="line"><span class="keyword">if</span> (memblock_double_array(type, obase, <span class="built_in">size</span>) < <span class="number">0</span>)</span><br><span class="line"><span class="keyword">return</span> -ENOMEM;</span><br><span class="line">insert = <span class="literal">true</span>;</span><br><span class="line"><span class="keyword">goto</span> repeat;</span><br><span class="line">} <span class="keyword">else</span> {</span><br><span class="line">memblock_merge_regions(type);</span><br><span class="line"><span class="keyword">return</span> <span class="number">0</span>;</span><br><span class="line">}</span><br><span class="line">}</span><br></pre></td></tr></table></figure><p>代码虽长，其实比较容易理解：</p><ul><li>首先如果对应类型类型还没有任何内存区域，便直接在对应分配所要求的的内存区域。</li><li>如果该类型区域非空，就需要遍历所有内存区域，确定待加入区域是否与已存在区域重合。据此会有三种处理：<ul><li>如果待加入区域与现存区域无重叠，则直接添加此区域</li><li>如果待加入区域与现存区域有重叠且并未被现存区域完整包含，则将待加入区域与现存区域进行合并</li><li>如果待加入区域被现存区域完全包含，则不用重新添加该区域</li></ul></li></ul><h4 id="释放"><a href="#释放" class="headerlink" title="释放"></a>释放</h4><p>另外可以从如下free系函数调用关系的简图看到，memblock_free*最终调用到memblock_remove_region和memblock_free_pages。</p><p><img src="/images/posts/20210923/Memblock_Free.png" alt="Memblock Free APIs" title="Figure 2. Memblock Free函数调用"></p><p>memblock_remove_region的主要作用是移除对应内存区，其代码如下：</p><figure class="highlight c"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br></pre></td><td class="code"><pre><span class="line"><span class="function"><span class="keyword">static</span> <span class="keyword">void</span> __init_memblock <span class="title">memblock_remove_region</span><span class="params">(struct memblock_type *type, <span class="keyword">unsigned</span> <span class="keyword">long</span> r)</span></span></span><br><span class="line"><span class="function"></span>{</span><br><span class="line">type->total_size -= type->regions[r].<span class="built_in">size</span>;</span><br><span class="line">memmove(&type->regions[r], &type->regions[r + <span class="number">1</span>],</span><br><span class="line">(type->cnt - (r + <span class="number">1</span>)) * <span class="keyword">sizeof</span>(type->regions[r]));</span><br><span class="line">type->cnt--;</span><br><span class="line"></span><br><span class="line"><span class="comment">/* Special case for empty arrays */</span></span><br><span class="line"><span class="keyword">if</span> (type->cnt == <span class="number">0</span>) {</span><br><span class="line">WARN_ON(type->total_size != <span class="number">0</span>);</span><br><span class="line">type->cnt = <span class="number">1</span>;</span><br><span class="line">type->regions[<span class="number">0</span>].base = <span class="number">0</span>;</span><br><span class="line">type->regions[<span class="number">0</span>].<span class="built_in">size</span> = <span class="number">0</span>;</span><br><span class="line">type->regions[<span class="number">0</span>].flags = <span class="number">0</span>;</span><br><span class="line">memblock_set_region_node(&type->regions[<span class="number">0</span>], MAX_NUMNODES);</span><br><span class="line">}</span><br><span class="line">}</span><br></pre></td></tr></table></figure><p>这里的主要作用就是将内存区域从对应类型的区域数组中移除，修改对应类型区域的长度，同时将该区域索引之后的区域依次向前移动一位。</p><p>memblock_free_pages的作用则是将对应页释放会给Buddy System：</p><figure class="highlight c"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br></pre></td><td class="code"><pre><span class="line"><span class="function"><span class="keyword">void</span> __init <span class="title">memblock_free_pages</span><span class="params">(struct page *page, <span class="keyword">unsigned</span> <span class="keyword">long</span> pfn,</span></span></span><br><span class="line"><span class="function"><span class="params"><span class="keyword">unsigned</span> <span class="keyword">int</span> order)</span></span></span><br><span class="line"><span class="function"></span>{</span><br><span class="line"><span class="keyword">if</span> (early_page_uninitialised(pfn))</span><br><span class="line"><span class="keyword">return</span>;</span><br><span class="line">__free_pages_core(page, order);</span><br><span class="line">}</span><br></pre></td></tr></table></figure><h2 id="物理内存模型"><a href="#物理内存模型" class="headerlink" title="物理内存模型"></a>物理内存模型</h2><p>在讲解伙伴系统之前，我们先讲解物理内存模型（<a href="https://www.kernel.org/doc/html/latest/vm/memory-model.html" target="_blank" rel="noopener">Physical Memory Model</a>），这是向伙伴系统过度的基础。简单一点讲，伙伴系统是按页对内存进行管理的，物理内存模型解决的是：</p><ul><li>页对应的描述符(<em>struct page</em>)如何与对应物理页匹配。</li><li>通过物理页帧号如何快速找到对应的页描述符。</li><li>处理内存地址不连续 （存在多个内存节点，或者同个内存节点内有空洞造成的不连续）</li></ul><p>Linux系统的物理内存管理模型有三种配置，通过KConfig选择：</p><ul><li>Flat：平坦内存模型是最简单的内存管理模型，适用于地址连续没有内存空洞的系统，也是Linux最早采用的内存模型。因为被管理的内存地址连续，因此可以方便地使用数组来管理。数组下标也可以直接和页帧号进行关联。</li><li>Discontiguous：随着处理器系统发展，有了非均匀内存访问模型（<a href="https://zh.wikipedia.org/wiki/%E9%9D%9E%E5%9D%87%E5%8C%80%E8%AE%BF%E5%AD%98%E6%A8%A1%E5%9E%8B" target="_blank" rel="noopener">NUMA</a>）。为了处理这种需求，内核就有了Discontiguous内存管理模型。这种管理模型因为在对页帧和对应页描述符映射不够有效，且不能很好适应一些嵌入式系统的需求，逐渐被Sparse模型替代。</li></ul><blockquote><p>也可以称为“非一致性内存访问”，但一致性内存往往会跟DMA一致性，Cache一致性等概念混淆。</p></blockquote><ul><li>Sparse：目前最常用且适配性最强的内存模型，它还支持内存的热拔插。管理方式如下：</li></ul><p><img src="/images/posts/20210923/Sparse_Memory_Model.PNG" alt="Sparse Memory Model" title="Figure 3. Sparse内存模型"></p><h3 id="数据结构-1"><a href="#数据结构-1" class="headerlink" title="数据结构"></a>数据结构</h3><p>从上图看出，mem_section的数据结构比较简单：</p><figure class="highlight c"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br><span class="line">21</span><br><span class="line">22</span><br><span class="line">23</span><br><span class="line">24</span><br><span class="line">25</span><br><span class="line">26</span><br><span class="line">27</span><br></pre></td><td class="code"><pre><span class="line"><span class="meta">#<span class="meta-keyword">ifdef</span> CONFIG_SPARSEMEM_EXTREME</span></span><br><span class="line"><span class="class"><span class="keyword">struct</span> <span class="title">mem_section</span> **<span class="title">mem_section</span>;</span></span><br><span class="line"><span class="meta">#<span class="meta-keyword">else</span></span></span><br><span class="line"><span class="class"><span class="keyword">struct</span> <span class="title">mem_section</span> <span class="title">mem_section</span>[<span class="title">NR_SECTION_ROOTS</span>][<span class="title">SECTIONS_PER_ROOT</span>]</span></span><br><span class="line"><span class="class">____<span class="title">cacheline_internodealigned_in_smp</span>;</span></span><br><span class="line"><span class="meta">#<span class="meta-keyword">endif</span></span></span><br><span class="line"><span class="class"><span class="keyword">struct</span> <span class="title">mem_section</span> {</span></span><br><span class="line"><span class="comment">/*</span></span><br><span class="line"><span class="comment"> * This is, logically, a pointer to an array of struct</span></span><br><span class="line"><span class="comment"> * pages. However, it is stored with some other magic.</span></span><br><span class="line"><span class="comment"> * (see sparse.c::sparse_init_one_section())</span></span><br><span class="line"><span class="comment"> *</span></span><br><span class="line"><span class="comment"> * Additionally during early boot we encode node id of</span></span><br><span class="line"><span class="comment"> * the location of the section here to guide allocation.</span></span><br><span class="line"><span class="comment"> * (see sparse.c::memory_present())</span></span><br><span class="line"><span class="comment"> *</span></span><br><span class="line"><span class="comment"> * Making it a UL at least makes someone do a cast</span></span><br><span class="line"><span class="comment"> * before using it wrong.</span></span><br><span class="line"><span class="comment"> */</span></span><br><span class="line"><span class="keyword">unsigned</span> <span class="keyword">long</span> section_mem_map;</span><br><span class="line"></span><br><span class="line"><span class="class"><span class="keyword">struct</span> <span class="title">mem_section_usage</span> *<span class="title">usage</span>;</span></span><br><span class="line"><span class="meta">#<span class="meta-keyword">ifdef</span> CONFIG_PAGE_EXTENSION</span></span><br><span class="line"><span class="class"><span class="keyword">struct</span> <span class="title">page_ext</span> *<span class="title">page_ext</span>;</span></span><br><span class="line"><span class="keyword">unsigned</span> <span class="keyword">long</span> pad;</span><br><span class="line"><span class="meta">#<span class="meta-keyword">endif</span></span></span><br><span class="line">};</span><br></pre></td></tr></table></figure><ul><li><em>section_mem_map</em>: 存的是指向对应struct page表的指针，以及一些标记性栏位（如该section是否是存在的）</li><li><em>usage</em>:</li><li><em>page_ext</em>:</li><li><em>pad</em>:</li></ul><h3 id="初始化"><a href="#初始化" class="headerlink" title="初始化"></a>初始化</h3><p>Sparse的每个mem_section管理一块连续的内存区域，它由多个物理页组成。mem_section和这些内存区域的映射关系在<strong>sparse_init</strong>函数建立。代码如下：</p><figure class="highlight c"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br><span class="line">21</span><br><span class="line">22</span><br><span class="line">23</span><br><span class="line">24</span><br><span class="line">25</span><br><span class="line">26</span><br><span class="line">27</span><br><span class="line">28</span><br><span class="line">29</span><br><span class="line">30</span><br></pre></td><td class="code"><pre><span class="line"><span class="function"><span class="keyword">void</span> __init <span class="title">sparse_init</span><span class="params">(<span class="keyword">void</span>)</span></span></span><br><span class="line"><span class="function"></span>{</span><br><span class="line"><span class="keyword">unsigned</span> <span class="keyword">long</span> pnum_end, pnum_begin, map_count = <span class="number">1</span>;</span><br><span class="line"><span class="keyword">int</span> nid_begin;</span><br><span class="line"></span><br><span class="line">memblocks_present();</span><br><span class="line"></span><br><span class="line">pnum_begin = first_present_section_nr();</span><br><span class="line">nid_begin = sparse_early_nid(__nr_to_section(pnum_begin));</span><br><span class="line"></span><br><span class="line"><span class="comment">/* Setup pageblock_order for HUGETLB_PAGE_SIZE_VARIABLE */</span></span><br><span class="line">set_pageblock_order();</span><br><span class="line"></span><br><span class="line">for_each_present_section_nr(pnum_begin + <span class="number">1</span>, pnum_end) {</span><br><span class="line"><span class="keyword">int</span> nid = sparse_early_nid(__nr_to_section(pnum_end));</span><br><span class="line"></span><br><span class="line"><span class="keyword">if</span> (nid == nid_begin) {</span><br><span class="line">map_count++;</span><br><span class="line"><span class="keyword">continue</span>;</span><br><span class="line">}</span><br><span class="line"><span class="comment">/* Init node with sections in range [pnum_begin, pnum_end) */</span></span><br><span class="line">sparse_init_nid(nid_begin, pnum_begin, pnum_end, map_count);</span><br><span class="line">nid_begin = nid;</span><br><span class="line">pnum_begin = pnum_end;</span><br><span class="line">map_count = <span class="number">1</span>;</span><br><span class="line">}</span><br><span class="line"><span class="comment">/* cover the last node */</span></span><br><span class="line">sparse_init_nid(nid_begin, pnum_begin, pnum_end, map_count);</span><br><span class="line">vmemmap_populate_print_last();</span><br><span class="line">}</span><br></pre></td></tr></table></figure><ul><li><em>memblocks_present</em> 作用是为Mem Block中标记的memory类型的内存分配mem_section，并对mem_section的section_mem_map自段做Present标记。如下：</li></ul><figure class="highlight c"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br></pre></td><td class="code"><pre><span class="line"><span class="keyword">if</span> (!ms->section_mem_map) {</span><br><span class="line">ms->section_mem_map = sparse_encode_early_nid(nid) |</span><br><span class="line">SECTION_IS_ONLINE;</span><br><span class="line">section_mark_present(ms);</span><br><span class="line">}</span><br></pre></td></tr></table></figure><blockquote><p>“分配mem_section”并不准确，当Kconfig不是 _CONFIG_SPARSEMEM_EXTREME_时，mem_section数组是静态定义的。</p></blockquote><ul><li>接着的循环就是遍历所有mem_section，分配struct page，修改mem_sesction的section_mem_map将该mem_section指向的page首地址与其关联。这里提一点是，section_mem_map主要存的是struct page表首地址减去该mem_section首个页的页帧号。这样后续可以快速的进行页帧号与对应struct page的相互转换。如下代码是封装，及页和页帧转换部分：</li></ul><figure class="highlight c"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br></pre></td><td class="code"><pre><span class="line"><span class="function"><span class="keyword">static</span> <span class="keyword">unsigned</span> <span class="keyword">long</span> <span class="title">sparse_encode_mem_map</span><span class="params">(struct page *mem_map, <span class="keyword">unsigned</span> <span class="keyword">long</span> pnum)</span></span></span><br><span class="line"><span class="function"></span>{</span><br><span class="line"><span class="keyword">unsigned</span> <span class="keyword">long</span> coded_mem_map =</span><br><span class="line">(<span class="keyword">unsigned</span> <span class="keyword">long</span>)(mem_map - (section_nr_to_pfn(pnum)));</span><br><span class="line">BUILD_BUG_ON(SECTION_MAP_LAST_BIT > (<span class="number">1U</span>L<<PFN_SECTION_SHIFT));</span><br><span class="line">BUG_ON(coded_mem_map & ~SECTION_MAP_MASK);</span><br><span class="line"><span class="keyword">return</span> coded_mem_map;</span><br><span class="line">}</span><br><span class="line"><span class="meta">#<span class="meta-keyword">define</span> __page_to_pfn(pg)\</span></span><br><span class="line">({<span class="keyword">const</span> struct page *__pg = (pg);\</span><br><span class="line"><span class="keyword">int</span> __sec = page_to_section(__pg);\</span><br><span class="line">(<span class="keyword">unsigned</span> <span class="keyword">long</span>)(__pg - __section_mem_map_addr(__nr_to_section(__sec)));\</span><br><span class="line">})</span><br><span class="line"></span><br><span class="line"><span class="meta">#<span class="meta-keyword">define</span> __pfn_to_page(pfn)\</span></span><br><span class="line">({<span class="keyword">unsigned</span> <span class="keyword">long</span> __pfn = (pfn);\</span><br><span class="line">struct mem_section *__sec = __pfn_to_section(__pfn);\</span><br><span class="line">__section_mem_map_addr(__sec) + __pfn;\</span><br><span class="line">})</span><br></pre></td></tr></table></figure><h2 id="小节"><a href="#小节" class="headerlink" title="小节"></a>小节</h2><p>本文先介绍到这，后续篇幅接着介绍Buddy System。</p>]]></content>
<summary type="html">
<blockquote>
<p>这是&lt;Linux内核内存管理&gt;系列的第四篇</p>
<blockquote>
<p><a href="/p/61304bbb.html">第一篇</a>为内核内存管理过程知识点的的简单梳理</p>
<p><a href="/p/1dc6adc2.html">第二篇</a>介绍了内核的数据结构</p>
<p><a href="/p/8f5117c5.html">第三篇</a>介绍了从内核第一行代码加载到跳转到C代码前的内存处理。</p>
<p><a href="/p/94b22c5d.html">第四篇</a>概览了初始化C代码中的内存处理</p>
</blockquote>
</blockquote><h2 id="概述"><a href="#概述" class="headerlink" title="概述"></a>概述</h2><p>参考<a href="https://www.kernel.org/doc/html/latest/core-api/boot-time-mm.html" target="_blank" rel="noopener">内核文档</a>，系统初始化早期是不能使用我们常用的 <em>kmalloc</em> ，<em>vmalloc</em> 等函数，这是因为此时对应的功能还没初始化好。</p>
</summary>
<category term="Linux" scheme="http://www.l2h.site/categories/Linux/"/>
<category term="Linux内存管理" scheme="http://www.l2h.site/categories/Linux/Linux%E5%86%85%E5%AD%98%E7%AE%A1%E7%90%86/"/>
<category term="Linux" scheme="http://www.l2h.site/tags/Linux/"/>
<category term="内存管理" scheme="http://www.l2h.site/tags/%E5%86%85%E5%AD%98%E7%AE%A1%E7%90%86/"/>
<category term="Buddy System" scheme="http://www.l2h.site/tags/Buddy-System/"/>
<category term="伙伴系统" scheme="http://www.l2h.site/tags/%E4%BC%99%E4%BC%B4%E7%B3%BB%E7%BB%9F/"/>
<category term="Memblock" scheme="http://www.l2h.site/tags/Memblock/"/>
</entry>
<entry>
<title>Linux内核内存管理 - 初始化C代码中的内存处理概览</title>
<link href="http://www.l2h.site/p/94b22c5d.html"/>
<id>http://www.l2h.site/p/94b22c5d.html</id>
<published>2021-09-20T02:27:33.000Z</published>
<updated>2023-06-01T13:14:04.480Z</updated>
<content type="html"><![CDATA[<blockquote><p>这是<Linux内核内存管理>系列的第四篇</p><blockquote><p><a href="/p/61304bbb.html">第一篇</a>为内核内存管理过程知识点的的简单梳理</p><p><a href="/p/1dc6adc2.html">第二篇</a>介绍了内核的数据结构</p><p><a href="/p/8f5117c5.html">第三篇</a>介绍了从内核第一行代码加载到跳转到C代码前的内存处理。</p></blockquote></blockquote><h2 id="x86-64体系结构C代码处理"><a href="#x86-64体系结构C代码处理" class="headerlink" title="x86_64体系结构C代码处理"></a>x86_64体系结构C代码处理</h2><p>前文我们从汇编代码跳转到了<strong>x86_64_start_kernel</strong>，该函数代码如下：</p><a id="more"></a><figure class="highlight c"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br><span class="line">21</span><br><span class="line">22</span><br><span class="line">23</span><br><span class="line">24</span><br><span class="line">25</span><br><span class="line">26</span><br><span class="line">27</span><br><span class="line">28</span><br></pre></td><td class="code"><pre><span class="line"><span class="function">asmlinkage __visible <span class="keyword">void</span> __init <span class="title">x86_64_start_kernel</span><span class="params">(<span class="keyword">char</span> * real_mode_data)</span></span></span><br><span class="line"><span class="function"></span>{</span><br><span class="line"></span><br><span class="line">BUILD_BUG_ON(MODULES_VADDR < __START_KERNEL_map);</span><br><span class="line">BUILD_BUG_ON(MODULES_VADDR - __START_KERNEL_map < KERNEL_IMAGE_SIZE);</span><br><span class="line">BUILD_BUG_ON(MODULES_LEN + KERNEL_IMAGE_SIZE > <span class="number">2</span>*PUD_SIZE);</span><br><span class="line">BUILD_BUG_ON((__START_KERNEL_map & ~PMD_MASK) != <span class="number">0</span>);</span><br><span class="line">BUILD_BUG_ON((MODULES_VADDR & ~PMD_MASK) != <span class="number">0</span>);</span><br><span class="line">BUILD_BUG_ON(!(MODULES_VADDR > __START_KERNEL));</span><br><span class="line">MAYBE_BUILD_BUG_ON(!(((MODULES_END - <span class="number">1</span>) & PGDIR_MASK) ==</span><br><span class="line">(__START_KERNEL & PGDIR_MASK)));</span><br><span class="line">BUILD_BUG_ON(__fix_to_virt(__end_of_fixed_addresses) <= MODULES_END);</span><br><span class="line"></span><br><span class="line">cr4_init_shadow();</span><br><span class="line">reset_early_page_tables();</span><br><span class="line">clear_bss();</span><br><span class="line">clear_page(init_top_pgt);</span><br><span class="line"></span><br><span class="line">sme_early_init();</span><br><span class="line">kasan_early_init();</span><br><span class="line">idt_setup_early_handler();</span><br><span class="line">copy_bootdata(__va(real_mode_data));</span><br><span class="line"></span><br><span class="line">load_ucode_bsp();</span><br><span class="line">init_top_pgt[<span class="number">511</span>] = early_top_pgt[<span class="number">511</span>];</span><br><span class="line"></span><br><span class="line">x86_64_start_reservations(real_mode_data);</span><br><span class="line">}</span><br></pre></td></tr></table></figure><p>首先几个<em>BUILD_BUG_ON</em> 用于检查潜在的配置错误，分别检查的是：</p><ul><li>内核模块的虚拟地址位于内核映像之后。</li><li>内核模块+内核映像所占空间小于 2^31 (即2GB)</li><li>内核映像和内核模块的地址为2MB对齐</li><li>固定映射结束地址大于内核模块结束地址</li></ul><p>接着初始化CR4 Shadow，<a href="https://lore.kernel.org/lkml/808fa43ef6e24d1df47c2666174d997105b9582d.1413323611.git.luto@amacapital.net/" target="_blank" rel="noopener">内核Check-in List</a>给出其作用如下：</p><blockquote><p>Context switches and TLB flushes can change individual bits of CR4. CR4 reads take several cycles, so store a shadow copy of CR4 in a per-cpu variable. To avoid wasting a cache line, I added the CR4 shadow to cpu_tlbstate, which is already touched during context switches.</p></blockquote><p>也就是说，CR4读取是需要多个CPU时钟周期的，所以将CR4存在一个per-cpu变量内。CR4 Shadow放置于cpu_tlbstate，因为cpu_tlbstate在上下文切换时会被CPU加载到Cache，由此可以节省Cache line的使用。</p><p><strong>reset_early_page_tables</strong>将early_top_pgt清除并重新加载其为内核页表。<strong>clear_bss</strong>清理BSS和init_top_pgt<strong>sme_early_init</strong>是初始化内存加密相关。<br><strong>kasan_early_init</strong>作用是初始化KASAN功能，后续会再具体介绍KASAN，这里略过不表。<br><strong>idt_setup_early_handler</strong>作用是加载<a href="https://elixir.bootlin.com/linux/v5.13.9/source/arch/x86/kernel/head_64.S#L363" target="_blank" rel="noopener">IDT Handler</a>，其代码如下：</p><figure class="highlight x86asm"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br><span class="line">21</span><br><span class="line">22</span><br><span class="line">23</span><br><span class="line">24</span><br><span class="line">25</span><br><span class="line">26</span><br><span class="line">27</span><br><span class="line">28</span><br><span class="line">29</span><br></pre></td><td class="code"><pre><span class="line">SYM_CODE_START_LOCAL(early_idt_handler_common)</span><br><span class="line"></span><br><span class="line"><span class="keyword">cld</span></span><br><span class="line"></span><br><span class="line">incl early_recursion_flag(%rip)</span><br><span class="line">pushq %rsi/* pt_regs-><span class="built_in">si</span> */</span><br><span class="line"><span class="keyword">movq</span> <span class="number">8</span>(%rsp), %rsi/* <span class="built_in">RSI</span> = vector number */</span><br><span class="line"><span class="keyword">movq</span> %rdi, <span class="number">8</span>(%rsp)/* pt_regs-><span class="built_in">di</span> = <span class="built_in">RDI</span> */</span><br><span class="line">pushq %rdx/* pt_regs-><span class="built_in">dx</span> */</span><br><span class="line">pushq %rcx/* pt_regs-><span class="built_in">cx</span> */</span><br><span class="line">pushq %rax/* pt_regs-><span class="built_in">ax</span> */</span><br><span class="line">pushq %r8/* pt_regs-><span class="built_in">r8</span> */</span><br><span class="line">pushq %r9/* pt_regs-><span class="built_in">r9</span> */</span><br><span class="line">pushq %r10/* pt_regs-><span class="built_in">r10</span> */</span><br><span class="line">pushq %r11/* pt_regs-><span class="built_in">r11</span> */</span><br><span class="line">pushq %rbx/* pt_regs-><span class="built_in">bx</span> */</span><br><span class="line">pushq %rbp/* pt_regs-><span class="built_in">bp</span> */</span><br><span class="line">pushq %r12/* pt_regs-><span class="built_in">r12</span> */</span><br><span class="line">pushq %r13/* pt_regs-><span class="built_in">r13</span> */</span><br><span class="line">pushq %r14/* pt_regs-><span class="built_in">r14</span> */</span><br><span class="line">pushq %r15/* pt_regs-><span class="built_in">r15</span> */</span><br><span class="line">UNWIND_HINT_REGS</span><br><span class="line"></span><br><span class="line"><span class="keyword">movq</span> %rsp,%rdi/* <span class="built_in">RDI</span> = pt_regs<span class="comment">; RSI is already trapnr */</span></span><br><span class="line"><span class="keyword">call</span> do_early_exception</span><br><span class="line"></span><br><span class="line">decl early_recursion_flag(%rip)</span><br><span class="line"><span class="keyword">jmp</span> restore_regs_and_return_to_kernel</span><br><span class="line">SYM_CODE_END(early_idt_handler_common)</span><br></pre></td></tr></table></figure><p>上述代码主要作用是寄存器状态保存，同时执行<strong>do_early_exception</strong>。<br><strong>copy_bootdata</strong>的主要作用是检查初始化参数，并将它们复制<em>boot_params</em>和<em>boot_command_line</em>内。同时将early_top_pgt页表的第512项赋值给init_top_pgt对应项。<br>最后<strong>x86_64_start_reservations</strong>执行一些特定平台相关的”quirks”后，开始执行<strong>start_kernel</strong>。</p><h2 id="start-kernel"><a href="#start-kernel" class="headerlink" title="start_kernel"></a>start_kernel</h2><p><strong>start_kernel</strong>执行所有内核初始化代码。本文仅分析与内存管理相关的步骤如下图：<br><img src="http://pic.l2h.site/start_kernel.png" alt="内核初始化的内存处理" title="Figure 1. 内核初始化的内存处理"></p><ul><li><strong>set_task_stack_end_magic</strong> 为内核栈底设置Magic Number，用于栈溢出的检查。</li><li><strong>page_address_init</strong> 初始化<em>page_address_htable</em>链表</li><li><strong>setup_arch</strong>为体系结构相关的初始化代码。X64系统对应的setup_arch定义在<a href="https://elixir.bootlin.com/linux/v5.13.9/source/arch/x86/kernel/setup.c#L790" target="_blank" rel="noopener">arch/x86/kernel/setup.c</a><ul><li><strong>early_ioremap_init</strong> 初始化数组 <strong><em>slot_virt</em></strong> 用于保存虚拟地址和外设物理地址的早期固定映射，其定义在<a href="https://elixir.bootlin.com/linux/latest/source/arch/x86/include/asm/fixmap.h#L127" target="_blank" rel="noopener">fixmap.h</a>。</li><li><strong>setup_olpc_ofw_pgd</strong> 为“<a href="https://en.wikipedia.org/wiki/One_Laptop_per_Child" target="_blank" rel="noopener">One Laptop Per Child</a>”公益项目相关设备初始化PGD。</li><li><strong>e820__memory_setup</strong> 执行 <strong>e820__memory_setup_default</strong> ，主要作用是从 <a href="https://en.wikipedia.org/wiki/E820" target="_blank" rel="noopener">E820</a>获取硬件内存布局，保存在全局变量<em>e820_table</em>。代码如下：<figure class="highlight c"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br></pre></td><td class="code"><pre><span class="line"><span class="keyword">static</span> <span class="class"><span class="keyword">struct</span> <span class="title">e820_table</span> <span class="title">e820_table_init</span>__<span class="title">initdata</span>;</span></span><br><span class="line"><span class="keyword">static</span> <span class="class"><span class="keyword">struct</span> <span class="title">e820_table</span> <span class="title">e820_table_kexec_init</span>__<span class="title">initdata</span>;</span></span><br><span class="line"><span class="keyword">static</span> <span class="class"><span class="keyword">struct</span> <span class="title">e820_table</span> <span class="title">e820_table_firmware_init</span>__<span class="title">initdata</span>;</span></span><br><span class="line"></span><br><span class="line"><span class="class"><span class="keyword">struct</span> <span class="title">e820_table</span> *<span class="title">e820_table</span> __<span class="title">refdata</span>= &<span class="title">e820_table_init</span>;</span></span><br><span class="line"><span class="class"><span class="keyword">struct</span> <span class="title">e820_table</span> *<span class="title">e820_table_kexec</span> __<span class="title">refdata</span>= &<span class="title">e820_table_kexec_init</span>;</span></span><br><span class="line"><span class="class"><span class="keyword">struct</span> <span class="title">e820_table</span> *<span class="title">e820_table_firmware</span> __<span class="title">refdata</span>= &<span class="title">e820_table_firmware_init</span>;</span></span><br></pre></td></tr></table></figure></li></ul></li></ul><blockquote><p>注意 <strong>initdata和</strong>refdata修饰作用在内核代码有说明,其中<strong>init的作用是为了标记初始化使用的数据以便内核初始化结束后释放对应的内存。而</strong>refdata的用于引用__initdata标记的数据。</p></blockquote><figure class="highlight c"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br></pre></td><td class="code"><pre><span class="line"><span class="meta">#<span class="meta-keyword">define</span> __init__section(<span class="meta-string">".init.text"</span>) __cold __latent_entropy __noinitretpoline __nocfi</span></span><br><span class="line"><span class="meta">#<span class="meta-keyword">define</span> __initdata__section(<span class="meta-string">".init.data"</span>)</span></span><br><span class="line"><span class="meta">#<span class="meta-keyword">define</span> __initconst__section(<span class="meta-string">".init.rodata"</span>)</span></span><br><span class="line"><span class="meta">#<span class="meta-keyword">define</span> __exitdata__section(<span class="meta-string">".exit.data"</span>)</span></span><br><span class="line"><span class="meta">#<span class="meta-keyword">define</span> __exit_call__used __section(<span class="meta-string">".exitcall.exit"</span>)</span></span><br><span class="line"></span><br><span class="line"><span class="meta">#<span class="meta-keyword">define</span> __ref __section(<span class="meta-string">".ref.text"</span>) noinline</span></span><br><span class="line"><span class="meta">#<span class="meta-keyword">define</span> __refdata __section(<span class="meta-string">".ref.data"</span>)</span></span><br><span class="line"><span class="meta">#<span class="meta-keyword">define</span> __refconst __section(<span class="meta-string">".ref.rodata"</span>)</span></span><br></pre></td></tr></table></figure><ul><li><p><strong>early_reserve_memory</strong> 作用是将已占用的内存区域标记为不可用。这样后续不允许被memblock或者伙伴系统分配器再分配。</p><ul><li>[_text, __end_of_kernel_reserve]</li><li>[0,64K]</li><li>setup_data: [hdr.setup_data, sizeof(setup_data)+hdr.setup_data]</li><li>initrd</li><li>ibft(<a href="https://docs.vmware.com/en/VMware-vSphere/6.7/com.vmware.vsphere.storage.doc/GUID-03E73408-076A-4A7F-9732-3BB816A16CA1.html" target="_blank" rel="noopener">iSCSI Boot Format Table</a>) 区域(如果有的话)</li><li>BIOS区域: [BIOS Start, 0x1000000]</li><li>etc.</li></ul></li><li><p><strong>memblock_set_bottom_up</strong> 标记memblock内存分配是从低地址到高地址</p><blockquote><p><a href="https://lwn.net/Articles/761215/" target="_blank" rel="noopener">memblock</a>是系统初始化初期，伙伴系统接管前的分配器，它取代了内核早期的bootmem分配器。</p></blockquote></li><li><p><strong>e820__reserve_setup_data</strong> 将Boot Loader扩展的数据区标记为内核保留区域，并为其分配内存映射。</p></li><li><p><strong>e820__finish_early_params</strong> 更新e820表。用户可以通过Loader传入内核CMD line来自定义内存区域映射。下图是在QEMU中E820扫描到的内存映射。<br><img src="/images/posts/20210921/e820_memory_mapping.png" alt="E820物理内存映射" title="Figure 2. E820物理内存映射"></p></li><li><p><strong>probe_roms</strong> 为ROM的分配IO资源</p></li><li><p><strong>insert_resource</strong> 将code、rodata、data和bss插入IOMEM资源</p></li><li><p><strong>e820_add_kernel_range</strong> 将内核_text 到 _end区域加入到e820表。</p></li><li><p><strong>trim_bios_range</strong> 处理一些BIOS识别内存的特殊情况</p><ul><li>0到4KB区域没有被BIOS加入到e820中，我们将这段区域加入到e820保留区域。</li><li>将BIOS区域中BIOS_BEGIN到BIOS_END(640Kb -> 1Mb) 从e820表中删除。这是因为有些BIOS会将这段区域识别为物理内存（但起始不是）。</li></ul></li><li><p><strong>early_gart_iommu_check</strong> 针对早期的AMD处理器中基于<a href="https://en.wikipedia.org/wiki/Graphics_address_remapping_table" target="_blank" rel="noopener">GART IOMMU</a>的支持。</p></li><li><p><strong>e820__end_of_ram_pfn</strong> 从e820获取最大物理页帧号</p></li><li><p><strong>init_cache_modes</strong> <font color="red">待确定</font></p></li><li><p><strong>kernel_randomize_memory</strong> 与KASLR相关，后续介绍，此处不表</p></li><li><p><strong>early_alloc_pgt_buf</strong> 为初始化过程中分配PGT预留堆空间</p></li><li><p><strong>reserve_brk</strong> 在Boot分配器Reserve堆空间</p></li><li><p><strong>e820__memblock_setup</strong> 将e820内存分布表的数据读出，并填写到Boot分配器管理</p><blockquote><p>关于memblock分配器memblock，系列后续文章专门介绍</p></blockquote></li><li><p><strong>e820__memblock_alloc_reserved_mpc_new</strong> 从Memblock为<a href="https://en.wikipedia.org/wiki/MultiProcessor_Specification" target="_blank" rel="noopener">MPC Table</a>分配内存。</p></li><li><p><strong>reserve_real_mode</strong> 从Memblock为实模式的[0, 1MB]分配内存。</p></li><li><p><strong>init_mem_mapping</strong> <font color="red">待确定</font></p></li><li><p><strong>memblock_set_current_limit</strong> 设置memblock.current_limit为membelcok管理的最大页帧号</p></li><li><p><strong>initmem_init</strong> 初始化NUMA（如果开启对应Kconfig的话），为memblock的现有区域分配NUMA节点ID号</p></li><li><p><strong>dma_contiguous_reserve</strong> 为DMA预留连续内存</p></li><li><p><strong>reserve_crashkernel</strong> 为kernel crash分配内存</p></li><li><p><strong>memblock_find_dma_reserve</strong> 计算DMA区域的大小</p></li><li><p><strong>x86_init.paging.pagetable_init</strong> 调用 <em>native_pagetable_init</em> 来初始化paging <font color="red">待确定</font></p></li><li><p><strong>kasan_init</strong> 初始化KASAN</p></li><li><p><strong>sync_initial_page_table</strong> <font color="red">待确定</font></p></li><li><p><strong>e820__reserve_resources</strong> 为e820表项分配IO resource （reserve标记的表项除外） </p></li><li><p><strong>x86_init.resources.reserve_resources</strong> 使用reserve_standard_io_resources 为下面硬件端口分配ioport resource</p></li></ul><figure class="highlight c"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br><span class="line">21</span><br><span class="line">22</span><br></pre></td><td class="code"><pre><span class="line"><span class="keyword">static</span> <span class="class"><span class="keyword">struct</span> <span class="title">resource</span> <span class="title">standard_io_resources</span>[] = {</span></span><br><span class="line">{ .name = <span class="string">"dma1"</span>, .start = <span class="number">0x00</span>, .<span class="built_in">end</span> = <span class="number">0x1f</span>,</span><br><span class="line">.flags = IORESOURCE_BUSY | IORESOURCE_IO },</span><br><span class="line">{ .name = <span class="string">"pic1"</span>, .start = <span class="number">0x20</span>, .<span class="built_in">end</span> = <span class="number">0x21</span>,</span><br><span class="line">.flags = IORESOURCE_BUSY | IORESOURCE_IO },</span><br><span class="line">{ .name = <span class="string">"timer0"</span>, .start = <span class="number">0x40</span>, .<span class="built_in">end</span> = <span class="number">0x43</span>,</span><br><span class="line">.flags = IORESOURCE_BUSY | IORESOURCE_IO },</span><br><span class="line">{ .name = <span class="string">"timer1"</span>, .start = <span class="number">0x50</span>, .<span class="built_in">end</span> = <span class="number">0x53</span>,</span><br><span class="line">.flags = IORESOURCE_BUSY | IORESOURCE_IO },</span><br><span class="line">{ .name = <span class="string">"keyboard"</span>, .start = <span class="number">0x60</span>, .<span class="built_in">end</span> = <span class="number">0x60</span>,</span><br><span class="line">.flags = IORESOURCE_BUSY | IORESOURCE_IO },</span><br><span class="line">{ .name = <span class="string">"keyboard"</span>, .start = <span class="number">0x64</span>, .<span class="built_in">end</span> = <span class="number">0x64</span>,</span><br><span class="line">.flags = IORESOURCE_BUSY | IORESOURCE_IO },</span><br><span class="line">{ .name = <span class="string">"dma page reg"</span>, .start = <span class="number">0x80</span>, .<span class="built_in">end</span> = <span class="number">0x8f</span>,</span><br><span class="line">.flags = IORESOURCE_BUSY | IORESOURCE_IO },</span><br><span class="line">{ .name = <span class="string">"pic2"</span>, .start = <span class="number">0xa0</span>, .<span class="built_in">end</span> = <span class="number">0xa1</span>,</span><br><span class="line">.flags = IORESOURCE_BUSY | IORESOURCE_IO },</span><br><span class="line">{ .name = <span class="string">"dma2"</span>, .start = <span class="number">0xc0</span>, .<span class="built_in">end</span> = <span class="number">0xdf</span>,</span><br><span class="line">.flags = IORESOURCE_BUSY | IORESOURCE_IO },</span><br><span class="line">{ .name = <span class="string">"fpu"</span>, .start = <span class="number">0xf0</span>, .<span class="built_in">end</span> = <span class="number">0xff</span>,</span><br><span class="line">.flags = IORESOURCE_BUSY | IORESOURCE_IO }</span><br><span class="line">};</span><br></pre></td></tr></table></figure><ul><li><strong>e820__setup_pci_gap</strong> 在0到4GB找到空闲的内存区域，用于PCI设备的IO映射。</li><li><strong>build_all_zonelists</strong> 将所有内存节点的区域加入到对应的zonelist，可以参考<a href="1dc6adc2">数据结构</a>的介绍</li><li><strong>page_alloc_init</strong> <font color="red">待确定</font></li><li><strong>mm_init</strong> 为内存管理初始化最核心的部分，由以下部分组成<ul><li><strong>page_ext_init_flatmem</strong></li><li><strong>init_mem_debugging_and_hardening</strong></li><li><strong>kfence_alloc_pool</strong></li><li><strong>report_meminit</strong></li><li><strong>stack_depot_init</strong> </li><li><strong>mem_init</strong> 回收memblock分配的内存（Reserve的除外），接着用<strong>mem_init_print_info</strong>打印内存信息</li><li><strong>kmem_cache_init</strong> 初始化SLAB分配器</li><li><strong>kmemleak_init</strong> 初始化Kmemleak</li><li><strong>pgtable_init</strong> 为启动后各进程的页表分配建立SLAB</li><li><strong>debug_objects_mem_init</strong> 为debugobject功能分配SLAB内存</li><li><strong>vmalloc_init</strong> </li></ul></li><li><strong>kmem_cache_init_late</strong> 为SLAB分配器的后续初始化。如果编译时采用的是SLUB，此处无操作。</li><li><strong>numa_policy_init</strong></li><li><strong>anon_vma_init</strong></li></ul><h2 id="总结"><a href="#总结" class="headerlink" title="总结"></a>总结</h2><p>本文是对初始化C代码中内存管理的概览，并没有介绍到每个子部分的细节，后续将会在专门的章节进行具体介绍。</p>]]></content>
<summary type="html">
<blockquote>
<p>这是&lt;Linux内核内存管理&gt;系列的第四篇</p>
<blockquote>
<p><a href="/p/61304bbb.html">第一篇</a>为内核内存管理过程知识点的的简单梳理</p>
<p><a href="/p/1dc6adc2.html">第二篇</a>介绍了内核的数据结构</p>
<p><a href="/p/8f5117c5.html">第三篇</a>介绍了从内核第一行代码加载到跳转到C代码前的内存处理。</p>
</blockquote>
</blockquote><h2 id="x86-64体系结构C代码处理"><a href="#x86-64体系结构C代码处理" class="headerlink" title="x86_64体系结构C代码处理"></a>x86_64体系结构C代码处理</h2><p>前文我们从汇编代码跳转到了<strong>x86_64_start_kernel</strong>，该函数代码如下：</p>
</summary>
<category term="Linux" scheme="http://www.l2h.site/categories/Linux/"/>
<category term="Linux内存管理" scheme="http://www.l2h.site/categories/Linux/Linux%E5%86%85%E5%AD%98%E7%AE%A1%E7%90%86/"/>
<category term="Linux" scheme="http://www.l2h.site/tags/Linux/"/>
<category term="内存管理" scheme="http://www.l2h.site/tags/%E5%86%85%E5%AD%98%E7%AE%A1%E7%90%86/"/>
</entry>
<entry>
<title>Linux内核内存管理 - 数据结构</title>
<link href="http://www.l2h.site/p/1dc6adc2.html"/>
<id>http://www.l2h.site/p/1dc6adc2.html</id>
<published>2021-09-10T00:25:32.000Z</published>
<updated>2023-06-01T13:14:04.479Z</updated>
<content type="html"><![CDATA[<p>内存管理是Linux内核最为复杂且最为重要的部分，本文从原理及代码角度对Linux内存管理机制进行分析。</p><h2 id="内存的划分"><a href="#内存的划分" class="headerlink" title="内存的划分"></a>内存的划分</h2><p>Linux将内存从大到小依次划分为Node（节点）->Zone（内存域）->Page（页）：</p><ul><li>节点：在大型结算及系统中，内存有不同的簇，依据对处理器距离的不同，访问这些簇有不同的代价。而这些簇就可以成为节点。例：在PC系统中可以理解为实际挂载的物理内存；在嵌入式系统中，有两块内存芯片A和B，分别代表一个节点。</li><li>内存域：内存域并不是物理存在的概念，是Linux系统对每个内存节点进行管理的单位，每个节点的内存域表示的是对该节点不同地址范围的划分。一般内存域有三种，分别为Normal、DMA和HighMem。</li><li>页：在每个内存域中，内存被划分为大小固定的块（32位系统一般为4K大小），为内核进行内存分配的基本单位（当然内核内存管理机制其实更为复杂，“基本单位”不代表每次分配内存最小就要分到4K。后边可以看到，当需要获取小于4K大小的内存时，内核有Slab分配器来满足要求）</li></ul><a id="more"></a><p>一张图说明Node、Zone和Page的关系如下：</p><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br></pre></td><td class="code"><pre><span class="line"> Node 1 Node 2 Node 3</span><br><span class="line"> +----------+ +----------+ +----------+</span><br><span class="line"> | | | | | |</span><br><span class="line"> |Zone_High | | | | |</span><br><span class="line"> | | | | | |</span><br><span class="line"> +----------+ | | | |</span><br><span class="line"> | | | | | |</span><br><span class="line"> | | | | | |</span><br><span class="line"> |Zone_Norm | | | | |</span><br><span class="line"> | | | | | |</span><br><span class="line"> | | | | | |</span><br><span class="line"> +----------+ | | | |</span><br><span class="line"> | | | | | |</span><br><span class="line"> page page |Zone_DMA | | | | |</span><br><span class="line">+-+--+--+--+--+ | | | | | |</span><br><span class="line">| | | | | | <--------+ | | | | |</span><br><span class="line">+-+--+--+--+--+ +----------+ +----------+ +----------+</span><br></pre></td></tr></table></figure><h2 id="数据结构"><a href="#数据结构" class="headerlink" title="数据结构"></a>数据结构</h2><p>构成上述三个内存划分的数据结构如下：</p><h3 id="内存节点"><a href="#内存节点" class="headerlink" title="内存节点"></a>内存节点</h3><p>Node对应的结构为pglist_data_t，定义如下（为方便理解，省略部分结构体成员）：</p><figure class="highlight c"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br><span class="line">21</span><br><span class="line">22</span><br><span class="line">23</span><br><span class="line">24</span><br><span class="line">25</span><br><span class="line">26</span><br><span class="line">27</span><br><span class="line">28</span><br><span class="line">29</span><br><span class="line">30</span><br><span class="line">31</span><br><span class="line">32</span><br><span class="line">33</span><br><span class="line">34</span><br><span class="line">35</span><br><span class="line">36</span><br><span class="line">37</span><br><span class="line">38</span><br><span class="line">39</span><br><span class="line">40</span><br><span class="line">41</span><br><span class="line">42</span><br><span class="line">43</span><br><span class="line">44</span><br><span class="line">45</span><br><span class="line">46</span><br><span class="line">47</span><br><span class="line">48</span><br><span class="line">49</span><br><span class="line">50</span><br><span class="line">51</span><br><span class="line">52</span><br><span class="line">53</span><br><span class="line">54</span><br><span class="line">55</span><br><span class="line">56</span><br><span class="line">57</span><br><span class="line">58</span><br><span class="line">59</span><br><span class="line">60</span><br><span class="line">61</span><br><span class="line">62</span><br><span class="line">63</span><br><span class="line">64</span><br><span class="line">65</span><br><span class="line">66</span><br><span class="line">67</span><br><span class="line">68</span><br><span class="line">69</span><br><span class="line">70</span><br><span class="line">71</span><br><span class="line">72</span><br><span class="line">73</span><br><span class="line">74</span><br><span class="line">75</span><br></pre></td><td class="code"><pre><span class="line"><span class="keyword">typedef</span> <span class="class"><span class="keyword">struct</span> <span class="title">pglist_data</span> {</span></span><br><span class="line"> <span class="class"><span class="keyword">struct</span> <span class="title">zone</span> <span class="title">node_zones</span>[<span class="title">MAX_NR_ZONES</span>];</span></span><br><span class="line"> <span class="class"><span class="keyword">struct</span> <span class="title">zonelist</span> <span class="title">node_zonelists</span>[<span class="title">MAX_ZONELISTS</span>];</span></span><br><span class="line"> <span class="keyword">int</span> nr_zones;</span><br><span class="line"><span class="meta">#<span class="meta-keyword">ifdef</span> CONFIG_FLAT_NODE_MEM_MAP <span class="comment">/* means !SPARSEMEM */</span></span></span><br><span class="line"> <span class="class"><span class="keyword">struct</span> <span class="title">page</span> *<span class="title">node_mem_map</span>;</span></span><br><span class="line"><span class="meta">#<span class="meta-keyword">ifdef</span> CONFIG_PAGE_EXTENSION</span></span><br><span class="line"> <span class="class"><span class="keyword">struct</span> <span class="title">page_ext</span> *<span class="title">node_page_ext</span>;</span></span><br><span class="line"><span class="meta">#<span class="meta-keyword">endif</span></span></span><br><span class="line"><span class="meta">#<span class="meta-keyword">endif</span></span></span><br><span class="line"><span class="meta">#<span class="meta-keyword">ifndef</span> CONFIG_NO_BOOTMEM</span></span><br><span class="line"> <span class="class"><span class="keyword">struct</span> <span class="title">bootmem_data</span> *<span class="title">bdata</span>;</span></span><br><span class="line"><span class="meta">#<span class="meta-keyword">endif</span></span></span><br><span class="line"><span class="meta">#<span class="meta-keyword">ifdef</span> CONFIG_MEMORY_HOTPLUG</span></span><br><span class="line"> <span class="keyword">spinlock_t</span> node_size_lock;</span><br><span class="line"><span class="meta">#<span class="meta-keyword">endif</span></span></span><br><span class="line"> <span class="keyword">unsigned</span> <span class="keyword">long</span> node_start_pfn;</span><br><span class="line"> <span class="keyword">unsigned</span> <span class="keyword">long</span> node_present_pages; <span class="comment">/* total number of physical pages */</span></span><br><span class="line"> <span class="keyword">unsigned</span> <span class="keyword">long</span> node_spanned_pages; <span class="comment">/* total size of physical page range, including holes */</span></span><br><span class="line"> <span class="keyword">int</span> node_id;</span><br><span class="line"> <span class="keyword">wait_queue_head_t</span> kswapd_wait;</span><br><span class="line"> <span class="keyword">wait_queue_head_t</span> pfmemalloc_wait;</span><br><span class="line"> <span class="class"><span class="keyword">struct</span> <span class="title">task_struct</span> *<span class="title">kswapd</span>;</span> <span class="comment">/* Protected by mem_hotplug_begin/end() */</span></span><br><span class="line"> <span class="keyword">int</span> kswapd_order;</span><br><span class="line"> <span class="keyword">enum</span> zone_type kswapd_classzone_idx;</span><br><span class="line"> <span class="keyword">int</span> kswapd_failures; <span class="comment">/* Number of 'reclaimed == 0' runs */</span></span><br><span class="line"><span class="meta">#<span class="meta-keyword">ifdef</span> CONFIG_COMPACTION</span></span><br><span class="line"> <span class="keyword">int</span> kcompactd_max_order;</span><br><span class="line"> <span class="keyword">enum</span> zone_type kcompactd_classzone_idx;</span><br><span class="line"> <span class="keyword">wait_queue_head_t</span> kcompactd_wait;</span><br><span class="line"> <span class="class"><span class="keyword">struct</span> <span class="title">task_struct</span> *<span class="title">kcompactd</span>;</span></span><br><span class="line"><span class="meta">#<span class="meta-keyword">endif</span></span></span><br><span class="line"><span class="meta">#<span class="meta-keyword">ifdef</span> CONFIG_NUMA_BALANCING</span></span><br><span class="line"> <span class="comment">/* Lock serializing the migrate rate limiting window */</span></span><br><span class="line"> <span class="keyword">spinlock_t</span> numabalancing_migrate_lock;</span><br><span class="line"> <span class="comment">/* Rate limiting time interval */</span></span><br><span class="line"> <span class="keyword">unsigned</span> <span class="keyword">long</span> numabalancing_migrate_next_window;</span><br><span class="line"> <span class="comment">/* Number of pages migrated during the rate limiting time interval */</span></span><br><span class="line"> <span class="keyword">unsigned</span> <span class="keyword">long</span> numabalancing_migrate_nr_pages;</span><br><span class="line"><span class="meta">#<span class="meta-keyword">endif</span></span></span><br><span class="line"> <span class="keyword">unsigned</span> <span class="keyword">long</span> totalreserve_pages;</span><br><span class="line"><span class="meta">#<span class="meta-keyword">ifdef</span> CONFIG_NUMA</span></span><br><span class="line"> <span class="comment">/*</span></span><br><span class="line"><span class="comment"> * zone reclaim becomes active if more unmapped pages exist.</span></span><br><span class="line"><span class="comment"> */</span></span><br><span class="line"> <span class="keyword">unsigned</span> <span class="keyword">long</span> min_unmapped_pages;</span><br><span class="line"> <span class="keyword">unsigned</span> <span class="keyword">long</span> min_slab_pages;</span><br><span class="line"><span class="meta">#<span class="meta-keyword">endif</span> <span class="comment">/* CONFIG_NUMA */</span></span></span><br><span class="line"></span><br><span class="line"> <span class="comment">/* Write-intensive fields used by page reclaim */</span></span><br><span class="line"> ZONE_PADDING(_pad1_)</span><br><span class="line"> <span class="keyword">spinlock_t</span> lru_lock;</span><br><span class="line"></span><br><span class="line"><span class="meta">#<span class="meta-keyword">ifdef</span> CONFIG_DEFERRED_STRUCT_PAGE_INIT</span></span><br><span class="line"> <span class="comment">/*</span></span><br><span class="line"><span class="comment"> * If memory initialisation on large machines is deferred then this</span></span><br><span class="line"><span class="comment"> * is the first PFN that needs to be initialised.</span></span><br><span class="line"><span class="comment"> */</span></span><br><span class="line"> <span class="keyword">unsigned</span> <span class="keyword">long</span> first_deferred_pfn;</span><br><span class="line"> <span class="comment">/* Number of non-deferred pages */</span></span><br><span class="line"> <span class="keyword">unsigned</span> <span class="keyword">long</span> static_init_pgcnt;</span><br><span class="line"><span class="meta">#<span class="meta-keyword">endif</span> <span class="comment">/* CONFIG_DEFERRED_STRUCT_PAGE_INIT */</span></span></span><br><span class="line"></span><br><span class="line"><span class="meta">#<span class="meta-keyword">ifdef</span> CONFIG_TRANSPARENT_HUGEPAGE</span></span><br><span class="line"> <span class="keyword">spinlock_t</span> split_queue_lock;</span><br><span class="line"> <span class="class"><span class="keyword">struct</span> <span class="title">list_head</span> <span class="title">split_queue</span>;</span></span><br><span class="line"> <span class="keyword">unsigned</span> <span class="keyword">long</span> split_queue_len;</span><br><span class="line"><span class="meta">#<span class="meta-keyword">endif</span></span></span><br><span class="line"> <span class="keyword">unsigned</span> <span class="keyword">int</span> inactive_ratio;</span><br><span class="line"> <span class="keyword">unsigned</span> <span class="keyword">long</span> flags;</span><br><span class="line"> ZONE_PADDING(_pad2_)</span><br><span class="line"> <span class="comment">/* Per-node vmstats */</span></span><br><span class="line"> <span class="class"><span class="keyword">struct</span> <span class="title">per_cpu_nodestat</span> __<span class="title">percpu</span> *<span class="title">per_cpu_nodestats</span>;</span></span><br><span class="line"> <span class="keyword">atomic_long_t</span> vm_stat_NR_VM_NODE_STAT_ITEMS];</span><br><span class="line">} <span class="keyword">pg_data_t</span>;</span><br></pre></td></tr></table></figure><ul><li><strong>node_zones</strong>: 内存节点上的内存域，分别为 ZONE_HIGHMEM, ZONE_NORMAL, ZONE_DMA。新版Linux还增加了ZONE_MOVABLE和ZONE_DEVICE。</li><li><strong>node_zonelists:</strong> 对内存域进行类别指定的优先级顺序。例，当ZONE_HIGHMEM分配失败时，会u退到ZONE_DMA类型后ZONE_NORMAL类型</li><li><strong>nr_zones:</strong> 该节点上的内存域数量</li><li><strong>node_mem_map:</strong> 节点中页面的映射图</li><li><strong>bdata:</strong> 与内核初始化内存分配器相关数据</li><li><strong>node_size_lock</strong>： 与内存热拔插相关</li><li><strong>node_start_pfn:</strong> 内存节点的起始页。</li><li><strong>node_present_pages:</strong> 物理页面数量<strong>.</strong></li><li><strong>node_spanned_pages:</strong> 内存节点物理页面的大小</li><li><strong>node_id:</strong> 节点编号</li><li><strong>kswapd_wait</strong>/<strong>pfmemalloc_wait</strong>/<strong>kswapd</strong>/<strong>kswapd_order</strong>/<strong>kswapd_classzone_idx/kswapd_failures:</strong> kswapd内核线程相关参数</li><li><strong>……..</strong></li></ul><h3 id="内存区域"><a href="#内存区域" class="headerlink" title="内存区域"></a>内存区域</h3><p>内存区域对应的结构体为struct zone，定义如下：</p><figure class="highlight c"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br><span class="line">21</span><br><span class="line">22</span><br><span class="line">23</span><br><span class="line">24</span><br><span class="line">25</span><br><span class="line">26</span><br><span class="line">27</span><br><span class="line">28</span><br><span class="line">29</span><br><span class="line">30</span><br><span class="line">31</span><br><span class="line">32</span><br><span class="line">33</span><br><span class="line">34</span><br><span class="line">35</span><br><span class="line">36</span><br></pre></td><td class="code"><pre><span class="line"><span class="class"><span class="keyword">struct</span> <span class="title">zone</span> {</span></span><br><span class="line"> <span class="keyword">unsigned</span> <span class="keyword">long</span> watermark_NR_WMARK];</span><br><span class="line"> <span class="keyword">unsigned</span> <span class="keyword">long</span> nr_reserved_highatomic;</span><br><span class="line"> <span class="keyword">long</span> lowmem_reserve_MAX_NR_ZONES];</span><br><span class="line"><span class="meta">#<span class="meta-keyword">ifdef</span> CONFIG_NUMA</span></span><br><span class="line"> <span class="keyword">int</span> node;</span><br><span class="line"><span class="meta">#<span class="meta-keyword">endif</span></span></span><br><span class="line"> <span class="class"><span class="keyword">struct</span> <span class="title">pglist_data</span> *<span class="title">zone_pgdat</span>;</span></span><br><span class="line"> <span class="class"><span class="keyword">struct</span> <span class="title">per_cpu_pageset</span> __<span class="title">percpu</span> *<span class="title">pageset</span>;</span></span><br><span class="line"></span><br><span class="line"><span class="meta">#<span class="meta-keyword">ifndef</span> CONFIG_SPARSEMEM</span></span><br><span class="line"> <span class="keyword">unsigned</span> <span class="keyword">long</span> *pageblock_flags;</span><br><span class="line"><span class="meta">#<span class="meta-keyword">endif</span> <span class="comment">/* CONFIG_SPARSEMEM */</span></span></span><br><span class="line"> <span class="keyword">unsigned</span> <span class="keyword">long</span> zone_start_pfn;</span><br><span class="line"> <span class="keyword">unsigned</span> <span class="keyword">long</span> managed_pages;</span><br><span class="line"> <span class="keyword">unsigned</span> <span class="keyword">long</span> spanned_pages;</span><br><span class="line"> <span class="keyword">unsigned</span> <span class="keyword">long</span> present_pages;</span><br><span class="line"></span><br><span class="line"> <span class="keyword">const</span> <span class="keyword">char</span> *name;</span><br><span class="line"><span class="meta">#<span class="meta-keyword">ifdef</span> CONFIG_MEMORY_ISOLATION</span></span><br><span class="line"> <span class="keyword">unsigned</span> <span class="keyword">long</span> nr_isolate_pageblock;</span><br><span class="line"><span class="meta">#<span class="meta-keyword">endif</span></span></span><br><span class="line"></span><br><span class="line"><span class="meta">#<span class="meta-keyword">ifdef</span> CONFIG_MEMORY_HOTPLUG</span></span><br><span class="line"> <span class="keyword">seqlock_t</span> span_seqlock;</span><br><span class="line"><span class="meta">#<span class="meta-keyword">endif</span></span></span><br><span class="line"> <span class="keyword">int</span> initialized;</span><br><span class="line"> ZONE_PADDING(_pad1_)</span><br><span class="line"> <span class="class"><span class="keyword">struct</span> <span class="title">free_area</span> <span class="title">free_area_MAX_ORDER</span>];</span></span><br><span class="line"> <span class="keyword">unsigned</span> <span class="keyword">long</span> flags;</span><br><span class="line"> <span class="keyword">spinlock_t</span> lock;</span><br><span class="line"> ZONE_PADDING(_pad2_)</span><br><span class="line"> <span class="keyword">unsigned</span> <span class="keyword">long</span> percpu_drift_mark;</span><br><span class="line"> ........</span><br><span class="line"> <span class="keyword">atomic_long_t</span> vm_stat_NR_VM_ZONE_STAT_ITEMS];</span><br><span class="line">} ____cacheline_internodealigned_in_smp;</span><br></pre></td></tr></table></figure><ul><li><strong>watermark</strong>: 内存域的水位</li><li><strong>nr_reserved_highatomic:</strong> 紧急内存大小，</li><li><strong>lowmem_reserve</strong>:内存域最少保留内存</li><li><strong>zone_pgdat</strong>: 所在内存节点指针</li><li><strong>pageset</strong>: 每个CPU维护的页面列表</li><li><strong>zone_start_pfn</strong>:内存域第一个页的索引</li><li><strong>managed_pages</strong>: 伙伴系统管理的所有页面数量</li><li><strong>spanned_pages</strong>: 内存域所跨越所有内存页数量</li><li><strong>present_pages</strong>: 内存域物理内存所有页数量(除去内存空洞后的部分)present_pages=spanned_pages-absent_pages</li><li><strong>name</strong>: 区域名</li><li><strong>free_area</strong>:所有空闲页面的数组</li><li><strong>flags</strong>:内存域标识</li><li><strong>lock</strong>:保护free_area的锁</li><li><strong>vm_stat</strong>:虚拟内存统计信息</li></ul><p>特别说明一下内存域的水位（Watermark），它表示几个阈值，用来管理内核线程kswapd唤起与休眠的。当域内可用内存水位较高时，kswapd不用起来工作，而水位较低时，kswapd需要唤起来回收内存。如下图（来自深入理解Linux虚拟内存管理）：</p><p><img src="http://pic.l2h.site/%E5%B1%8F%E5%B9%95%E5%BF%AB%E7%85%A7-2019-08-13-%E4%B8%8A%E5%8D%887.40.53.png" alt></p><h3 id="页面"><a href="#页面" class="headerlink" title="页面"></a>页面</h3><p>系统中每个物理页面都有数据结构struct page与其关联，用于管理页面的使用。结构如下：</p><figure class="highlight c"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br><span class="line">21</span><br><span class="line">22</span><br><span class="line">23</span><br><span class="line">24</span><br><span class="line">25</span><br><span class="line">26</span><br><span class="line">27</span><br><span class="line">28</span><br><span class="line">29</span><br><span class="line">30</span><br><span class="line">31</span><br><span class="line">32</span><br><span class="line">33</span><br><span class="line">34</span><br><span class="line">35</span><br><span class="line">36</span><br><span class="line">37</span><br><span class="line">38</span><br><span class="line">39</span><br><span class="line">40</span><br><span class="line">41</span><br><span class="line">42</span><br><span class="line">43</span><br><span class="line">44</span><br><span class="line">45</span><br><span class="line">46</span><br><span class="line">47</span><br><span class="line">48</span><br><span class="line">49</span><br><span class="line">50</span><br><span class="line">51</span><br><span class="line">52</span><br><span class="line">53</span><br><span class="line">54</span><br><span class="line">55</span><br><span class="line">56</span><br><span class="line">57</span><br><span class="line">58</span><br><span class="line">59</span><br><span class="line">60</span><br><span class="line">61</span><br><span class="line">62</span><br><span class="line">63</span><br><span class="line">64</span><br><span class="line">65</span><br><span class="line">66</span><br><span class="line">67</span><br><span class="line">68</span><br><span class="line">69</span><br><span class="line">70</span><br><span class="line">71</span><br><span class="line">72</span><br><span class="line">73</span><br><span class="line">74</span><br><span class="line">75</span><br><span class="line">76</span><br><span class="line">77</span><br><span class="line">78</span><br><span class="line">79</span><br><span class="line">80</span><br><span class="line">81</span><br><span class="line">82</span><br><span class="line">83</span><br><span class="line">84</span><br><span class="line">85</span><br><span class="line">86</span><br><span class="line">87</span><br><span class="line">88</span><br><span class="line">89</span><br><span class="line">90</span><br><span class="line">91</span><br><span class="line">92</span><br><span class="line">93</span><br><span class="line">94</span><br><span class="line">95</span><br><span class="line">96</span><br><span class="line">97</span><br><span class="line">98</span><br><span class="line">99</span><br><span class="line">100</span><br><span class="line">101</span><br></pre></td><td class="code"><pre><span class="line"><span class="class"><span class="keyword">struct</span> <span class="title">page</span> {</span></span><br><span class="line"> <span class="comment">/* First double word block */</span></span><br><span class="line"> <span class="keyword">unsigned</span> <span class="keyword">long</span> flags; </span><br><span class="line"> <span class="keyword">union</span> {</span><br><span class="line"> <span class="class"><span class="keyword">struct</span> <span class="title">address_space</span> *<span class="title">mapping</span>;</span> </span><br><span class="line"> <span class="keyword">void</span> *s_mem; <span class="comment">/* slab first object */</span></span><br><span class="line"> <span class="keyword">atomic_t</span> compound_mapcount; <span class="comment">/* first tail page */</span></span><br><span class="line"> };</span><br><span class="line"></span><br><span class="line"> <span class="comment">/* Second double word */</span></span><br><span class="line"> <span class="keyword">union</span> {</span><br><span class="line"> <span class="keyword">pgoff_t</span> index; <span class="comment">/* Our offset within mapping. */</span></span><br><span class="line"> <span class="keyword">void</span> *freelist; <span class="comment">/* slub first free object */</span></span><br><span class="line"> };</span><br><span class="line"></span><br><span class="line"> <span class="keyword">union</span> {</span><br><span class="line"><span class="meta">#<span class="meta-keyword">if</span> defined(CONFIG_HAVE_CMPXCHG_DOUBLE) && \</span></span><br><span class="line"> defined(CONFIG_HAVE_ALIGNED_STRUCT_PAGE)</span><br><span class="line"> <span class="comment">/* Used for cmpxchg_double in slub */</span></span><br><span class="line"> <span class="keyword">unsigned</span> <span class="keyword">long</span> counters;</span><br><span class="line"><span class="meta">#<span class="meta-keyword">else</span></span></span><br><span class="line"> <span class="keyword">unsigned</span> counters;</span><br><span class="line"><span class="meta">#<span class="meta-keyword">endif</span></span></span><br><span class="line"> <span class="class"><span class="keyword">struct</span> {</span></span><br><span class="line"> <span class="keyword">union</span> {</span><br><span class="line"> <span class="keyword">atomic_t</span> _mapcount;</span><br><span class="line"> <span class="keyword">unsigned</span> <span class="keyword">int</span> active; <span class="comment">/* SLAB */</span></span><br><span class="line"> <span class="class"><span class="keyword">struct</span> {</span> <span class="comment">/* SLUB */</span></span><br><span class="line"> <span class="keyword">unsigned</span> inuse:<span class="number">16</span>;</span><br><span class="line"> <span class="keyword">unsigned</span> objects:<span class="number">15</span>;</span><br><span class="line"> <span class="keyword">unsigned</span> frozen:<span class="number">1</span>;</span><br><span class="line"> };</span><br><span class="line"> <span class="keyword">int</span> units; <span class="comment">/* SLOB */</span></span><br><span class="line"> };</span><br><span class="line"> <span class="keyword">atomic_t</span> _refcount;</span><br><span class="line"> };</span><br><span class="line"> };</span><br><span class="line"></span><br><span class="line"> <span class="comment">/* Third double word block */</span></span><br><span class="line"> <span class="keyword">union</span> {</span><br><span class="line"> <span class="class"><span class="keyword">struct</span> <span class="title">list_head</span> <span class="title">lru</span>;</span> </span><br><span class="line"> <span class="class"><span class="keyword">struct</span> <span class="title">dev_pagemap</span> *<span class="title">pgmap</span>;</span> </span><br><span class="line"> <span class="class"><span class="keyword">struct</span> {</span> <span class="comment">/* slub per cpu partial pages */</span></span><br><span class="line"> <span class="class"><span class="keyword">struct</span> <span class="title">page</span> *<span class="title">next</span>;</span> <span class="comment">/* Next partial slab */</span></span><br><span class="line"><span class="meta">#<span class="meta-keyword">ifdef</span> CONFIG_64BIT</span></span><br><span class="line"> <span class="keyword">int</span> pages; <span class="comment">/* Nr of partial slabs left */</span></span><br><span class="line"> <span class="keyword">int</span> pobjects; <span class="comment">/* Approximate # of objects */</span></span><br><span class="line"><span class="meta">#<span class="meta-keyword">else</span></span></span><br><span class="line"> short <span class="keyword">int</span> pages;</span><br><span class="line"> short <span class="keyword">int</span> pobjects;</span><br><span class="line"><span class="meta">#<span class="meta-keyword">endif</span></span></span><br><span class="line"> };</span><br><span class="line"></span><br><span class="line"> <span class="class"><span class="keyword">struct</span> <span class="title">rcu_head</span> <span class="title">rcu_head</span>;</span> </span><br><span class="line"> <span class="class"><span class="keyword">struct</span> {</span></span><br><span class="line"> <span class="keyword">unsigned</span> <span class="keyword">long</span> compound_head; </span><br><span class="line"><span class="meta">#<span class="meta-keyword">ifdef</span> CONFIG_64BIT</span></span><br><span class="line"> <span class="keyword">unsigned</span> <span class="keyword">int</span> compound_dtor;</span><br><span class="line"> <span class="keyword">unsigned</span> <span class="keyword">int</span> compound_order;</span><br><span class="line"><span class="meta">#<span class="meta-keyword">else</span></span></span><br><span class="line"> <span class="keyword">unsigned</span> short <span class="keyword">int</span> compound_dtor;</span><br><span class="line"> <span class="keyword">unsigned</span> short <span class="keyword">int</span> compound_order;</span><br><span class="line"><span class="meta">#<span class="meta-keyword">endif</span></span></span><br><span class="line"> };</span><br><span class="line"></span><br><span class="line"><span class="meta">#<span class="meta-keyword">if</span> defined(CONFIG_TRANSPARENT_HUGEPAGE) && USE_SPLIT_PMD_PTLOCKS</span></span><br><span class="line"> <span class="class"><span class="keyword">struct</span> {</span></span><br><span class="line"> <span class="keyword">unsigned</span> <span class="keyword">long</span> __pad; </span><br><span class="line"> <span class="keyword">pgtable_t</span> pmd_huge_pte; <span class="comment">/* protected by page->ptl */</span></span><br><span class="line"> };</span><br><span class="line"><span class="meta">#<span class="meta-keyword">endif</span></span></span><br><span class="line"> };</span><br><span class="line"></span><br><span class="line"> <span class="comment">/* Remainder is not double word aligned */</span></span><br><span class="line"> <span class="keyword">union</span> {</span><br><span class="line"> <span class="keyword">unsigned</span> <span class="keyword">long</span> <span class="keyword">private</span>; </span><br><span class="line"><span class="meta">#<span class="meta-keyword">if</span> USE_SPLIT_PTE_PTLOCKS</span></span><br><span class="line"><span class="meta">#<span class="meta-keyword">if</span> ALLOC_SPLIT_PTLOCKS</span></span><br><span class="line"> <span class="keyword">spinlock_t</span> *ptl;</span><br><span class="line"><span class="meta">#<span class="meta-keyword">else</span></span></span><br><span class="line"> <span class="keyword">spinlock_t</span> ptl;</span><br><span class="line"><span class="meta">#<span class="meta-keyword">endif</span></span></span><br><span class="line"><span class="meta">#<span class="meta-keyword">endif</span></span></span><br><span class="line"> <span class="class"><span class="keyword">struct</span> <span class="title">kmem_cache</span> *<span class="title">slab_cache</span>;</span> <span class="comment">/* SLuB: Pointer to slab */</span></span><br><span class="line"> };</span><br><span class="line"></span><br><span class="line"><span class="meta">#<span class="meta-keyword">ifdef</span> CONFIG_MEMCG</span></span><br><span class="line"> <span class="class"><span class="keyword">struct</span> <span class="title">mem_cgroup</span> *<span class="title">mem_cgroup</span>;</span></span><br><span class="line"><span class="meta">#<span class="meta-keyword">endif</span></span></span><br><span class="line"><span class="meta">#<span class="meta-keyword">if</span> defined(WANT_PAGE_VIRTUAL)</span></span><br><span class="line"> <span class="keyword">void</span> *<span class="keyword">virtual</span>; <span class="comment">/* Kernel virtual address (NULL if not kmapped, ie. highmem) */</span></span><br><span class="line"><span class="meta">#<span class="meta-keyword">endif</span> <span class="comment">/* WANT_PAGE_VIRTUAL */</span></span></span><br><span class="line"></span><br><span class="line"><span class="meta">#<span class="meta-keyword">ifdef</span> CONFIG_KMEMCHECK</span></span><br><span class="line"> <span class="keyword">void</span> *shadow;</span><br><span class="line"><span class="meta">#<span class="meta-keyword">endif</span></span></span><br><span class="line"></span><br><span class="line"><span class="meta">#<span class="meta-keyword">ifdef</span> LAST_CPUPID_NOT_IN_PAGE_FLAGS</span></span><br><span class="line"> <span class="keyword">int</span> _last_cpupid;</span><br><span class="line"><span class="meta">#<span class="meta-keyword">endif</span></span></span><br><span class="line">}</span><br></pre></td></tr></table></figure><p>页面结构体使用双字块来划分：</p><ul><li><p>第一个双字</p><ul><li><p>flags: 页面状态，脏页、上锁等院子标记</p></li><li><p>联合体</p><ul><li><p>mapping：指向inode address_space</p></li><li><p>s_mem：slab首对象</p></li><li><p>compound_mapcount：</p></li></ul></li></ul></li><li><p>第二个双字：</p><ul><li><p>联合体</p><ul><li><p>index：页面偏移</p></li><li><p>freelist：slab/slob的首个可用对象</p></li></ul></li><li><p>联合体：slab/slub/slob相关的记数（取决于编译内核时选择的管理方式）</p></li></ul></li><li><p>第三个双字：</p><ul><li><p>lru：换出页列表</p></li><li><p>pgmap:</p></li><li><p>rcu_head</p></li><li><p>结构体，用于slub管理</p></li><li><p>结构体，用于复合页管理</p></li></ul></li><li><p>联合体(ptl/slab_cache): slab指针，或者PTE自旋锁</p></li><li><p>virtual: 内核虚拟地址。用于高端内存中的页，即无法直接映射到内核内存中的页</p></li></ul><h2 id="页表"><a href="#页表" class="headerlink" title="页表"></a>页表</h2><p>Linux进行内存寻址时，往往不会直接内存物理地址，需要经过虚拟地址到物理地址的转化。使用虚拟地址的好处是可以避免进程与进程间互踩内存（除非特别指定共享内存），同时虚拟内存的换出换入使得进程使用超过物理内存大小的内存范围。</p><p>CPU中内存管理单元（MMU）作用就是根据内存中特定的转化表格（不错，页表本身也是需要内存存储的），将虚拟地址转化为真正的物理地址。而这个表格就是我们所讲的页表。</p><p>取决于体系结构，Linux采用三级或者四级页表机制：</p><ul><li>PGD：Page Global Directory，全局页表目录</li><li>PUD：Page Upper Directory，上级页表目录</li><li>PMD：Page Middle Directory，中级页表目录</li><li>PTE：Page Table Entry，页表表项</li></ul><p>每级表项所占位数，取决于我们编译内核时的选择。一般情况下，取决于寻址宽度，以及CPU体系结构每级页表所占位数是有约定俗成的。</p><p><img src="http://pic.l2h.site/%E5%B1%8F%E5%B9%95%E5%BF%AB%E7%85%A7-2019-08-13-%E4%B8%8A%E5%8D%887.56.58.png" alt></p><p>内核在arch/xxx/include/asm/page.h（其中xxx表示CPU体系结构）定义了一系列的类型、函数和宏来方便对每级页表进行操作。</p><p>如上图我们看到的几个SHIFT宏定义，是为了方便通过位移操作来快速获取对应等级页表。</p><p>在IA64中用来表示以上各级页表目录的数据结构定义如下：</p><figure class="highlight c"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br></pre></td><td class="code"><pre><span class="line"> <span class="keyword">typedef</span> <span class="class"><span class="keyword">struct</span> {</span> <span class="keyword">unsigned</span> <span class="keyword">long</span> pte; } <span class="keyword">pte_t</span>;</span><br><span class="line"> <span class="keyword">typedef</span> <span class="class"><span class="keyword">struct</span> {</span> <span class="keyword">unsigned</span> <span class="keyword">long</span> pmd; } <span class="keyword">pmd_t</span>;</span><br><span class="line"><span class="meta">#<span class="meta-keyword">if</span> CONFIG_PGTABLE_LEVELS == 4</span></span><br><span class="line"> <span class="keyword">typedef</span> <span class="class"><span class="keyword">struct</span> {</span> <span class="keyword">unsigned</span> <span class="keyword">long</span> pud; } <span class="keyword">pud_t</span>;</span><br><span class="line"><span class="meta">#<span class="meta-keyword">endif</span></span></span><br><span class="line"> <span class="keyword">typedef</span> <span class="class"><span class="keyword">struct</span> {</span> <span class="keyword">unsigned</span> <span class="keyword">long</span> pgd; } <span class="keyword">pgd_t</span>;</span><br></pre></td></tr></table></figure><p>与页表相关的宏或者函数定义有<a href="https://elixir.bootlin.com/linux/latest/ident/pmd_alloc" target="_blank" rel="noopener">pmd/pte/pgd_alloc</a>/free()等等，具体可以参考include/linux/mm.h。</p><h2 id="结语"><a href="#结语" class="headerlink" title="结语"></a>结语</h2><p>本文介绍了Linux内核内存管理的基本单位划分Node、Zone和Page及对应的数据结构，同时对页表的基本概念进行了介绍。将在下一文分析Linux初始化流程中对内存的管理。</p>]]></content>
<summary type="html">
<p>内存管理是Linux内核最为复杂且最为重要的部分，本文从原理及代码角度对Linux内存管理机制进行分析。</p><h2 id="内存的划分"><a href="#内存的划分" class="headerlink" title="内存的划分"></a>内存的划分</h2><p>Linux将内存从大到小依次划分为Node（节点）-&gt;Zone（内存域）-&gt;Page（页）：</p><ul>
<li>节点：在大型结算及系统中，内存有不同的簇，依据对处理器距离的不同，访问这些簇有不同的代价。而这些簇就可以成为节点。例：在PC系统中可以理解为实际挂载的物理内存；在嵌入式系统中，有两块内存芯片A和B，分别代表一个节点。</li>
<li>内存域：内存域并不是物理存在的概念，是Linux系统对每个内存节点进行管理的单位，每个节点的内存域表示的是对该节点不同地址范围的划分。一般内存域有三种，分别为Normal、DMA和HighMem。</li>
<li>页：在每个内存域中，内存被划分为大小固定的块（32位系统一般为4K大小），为内核进行内存分配的基本单位（当然内核内存管理机制其实更为复杂，“基本单位”不代表每次分配内存最小就要分到4K。后边可以看到，当需要获取小于4K大小的内存时，内核有Slab分配器来满足要求）</li>
</ul>
</summary>
<category term="Linux" scheme="http://www.l2h.site/categories/Linux/"/>
<category term="Linux内存管理" scheme="http://www.l2h.site/categories/Linux/Linux%E5%86%85%E5%AD%98%E7%AE%A1%E7%90%86/"/>
<category term="Linux" scheme="http://www.l2h.site/tags/Linux/"/>
<category term="内存管理" scheme="http://www.l2h.site/tags/%E5%86%85%E5%AD%98%E7%AE%A1%E7%90%86/"/>
<category term="内核" scheme="http://www.l2h.site/tags/%E5%86%85%E6%A0%B8/"/>
</entry>
<entry>
<title>Linux内核内存管理 - 从内核启动过程透视内存管理</title>
<link href="http://www.l2h.site/p/8f5117c5.html"/>
<id>http://www.l2h.site/p/8f5117c5.html</id>
<published>2021-09-05T09:56:48.000Z</published>
<updated>2023-06-01T13:14:04.480Z</updated>
<content type="html"><![CDATA[<blockquote><p>这是<Linux内核内存管理>系列的第三篇</p><blockquote><p><a href="/p/61304bbb.html">第一篇</a>为内核内存管理过程知识点的的简单梳理</p><p><a href="/p/1dc6adc2.html">第二篇</a>介绍了内核的数据结构</p></blockquote></blockquote><h2 id="前言"><a href="#前言" class="headerlink" title="前言"></a>前言</h2><p>以Intel X64 CPU为例，Linux的初始化可大致分为如下几个过程：</p><ol><li>Loader跳转到内核后的实模式(Real Mode)</li><li>32位保护模式跳转到64位长模式</li><li>64位长模式下解压内核</li><li>解压内核后，建立新的页表映射，并跳转到Arch（平台）相关C代码</li><li>执行平台无关初始化代码</li></ol><a id="more"></a><p>内存管理占据了以上过程的重要角色。包括了内存布局规划、分段管理、页表配置、内核移动等。</p><p>本文使用Qemu模拟，基于Linux v5.13.9版本，按顺序介绍以上过程中的内存管理。</p><h2 id="实模式-Real-Mode"><a href="#实模式-Real-Mode" class="headerlink" title="实模式(Real Mode)"></a>实模式(Real Mode)</h2><p>使用如下命令启动编译好的64位内核：</p><figure class="highlight shell"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><span class="line">qemu-system-x86_64 -kernel arch/x86/boot/bzImage -nographic -append "console=ttyS0 nokaslr" -s -S</span><br></pre></td></tr></table></figure><p>其中：</p><ul><li>内核参数”<strong>console=ttyS0 nokaslr</strong>“的主要作用为指定内核控制台，以及关闭KASLR功能（主要原因是为了调试方便，开启KASLR后每次开机内核解压的地址都是随机的）。</li><li>而-s和-S参数主要是为了GDB调试Qemu之用。 </li></ul><p>执行上述命令后，便得到如下图的内核地址分布。<br><img src="http://pic.l2h.site/Linux%20Boot%20Layout-1.png" alt="实模式内存分布" title="Figure 1. 实模式下内存分布"></p><p>根据内核文档<a href="https://www.kernel.org/doc/html/latest/x86/boot.html" target="_blank" rel="noopener">Linux/x86 Boot Protocol</a>，任何Boot Loader(Grub/Lilo/…)加载X86内核，均要遵守该协议。内核发展至今，该协议版本已经发展到了2.15。图中<em>X</em>为Boot Loader加载内核的起始偏移，在Qemu平台上该偏移为 <strong>0x10000</strong>。 加载后，内核Boot Sector开始执行，执行入口点为 <strong>_start</strong>。参考Linker Script <strong>arch/x86/boot/setup.ld</strong>。</p><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br></pre></td><td class="code"><pre><span class="line">OUTPUT_FORMAT("elf32-i386")</span><br><span class="line">OUTPUT_ARCH(i386)</span><br><span class="line">ENTRY(_start)</span><br><span class="line"></span><br><span class="line">SECTIONS</span><br><span class="line">{</span><br><span class="line">. = 0;</span><br><span class="line">.bstext: { *(.bstext) }</span><br><span class="line">.bsdata: { *(.bsdata) }</span><br><span class="line">....</span><br></pre></td></tr></table></figure><p>这里会直接跳转到<strong>start_of_setup</strong>开始执行。</p><figure class="highlight x86asm"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br><span class="line">21</span><br><span class="line">22</span><br><span class="line">23</span><br><span class="line">24</span><br><span class="line">25</span><br><span class="line">26</span><br><span class="line">27</span><br><span class="line">28</span><br><span class="line">29</span><br><span class="line">30</span><br><span class="line">31</span><br><span class="line">32</span><br><span class="line">33</span><br><span class="line">34</span><br><span class="line">35</span><br><span class="line">36</span><br><span class="line">37</span><br><span class="line">38</span><br><span class="line">39</span><br><span class="line">40</span><br><span class="line">41</span><br><span class="line">42</span><br><span class="line">43</span><br><span class="line">44</span><br><span class="line">45</span><br><span class="line">46</span><br><span class="line">47</span><br><span class="line">48</span><br><span class="line">49</span><br><span class="line">50</span><br><span class="line">51</span><br><span class="line">52</span><br><span class="line">53</span><br><span class="line">54</span><br></pre></td><td class="code"><pre><span class="line">#arch/x86/boot/header<span class="number">.</span>S</span><br><span class="line"><span class="meta">.globl</span>_start</span><br><span class="line"><span class="symbol">_start:</span></span><br><span class="line"><span class="meta">.byte</span><span class="number">0xeb</span># short (<span class="number">2</span>-<span class="built_in">byte</span>) jump</span><br><span class="line"><span class="meta">.byte</span>start_of_setup-1f</span><br><span class="line"></span><br><span class="line"><span class="meta">.section</span> <span class="string">".entrytext"</span>, <span class="string">"ax"</span></span><br><span class="line"><span class="symbol">start_of_setup:</span></span><br><span class="line"># Force %es = %ds</span><br><span class="line">movw%ds, %ax</span><br><span class="line">movw%ax, %es</span><br><span class="line"><span class="keyword">cld</span></span><br><span class="line"></span><br><span class="line">movw%ss, %dx</span><br><span class="line">cmpw%ax, %dx# %ds == %ss?</span><br><span class="line">movw%sp, %dx</span><br><span class="line"><span class="keyword">je</span>2f# -> <span class="meta">assume</span> %sp is reasonably set</span><br><span class="line"></span><br><span class="line"># Invalid %ss, make <span class="meta">up</span> a new stack</span><br><span class="line">movw$_end, %dx</span><br><span class="line">testb$CAN_USE_HEAP, loadflags</span><br><span class="line"><span class="keyword">jz</span>1f</span><br><span class="line">movwheap_end_ptr, %dx</span><br><span class="line"><span class="number">1</span>:addw$STACK_SIZE, %dx</span><br><span class="line"><span class="keyword">jnc</span>2f</span><br><span class="line">xorw%dx, %dx# Prevent wraparound</span><br><span class="line"></span><br><span class="line"><span class="number">2</span>:# Now %dx should point to the end of our stack space</span><br><span class="line">andw$~<span class="number">3</span>, %dx# <span class="built_in">dword</span> <span class="meta">align</span> (might as well...)</span><br><span class="line"><span class="keyword">jnz</span>3f</span><br><span class="line">movw<span class="number">$0</span>xfffc, %dx# Make sure we<span class="string">'re not zero</span></span><br><span class="line"><span class="string">3:movw%ax, %ss</span></span><br><span class="line"><span class="string">movzwl%dx, %esp# Clear upper half of %esp</span></span><br><span class="line"><span class="string">sti# Now we should have a working stack</span></span><br><span class="line"><span class="string"></span></span><br><span class="line"><span class="string"># We will have entered with %cs = %ds+0x20, normalize %cs so it is on par with the other segments.</span></span><br><span class="line"><span class="string">pushw%ds</span></span><br><span class="line"><span class="string">pushw$6f</span></span><br><span class="line"><span class="string">lretw</span></span><br><span class="line"><span class="string">6:</span></span><br><span class="line"><span class="string"># Check signature at end of setup</span></span><br><span class="line"><span class="string">cmpl$0x5a5aaa55, setup_sig</span></span><br><span class="line"><span class="string">jnesetup_bad</span></span><br><span class="line"><span class="string"></span></span><br><span class="line"><span class="string"># Zero the bss</span></span><br><span class="line"><span class="string">movw$__bss_start, %di</span></span><br><span class="line"><span class="string">movw$_end+3, %cx</span></span><br><span class="line"><span class="string">xorl%eax, %eax</span></span><br><span class="line"><span class="string">subw%di, %cx</span></span><br><span class="line"><span class="string">shrw$2, %cx</span></span><br><span class="line"><span class="string">rep; stosl</span></span><br><span class="line"><span class="string"></span></span><br><span class="line"><span class="string"># Jump to C code (should not return)</span></span><br><span class="line"><span class="string">calllmain</span></span><br></pre></td></tr></table></figure><p>以上代码会为实模式代码执行清理方向位，并未C代码的执行分配堆空间和栈空间。接着跳转到6执行，检查内核代码加载的正确性。这里说明一下， <strong>lretw</strong>及之前两行汇编语句的作用是调用返回，之前两行是将返回地址保存在栈内，参考<<a href="https://software.intel.com/content/www/us/en/develop/download/intel-64-and-ia-32-architectures-sdm-combined-volumes-1-2a-2b-2c-2d-3a-3b-3c-3d-and-4.html" target="_blank" rel="noopener">Intel® 64 and IA-32 Architectures Software Developer’s Manual</a>>。如注释，使用lret的目的是为了重置CS寄存器的值，确保与其他段寄存器一致。可参考Intel手册，ret指令的说明：</p><blockquote><p>When executing a far return, the processor pops the return instruction pointer from the top of the stack into the EIP<br>register, <strong>then pops the segment selector from the top of the stack into the CS register</strong>. The processor then begins<br>program execution in the new code segment at the new instruction pointer.</p></blockquote><p>接着清空BSS段后跳转到main函数执行。</p><figure class="highlight c"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br><span class="line">21</span><br><span class="line">22</span><br><span class="line">23</span><br></pre></td><td class="code"><pre><span class="line"><span class="comment">/* First, copy the boot header into the "zeropage" */</span></span><br><span class="line">copy_boot_params();</span><br><span class="line">console_init();</span><br><span class="line"><span class="keyword">if</span> (cmdline_find_option_bool(<span class="string">"debug"</span>))</span><br><span class="line"><span class="built_in">puts</span>(<span class="string">"early console in setup code\n"</span>);</span><br><span class="line">init_heap();</span><br><span class="line"><span class="keyword">if</span> (validate_cpu()) {</span><br><span class="line"><span class="built_in">puts</span>(<span class="string">"Unable to boot - please use a kernel appropriate "</span></span><br><span class="line"> <span class="string">"for your CPU.\n"</span>);</span><br><span class="line">die();</span><br><span class="line">}</span><br><span class="line">set_bios_mode();</span><br><span class="line">detect_memory();</span><br><span class="line">keyboard_init();</span><br><span class="line">query_ist();</span><br><span class="line"><span class="meta">#<span class="meta-keyword">if</span> defined(CONFIG_APM) || defined(CONFIG_APM_MODULE)</span></span><br><span class="line">query_apm_bios();</span><br><span class="line"><span class="meta">#<span class="meta-keyword">endif</span></span></span><br><span class="line"><span class="meta">#<span class="meta-keyword">if</span> defined(CONFIG_EDD) || defined(CONFIG_EDD_MODULE)</span></span><br><span class="line">query_edd();</span><br><span class="line"><span class="meta">#<span class="meta-keyword">endif</span></span></span><br><span class="line">set_video();</span><br><span class="line">go_to_protected_mode();</span><br></pre></td></tr></table></figure><p><strong>main</strong>函数的注释比较清楚,我们这里只讲一下copy_boot_param/detect_memory/go_to_protected_mode:</p><ul><li>copy_boot_param将内存中boot_param的信息（见图“实模式内存分布”）拷贝到全局变量boot_params内。boot_params存放的就是Linux Boot Protocol定义的哪些参数。有些栏位是编译过程中改写，有些烂尾由Boot Loader填写。boot_param包括内核cmdline会穿插内核初始化的各个子过程中</li><li>detect_memory主要是使用<a href="https://en.wikipedia.org/wiki/E820" target="_blank" rel="noopener">e820</a>获取内存的基础布局，存储到boot_param指定区域（<strong>boot_params.e820_table</strong>和<strong>boot_params.e820_entries</strong>）。</li><li>go_to_protected_mode主要是打开32位地址线（<a href="https://en.wikipedia.org/wiki/A20_line" target="_blank" rel="noopener">A20 Gate</a>），做GDT/IDT表的配置，关中断，打开保护模式，并跳转到32位代码开始执行。代码如下：</li></ul><figure class="highlight c"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br></pre></td><td class="code"><pre><span class="line"><span class="comment">//arch/x86/boot/pm.c</span></span><br><span class="line"><span class="function"><span class="keyword">void</span> <span class="title">go_to_protected_mode</span><span class="params">(<span class="keyword">void</span>)</span></span></span><br><span class="line"><span class="function"></span>{</span><br><span class="line">realmode_switch_hook();</span><br><span class="line"></span><br><span class="line"><span class="comment">/* Enable the A20 gate */</span></span><br><span class="line"><span class="keyword">if</span> (enable_a20()) {</span><br><span class="line"><span class="built_in">puts</span>(<span class="string">"A20 gate not responding, unable to boot...\n"</span>);</span><br><span class="line">die();</span><br><span class="line">}</span><br><span class="line"></span><br><span class="line">reset_coprocessor();</span><br><span class="line"></span><br><span class="line">mask_all_interrupts();</span><br><span class="line"></span><br><span class="line">setup_idt();</span><br><span class="line">setup_gdt();</span><br><span class="line">protected_mode_jump(boot_params.hdr.code32_start,</span><br><span class="line"> (u32)&boot_params + (ds() << <span class="number">4</span>));</span><br><span class="line">}</span><br></pre></td></tr></table></figure><p><strong>protected_mode_jump</strong>是一段汇编代码，定义在arch/x86/boot/pmjump.S，这里不多过多分析。其主要就是修改CR0寄存器的PE（Protect Enable）位，并执行跳转指令跳转到32位代码(.Lin_pm32标号)处执行。</p><figure class="highlight x86asm"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br><span class="line">21</span><br><span class="line">22</span><br><span class="line">23</span><br><span class="line">24</span><br><span class="line">25</span><br><span class="line">26</span><br><span class="line">27</span><br><span class="line">28</span><br><span class="line">29</span><br><span class="line">30</span><br><span class="line">31</span><br><span class="line">32</span><br><span class="line">33</span><br><span class="line">34</span><br><span class="line">35</span><br><span class="line">36</span><br><span class="line">37</span><br><span class="line">38</span><br><span class="line">39</span><br><span class="line">40</span><br><span class="line">41</span><br><span class="line">42</span><br><span class="line">43</span><br><span class="line">44</span><br></pre></td><td class="code"><pre><span class="line">#arch/x86/boot/pmjump<span class="number">.</span>S</span><br><span class="line">/*</span><br><span class="line"> * void protected_mode_jump(u32 entrypoint, u32 bootparams)<span class="comment">;</span></span><br><span class="line"> */</span><br><span class="line">SYM_FUNC_START_NOALIGN(protected_mode_jump)</span><br><span class="line">........</span><br><span class="line"></span><br><span class="line">movl%cr0, %edx</span><br><span class="line">orb$X86_CR0_PE, %dl# Protected mode</span><br><span class="line">movl%edx, %cr0</span><br><span class="line"></span><br><span class="line"># Transition to <span class="number">32</span>-bit mode</span><br><span class="line"><span class="meta">.byte</span><span class="number">0x66</span>, <span class="number">0xea</span># ljmpl opcode</span><br><span class="line"><span class="number">2</span>:.long.Lin_pm32# offset</span><br><span class="line"><span class="meta">.word</span>__BOOT_CS# <span class="meta">segment</span></span><br><span class="line">SYM_FUNC_END(protected_mode_jump)</span><br><span class="line"></span><br><span class="line">SYM_FUNC_START_LOCAL_NOALIGN(.Lin_pm32)</span><br><span class="line"># Set <span class="meta">up</span> data segments for flat <span class="number">32</span>-bit mode</span><br><span class="line">movl%ecx, %ds</span><br><span class="line">movl%ecx, %es</span><br><span class="line">movl%ecx, %fs</span><br><span class="line">movl%ecx, %gs</span><br><span class="line">movl%ecx, %ss</span><br><span class="line"># The <span class="number">32</span>-bit code <span class="keyword">sets</span> <span class="meta">up</span> its own stack, but this way we <span class="built_in">do</span> have</span><br><span class="line"># a valid stack if some debugging hack wants to use it.</span><br><span class="line">addl%ebx, %esp</span><br><span class="line"></span><br><span class="line"># Set <span class="meta">up</span> TR to make Intel VT happy</span><br><span class="line"><span class="keyword">ltr</span>%di</span><br><span class="line"></span><br><span class="line"># Clear registers to allow for future extensions to the</span><br><span class="line"># <span class="number">32</span>-bit boot protocol</span><br><span class="line">xorl%ecx, %ecx</span><br><span class="line">xorl%edx, %edx</span><br><span class="line">xorl%ebx, %ebx</span><br><span class="line">xorl%ebp, %ebp</span><br><span class="line">xorl%edi, %edi</span><br><span class="line"></span><br><span class="line"># Set <span class="meta">up</span> LDTR to make Intel VT happy</span><br><span class="line"><span class="keyword">lldt</span>%cx</span><br><span class="line"></span><br><span class="line">jmpl*%eax# Jump to the <span class="number">32</span>-bit entrypoint</span><br><span class="line">SYM_FUNC_END(.Lin_pm32)</span><br></pre></td></tr></table></figure><p>32位代码伊始就是重建各个段寄存器为<strong>BOOT_DS。段寄存器内容为向GDT某项的段选择子，而</strong>BOOT_DS即为GDT的第三个表项。 此时GDT的表项可以到arch/x86/boot/pm.c查找，大概定义了Base为0大小为4G的段，这足以覆盖内核初始化32位代码执行的区域。有关GDT表及段选择相关知识，可以查阅<<a href="https://software.intel.com/content/www/us/en/develop/download/intel-64-and-ia-32-architectures-sdm-combined-volumes-1-2a-2b-2c-2d-3a-3b-3c-3d-and-4.html" target="_blank" rel="noopener">Intel® 64 and IA-32 Architectures Software Developer’s Manual</a>>中Volume 3，CHAPTER 3 PROTECTED-MODE MEMORY MANAGEMENT一节。做一些寄存器内容的清理，就跳转到32位内核的起始地址执行。 </p><p>该起始地址，是protected_mode_jump函数的第一个参数-boot_params.hdr.code32_start。在我们的QEMU环境中这个值为<strong>0x100000</strong></p><blockquote><p>为什么是存储在eax寄存器呢，这里就需要了解<a href="https://uclibc.org/docs/psABI-x86_64.pdf" target="_blank" rel="noopener">System V Application Binary Interface AMD64</a>中有关<strong>calling convention</strong>的知识,Linux内核也是遵守System V ABI的。ABI指的是Application Binary Interface，根据程序运行的Arch不同而有不同的定义。</p></blockquote><h2 id="32位保护模式跳转到64位长模式"><a href="#32位保护模式跳转到64位长模式" class="headerlink" title="32位保护模式跳转到64位长模式"></a>32位保护模式跳转到64位长模式</h2><h3 id="startup-32"><a href="#startup-32" class="headerlink" title="startup_32"></a>startup_32</h3><p>_<br><strong>0x100000</strong>存放的是32位代码起始地址，具体布局可以参考链接脚本：<a href="https://elixir.bootlin.com/linux/v5.13.9/source/arch/x86/boot/compressed/vmlinux.lds.S" target="_blank" rel="noopener">vmlinux.lds</a></p><blockquote><p>链接脚本，即<a href="https://ftp.gnu.org/old-gnu/Manuals/ld-2.9.1/html_chapter/ld_3.html" target="_blank" rel="noopener">Linker Script</a>，这是告诉链接器目标文件该如何链接的脚本。一般GCC编译我们不会指定链接脚本，这是因为其有默认的链接脚本。</p></blockquote><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br><span class="line">21</span><br><span class="line">22</span><br><span class="line">23</span><br><span class="line">24</span><br><span class="line">25</span><br><span class="line">26</span><br><span class="line">27</span><br><span class="line">28</span><br><span class="line">29</span><br><span class="line">30</span><br><span class="line">31</span><br><span class="line">32</span><br><span class="line">33</span><br><span class="line">34</span><br><span class="line">35</span><br><span class="line">36</span><br><span class="line">37</span><br><span class="line">38</span><br><span class="line">39</span><br><span class="line">40</span><br><span class="line">41</span><br><span class="line">42</span><br><span class="line">43</span><br><span class="line">44</span><br><span class="line">45</span><br><span class="line">46</span><br><span class="line">47</span><br><span class="line">48</span><br><span class="line">49</span><br><span class="line">50</span><br><span class="line">51</span><br><span class="line">52</span><br><span class="line">53</span><br><span class="line">54</span><br><span class="line">55</span><br><span class="line">56</span><br><span class="line">57</span><br><span class="line">58</span><br><span class="line">59</span><br><span class="line">60</span><br></pre></td><td class="code"><pre><span class="line">#ifdef CONFIG_X86_64</span><br><span class="line">OUTPUT_ARCH(i386:x86-64)</span><br><span class="line">ENTRY(startup_64)</span><br><span class="line">#else</span><br><span class="line">OUTPUT_ARCH(i386)</span><br><span class="line">ENTRY(startup_32)</span><br><span class="line">#endif</span><br><span class="line"></span><br><span class="line">SECTIONS</span><br><span class="line">{</span><br><span class="line">/* Be careful parts of head_64.S assume startup_32 is at</span><br><span class="line"> * address 0.</span><br><span class="line"> */</span><br><span class="line">. = 0;</span><br><span class="line">.head.text : {</span><br><span class="line">_head = . ;</span><br><span class="line">HEAD_TEXT</span><br><span class="line">_ehead = . ;</span><br><span class="line">}</span><br><span class="line">.rodata..compressed : {</span><br><span class="line">*(.rodata..compressed)</span><br><span class="line">}</span><br><span class="line">.text :{</span><br><span class="line">_text = .; /* Text */</span><br><span class="line">*(.text)</span><br><span class="line">*(.text.*)</span><br><span class="line">_etext = . ;</span><br><span class="line">}</span><br><span class="line">.rodata : {</span><br><span class="line">_rodata = . ;</span><br><span class="line">*(.rodata) /* read-only data */</span><br><span class="line">*(.rodata.*)</span><br><span class="line">_erodata = . ;</span><br><span class="line">}</span><br><span class="line">.data :{</span><br><span class="line">_data = . ;</span><br><span class="line">*(.data)</span><br><span class="line">*(.data.*)</span><br><span class="line">*(.bss.efistub)</span><br><span class="line">_edata = . ;</span><br><span class="line">}</span><br><span class="line">. = ALIGN(L1_CACHE_BYTES);</span><br><span class="line">.bss : {</span><br><span class="line">_bss = . ;</span><br><span class="line">*(.bss)</span><br><span class="line">*(.bss.*)</span><br><span class="line">*(COMMON)</span><br><span class="line">. = ALIGN(8);/* For convenience during zeroing */</span><br><span class="line">_ebss = .;</span><br><span class="line">}</span><br><span class="line">#ifdef CONFIG_X86_64</span><br><span class="line"> . = ALIGN(PAGE_SIZE);</span><br><span class="line"> .pgtable : {</span><br><span class="line">_pgtable = . ;</span><br><span class="line">*(.pgtable)</span><br><span class="line">_epgtable = . ;</span><br><span class="line">}</span><br><span class="line">#endif</span><br><span class="line">. = ALIGN(PAGE_SIZE);/* keep ZO size page aligned */</span><br><span class="line">_end = .;</span><br></pre></td></tr></table></figure><p>经过ld链接、且qemu加载后，得到下图左侧的内存布局。从地址0x100000开始，首先是32位保护模式入口代码、解压缩代码等，之后摆放了压缩的内核。其后分别是解压后内核的代码段、只读数据段、数据段、未初始化数据段和32位代码页表。<br><img src="http://pic.l2h.site/Linux-Boot-Layout-2.png" alt="32位下内存分布" title="Figure 2. 32位下内存分布"></p><p>从链接脚本可以看出：32位代码的入口地址是<strong>startup_32</strong>。代码首先清中断，加载新GDT表，同时重置各段寄存器，建立堆栈。</p><blockquote><p>需要注意代码定义了一个宏<strong>rva</strong>,它的主要作用是为了计算段内相对地址，这样可以避免内核加载到不同位置时，同样的代码皆可执行。</p></blockquote><figure class="highlight x86asm"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br><span class="line">21</span><br><span class="line">22</span><br><span class="line">23</span><br><span class="line">24</span><br><span class="line">25</span><br><span class="line">26</span><br><span class="line">27</span><br><span class="line">28</span><br><span class="line">29</span><br><span class="line">30</span><br><span class="line">31</span><br><span class="line">32</span><br><span class="line">33</span><br><span class="line">34</span><br><span class="line">35</span><br><span class="line">36</span><br><span class="line">37</span><br><span class="line">38</span><br><span class="line">39</span><br><span class="line">40</span><br><span class="line">41</span><br><span class="line">42</span><br><span class="line">43</span><br><span class="line">44</span><br><span class="line">45</span><br><span class="line">46</span><br><span class="line">47</span><br><span class="line">48</span><br><span class="line">49</span><br><span class="line">50</span><br><span class="line">51</span><br><span class="line">52</span><br><span class="line">53</span><br><span class="line">54</span><br><span class="line">55</span><br><span class="line">56</span><br><span class="line">57</span><br><span class="line">58</span><br></pre></td><td class="code"><pre><span class="line">#arch/x86/boot/compressed/head_64<span class="number">.</span>S</span><br><span class="line">#define rva(X) ((X) - startup_32)</span><br><span class="line"></span><br><span class="line"><span class="meta">.code32</span></span><br><span class="line">SYM_FUNC_START(startup_32)</span><br><span class="line"><span class="keyword">cld</span></span><br><span class="line"><span class="keyword">cli</span></span><br><span class="line"></span><br><span class="line">leal(BP_scratch+<span class="number">4</span>)(%esi), %esp</span><br><span class="line"><span class="keyword">call</span>1f</span><br><span class="line"><span class="number">1</span>:popl%ebp</span><br><span class="line">subl$ rva(<span class="number">1b</span>), %ebp</span><br><span class="line"></span><br><span class="line">lealrva(gdt)(%ebp), %eax</span><br><span class="line">movl%eax, <span class="number">2</span>(%eax)</span><br><span class="line"><span class="keyword">lgdt</span>(%eax)</span><br><span class="line"></span><br><span class="line">/* Load <span class="meta">segment</span> registers with our descriptors */</span><br><span class="line">movl$__BOOT_DS, %eax</span><br><span class="line">movl%eax, %ds</span><br><span class="line">movl%eax, %es</span><br><span class="line">movl%eax, %fs</span><br><span class="line">movl%eax, %gs</span><br><span class="line">movl%eax, %ss</span><br><span class="line"></span><br><span class="line">lealrva(boot_stack_end)(%ebp), %esp</span><br><span class="line"></span><br><span class="line">pushl$__KERNEL32_CS</span><br><span class="line">lealrva(1f)(%ebp), %eax</span><br><span class="line">pushl%eax</span><br><span class="line">lretl</span><br><span class="line"><span class="number">1</span>:</span><br><span class="line"><span class="keyword">call</span>startup32_load_idt</span><br><span class="line"><span class="keyword">call</span>verify_cpu</span><br><span class="line">testl%eax, %eax</span><br><span class="line"><span class="keyword">jnz</span>.Lno_longmode</span><br><span class="line"></span><br><span class="line">#ifdef CONFIG_RELOCATABLE</span><br><span class="line">movl%ebp, %ebx</span><br><span class="line">......</span><br><span class="line">movlBP_kernel_alignment(%esi), %eax</span><br><span class="line">decl%eax</span><br><span class="line">addl%eax, %ebx</span><br><span class="line">notl%eax</span><br><span class="line">andl%eax, %ebx</span><br><span class="line">cmpl$LOAD_PHYSICAL_ADDR, %ebx</span><br><span class="line"><span class="keyword">jae</span>1f</span><br><span class="line">#endif</span><br><span class="line">movl$LOAD_PHYSICAL_ADDR, %ebx</span><br><span class="line"><span class="number">1</span>:</span><br><span class="line"></span><br><span class="line">addlBP_init_size(%esi), %ebx</span><br><span class="line">subl$ rva(_end), %ebx</span><br><span class="line"></span><br><span class="line">/* Enable PAE mode */</span><br><span class="line">movl%cr4, %eax</span><br><span class="line">orl$X86_CR4_PAE, %eax</span><br><span class="line">movl%eax, %cr4</span><br></pre></td></tr></table></figure><p>加载IDT后，打开PAE模式。 然后会计算出将压缩内核摆放的位置放到ebx，用于原地(<strong>in-place</strong>)解压。上面代码中<strong>BP_kernel_alignment(%esi)</strong> 主要作用是从boot_param对应区域取出对应的值。我们再次打开<a href="https://www.kernel.org/doc/html/latest/x86/boot.html" target="_blank" rel="noopener">Linux/x86 Boot Protocol</a>和<a href="https://www.kernel.org/doc/html/v5.6/x86/zero-page.html" target="_blank" rel="noopener">Boot Protocol附属栏位</a>查看这些栏位的说明:</p><table><thead><tr><th>偏移/所占字节数</th><th>参数</th><th>描述</th></tr></thead><tbody><tr><td>0230/4</td><td>kernel_alignment</td><td>Physical addr alignment required for kernel</td></tr><tr><td>0260/4</td><td>init_size</td><td>Linear memory required during initialization</td></tr><tr><td>01E4/4</td><td>scratch</td><td>Scratch field for the kernel setup code</td></tr></tbody></table><p>其中init_size存放的是内核初始化、解压所需要的空间，这是根据内核压缩In-place解压预留足够的空间。这部分大小的计算可以参考内核源码<a href="https://elixir.bootlin.com/linux/v5.13.9/source/arch/x86/boot/header.S#L494" target="_blank" rel="noopener">arch/x86/boot/header.S</a>的说明（<strong><em>本人也还没吃透,待补充</em></strong>）。<br>紧接着内核为4GB大小的内存建立每页大小为2MB的内核页表（见Figure 2图右）并加载页表目录地址（<strong>pgtable</strong>）到CR3寄存器，并开启64位长模式。参考<a href="https://en.wikipedia.org/wiki/Long_mode" target="_blank" rel="noopener">Wiki</a>:</p><blockquote><p>当处于长模式（Long mode）时，64位应用程序（或者是操作系统）可以使用64位指令和寄存器，而32位程序将以一种兼容子模式运行。</p></blockquote><p>4GB大小足以执行内核解压等动作。接着内核将64位地址<strong>startup_64</strong>压入栈，开启分页，并执行<em>lret</em>指令跳转到<strong>startup_64</strong>处执行。</p><blockquote><p>此处我们省略了SEV功能的检查，这是AMD CPU的特性。此处不做分析。</p></blockquote><h3 id="startup-64"><a href="#startup-64" class="headerlink" title="startup_64"></a>startup_64</h3><p><strong>startup_64</strong> 的开始同样会清中断，清理各段寄存器。同时计算压缩内核要移动到的地址，即LOAD_PHYSICAL_ADDR + INIT_SIZE - 压缩内核的长度(<strong>rva（_end）</strong>)。此处处理与<strong>startup_32</strong>相同</p><blockquote><p>可能大家会疑惑，为什么这段代码在startup_32做了，此处还要做一遍。主要原因代码内有描述，内核可能会被64位Loader直接加载并从<strong>startup_64</strong>处执行。</p></blockquote><p>接着内核加载空的IDT表，检查是否需要开启五级页表，并做对应处理。紧接着清除EFLAGS寄存器后，将压缩内核移动到In-place解压的位置(LOAD_PHYSICAL_ADDR + INIT_SIZE - 压缩内核的长度)，紧接着重新加载移动过位置的GDT表。之后跳转到移动后的 <strong>.Lrelocated</strong> 地址处开始执行。</p><h3 id="Lrelocated"><a href="#Lrelocated" class="headerlink" title=".Lrelocated"></a>.Lrelocated</h3><p><strong>.Lrelocated</strong> 代码最主要的作用有三个：</p><ul><li><strong>加载IDT</strong>：此时IDT的内容只开启了Page Fault Trap，对应的处理函数是<em>boot_page_fault</em>，其实现在<a href="https://elixir.bootlin.com/linux/v5.13.9/source/arch/x86/boot/compressed/ident_map_64.c#L324" target="_blank" rel="noopener">arch/x86/boot/compressed/ident_map_64.c</a>, 主要作用就是做一些基础检查后，为对应缺页的地址建立一致性映射。</li><li><strong>创建一致性映射</strong>： 主要为[_head, _end], bootparam 和 boot cmdline三个区域建立一致性映射。</li><li><strong>解压内核</strong>： 解压内核本文不做分析。提及一点就是如果开启了KASLR，解压内核前，会计算一个随机偏移生成内核真正的解压地址。</li></ul><p>解压完内核后跳转到加压后内核的入口地址,即arch/x86/kernel/head_64.S的<a href="https://elixir.bootlin.com/linux/v5.13.9/source/arch/x86/kernel/head_64.S#L44" target="_blank" rel="noopener">startup_64标号处</a></p><h2 id="内核解压后"><a href="#内核解压后" class="headerlink" title="内核解压后"></a>内核解压后</h2><p><strong>startup_64</strong> 代码如下：</p><figure class="highlight x86asm"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br><span class="line">21</span><br><span class="line">22</span><br><span class="line">23</span><br><span class="line">24</span><br><span class="line">25</span><br><span class="line">26</span><br><span class="line">27</span><br><span class="line">28</span><br></pre></td><td class="code"><pre><span class="line">SYM_CODE_START_NOALIGN(startup_64)</span><br><span class="line">UNWIND_HINT_EMPTY</span><br><span class="line">leaq(__end_init_task - SIZEOF_PTREGS)(%rip), %rsp</span><br><span class="line"></span><br><span class="line">leaq_text(%rip), %rdi</span><br><span class="line">pushq%rsi</span><br><span class="line"><span class="keyword">call</span>startup_64_setup_env</span><br><span class="line">popq%rsi</span><br><span class="line"></span><br><span class="line">pushq$__KERNEL_CS</span><br><span class="line">leaq.Lon_kernel_cs(%rip), %rax</span><br><span class="line">pushq%rax</span><br><span class="line">lretq</span><br><span class="line"></span><br><span class="line"><span class="symbol">.Lon_kernel_cs:</span></span><br><span class="line">UNWIND_HINT_EMPTY</span><br><span class="line"></span><br><span class="line">/* Sanitize <span class="meta">CPU</span> configuration */</span><br><span class="line"><span class="keyword">call</span> verify_cpu</span><br><span class="line"></span><br><span class="line">leaq_text(%rip), %rdi</span><br><span class="line">pushq%rsi</span><br><span class="line"><span class="keyword">call</span>__startup_64</span><br><span class="line">popq%rsi</span><br><span class="line"></span><br><span class="line">addq$(early_top_pgt - __START_KERNEL_map), %rax</span><br><span class="line"><span class="keyword">jmp</span> 1f</span><br><span class="line">SYM_CODE_END(startup_64)</span><br></pre></td></tr></table></figure><p>以上代码会配置栈之后，调用<strong>startup_64_setup_env</strong>配置Startup GDT和IDT。GDT表的内容如下：</p><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br></pre></td><td class="code"><pre><span class="line">static struct desc_struct startup_gdt[GDT_ENTRIES] = {</span><br><span class="line">[GDT_ENTRY_KERNEL32_CS] = GDT_ENTRY_INIT(0xc09b, 0, 0xfffff),</span><br><span class="line">[GDT_ENTRY_KERNEL_CS] = GDT_ENTRY_INIT(0xa09b, 0, 0xfffff),</span><br><span class="line">[GDT_ENTRY_KERNEL_DS] = GDT_ENTRY_INIT(0xc093, 0, 0xfffff),</span><br><span class="line">};</span><br></pre></td></tr></table></figure><p>Startup GDT中的段描述符,都是0地址开始的4GB大小。Startup IDT（也叫binrgup IDT）主要处理AMD 架构下VMM Communication异常，该异常与虚拟机有关。<br>之后内核继续执行到<strong>verify_cpu</strong>这个汇编函数，其定义在<a href="https://elixir.bootlin.com/linux/v5.13.9/source/arch/x86/kernel/verify_cpu.S#L34" target="_blank" rel="noopener">verify_cpu.S</a>,其主要是使用<em>cpuid</em>指令得到CPU对长模式和SSE指令集的支持状况。<br>检查完后，内核跳转执行 <strong>__startup_64</strong>,其主要作用是重新建立内核早期4级或者5级页表，此时需要考虑KASLR产生的随机偏移，因此我们可以看到此函数调用了多次<strong>fixup_pointer</strong>函数进行页表项纠正。<br>页表定义在<a href="https://elixir.bootlin.com/linux/v5.13.9/source/arch/x86/kernel/head_64.S#L472" target="_blank" rel="noopener">head_64.s</a>，如下：</p><figure class="highlight x86asm"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br><span class="line">21</span><br><span class="line">22</span><br><span class="line">23</span><br><span class="line">24</span><br><span class="line">25</span><br><span class="line">26</span><br><span class="line">27</span><br><span class="line">28</span><br><span class="line">29</span><br><span class="line">30</span><br><span class="line">31</span><br><span class="line">32</span><br><span class="line">33</span><br><span class="line">34</span><br><span class="line">35</span><br><span class="line">36</span><br><span class="line">37</span><br><span class="line">38</span><br><span class="line">39</span><br><span class="line">40</span><br><span class="line">41</span><br><span class="line">42</span><br><span class="line">43</span><br><span class="line">44</span><br><span class="line">45</span><br><span class="line">46</span><br><span class="line">47</span><br><span class="line">48</span><br><span class="line">49</span><br><span class="line">50</span><br><span class="line">51</span><br><span class="line">52</span><br><span class="line">53</span><br><span class="line">54</span><br><span class="line">55</span><br><span class="line">56</span><br><span class="line">57</span><br><span class="line">58</span><br><span class="line">59</span><br><span class="line">60</span><br><span class="line">61</span><br><span class="line">62</span><br><span class="line">63</span><br><span class="line">64</span><br><span class="line">65</span><br><span class="line">66</span><br><span class="line">67</span><br><span class="line">68</span><br><span class="line">69</span><br><span class="line">70</span><br><span class="line">71</span><br><span class="line">72</span><br><span class="line">73</span><br></pre></td><td class="code"><pre><span class="line">SYM_DATA_START_PTI_ALIGNED(early_top_pgt)</span><br><span class="line"><span class="meta">.fill</span><span class="number">512</span>,<span class="number">8</span>,<span class="number">0</span></span><br><span class="line"><span class="meta">.fill</span>PTI_USER_PGD_FILL,<span class="number">8</span>,<span class="number">0</span></span><br><span class="line">SYM_DATA_END(early_top_pgt)</span><br><span class="line"></span><br><span class="line">SYM_DATA_START_PAGE_ALIGNED(early_dynamic_pgts)</span><br><span class="line"><span class="meta">.fill</span><span class="number">512</span>*EARLY_DYNAMIC_PAGE_TABLES,<span class="number">8</span>,<span class="number">0</span></span><br><span class="line">SYM_DATA_END(early_dynamic_pgts)</span><br><span class="line"></span><br><span class="line">SYM_DATA(early_recursion_flag, .long <span class="number">0</span>)</span><br><span class="line"></span><br><span class="line"><span class="meta">.data</span></span><br><span class="line"></span><br><span class="line">#if defined(CONFIG_XEN_PV) || defined(CONFIG_PVH)</span><br><span class="line">SYM_DATA_START_PTI_ALIGNED(init_top_pgt)</span><br><span class="line"><span class="meta">.quad</span> level3_ident_pgt - __START_KERNEL_map + _KERNPG_TABLE_NOENC</span><br><span class="line"><span class="meta">.org</span> init_top_pgt + L4_PAGE_OFFSET*<span class="number">8</span>, <span class="number">0</span></span><br><span class="line"><span class="meta">.quad</span> level3_ident_pgt - __START_KERNEL_map + _KERNPG_TABLE_NOENC</span><br><span class="line"><span class="meta">.org</span> init_top_pgt + L4_START_KERNEL*<span class="number">8</span>, <span class="number">0</span></span><br><span class="line">/* (<span class="number">2</span>^<span class="number">48</span>-(<span class="number">2</span>*<span class="number">1024</span>*<span class="number">1024</span>*<span class="number">1024</span>))/(<span class="number">2</span>^<span class="number">39</span>) = <span class="number">511</span> */</span><br><span class="line"><span class="meta">.quad</span> level3_kernel_pgt - __START_KERNEL_map + _PAGE_TABLE_NOENC</span><br><span class="line"><span class="meta">.fill</span>PTI_USER_PGD_FILL,<span class="number">8</span>,<span class="number">0</span></span><br><span class="line">SYM_DATA_END(init_top_pgt)</span><br><span class="line"></span><br><span class="line">SYM_DATA_START_PAGE_ALIGNED(level3_ident_pgt)</span><br><span class="line"><span class="meta">.quad</span>level2_ident_pgt - __START_KERNEL_map + _KERNPG_TABLE_NOENC</span><br><span class="line"><span class="meta">.fill</span><span class="number">511</span>, <span class="number">8</span>, <span class="number">0</span></span><br><span class="line">SYM_DATA_END(level3_ident_pgt)</span><br><span class="line">SYM_DATA_START_PAGE_ALIGNED(level2_ident_pgt)</span><br><span class="line">PMDS(<span class="number">0</span>, __PAGE_KERNEL_IDENT_LARGE_EXEC, PTRS_PER_PMD)</span><br><span class="line">SYM_DATA_END(level2_ident_pgt)</span><br><span class="line">#else</span><br><span class="line">SYM_DATA_START_PTI_ALIGNED(init_top_pgt)</span><br><span class="line"><span class="meta">.fill</span><span class="number">512</span>,<span class="number">8</span>,<span class="number">0</span></span><br><span class="line"><span class="meta">.fill</span>PTI_USER_PGD_FILL,<span class="number">8</span>,<span class="number">0</span></span><br><span class="line">SYM_DATA_END(init_top_pgt)</span><br><span class="line">#endif</span><br><span class="line"></span><br><span class="line">#ifdef CONFIG_X86_5LEVEL</span><br><span class="line">SYM_DATA_START_PAGE_ALIGNED(level4_kernel_pgt)</span><br><span class="line"><span class="meta">.fill</span><span class="number">511</span>,<span class="number">8</span>,<span class="number">0</span></span><br><span class="line"><span class="meta">.quad</span>level3_kernel_pgt - __START_KERNEL_map + _PAGE_TABLE_NOENC</span><br><span class="line">SYM_DATA_END(level4_kernel_pgt)</span><br><span class="line">#endif</span><br><span class="line"></span><br><span class="line">SYM_DATA_START_PAGE_ALIGNED(level3_kernel_pgt)</span><br><span class="line"><span class="meta">.fill</span>L3_START_KERNEL,<span class="number">8</span>,<span class="number">0</span></span><br><span class="line">/* (<span class="number">2</span>^<span class="number">48</span>-(<span class="number">2</span>*<span class="number">1024</span>*<span class="number">1024</span>*<span class="number">1024</span>)-((<span class="number">2</span>^<span class="number">39</span>)*<span class="number">511</span>))/(<span class="number">2</span>^<span class="number">30</span>) = <span class="number">510</span> */</span><br><span class="line"><span class="meta">.quad</span>level2_kernel_pgt - __START_KERNEL_map + _KERNPG_TABLE_NOENC</span><br><span class="line"><span class="meta">.quad</span>level2_fixmap_pgt - __START_KERNEL_map + _PAGE_TABLE_NOENC</span><br><span class="line">SYM_DATA_END(level3_kernel_pgt)</span><br><span class="line"></span><br><span class="line">SYM_DATA_START_PAGE_ALIGNED(level2_kernel_pgt)</span><br><span class="line">PMDS(<span class="number">0</span>, __PAGE_KERNEL_LARGE_EXEC, KERNEL_IMAGE_SIZE/PMD_SIZE)</span><br><span class="line">SYM_DATA_END(level2_kernel_pgt)</span><br><span class="line"></span><br><span class="line">SYM_DATA_START_PAGE_ALIGNED(level2_fixmap_pgt)</span><br><span class="line"><span class="meta">.fill</span>(<span class="number">512</span> - <span class="number">4</span> - FIXMAP_PMD_NUM),<span class="number">8</span>,<span class="number">0</span></span><br><span class="line">pgtno = <span class="number">0</span></span><br><span class="line"><span class="meta">.rept</span> (FIXMAP_PMD_NUM)</span><br><span class="line"><span class="meta">.quad</span> level1_fixmap_pgt + (pgtno << PAGE_SHIFT) - __START_KERNEL_map \</span><br><span class="line">+ _PAGE_TABLE_NOENC<span class="comment">;</span></span><br><span class="line">pgtno = pgtno + <span class="number">1</span></span><br><span class="line"><span class="meta">.endr</span></span><br><span class="line">/* <span class="number">6</span> MB reserved space + a 2MB hole */</span><br><span class="line"><span class="meta">.fill</span><span class="number">4</span>,<span class="number">8</span>,<span class="number">0</span></span><br><span class="line">SYM_DATA_END(level2_fixmap_pgt)</span><br><span class="line"></span><br><span class="line">SYM_DATA_START_PAGE_ALIGNED(level1_fixmap_pgt)</span><br><span class="line"><span class="meta">.rept</span> (FIXMAP_PMD_NUM)</span><br><span class="line"><span class="meta">.fill</span><span class="number">512</span>,<span class="number">8</span>,<span class="number">0</span></span><br><span class="line"><span class="meta">.endr</span></span><br><span class="line">SYM_DATA_END(level1_fixmap_pgt)</span><br></pre></td></tr></table></figure><p>比较难理解，我们用图翻译一下：<br><img src="http://pic.l2h.site/early_page_table.png" alt="内核早期页表" title="Figure 3. 内核早期页表"></p><p>图中为内核代码建立了早期映射，这样，就可以愉快地执行内核代码了。（当然，也并不一定是愉快执行内核代码，后面我们也会看到，内核需要注册IDT表项来处理Page Fault Trap）。</p><figure class="highlight x86asm"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br></pre></td><td class="code"><pre><span class="line">/* Switch to new page-table */</span><br><span class="line"><span class="keyword">movq</span>%rax, %cr3</span><br><span class="line"></span><br><span class="line">/* Ensure I am executing from virtual addresses */</span><br><span class="line"><span class="keyword">movq</span><span class="number">$1f</span>, %rax</span><br><span class="line">ANNOTATE_RETPOLINE_SAFE</span><br><span class="line"><span class="keyword">jmp</span>*%rax</span><br></pre></td></tr></table></figure><p> <strong>__startup_64</strong>执行过后我们跳过一些SEV的处理，便开始使用新的内核页表。此后我们就跳转到__START_KERNEL_map开始的虚拟地址执行了。紧接着重新初始化GDT、设置段寄存器、建立初始化运营时的栈、建立IDT。这中间有一段代码：</p><figure class="highlight x86asm"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br><span class="line">21</span><br></pre></td><td class="code"><pre><span class="line">/* Set <span class="meta">up</span> %gs.</span><br><span class="line"> *</span><br><span class="line"> * The base of %gs always points to fixed_percpu_data. If the</span><br><span class="line"> * stack protector canary is enabled, it is located <span class="meta">at</span> %gs:<span class="number">40</span>.</span><br><span class="line"> * Note that, on SMP, the boot <span class="meta">cpu</span> uses init data <span class="meta">section</span> until</span><br><span class="line"> * the per <span class="meta">cpu</span> areas are set <span class="meta">up</span>.</span><br><span class="line"> */</span><br><span class="line">movl$MSR_GS_BASE,%ecx</span><br><span class="line">movlinitial_gs(%rip),%eax</span><br><span class="line">movlinitial_gs+<span class="number">4</span>(%rip),%edx</span><br><span class="line"><span class="keyword">wrmsr</span></span><br><span class="line">..................</span><br><span class="line">pushq$.Lafter_lret# put return address on stack for unwinder</span><br><span class="line">xorl%ebp, %ebp# clear frame pointer</span><br><span class="line"><span class="keyword">movq</span>initial_code(%rip), %rax</span><br><span class="line">pushq$__KERNEL_CS# set correct <span class="built_in">cs</span></span><br><span class="line">pushq%rax# target address <span class="keyword">in</span> negative space</span><br><span class="line">lretq</span><br><span class="line"></span><br><span class="line">......</span><br><span class="line">SYM_DATA(initial_code,.quad x86_64_start_kernel)</span><br></pre></td></tr></table></figure><p>它的作用是为多处理器系统保存per CPU变量的地址，保存到64-bit model specific register (MSR)。接着跳转到初始化c代码, 即<strong>x86_64_start_kernel</strong>。</p><h2 id="总结"><a href="#总结" class="headerlink" title="总结"></a>总结</h2><p>本文重点分析了从内核被Loader加载一直执行到C代码入口的内存管理。一些主要的步骤：</p><ul><li>开启保护模式</li><li>开启长模式</li><li>内核解压同时添加随机偏移</li><li>建立内核页表并跳转到虚拟地址执行</li></ul><p>系列后续我们将分析执行到C代码入口之后的处理</p>]]></content>
<summary type="html">
<blockquote>
<p>这是&lt;Linux内核内存管理&gt;系列的第三篇</p>
<blockquote>
<p><a href="/p/61304bbb.html">第一篇</a>为内核内存管理过程知识点的的简单梳理</p>
<p><a href="/p/1dc6adc2.html">第二篇</a>介绍了内核的数据结构</p>
</blockquote>
</blockquote><h2 id="前言"><a href="#前言" class="headerlink" title="前言"></a>前言</h2><p>以Intel X64 CPU为例，Linux的初始化可大致分为如下几个过程：</p><ol>
<li>Loader跳转到内核后的实模式(Real Mode)</li>
<li>32位保护模式跳转到64位长模式</li>
<li>64位长模式下解压内核</li>
<li>解压内核后，建立新的页表映射，并跳转到Arch（平台）相关C代码</li>
<li>执行平台无关初始化代码</li>
</ol>
</summary>
<category term="Linux" scheme="http://www.l2h.site/categories/Linux/"/>
<category term="Linux内存管理" scheme="http://www.l2h.site/categories/Linux/Linux%E5%86%85%E5%AD%98%E7%AE%A1%E7%90%86/"/>
<category term="Linux" scheme="http://www.l2h.site/tags/Linux/"/>
<category term="内存管理" scheme="http://www.l2h.site/tags/%E5%86%85%E5%AD%98%E7%AE%A1%E7%90%86/"/>
</entry>
<entry>
<title>Linux内核内存管理 - 内核职责</title>
<link href="http://www.l2h.site/p/61304bbb.html"/>
<id>http://www.l2h.site/p/61304bbb.html</id>
<published>2021-09-04T01:21:04.000Z</published>
<updated>2023-06-01T13:14:04.479Z</updated>
<content type="html"><![CDATA[<blockquote><p>本系列是本人对Linux内核内存管理的学习持续总结。</p></blockquote><h2 id="序"><a href="#序" class="headerlink" title="序"></a>序</h2><p>内存系统是操作系统最复杂的子系统之一，内存管理穿插着内核的方方面面。做驱动开发有2年多了，之前写过Linux内核<a href="d209b5b5.html">内存管理的博客</a>。现在回头看，之前的理解并不到位，也不完整。希望用本系列对Linux内核内存管理的知识做重新梳理，增强自己的理解，也能给对这个复杂功能一头雾水的朋友提供一些思路。 </p><a id="more"></a><h2 id="平台"><a href="#平台" class="headerlink" title="平台"></a>平台</h2><p>为了阐述方便，本文（系列）会基于Intel 64位平台做讨论。所涉及内核代码主要位于如下目录：</p><ul><li>mm</li><li>arch/x86</li></ul><h2 id="内核职责"><a href="#内核职责" class="headerlink" title="内核职责"></a>内核职责</h2><p>内存是系统得以运行的最基本保证。为了将内存进行有效管理，内核需要做如下考虑：</p><h3 id="物理内存管理"><a href="#物理内存管理" class="headerlink" title="物理内存管理"></a>物理内存管理</h3><ul><li><strong>非一致性内存访问（NUMA）</strong>： 多处理器系统中有多个内存节点。每个处理器和每个内存节点距离并不相同，因此访问不同内存节点的距离和开销并不相同。OS需要考虑如何有效管理处理器对内存的访问，使该处理器更多地距其更近的处理器（一般称为Local）节点。 当然实际内核NUMA内存管理策略并非所述这么简单，Linux有一系列配置NUMA访问策略的方法，详见<a href="https://elixir.bootlin.com/linux/latest/source/Documentation/admin-guide/mm/numa_memory_policy.rst" target="_blank" rel="noopener">NUMA Policy Guide</a>。</li><li><strong>内存热拔插</strong>: 顾名思义,系统运行时内存热拔插的处理。</li></ul><h3 id="内核物理地址的规划"><a href="#内核物理地址的规划" class="headerlink" title="内核物理地址的规划"></a>内核物理地址的规划</h3><p>在Boot Loader加载内核后，如何分别摆放16 Bit, 32 Bit代码区域。压缩内核，以及如何解压内核，解压还要考虑KASLR（内核地址随机化）等因素。同时，每个启动阶段使用的堆和栈如何划分。</p><h3 id="虚拟内存管理"><a href="#虚拟内存管理" class="headerlink" title="虚拟内存管理"></a>虚拟内存管理</h3><ul><li><strong>内核初始化前</strong>：可大致分为32 Bit页表映射，64 Bit内存页表映射（解压内核前），64 Bit内存页表映射（解压内核后）</li><li><strong>内核初始化后</strong>: 各个进程独立页表的控制，同时需要保证内核空间态地址对所有进程是一致的。同时需要考虑用户态和内核态对不同内存区域访问权限的控制。</li></ul><h3 id="外设访问"><a href="#外设访问" class="headerlink" title="外设访问"></a>外设访问</h3><p>通过虚拟地址访问外设IO端口或者MMIO端口，这需要内核为其建立对应的页表项，同时为了保证特定IO区域只能有一个主体来管理，内核需要以树状结构来管理IO区域。<br>同时，外设要访问的内存空间，需要考虑CPU和外设访问内存一致性问题（DMA一致性）。</p><h3 id="内存分配"><a href="#内存分配" class="headerlink" title="内存分配"></a>内存分配</h3><p>内存分配和释放是操作系统内最为频繁的操作。保证内存分配和释放的同时，也需要考虑避免系统内存的碎片化，避免系统运行到一段时间后，程序需要一块大内存的的时候无法分配到。其中：</p><ul><li><strong>伙伴系统(Buddy System)</strong>: 按页管理内存分配和释放</li><li><strong>SLAB系列</strong>: 页内小内存分配的管理和释放</li></ul><h3 id="内存回收和换页"><a href="#内存回收和换页" class="headerlink" title="内存回收和换页"></a>内存回收和换页</h3><p>当系统物理内存紧张时，系统会将一些内存换出到硬盘上。而当系统访问该内存页产生Page Fault时，操作系统需要负责将该内存页的内容换回到内存上。</p><h2 id="内存使用检测和Debug"><a href="#内存使用检测和Debug" class="headerlink" title="内存使用检测和Debug"></a>内存使用检测和Debug</h2><p>除了上述职责，内核也需要提供方法对内存使用进行检测和调试。例： </p><ul><li>程序可能会对内存做不当使用，当这种不当使用发生在内核态，往往会产生严重的后果。因此Linux提供了KASAN、Kmemleak等工具方便开发人员进行检查。</li><li>同时内核内存管理子系统也提供了一系列接口和方法供系统运维管理人员对内存使用进行调试和检测。</li></ul><h2 id="结语"><a href="#结语" class="headerlink" title="结语"></a>结语</h2><p>以上为本人对内核内存管理功能的梳理。因为内核内存管理功能复杂，以上理解并不一定准确，因此本文也需要持续更新。系列文章后续也将会对本文提及的内容进行具体的分析和介绍。</p>]]></content>
<summary type="html">
<blockquote>
<p>本系列是本人对Linux内核内存管理的学习持续总结。</p>
</blockquote><h2 id="序"><a href="#序" class="headerlink" title="序"></a>序</h2><p>内存系统是操作系统最复杂的子系统之一，内存管理穿插着内核的方方面面。做驱动开发有2年多了，之前写过Linux内核<a href="d209b5b5.html">内存管理的博客</a>。现在回头看，之前的理解并不到位，也不完整。希望用本系列对Linux内核内存管理的知识做重新梳理，增强自己的理解，也能给对这个复杂功能一头雾水的朋友提供一些思路。 </p>
</summary>
<category term="Linux" scheme="http://www.l2h.site/categories/Linux/"/>
<category term="Linux内存管理" scheme="http://www.l2h.site/categories/Linux/Linux%E5%86%85%E5%AD%98%E7%AE%A1%E7%90%86/"/>
<category term="Linux" scheme="http://www.l2h.site/tags/Linux/"/>
<category term="内存管理" scheme="http://www.l2h.site/tags/%E5%86%85%E5%AD%98%E7%AE%A1%E7%90%86/"/>
</entry>
<entry>
<title>哎，Notepad++</title>
<link href="http://www.l2h.site/p/8b759bb.html"/>
<id>http://www.l2h.site/p/8b759bb.html</id>
<published>2020-10-06T23:26:21.000Z</published>
<updated>2023-06-01T13:14:04.478Z</updated>
<content type="html"><![CDATA[<p>一直以为技术是无国界的，也是不带政治观点的。没想到被NotePad++ 疯狂打脸。<br>来看看其作者如何表达政治观点的。<br><img src="https://l2h.site/images/posts/20201007/1.png" alt="NoteFuxk++"></p><p>至于为什么这么多对华内容，后来了解到这是一个台湾人。<br>在台湾公司呆了这么多年，从来不会也不可能表达政治观点。同事也都是客客气气的。<br>但是相信无论是我们或者台湾同事，也都能感受到对方的想法。例如，</p><a id="more"></a><ul><li>台湾同事有意无意将中国、台湾放在并列的层面讨论。Nice一点的台湾同事发现自己讲错了以后也会改成“大陆”这种讲法</li><li>而我又习惯用大陆、台湾这样的逻辑和方式</li></ul><p>相信看到作者这些政治宣言后，个人不会再去下载该软件，也会跟同事朋友提醒，远离这个软件!</p>]]></content>
<summary type="html">
<p>一直以为技术是无国界的，也是不带政治观点的。没想到被NotePad++ 疯狂打脸。<br>来看看其作者如何表达政治观点的。<br><img src="https://l2h.site/images/posts/20201007/1.png" alt="NoteFuxk++"></p><p>至于为什么这么多对华内容，后来了解到这是一个台湾人。<br>在台湾公司呆了这么多年，从来不会也不可能表达政治观点。同事也都是客客气气的。<br>但是相信无论是我们或者台湾同事，也都能感受到对方的想法。例如，</p>
</summary>
<category term="L&amp;H生活" scheme="http://www.l2h.site/categories/L-amp-H%E7%94%9F%E6%B4%BB/"/>
<category term="L&amp;H随感" scheme="http://www.l2h.site/categories/L-amp-H%E7%94%9F%E6%B4%BB/L-amp-H%E9%9A%8F%E6%84%9F/"/>
</entry>
<entry>
<title>无前</title>
<link href="http://www.l2h.site/p/89f375ed.html"/>
<id>http://www.l2h.site/p/89f375ed.html</id>
<published>2020-08-23T05:45:18.000Z</published>
<updated>2023-06-01T13:14:04.478Z</updated>
<content type="html"><![CDATA[<p>距离上次写博客已经有约5个月了。一直没有时间和动力来更新博客。</p><p>最近半年，工作异常忙碌，不止一次跟XH表示，这是我工作以来最为忙碌的时光。所幸所做之工作，又是个人认为比较前言且具有挑战的项目。虽然充实，但也持续迷茫着。</p><p>外面的世界，贸易战、洪水、新冠肆虐。家里却也非绝对宁静。</p><p>去年八月一年来，搬了三次家。上次博客更新时，还在20公里之外的出租房里。现在，也幸得终于可以回到自己的寒舍。房子虽小，也有雨季漏水的问题。却也增添了几分方便，更多了归属感。</p><a id="more"></a><p>预见还会有的动迁和变化，希望一切顺利。也希望一直以来的愿望还是可以达成。</p><p>努力！</p>]]></content>
<summary type="html">
<p>距离上次写博客已经有约5个月了。一直没有时间和动力来更新博客。</p><p>最近半年，工作异常忙碌，不止一次跟XH表示，这是我工作以来最为忙碌的时光。所幸所做之工作，又是个人认为比较前言且具有挑战的项目。虽然充实，但也持续迷茫着。</p><p>外面的世界，贸易战、洪水、新冠肆虐。家里却也非绝对宁静。</p><p>去年八月一年来，搬了三次家。上次博客更新时，还在20公里之外的出租房里。现在，也幸得终于可以回到自己的寒舍。房子虽小，也有雨季漏水的问题。却也增添了几分方便，更多了归属感。</p>
</summary>
<category term="L&amp;H生活" scheme="http://www.l2h.site/categories/L-amp-H%E7%94%9F%E6%B4%BB/"/>
<category term="L&amp;H随感" scheme="http://www.l2h.site/categories/L-amp-H%E7%94%9F%E6%B4%BB/L-amp-H%E9%9A%8F%E6%84%9F/"/>
</entry>
<entry>
<title>在家办公记</title>
<link href="http://www.l2h.site/p/f790099a.html"/>
<id>http://www.l2h.site/p/f790099a.html</id>
<published>2020-02-02T08:34:10.000Z</published>
<updated>2023-06-01T13:14:04.478Z</updated>
<content type="html"><![CDATA[<p>今天是20200202，难得的日期对称日，有必要记上一篇。</p><p>四川省最后还是刚起，仅仅发了一个不痛不痒的通知说企业可自行安排复工时间，成为遵守国务院命令2月3日准时上班的唯一一个省份。</p><p><img src="https://l2h.site/images/posts/20200202/1.jpeg" alt="全国开复工时间表"></p><p>看着最新的疫情地图，加上查到的1600多万的成都人口，心理慌得不要不要的。料医学博士的省长也没办法抑制得了新病毒的传播，希望成都不要学武汉的各种神操作。幸好公司人性化，允许在家办公。</p><a id="more"></a><p><img src="https://l2h.site/images/posts/20200202/2.png" alt="疫情地图"></p><p>仔细想想，如果可以解决公司信息泄露问题，IT行业在家办公其实挺好的。最起码有以下几点优势：</p><ul><li>节省时间（各种通勤时间）</li><li>对企业来讲节省各种水电物业等开销。</li><li>居家环境，压力更小，做事效率应也会更高。</li></ul><p>明天先这样工作第一天试试看，如果可行，强烈建议公司在疫情结束后也继续推广！</p>]]></content>
<summary type="html">
<p>今天是20200202，难得的日期对称日，有必要记上一篇。</p><p>四川省最后还是刚起，仅仅发了一个不痛不痒的通知说企业可自行安排复工时间，成为遵守国务院命令2月3日准时上班的唯一一个省份。</p><p><img src="https://l2h.site/images/posts/20200202/1.jpeg" alt="全国开复工时间表"></p><p>看着最新的疫情地图，加上查到的1600多万的成都人口，心理慌得不要不要的。料医学博士的省长也没办法抑制得了新病毒的传播，希望成都不要学武汉的各种神操作。幸好公司人性化，允许在家办公。</p>
</summary>
<category term="L&amp;H生活" scheme="http://www.l2h.site/categories/L-amp-H%E7%94%9F%E6%B4%BB/"/>
</entry>
<entry>
<title>新SARS</title>
<link href="http://www.l2h.site/p/f4ebc93a.html"/>
<id>http://www.l2h.site/p/f4ebc93a.html</id>
<published>2020-01-23T06:16:33.000Z</published>
<updated>2023-06-01T13:14:04.520Z</updated>
<content type="html"><![CDATA[<p>人类自以为是世界的主宰，实则不然。2020新年伊始，新冠状病毒从武汉开始蔓延。</p><p>强大的传染力，加上春节的人口流动速率，病毒感染的人口可能远非官方所报道那么多。</p><p>想起2013年的SARS，我高三，生活在小县城也能深切感受到这传染病的威力。<br>每天早读几个温度计班里传递量体温，诊所里板蓝根早早卖断货。<br>晚上自习新闻时间打开电视随时关心病情进展，看到又是几个全身防护的医务人员感染甚至病逝的消息。</p><a id="more"></a><p>目前能做的，就是相信国家，好好待在家里，不出去添乱。希望大家一切都好。</p>]]></content>
<summary type="html">
<p>人类自以为是世界的主宰，实则不然。2020新年伊始，新冠状病毒从武汉开始蔓延。</p><p>强大的传染力，加上春节的人口流动速率，病毒感染的人口可能远非官方所报道那么多。</p><p>想起2013年的SARS，我高三，生活在小县城也能深切感受到这传染病的威力。<br>每天早读几个温度计班里传递量体温，诊所里板蓝根早早卖断货。<br>晚上自习新闻时间打开电视随时关心病情进展，看到又是几个全身防护的医务人员感染甚至病逝的消息。</p>
</summary>
<category term="L&amp;H生活" scheme="http://www.l2h.site/categories/L-amp-H%E7%94%9F%E6%B4%BB/"/>
<category term="L&amp;H随感" scheme="http://www.l2h.site/categories/L-amp-H%E7%94%9F%E6%B4%BB/L-amp-H%E9%9A%8F%E6%84%9F/"/>
<category term="SARS" scheme="http://www.l2h.site/tags/SARS/"/>
<category term="冠状病毒" scheme="http://www.l2h.site/tags/%E5%86%A0%E7%8A%B6%E7%97%85%E6%AF%92/"/>
</entry>
<entry>
<title>小米手表使用ADB安装应用</title>
<link href="http://www.l2h.site/p/2639ab6d.html"/>
<id>http://www.l2h.site/p/2639ab6d.html</id>
<published>2019-12-01T02:48:35.000Z</published>
<updated>2023-06-01T13:14:04.485Z</updated>
<content type="html"><![CDATA[<p>一直想入手一个智能手表，可惜Apple Watch需要连接苹果手机，遂未能如愿。最近小米终于发布一款智能手表，支持ESIM，使用Wear OS，果断入手。 </p><p>实际使用发现Bug不少，软件不多。Bug可以等系统更新来解决。不过软件不多是生态链的问题，等生态链慢慢建起来实在太慢。Wear OS是Android的一个分支，理论上应该支持安装安卓应用才对。翻了下小米手表的设置，也是有看到可以做ADB调试，因此理应可以用ADB安装Android应用。 </p><a id="more"></a><p>步骤：</p><ol><li>电脑上安装ADB（可根据您自己的电脑系统自行百度安装方法）</li><li>手表上“设置–开发者选项”，打开ADB调试。如图：<br><img src="http://pic.l2h.site/xiaomiwatch1.jpg" alt="开启ADB调试"></li><li>将手表连上充电底座，并将USB连接电源插头端直接连上电脑。如图：<br><img src="http://pic.l2h.site/xiaomiwatch2.jpg" alt="连接电脑"></li><li>连上电脑后，手表弹出“是否允许调试”选择框。直接选择允许或者始终允许。<br><img src="http://pic.l2h.site/xiaomiwatch3.jpg" alt="允许调试"></li><li>电脑端执行ADB Install安装应用,待显示成功，便安装完成。例如：</li></ol><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><span class="line">adb install qqmail_android_5.7.1.10141908.2480_0.apk</span><br></pre></td></tr></table></figure><p>注意：</p><ol><li>手表内存小，APK程序越大，其所需的内存越大。因此尽量选择程序本身小于10MB的程序</li><li>大部分APP并未针对小屏幕做优化，目前提供方法和思路供试用。如上例子QQMAIL安装后就因其无法支持手表屏幕大小，因此无法做有效操作。</li></ol><p>最后，收集可支持手表的应用如下：</p><table><thead><tr><th>应用</th><th>版本</th><th>APK下载链接</th></tr></thead></table>]]></content>
<summary type="html">
<p>一直想入手一个智能手表，可惜Apple Watch需要连接苹果手机，遂未能如愿。最近小米终于发布一款智能手表，支持ESIM，使用Wear OS，果断入手。 </p><p>实际使用发现Bug不少，软件不多。Bug可以等系统更新来解决。不过软件不多是生态链的问题，等生态链慢慢建起来实在太慢。Wear OS是Android的一个分支，理论上应该支持安装安卓应用才对。翻了下小米手表的设置，也是有看到可以做ADB调试，因此理应可以用ADB安装Android应用。 </p>
</summary>
<category term="L&amp;H生活" scheme="http://www.l2h.site/categories/L-amp-H%E7%94%9F%E6%B4%BB/"/>
<category term="小米手表" scheme="http://www.l2h.site/tags/%E5%B0%8F%E7%B1%B3%E6%89%8B%E8%A1%A8/"/>
</entry>
<entry>
<title>Zephyr线程管理 - 数据结构与API</title>
<link href="http://www.l2h.site/p/5b9388b6.html"/>
<id>http://www.l2h.site/p/5b9388b6.html</id>
<published>2019-11-16T14:16:35.000Z</published>
<updated>2023-06-01T13:14:04.477Z</updated>
<content type="html"><![CDATA[<p>本文介绍和Zephyr线程的数据结构，及相应的API。 </p><h2 id="数据结构"><a href="#数据结构" class="headerlink" title="数据结构"></a>数据结构</h2><p>Zephyr数据结构使用k_thread定义，如下所示：</p><figure class="highlight c"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br><span class="line">21</span><br><span class="line">22</span><br><span class="line">23</span><br><span class="line">24</span><br><span class="line">25</span><br><span class="line">26</span><br><span class="line">27</span><br><span class="line">28</span><br><span class="line">29</span><br><span class="line">30</span><br><span class="line">31</span><br><span class="line">32</span><br><span class="line">33</span><br><span class="line">34</span><br><span class="line">35</span><br><span class="line">36</span><br><span class="line">37</span><br><span class="line">38</span><br><span class="line">39</span><br></pre></td><td class="code"><pre><span class="line"><span class="class"><span class="keyword">struct</span> <span class="title">k_thread</span> {</span></span><br><span class="line"><span class="class"><span class="keyword">struct</span> _<span class="title">thread_base</span> <span class="title">base</span>;</span></span><br><span class="line"><span class="class"><span class="keyword">struct</span> _<span class="title">callee_saved</span> <span class="title">callee_saved</span>;</span></span><br><span class="line"><span class="keyword">void</span> *init_data;</span><br><span class="line"><span class="keyword">void</span> (*fn_abort)(<span class="keyword">void</span>);</span><br><span class="line"></span><br><span class="line"><span class="meta">#<span class="meta-keyword">if</span> defined(CONFIG_THREAD_MONITOR)</span></span><br><span class="line"><span class="class"><span class="keyword">struct</span> __<span class="title">thread_entry</span> <span class="title">entry</span>;</span></span><br><span class="line"><span class="class"><span class="keyword">struct</span> <span class="title">k_thread</span> *<span class="title">next_thread</span>;</span></span><br><span class="line"><span class="meta">#<span class="meta-keyword">endif</span></span></span><br><span class="line"><span class="meta">#<span class="meta-keyword">if</span> defined(CONFIG_THREAD_NAME)</span></span><br><span class="line"><span class="keyword">char</span> name[CONFIG_THREAD_MAX_NAME_LEN];</span><br><span class="line"><span class="meta">#<span class="meta-keyword">endif</span></span></span><br><span class="line"><span class="meta">#<span class="meta-keyword">ifdef</span> CONFIG_THREAD_CUSTOM_DATA</span></span><br><span class="line"><span class="keyword">void</span> *custom_data;</span><br><span class="line"><span class="meta">#<span class="meta-keyword">endif</span></span></span><br><span class="line"><span class="meta">#<span class="meta-keyword">ifdef</span> CONFIG_THREAD_USERSPACE_LOCAL_DATA</span></span><br><span class="line"><span class="class"><span class="keyword">struct</span> _<span class="title">thread_userspace_local_data</span> *<span class="title">userspace_local_data</span>;</span></span><br><span class="line"><span class="meta">#<span class="meta-keyword">endif</span></span></span><br><span class="line"><span class="meta">#<span class="meta-keyword">ifdef</span> CONFIG_ERRNO</span></span><br><span class="line"><span class="meta">#<span class="meta-keyword">ifndef</span> CONFIG_USERSPACE</span></span><br><span class="line"><span class="keyword">int</span> errno_var;</span><br><span class="line"><span class="meta">#<span class="meta-keyword">endif</span></span></span><br><span class="line"><span class="meta">#<span class="meta-keyword">endif</span></span></span><br><span class="line"></span><br><span class="line"><span class="meta">#<span class="meta-keyword">if</span> defined(CONFIG_THREAD_STACK_INFO)</span></span><br><span class="line"><span class="class"><span class="keyword">struct</span> _<span class="title">thread_stack_info</span> <span class="title">stack_info</span>;</span></span><br><span class="line"><span class="meta">#<span class="meta-keyword">endif</span> <span class="comment">/* CONFIG_THREAD_STACK_INFO */</span></span></span><br><span class="line"><span class="meta">#<span class="meta-keyword">if</span> defined(CONFIG_USERSPACE)</span></span><br><span class="line"><span class="class"><span class="keyword">struct</span> _<span class="title">mem_domain_info</span> <span class="title">mem_domain_info</span>;</span></span><br><span class="line"><span class="keyword">k_thread_stack_t</span> *stack_obj;</span><br><span class="line"><span class="meta">#<span class="meta-keyword">endif</span> <span class="comment">/* CONFIG_USERSPACE */</span></span></span><br><span class="line"><span class="meta">#<span class="meta-keyword">if</span> defined(CONFIG_USE_SWITCH)</span></span><br><span class="line"><span class="keyword">int</span> swap_retval;</span><br><span class="line"><span class="keyword">void</span> *switch_handle;</span><br><span class="line"><span class="meta">#<span class="meta-keyword">endif</span></span></span><br><span class="line"><span class="class"><span class="keyword">struct</span> <span class="title">k_mem_pool</span> *<span class="title">resource_pool</span>;</span></span><br><span class="line"><span class="class"><span class="keyword">struct</span> _<span class="title">thread_arch</span> <span class="title">arch</span>;</span></span><br><span class="line">};</span><br></pre></td></tr></table></figure><a id="more"></a><p>参数说明：</p><ul><li><strong>base</strong>： 存储Thread的基础调度信息结构体。具体如下：<ul><li><strong>qnode_dlist/qnode_rb</strong>: 指向线程在等待/就绪队列的位置</li><li><strong>pended_on</strong>:指向线程所在的等待队列（仅对红黑树等待队列有效）</li><li><strong>user_options</strong>:线程参数</li><li><strong>thread_state</strong>:线程状态</li><li><strong>union(prio/sched_locked/preemt)</strong>: 线程的抢占优先级相关参数</li><li><strong>order_key</strong>:被调度器使用来比对优先级</li><li>SMP相关参数先忽略</li><li><strong>timeout</strong>:指向超时队列中该线程的位置</li></ul></li></ul><figure class="highlight c"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br><span class="line">21</span><br><span class="line">22</span><br><span class="line">23</span><br><span class="line">24</span><br><span class="line">25</span><br><span class="line">26</span><br><span class="line">27</span><br><span class="line">28</span><br><span class="line">29</span><br><span class="line">30</span><br></pre></td><td class="code"><pre><span class="line"><span class="class"><span class="keyword">struct</span> _<span class="title">thread_base</span> {</span></span><br><span class="line"><span class="keyword">union</span> {</span><br><span class="line"><span class="keyword">sys_dnode_t</span> qnode_dlist;</span><br><span class="line"><span class="class"><span class="keyword">struct</span> <span class="title">rbnode</span> <span class="title">qnode_rb</span>;</span></span><br><span class="line">};</span><br><span class="line"><span class="keyword">_wait_q_t</span> *pended_on;</span><br><span class="line"><span class="keyword">u8_t</span> user_options;</span><br><span class="line"><span class="keyword">u8_t</span> thread_state;</span><br><span class="line"><span class="keyword">union</span> {</span><br><span class="line"><span class="class"><span class="keyword">struct</span> {</span></span><br><span class="line"><span class="meta">#<span class="meta-keyword">if</span> __BYTE_ORDER__ == __ORDER_BIG_ENDIAN__</span></span><br><span class="line"><span class="keyword">u8_t</span> sched_locked;</span><br><span class="line"><span class="keyword">s8_t</span> prio;</span><br><span class="line"><span class="meta">#<span class="meta-keyword">else</span> <span class="comment">/* LITTLE and PDP */</span></span></span><br><span class="line"><span class="keyword">s8_t</span> prio;</span><br><span class="line"><span class="keyword">u8_t</span> sched_locked;</span><br><span class="line"><span class="meta">#<span class="meta-keyword">endif</span></span></span><br><span class="line">};</span><br><span class="line"><span class="keyword">u16_t</span> preempt;</span><br><span class="line">};</span><br><span class="line"><span class="meta">#<span class="meta-keyword">ifdef</span> CONFIG_SCHED_DEADLINE</span></span><br><span class="line"><span class="keyword">int</span> prio_deadline;</span><br><span class="line"><span class="meta">#<span class="meta-keyword">endif</span></span></span><br><span class="line"><span class="keyword">u32_t</span> order_key;</span><br><span class="line"><span class="comment">/*SMP 相关*/</span></span><br><span class="line"><span class="keyword">void</span> *swap_data;</span><br><span class="line"><span class="meta">#<span class="meta-keyword">ifdef</span> CONFIG_SYS_CLOCK_EXISTS</span></span><br><span class="line"><span class="class"><span class="keyword">struct</span> _<span class="title">timeout</span> <span class="title">timeout</span>;</span></span><br><span class="line"><span class="meta">#<span class="meta-keyword">endif</span></span></span><br><span class="line">};</span><br></pre></td></tr></table></figure><ul><li><strong>_callee_saved</strong>: 与平台相关的一些参数，主要保存一些线程相关寄存器信息。</li><li><strong>init_data</strong>: 线程静态初始化数据</li><li><strong>fn_abort</strong>: 线程取消回调函数</li><li><strong>entry</strong>:Thread入口函数</li><li><strong>next_thread</strong>: Thread链表中下一个线程指针</li><li><strong>name</strong>: Thread名称</li><li><strong>custom_data</strong>: Thread自定义数据</li><li><strong>userspace_local_data</strong>:用户空间态数据</li><li><strong>errno_var</strong>: 错误编号（类似Linux的Errno）</li><li><strong>stack_info</strong>: 栈信息，主要记录栈的开始地址与大小</li><li><strong>mem_domain_info</strong>: 线程内存域信息（用户空间态线程使用）</li><li><strong>stack_obj</strong>: 线程栈地址</li><li><strong>swap_retval</strong>: TBD</li><li><strong>switch_handle</strong>: TBD</li><li><strong>resource_pool</strong>:线程所占用的资源池</li><li><strong>arch</strong>: 体系结构相关的线程数据。因为体系结构不同，所占大小也不相同，为保证其他参数的位置相同，Zephyr将此参数置于线程末尾</li></ul><h2 id="API"><a href="#API" class="headerlink" title="API"></a>API</h2><p>Zephyr定义了一系列的API来使用线程：</p><h3 id="线程创建"><a href="#线程创建" class="headerlink" title="线程创建"></a>线程创建</h3><p>线程创建的函数定义如下：</p><figure class="highlight c"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br></pre></td><td class="code"><pre><span class="line">K_THREAD_DEFINE(my_tid, MY_STACK_SIZE, my_entry_point, <span class="literal">NULL</span>, <span class="literal">NULL</span>, <span class="literal">NULL</span>, MY_PRIORITY, <span class="number">0</span>, K_NO_WAIT);</span><br><span class="line"><span class="function"><span class="keyword">k_tid_t</span> <span class="title">k_thread_create</span><span class="params">(structk_thread *new_thread, <span class="keyword">k_thread_stack_t</span> *<span class="built_in">stack</span>, <span class="keyword">size_t</span> stack_size, <span class="keyword">k_thread_entry_t</span> entry, <span class="keyword">void</span> *p1, <span class="keyword">void</span> *p2, <span class="keyword">void</span> *p3, <span class="keyword">int</span> prio, <span class="keyword">u32_t</span> options, <span class="keyword">s32_t</span> <span class="built_in">delay</span>)</span></span></span><br></pre></td></tr></table></figure><p>前者主要用于静态定义线程，而后者主要用于运行时创建线程。线程的静态定义主要就是定义如下结构体（参数明确，不做一一解释）：</p><figure class="highlight c"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br></pre></td><td class="code"><pre><span class="line"><span class="class"><span class="keyword">struct</span> _<span class="title">static_thread_data</span> {</span></span><br><span class="line"><span class="class"><span class="keyword">struct</span> <span class="title">k_thread</span> *<span class="title">init_thread</span>;</span></span><br><span class="line"><span class="keyword">k_thread_stack_t</span> *init_stack;</span><br><span class="line"><span class="keyword">unsigned</span> <span class="keyword">int</span> init_stack_size;</span><br><span class="line"><span class="keyword">k_thread_entry_t</span> init_entry;</span><br><span class="line"><span class="keyword">void</span> *init_p1;</span><br><span class="line"><span class="keyword">void</span> *init_p2;</span><br><span class="line"><span class="keyword">void</span> *init_p3;</span><br><span class="line"><span class="keyword">int</span> init_prio;</span><br><span class="line"><span class="keyword">u32_t</span> init_options;</span><br><span class="line"><span class="keyword">s32_t</span> init_delay;</span><br><span class="line"><span class="keyword">void</span> (*init_abort)(<span class="keyword">void</span>);</span><br><span class="line"><span class="keyword">const</span> <span class="keyword">char</span> *init_name;</span><br><span class="line">};</span><br></pre></td></tr></table></figure><p>k_thread_create的源码如下：</p><figure class="highlight c"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br></pre></td><td class="code"><pre><span class="line"><span class="function"><span class="keyword">k_tid_t</span> <span class="title">z_impl_k_thread_create</span><span class="params">(struct k_thread *new_thread, <span class="comment">//代码注解1</span></span></span></span><br><span class="line"><span class="function"><span class="params"> <span class="keyword">k_thread_stack_t</span> *<span class="built_in">stack</span>,</span></span></span><br><span class="line"><span class="function"><span class="params"> <span class="keyword">size_t</span> stack_size, <span class="keyword">k_thread_entry_t</span> entry,</span></span></span><br><span class="line"><span class="function"><span class="params"> <span class="keyword">void</span> *p1, <span class="keyword">void</span> *p2, <span class="keyword">void</span> *p3,</span></span></span><br><span class="line"><span class="function"><span class="params"> <span class="keyword">int</span> prio, <span class="keyword">u32_t</span> options, <span class="keyword">s32_t</span> <span class="built_in">delay</span>)</span></span></span><br><span class="line"><span class="function"></span>{</span><br><span class="line">__ASSERT(!z_arch_is_in_isr(), <span class="string">"Threads may not be created in ISRs"</span>); <span class="comment">//代码注解2</span></span><br><span class="line"></span><br><span class="line">z_setup_new_thread(new_thread, <span class="built_in">stack</span>, stack_size, entry, p1, p2, p3, prio, options, <span class="literal">NULL</span>); <span class="comment">//代码注解3</span></span><br><span class="line"></span><br><span class="line"><span class="keyword">if</span> (<span class="built_in">delay</span> != K_FOREVER) {</span><br><span class="line">schedule_new_thread(new_thread, <span class="built_in">delay</span>); <span class="comment">//代码注解4</span></span><br><span class="line">}</span><br><span class="line"></span><br><span class="line"><span class="keyword">return</span> new_thread;</span><br><span class="line">}</span><br></pre></td></tr></table></figure><p>其中：</p><ul><li><strong>代码注解1</strong>： 这里函数名为何是z_impl_k_thread_create而不是k_thread_create。因为Zephyr在编译时使用了脚本来生成系统中API调用。所以在搜索源码时大部分时间只能搜索到z_impl_xxx形式的系统函数定义。不过编译连接前，是可以看得到不带<em>z_impl_</em>形式的API，最终调用的即是z_impl_k_thread_create</li><li><strong>代码注解2</strong>：中断上下文不能创建线程，因为创建线程会需要申请系统资源，也会引起重新调度，带来不可预料的后果</li><li><strong>代码注解3</strong>：创建新线程</li><li><strong>代码注解4</strong>：将线程加入调度</li></ul><p>z_setup_new_thread定义如下(为方便分析，代码做一定简化)：</p><figure class="highlight c"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br><span class="line">21</span><br><span class="line">22</span><br><span class="line">23</span><br><span class="line">24</span><br><span class="line">25</span><br><span class="line">26</span><br><span class="line">27</span><br><span class="line">28</span><br><span class="line">29</span><br><span class="line">30</span><br><span class="line">31</span><br><span class="line">32</span><br><span class="line">33</span><br><span class="line">34</span><br><span class="line">35</span><br></pre></td><td class="code"><pre><span class="line"><span class="function"><span class="keyword">void</span> <span class="title">z_setup_new_thread</span><span class="params">(struct k_thread *new_thread, <span class="keyword">k_thread_stack_t</span> *<span class="built_in">stack</span>, <span class="keyword">size_t</span> stack_size, <span class="keyword">k_thread_entry_t</span> entry, <span class="keyword">void</span> *p1, <span class="keyword">void</span> *p2, <span class="keyword">void</span> *p3, <span class="keyword">int</span> prio, <span class="keyword">u32_t</span> options, <span class="keyword">const</span> <span class="keyword">char</span> *name)</span></span></span><br><span class="line"><span class="function"></span>{</span><br><span class="line">stack_size = adjust_stack_size(stack_size); <span class="comment">//代码注解1</span></span><br><span class="line"></span><br><span class="line">z_arch_new_thread(new_thread, <span class="built_in">stack</span>, stack_size, entry, p1, p2, p3, prio, options); <span class="comment">//代码注解2</span></span><br><span class="line"></span><br><span class="line"><span class="meta">#<span class="meta-keyword">ifdef</span> CONFIG_THREAD_MONITOR</span></span><br><span class="line">new_thread->entry.pEntry = entry;</span><br><span class="line">new_thread->entry.parameter1 = p1;</span><br><span class="line">new_thread->entry.parameter2 = p2;</span><br><span class="line">new_thread->entry.parameter3 = p3;</span><br><span class="line"></span><br><span class="line"><span class="keyword">k_spinlock_key_t</span> key = k_spin_lock(&lock);</span><br><span class="line"></span><br><span class="line">new_thread->next_thread = _kernel.threads;</span><br><span class="line">_kernel.threads = new_thread;</span><br><span class="line">k_spin_unlock(&lock, key); <span class="comment">//代码注解3</span></span><br><span class="line"><span class="meta">#<span class="meta-keyword">endif</span></span></span><br><span class="line"><span class="meta">#<span class="meta-keyword">ifdef</span> CONFIG_THREAD_NAME</span></span><br><span class="line"><span class="keyword">if</span> (name != <span class="literal">NULL</span>) {</span><br><span class="line"><span class="built_in">strncpy</span>(new_thread->name, name,</span><br><span class="line">CONFIG_THREAD_MAX_NAME_LEN - <span class="number">1</span>);</span><br><span class="line">new_thread->name[CONFIG_THREAD_MAX_NAME_LEN - <span class="number">1</span>] = <span class="string">'\0'</span>; <span class="comment">//代码注解4</span></span><br><span class="line">}</span><br><span class="line"><span class="meta">#<span class="meta-keyword">endif</span></span></span><br><span class="line"></span><br><span class="line"><span class="meta">#<span class="meta-keyword">ifdef</span> CONFIG_ARCH_HAS_CUSTOM_SWAP_TO_MAIN</span></span><br><span class="line"><span class="keyword">if</span> (!_current) {</span><br><span class="line">new_thread->resource_pool = <span class="literal">NULL</span>;</span><br><span class="line"><span class="keyword">return</span>;</span><br><span class="line">}</span><br><span class="line"><span class="meta">#<span class="meta-keyword">endif</span></span></span><br><span class="line">new_thread->resource_pool = _current->resource_pool; <span class="comment">//代码注解5</span></span><br><span class="line">sys_trace_thread_create(new_thread);</span><br><span class="line">}</span><br></pre></td></tr></table></figure><ul><li><strong>代码注解1</strong>: 修正栈大小，主要为线程栈地址随机化功能使用（算是简单的地址随机化方案，感兴趣大家可以搜索一下ASLR相关资料），目的为的是防止破解攻击</li><li><strong>代码注解2</strong>: 执行体系结构相关的线程创建过程（本文接下来以ARM Cortex-M为例介绍）</li><li><strong>代码注解3/代码注解4</strong>: 线程一些公共参数的初始化，用于系统监视器监视线程状态。</li><li><strong>代码注解5</strong>: 继承父线程的资源池。</li></ul><p>其中z_arch_new_thread代码如下(省去部分代码)：</p><figure class="highlight c"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br><span class="line">21</span><br><span class="line">22</span><br><span class="line">23</span><br><span class="line">24</span><br><span class="line">25</span><br><span class="line">26</span><br><span class="line">27</span><br><span class="line">28</span><br><span class="line">29</span><br><span class="line">30</span><br><span class="line">31</span><br><span class="line">32</span><br><span class="line">33</span><br><span class="line">34</span><br><span class="line">35</span><br><span class="line">36</span><br><span class="line">37</span><br><span class="line">38</span><br><span class="line">39</span><br><span class="line">40</span><br><span class="line">41</span><br><span class="line">42</span><br><span class="line">43</span><br><span class="line">44</span><br><span class="line">45</span><br><span class="line">46</span><br><span class="line">47</span><br><span class="line">48</span><br><span class="line">49</span><br><span class="line">50</span><br><span class="line">51</span><br><span class="line">52</span><br><span class="line">53</span><br><span class="line">54</span><br><span class="line">55</span><br><span class="line">56</span><br><span class="line">57</span><br><span class="line">58</span><br><span class="line">59</span><br><span class="line">60</span><br><span class="line">61</span><br><span class="line">62</span><br><span class="line">63</span><br><span class="line">64</span><br><span class="line">65</span><br><span class="line">66</span><br><span class="line">67</span><br><span class="line">68</span><br><span class="line">69</span><br><span class="line">70</span><br><span class="line">71</span><br><span class="line">72</span><br><span class="line">73</span><br><span class="line">74</span><br></pre></td><td class="code"><pre><span class="line"><span class="function"><span class="keyword">void</span> <span class="title">z_arch_new_thread</span><span class="params">(struct k_thread *thread, <span class="keyword">k_thread_stack_t</span> *<span class="built_in">stack</span>, <span class="keyword">size_t</span> stackSize, <span class="keyword">k_thread_entry_t</span> pEntry, <span class="keyword">void</span> *parameter1, <span class="keyword">void</span> *parameter2, <span class="keyword">void</span> *parameter3, <span class="keyword">int</span> priority, <span class="keyword">unsigned</span> <span class="keyword">int</span> options)</span></span></span><br><span class="line"><span class="function"></span>{</span><br><span class="line"><span class="keyword">char</span> *pStackMem = Z_THREAD_STACK_BUFFER(<span class="built_in">stack</span>);</span><br><span class="line"><span class="keyword">char</span> *stackEnd;</span><br><span class="line"></span><br><span class="line"><span class="keyword">u32_t</span> top_of_stack_offset = <span class="number">0U</span>;</span><br><span class="line"></span><br><span class="line">Z_ASSERT_VALID_PRIO(priority, pEntry);</span><br><span class="line"></span><br><span class="line"></span><br><span class="line"><span class="meta">#<span class="meta-keyword">if</span> defined(CONFIG_MPU_REQUIRES_POWER_OF_TWO_ALIGNMENT) \</span></span><br><span class="line">&& defined(CONFIG_USERSPACE)</span><br><span class="line">stackSize -= MPU_GUARD_ALIGN_AND_SIZE;</span><br><span class="line"><span class="meta">#<span class="meta-keyword">endif</span></span></span><br><span class="line"></span><br><span class="line"><span class="meta">#<span class="meta-keyword">if</span> defined(CONFIG_FLOAT) && defined(CONFIG_FP_SHARING) \</span></span><br><span class="line">&& defined(CONFIG_MPU_STACK_GUARD)</span><br><span class="line"><span class="keyword">if</span> ((options & K_FP_REGS) != <span class="number">0</span>) {</span><br><span class="line">pStackMem += MPU_GUARD_ALIGN_AND_SIZE_FLOAT</span><br><span class="line">- MPU_GUARD_ALIGN_AND_SIZE;</span><br><span class="line">stackSize -= MPU_GUARD_ALIGN_AND_SIZE_FLOAT</span><br><span class="line">- MPU_GUARD_ALIGN_AND_SIZE;</span><br><span class="line">}</span><br><span class="line"><span class="meta">#<span class="meta-keyword">endif</span></span></span><br><span class="line">stackEnd = pStackMem + stackSize;</span><br><span class="line"></span><br><span class="line"><span class="class"><span class="keyword">struct</span> __<span class="title">esf</span> *<span class="title">pInitCtx</span>;</span></span><br><span class="line"></span><br><span class="line">z_new_thread_init(thread, pStackMem, stackSize, priority, options);</span><br><span class="line"></span><br><span class="line">pInitCtx = (struct __esf *)(STACK_ROUND_DOWN(stackEnd -</span><br><span class="line">(<span class="keyword">char</span> *)top_of_stack_offset - <span class="keyword">sizeof</span>(struct __basic_sf)));</span><br><span class="line"> </span><br><span class="line">pInitCtx->basic.pc = (<span class="keyword">u32_t</span>)z_thread_entry;</span><br><span class="line"></span><br><span class="line"><span class="meta">#<span class="meta-keyword">if</span> defined(CONFIG_CPU_CORTEX_M)</span></span><br><span class="line">pInitCtx->basic.pc &= <span class="number">0xfffffffe</span>;</span><br><span class="line"><span class="meta">#<span class="meta-keyword">endif</span></span></span><br><span class="line">pInitCtx->basic.a1 = (<span class="keyword">u32_t</span>)pEntry;</span><br><span class="line">pInitCtx->basic.a2 = (<span class="keyword">u32_t</span>)parameter1;</span><br><span class="line">pInitCtx->basic.a3 = (<span class="keyword">u32_t</span>)parameter2;</span><br><span class="line">pInitCtx->basic.a4 = (<span class="keyword">u32_t</span>)parameter3;</span><br><span class="line">pInitCtx->basic.xpsr = <span class="number">0x01000000</span>UL; </span><br><span class="line"></span><br><span class="line">thread->callee_saved.psp = (<span class="keyword">u32_t</span>)pInitCtx;</span><br><span class="line">thread->arch.basepri = <span class="number">0</span>;</span><br><span class="line"></span><br><span class="line"><span class="meta">#<span class="meta-keyword">if</span> defined(CONFIG_USERSPACE) || defined(CONFIG_FP_SHARING)</span></span><br><span class="line">thread->arch.mode = <span class="number">0</span>;</span><br><span class="line"><span class="meta">#<span class="meta-keyword">endif</span></span></span><br><span class="line">}</span><br><span class="line"></span><br><span class="line"><span class="function"><span class="keyword">static</span> ALWAYS_INLINE <span class="keyword">void</span> <span class="title">z_new_thread_init</span><span class="params">(struct k_thread *thread, <span class="keyword">char</span> *pStack, <span class="keyword">size_t</span> stackSize, <span class="keyword">int</span> prio, <span class="keyword">unsigned</span> <span class="keyword">int</span> options)</span></span></span><br><span class="line"><span class="function"></span>{</span><br><span class="line"><span class="meta">#<span class="meta-keyword">ifdef</span> CONFIG_INIT_STACKS</span></span><br><span class="line"><span class="built_in">memset</span>(pStack, <span class="number">0xaa</span>, stackSize);</span><br><span class="line"><span class="meta">#<span class="meta-keyword">endif</span></span></span><br><span class="line"><span class="meta">#<span class="meta-keyword">ifdef</span> CONFIG_STACK_SENTINEL</span></span><br><span class="line">*((<span class="keyword">u32_t</span> *)pStack) = STACK_SENTINEL;</span><br><span class="line"><span class="meta">#<span class="meta-keyword">endif</span> <span class="comment">/* CONFIG_STACK_SENTINEL */</span></span></span><br><span class="line">z_init_thread_base(&thread->base, prio, _THREAD_PRESTART, options);</span><br><span class="line">thread->init_data = <span class="literal">NULL</span>;</span><br><span class="line">thread->fn_abort = <span class="literal">NULL</span>;</span><br><span class="line"><span class="meta">#<span class="meta-keyword">ifdef</span> CONFIG_THREAD_CUSTOM_DATA</span></span><br><span class="line">thread->custom_data = <span class="literal">NULL</span>;</span><br><span class="line"><span class="meta">#<span class="meta-keyword">endif</span></span></span><br><span class="line"><span class="meta">#<span class="meta-keyword">ifdef</span> CONFIG_THREAD_NAME</span></span><br><span class="line">thread->name[<span class="number">0</span>] = <span class="string">'\0'</span>;</span><br><span class="line"><span class="meta">#<span class="meta-keyword">endif</span></span></span><br><span class="line"><span class="meta">#<span class="meta-keyword">if</span> defined(CONFIG_THREAD_STACK_INFO)</span></span><br><span class="line">thread->stack_info.start = (<span class="keyword">uintptr_t</span>)pStack;</span><br><span class="line">thread->stack_info.<span class="built_in">size</span> = (<span class="keyword">u32_t</span>)stackSize;</span><br><span class="line"><span class="meta">#<span class="meta-keyword">endif</span> <span class="comment">/* CONFIG_THREAD_STACK_INFO */</span></span></span><br><span class="line">}</span><br></pre></td></tr></table></figure><p>以上初始化过程执行后，线程的栈空间如图所示：</p><p><img src="http://pic.l2h.site/QQMail_0.png" alt="http://pic.l2h.site/QQMail_0.png"></p><p>以上为线程的初始化过程</p><h3 id="线程执行"><a href="#线程执行" class="headerlink" title="线程执行"></a>线程执行</h3><p>线程开始执行调用<em>z_impl_k_thread_start</em>函数。该函数也会被k_thread_create–>schedule_new_thread间接调用，其源码如下：</p><figure class="highlight c"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br></pre></td><td class="code"><pre><span class="line"><span class="function"><span class="keyword">void</span> <span class="title">z_impl_k_thread_start</span><span class="params">(struct k_thread *thread)</span></span></span><br><span class="line"><span class="function"></span>{</span><br><span class="line"><span class="keyword">k_spinlock_key_t</span> key = k_spin_lock(&lock); </span><br><span class="line"></span><br><span class="line"><span class="keyword">if</span> (z_has_thread_started(thread)) {</span><br><span class="line">k_spin_unlock(&lock, key);</span><br><span class="line"><span class="keyword">return</span>;</span><br><span class="line">}</span><br><span class="line"></span><br><span class="line">z_mark_thread_as_started(thread); <span class="comment">//代码注解1</span></span><br><span class="line">z_ready_thread(thread); <span class="comment">//代码注解2</span></span><br><span class="line">z_reschedule(&lock, key); <span class="comment">//代码注解3</span></span><br><span class="line">}</span><br></pre></td></tr></table></figure><ul><li><strong>代码注解1</strong>：仅仅将thread_state标记上_THREAD_PRESTART </li><li><strong>代码注解2</strong>：将Thread加入调度的ready Q，关于线程调度之后文章进行介绍</li><li><strong>代码注解3</strong>：调用系统的reschedule函数触发系统调度。对Cortex-M3来讲，将会调用系统的PENDSV系统指令，悬起系统切换异常。下图为引用自《Cortex-M3权威指南》的一张PendSV使用的示例：</li></ul><p><img src="http://pic.l2h.site/PendSV.png" alt="PendSV"></p><h3 id="线程挂起"><a href="#线程挂起" class="headerlink" title="线程挂起"></a>线程挂起</h3><p>线程挂起使用<em>z_impl_k_thread_suspend</em>，代码如下：</p><figure class="highlight c"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br></pre></td><td class="code"><pre><span class="line"><span class="function"><span class="keyword">void</span> <span class="title">z_impl_k_thread_suspend</span><span class="params">(struct k_thread *thread)</span></span></span><br><span class="line"><span class="function"></span>{</span><br><span class="line"><span class="keyword">k_spinlock_key_t</span> key = k_spin_lock(&lock);</span><br><span class="line">z_thread_single_suspend(thread);</span><br><span class="line">sys_trace_thread_suspend(thread);</span><br><span class="line"></span><br><span class="line"><span class="keyword">if</span> (thread == _current) {</span><br><span class="line">z_reschedule(&lock, key); <span class="comment">//代码注解3</span></span><br><span class="line">} <span class="keyword">else</span> {</span><br><span class="line">k_spin_unlock(&lock, key);</span><br><span class="line">}</span><br><span class="line">}</span><br><span class="line"><span class="function"><span class="keyword">void</span> <span class="title">z_thread_single_suspend</span><span class="params">(struct k_thread *thread)</span></span></span><br><span class="line"><span class="function"></span>{</span><br><span class="line"><span class="keyword">if</span> (z_is_thread_ready(thread)) {</span><br><span class="line">z_remove_thread_from_ready_q(thread); <span class="comment">//代码注解1</span></span><br><span class="line">}</span><br><span class="line">z_mark_thread_as_suspended(thread); <span class="comment">//代码注解2</span></span><br><span class="line">}</span><br></pre></td></tr></table></figure><ul><li><strong>代码注解3</strong>：若该线程正在运行，调用reschedule重新调度</li><li><strong>代码注解1</strong>：将Thread从调度的ready Q移除</li><li><strong>代码注解2</strong>：对线程状态做挂起标记</li></ul><h3 id="线程继续（resume）"><a href="#线程继续（resume）" class="headerlink" title="线程继续（resume）"></a>线程继续（resume）</h3><p>线程Resume仅做线程的标记，不再进行分析</p><h3 id="线程取消（Abort）"><a href="#线程取消（Abort）" class="headerlink" title="线程取消（Abort）"></a>线程取消（Abort）</h3><p>线程取消使用API <em>z_impl_k_thread_abort</em>,代码如下：</p><figure class="highlight c"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br><span class="line">21</span><br><span class="line">22</span><br><span class="line">23</span><br><span class="line">24</span><br><span class="line">25</span><br><span class="line">26</span><br><span class="line">27</span><br><span class="line">28</span><br><span class="line">29</span><br><span class="line">30</span><br><span class="line">31</span><br><span class="line">32</span><br><span class="line">33</span><br><span class="line">34</span><br><span class="line">35</span><br><span class="line">36</span><br><span class="line">37</span><br><span class="line">38</span><br><span class="line">39</span><br><span class="line">40</span><br><span class="line">41</span><br><span class="line">42</span><br><span class="line">43</span><br></pre></td><td class="code"><pre><span class="line"><span class="function"><span class="keyword">void</span> <span class="title">z_impl_k_thread_abort</span><span class="params">(<span class="keyword">k_tid_t</span> thread)</span></span></span><br><span class="line"><span class="function"></span>{</span><br><span class="line"><span class="keyword">unsigned</span> <span class="keyword">int</span> key;</span><br><span class="line"></span><br><span class="line">key = irq_lock();</span><br><span class="line">__ASSERT(!(thread->base.user_options & K_ESSENTIAL),</span><br><span class="line"> <span class="string">"essential thread aborted"</span>);</span><br><span class="line"></span><br><span class="line">z_thread_single_abort(thread);</span><br><span class="line">z_thread_monitor_exit(thread);</span><br><span class="line"></span><br><span class="line"><span class="keyword">if</span> (_current == thread) {</span><br><span class="line"><span class="keyword">if</span> ((SCB->ICSR & SCB_ICSR_VECTACTIVE_Msk) == <span class="number">0</span>) {</span><br><span class="line">(<span class="keyword">void</span>)z_swap_irqlock(key);</span><br><span class="line">CODE_UNREACHABLE; <span class="comment">//代码注解7</span></span><br><span class="line">} <span class="keyword">else</span> {</span><br><span class="line">SCB->ICSR |= SCB_ICSR_PENDSVSET_Msk; <span class="comment">//代码注解7</span></span><br><span class="line">}</span><br><span class="line">}</span><br><span class="line"></span><br><span class="line">z_reschedule_irqlock(key);</span><br><span class="line">}</span><br><span class="line"><span class="function"><span class="keyword">void</span> <span class="title">z_thread_single_abort</span><span class="params">(struct k_thread *thread)</span></span></span><br><span class="line"><span class="function"></span>{</span><br><span class="line"><span class="keyword">if</span> (thread->fn_abort != <span class="literal">NULL</span>) {</span><br><span class="line">thread->fn_abort(); <span class="comment">//代码注解1</span></span><br><span class="line">}</span><br><span class="line"><span class="keyword">if</span> (IS_ENABLED(CONFIG_SMP)) {</span><br><span class="line">z_sched_abort(thread); <span class="comment">//代码注解2</span></span><br><span class="line">}</span><br><span class="line"><span class="keyword">if</span> (z_is_thread_ready(thread)) {</span><br><span class="line">z_remove_thread_from_ready_q(thread); <span class="comment">//代码注解3</span></span><br><span class="line">} <span class="keyword">else</span> {</span><br><span class="line"><span class="keyword">if</span> (z_is_thread_pending(thread)) {</span><br><span class="line">z_unpend_thread_no_timeout(thread); <span class="comment">//代码注解4</span></span><br><span class="line">}</span><br><span class="line"><span class="keyword">if</span> (z_is_thread_timeout_active(thread)) {</span><br><span class="line">(<span class="keyword">void</span>)z_abort_thread_timeout(thread); <span class="comment">//代码注解5</span></span><br><span class="line">}</span><br><span class="line">}</span><br><span class="line">thread->base.thread_state |= _THREAD_DEAD; <span class="comment">//代码注解6</span></span><br><span class="line">sys_trace_thread_abort(thread);</span><br><span class="line">}</span><br></pre></td></tr></table></figure><ul><li><strong>代码注解1</strong>：先执行该线程的退出回调函数（供应用程序做特殊处理）</li><li><strong>代码注解2</strong>：SMP相关通知，此处不做深入分析</li><li><strong>代码注解3</strong>：将Thread从调度的ready Q移除</li><li><strong>代码注解4</strong>：将Thread从Pending Q也移除</li><li><strong>代码注解5</strong>：若Thread正在等待一定超时后调度，也将其从超时列表中移除</li><li><strong>代码注解6</strong>：对线程状态做死亡（Dead）标记</li><li><strong>代码注解7</strong>：待添加</li><li><strong>代码注解8</strong>：置起PendSV，进行重新调度</li></ul><h2 id="结语"><a href="#结语" class="headerlink" title="结语"></a>结语</h2><p>以上为线程使用相关结构体、API及对应的分析。可以看出，线程相关函数的实现足够简洁明了，这也正应了Zephyr的设计思想。有任何问题，欢迎留言讨论。</p>]]></content>
<summary type="html">
<p>本文介绍和Zephyr线程的数据结构，及相应的API。 </p><h2 id="数据结构"><a href="#数据结构" class="headerlink" title="数据结构"></a>数据结构</h2><p>Zephyr数据结构使用k_thread定义，如下所示：</p><figure class="highlight c"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br><span class="line">21</span><br><span class="line">22</span><br><span class="line">23</span><br><span class="line">24</span><br><span class="line">25</span><br><span class="line">26</span><br><span class="line">27</span><br><span class="line">28</span><br><span class="line">29</span><br><span class="line">30</span><br><span class="line">31</span><br><span class="line">32</span><br><span class="line">33</span><br><span class="line">34</span><br><span class="line">35</span><br><span class="line">36</span><br><span class="line">37</span><br><span class="line">38</span><br><span class="line">39</span><br></pre></td><td class="code"><pre><span class="line"><span class="class"><span class="keyword">struct</span> <span class="title">k_thread</span> &#123;</span></span><br><span class="line"> <span class="class"><span class="keyword">struct</span> _<span class="title">thread_base</span> <span class="title">base</span>;</span></span><br><span class="line"> <span class="class"><span class="keyword">struct</span> _<span class="title">callee_saved</span> <span class="title">callee_saved</span>;</span></span><br><span class="line"> <span class="keyword">void</span> *init_data;</span><br><span class="line"> <span class="keyword">void</span> (*fn_abort)(<span class="keyword">void</span>);</span><br><span class="line"></span><br><span class="line"><span class="meta">#<span class="meta-keyword">if</span> defined(CONFIG_THREAD_MONITOR)</span></span><br><span class="line"> <span class="class"><span class="keyword">struct</span> __<span class="title">thread_entry</span> <span class="title">entry</span>;</span></span><br><span class="line"> <span class="class"><span class="keyword">struct</span> <span class="title">k_thread</span> *<span class="title">next_thread</span>;</span></span><br><span class="line"><span class="meta">#<span class="meta-keyword">endif</span></span></span><br><span class="line"><span class="meta">#<span class="meta-keyword">if</span> defined(CONFIG_THREAD_NAME)</span></span><br><span class="line"> <span class="keyword">char</span> name[CONFIG_THREAD_MAX_NAME_LEN];</span><br><span class="line"><span class="meta">#<span class="meta-keyword">endif</span></span></span><br><span class="line"><span class="meta">#<span class="meta-keyword">ifdef</span> CONFIG_THREAD_CUSTOM_DATA</span></span><br><span class="line"> <span class="keyword">void</span> *custom_data;</span><br><span class="line"><span class="meta">#<span class="meta-keyword">endif</span></span></span><br><span class="line"><span class="meta">#<span class="meta-keyword">ifdef</span> CONFIG_THREAD_USERSPACE_LOCAL_DATA</span></span><br><span class="line"> <span class="class"><span class="keyword">struct</span> _<span class="title">thread_userspace_local_data</span> *<span class="title">userspace_local_data</span>;</span></span><br><span class="line"><span class="meta">#<span class="meta-keyword">endif</span></span></span><br><span class="line"><span class="meta">#<span class="meta-keyword">ifdef</span> CONFIG_ERRNO</span></span><br><span class="line"><span class="meta">#<span class="meta-keyword">ifndef</span> CONFIG_USERSPACE</span></span><br><span class="line"> <span class="keyword">int</span> errno_var;</span><br><span class="line"><span class="meta">#<span class="meta-keyword">endif</span></span></span><br><span class="line"><span class="meta">#<span class="meta-keyword">endif</span></span></span><br><span class="line"></span><br><span class="line"><span class="meta">#<span class="meta-keyword">if</span> defined(CONFIG_THREAD_STACK_INFO)</span></span><br><span class="line"> <span class="class"><span class="keyword">struct</span> _<span class="title">thread_stack_info</span> <span class="title">stack_info</span>;</span></span><br><span class="line"><span class="meta">#<span class="meta-keyword">endif</span> <span class="comment">/* CONFIG_THREAD_STACK_INFO */</span></span></span><br><span class="line"><span class="meta">#<span class="meta-keyword">if</span> defined(CONFIG_USERSPACE)</span></span><br><span class="line"> <span class="class"><span class="keyword">struct</span> _<span class="title">mem_domain_info</span> <span class="title">mem_domain_info</span>;</span></span><br><span class="line"> <span class="keyword">k_thread_stack_t</span> *stack_obj;</span><br><span class="line"><span class="meta">#<span class="meta-keyword">endif</span> <span class="comment">/* CONFIG_USERSPACE */</span></span></span><br><span class="line"><span class="meta">#<span class="meta-keyword">if</span> defined(CONFIG_USE_SWITCH)</span></span><br><span class="line"> <span class="keyword">int</span> swap_retval;</span><br><span class="line"> <span class="keyword">void</span> *switch_handle;</span><br><span class="line"><span class="meta">#<span class="meta-keyword">endif</span></span></span><br><span class="line"> <span class="class"><span class="keyword">struct</span> <span class="title">k_mem_pool</span> *<span class="title">resource_pool</span>;</span></span><br><span class="line"> <span class="class"><span class="keyword">struct</span> _<span class="title">thread_arch</span> <span class="title">arch</span>;</span></span><br><span class="line">&#125;;</span><br></pre></td></tr></table></figure>
</summary>
<category term="RTOS" scheme="http://www.l2h.site/categories/RTOS/"/>
</entry>
<entry>
<title>Zephyr线程管理 - 概念</title>
<link href="http://www.l2h.site/p/42e2c2a3.html"/>
<id>http://www.l2h.site/p/42e2c2a3.html</id>
<published>2019-11-16T12:56:38.000Z</published>
<updated>2023-06-01T13:14:04.477Z</updated>
<content type="html"><![CDATA[<h2 id="前言"><a href="#前言" class="headerlink" title="前言"></a>前言</h2><p>一直想写一些RTOS的技术资料，算作对自己之前一些相关技术调研的总结。无奈懒癌发作，一拖再拖。然今日灌上鸡血，笃定主意，从最基本的调度相关内容开始。 </p><p>简单讲，Zephyr是一个开源实时操作系统。相较Linux，其对系统资源的使用量更小，当然也牺牲了许多复杂且完善的功能（如，系统Debug易用性，线程的堆栈保护）。与此同时，因其是开源社区开发，也多少继承和保留了许多Linux系统的优秀思想和功能（例如，Workqueue、设备树等）。其定位为万物互联时代各种各样的嵌入式设备，目光长远。</p><a id="more"></a><p>Zephyr里没有管程、进程、线程之分。除了中断响应例程，其余所有可执行调度单位皆是线程。系统中应用程序可以定义一个或者多个线程，这种情况下，每个线程也都有自己独立的调度信息和线程ID。</p><p>虽然也有区分用户空间和内核空间态，Zephyr线程却没有自己独立的地址转换表，皆使用相同的地址空间，这是由Zephyr的内存管理方式决定的。</p><h2 id="线程状态"><a href="#线程状态" class="headerlink" title="线程状态"></a>线程状态</h2><p>线程状态即转换关系如下图所示：<br><img src="https://docs.zephyrproject.org/latest/_images/thread_states.svg" alt="https://docs.zephyrproject.org/latest/_images/thread_states.svg"><br>可以看出，线程分为如下状态： </p><ul><li>New：表示线程新创建</li><li>Ready: 线程就绪状态</li><li>Waiting: 线程等待某种IO资源</li><li>Running: 线程执行中</li><li>Suspended: 线程挂起状态，非等待资源原因被停止执行</li><li>Terminated: 线程退出</li></ul><p>以上线程状态转换关系见图，后续进行代码分析将进一步介绍。</p><h2 id="线程优先级"><a href="#线程优先级" class="headerlink" title="线程优先级"></a>线程优先级</h2><p>线程的优先级使用数字表示，数字越小，线程的优先级越高。Zephyr系统的线程可以分为两类: </p><ul><li>Cooperative线程：一旦开始执行，除非中断或者线程自行让出CPU，将会一直执行</li><li>可抢占线程: 普通线程，可被更高优先级的线程抢占执行</li></ul><p>Zephyr默认给Cooperative线程分配小于0的优先级数值，而可抢占线程分配为正值。用户可修改编译选项来更改这两种线程的优先级区间，当然要保证Cooperative线程的优先级数值区间小于可抢占线程。线程运行过程中，其优先级可以被更改。一张图表示线程优先级关系：</p><p><img src="https://docs.zephyrproject.org/latest/_images/priorities.svg" alt="https://docs.zephyrproject.org/latest/_images/priorities.svg"></p><h2 id="线程属性"><a href="#线程属性" class="headerlink" title="线程属性"></a>线程属性</h2><p>线程有一列属性，根据这些不同的属性，线程的执行方式也会有相应的差异。一些主要的属性如下: </p><table><thead><tr><th>属性</th><th>说明</th></tr></thead><tbody><tr><td>K_ESSENTIAL</td><td>表示线程是核心线程，该类线程若有退出或者取消是系统不允许的，系统会断言严重错误</td></tr><tr><td>K_SSE_REGS</td><td>X86独有属性，表示是否使用CPU的SSE功能</td></tr><tr><td>K_FP_REGS</td><td>表示是否使用CPU的浮点计算寄存器</td></tr><tr><td>K_USER</td><td>表示用户空间态线程，只有当系统编译选项CONFIG_USERSPACE打开时才有效</td></tr><tr><td>K_INHERIT_PERMS</td><td>对USERSPACE线程有效，表示是否继承父进程的权限属性</td></tr></tbody></table><h2 id="线程自定义数据"><a href="#线程自定义数据" class="headerlink" title="线程自定义数据"></a>线程自定义数据</h2><p>线程可以自定义数据，使得应用可以对线程功能做一定程度的扩展。</p><h2 id="特殊线程"><a href="#特殊线程" class="headerlink" title="特殊线程"></a>特殊线程</h2><h3 id="系统线程"><a href="#系统线程" class="headerlink" title="系统线程"></a>系统线程</h3><p>Zephyr内核初始化会创建的一些初始线程。主要分为主线程和空闲线程。</p><h3 id="Workqueue"><a href="#Workqueue" class="headerlink" title="Workqueue"></a>Workqueue</h3><p>与Linux系统的Workqueue相似，主要用于中断下半部使用。</p><h2 id="总结"><a href="#总结" class="headerlink" title="总结"></a>总结</h2><p>本文为Zephyr系统，线程的基本概念。主要介绍了线程的状态、分类、优先级等基础思想。</p>]]></content>
<summary type="html">
<h2 id="前言"><a href="#前言" class="headerlink" title="前言"></a>前言</h2><p>一直想写一些RTOS的技术资料，算作对自己之前一些相关技术调研的总结。无奈懒癌发作，一拖再拖。然今日灌上鸡血，笃定主意，从最基本的调度相关内容开始。 </p><p>简单讲，Zephyr是一个开源实时操作系统。相较Linux，其对系统资源的使用量更小，当然也牺牲了许多复杂且完善的功能（如，系统Debug易用性，线程的堆栈保护）。与此同时，因其是开源社区开发，也多少继承和保留了许多Linux系统的优秀思想和功能（例如，Workqueue、设备树等）。其定位为万物互联时代各种各样的嵌入式设备，目光长远。</p>
</summary>
<category term="RTOS" scheme="http://www.l2h.site/categories/RTOS/"/>
</entry>
</feed>