forked from linux-kernel-labs/linux-kernel-labs.github.io
-
Notifications
You must be signed in to change notification settings - Fork 0
Expand file tree
/
Copy pathkernel_modules.html
More file actions
1474 lines (1322 loc) · 96.5 KB
/
Copy pathkernel_modules.html
File metadata and controls
1474 lines (1322 loc) · 96.5 KB
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
400
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
424
425
426
427
428
429
430
431
432
433
434
435
436
437
438
439
440
441
442
443
444
445
446
447
448
449
450
451
452
453
454
455
456
457
458
459
460
461
462
463
464
465
466
467
468
469
470
471
472
473
474
475
476
477
478
479
480
481
482
483
484
485
486
487
488
489
490
491
492
493
494
495
496
497
498
499
500
501
502
503
504
505
506
507
508
509
510
511
512
513
514
515
516
517
518
519
520
521
522
523
524
525
526
527
528
529
530
531
532
533
534
535
536
537
538
539
540
541
542
543
544
545
546
547
548
549
550
551
552
553
554
555
556
557
558
559
560
561
562
563
564
565
566
567
568
569
570
571
572
573
574
575
576
577
578
579
580
581
582
583
584
585
586
587
588
589
590
591
592
593
594
595
596
597
598
599
600
601
602
603
604
605
606
607
608
609
610
611
612
613
614
615
616
617
618
619
620
621
622
623
624
625
626
627
628
629
630
631
632
633
634
635
636
637
638
639
640
641
642
643
644
645
646
647
648
649
650
651
652
653
654
655
656
657
658
659
660
661
662
663
664
665
666
667
668
669
670
671
672
673
674
675
676
677
678
679
680
681
682
683
684
685
686
687
688
689
690
691
692
693
694
695
696
697
698
699
700
701
702
703
704
705
706
707
708
709
710
711
712
713
714
715
716
717
718
719
720
721
722
723
724
725
726
727
728
729
730
731
732
733
734
735
736
737
738
739
740
741
742
743
744
745
746
747
748
749
750
751
752
753
754
755
756
757
758
759
760
761
762
763
764
765
766
767
768
769
770
771
772
773
774
775
776
777
778
779
780
781
782
783
784
785
786
787
788
789
790
791
792
793
794
795
796
797
798
799
800
801
802
803
804
805
806
807
808
809
810
811
812
813
814
815
816
817
818
819
820
821
822
823
824
825
826
827
828
829
830
831
832
833
834
835
836
837
838
839
840
841
842
843
844
845
846
847
848
849
850
851
852
853
854
855
856
857
858
859
860
861
862
863
864
865
866
867
868
869
870
871
872
873
874
875
876
877
878
879
880
881
882
883
884
885
886
887
888
889
890
891
892
893
894
895
896
897
898
899
900
901
902
903
904
905
906
907
908
909
910
911
912
913
914
915
916
917
918
919
920
921
922
923
924
925
926
927
928
929
930
931
932
933
934
935
936
937
938
939
940
941
942
943
944
945
946
947
948
949
950
951
952
953
954
955
956
957
958
959
960
961
962
963
964
965
966
967
968
969
970
971
972
973
974
975
976
977
978
979
980
981
982
983
984
985
986
987
988
989
990
991
992
993
994
995
996
997
998
999
1000
<!DOCTYPE html>
<!--[if IE 8]><html class="no-js lt-ie9" lang="en" > <![endif]-->
<!--[if gt IE 8]><!--> <html class="no-js" lang="en" > <!--<![endif]-->
<head>
<meta charset="utf-8">
<meta name="viewport" content="width=device-width, initial-scale=1.0">
<title>Kernel modules — The Linux Kernel documentation</title>
<script type="text/javascript" src="../_static/js/modernizr.min.js"></script>
<script type="text/javascript">
var DOCUMENTATION_OPTIONS = {
URL_ROOT:'../',
VERSION:'',
LANGUAGE:'None',
COLLAPSE_INDEX:false,
FILE_SUFFIX:'.html',
HAS_SOURCE: true,
SOURCELINK_SUFFIX: '.txt'
};
</script>
<script type="text/javascript" src="../_static/jquery.js"></script>
<script type="text/javascript" src="../_static/underscore.js"></script>
<script type="text/javascript" src="../_static/doctools.js"></script>
<script type="text/javascript" src="../_static/asciinema-player.js"></script>
<script type="text/javascript" src="../_static/js/theme.js"></script>
<link rel="stylesheet" href="../_static/css/theme.css" type="text/css" />
<link rel="stylesheet" href="../_static/pygments.css" type="text/css" />
<link rel="stylesheet" href="../_static/asciinema-player.css" type="text/css" />
<link rel="stylesheet" href="../_static/theme_overrides.css" type="text/css" />
<link rel="index" title="Index" href="../genindex.html" />
<link rel="search" title="Search" href="../search.html" />
<link rel="next" title="Kernel API" href="kernel_api.html" />
<link rel="prev" title="Introduction" href="introduction.html" />
</head>
<body class="wy-body-for-nav">
<div class="wy-grid-for-nav">
<nav data-toggle="wy-nav-shift" class="wy-nav-side">
<div class="wy-side-scroll">
<div class="wy-side-nav-search" >
<a href="../index.html" class="icon icon-home"> The Linux Kernel
</a>
<div class="version">
4.19.0
</div>
<div role="search">
<form id="rtd-search-form" class="wy-form" action="../search.html" method="get">
<input type="text" name="q" placeholder="Search docs" />
<input type="hidden" name="check_keywords" value="yes" />
<input type="hidden" name="area" value="default" />
</form>
</div>
</div>
<div class="wy-menu wy-menu-vertical" data-spy="affix" role="navigation" aria-label="main navigation">
<p class="caption"><span class="caption-text">Lectures</span></p>
<ul>
<li class="toctree-l1"><a class="reference internal" href="../lectures/so2.cs.pub.ro.html">Sisteme de operare 2</a></li>
<li class="toctree-l1"><a class="reference internal" href="../lectures/intro.html">Introduction</a></li>
<li class="toctree-l1"><a class="reference internal" href="../lectures/syscalls.html">System Calls</a></li>
<li class="toctree-l1"><a class="reference internal" href="../lectures/interrupts.html">Interrupts</a></li>
<li class="toctree-l1"><a class="reference internal" href="../lectures/smp.html">Symmetric Multi-Processing</a></li>
<li class="toctree-l1"><a class="reference internal" href="../lectures/debugging.html">Debugging</a></li>
<li class="toctree-l1"><a class="reference internal" href="vm.html">Virtual Machine Setup</a></li>
</ul>
<p class="caption"><span class="caption-text">Labs</span></p>
<ul class="current">
<li class="toctree-l1"><a class="reference internal" href="infrastructure.html">Infrastructure</a></li>
<li class="toctree-l1"><a class="reference internal" href="introduction.html">Introduction</a></li>
<li class="toctree-l1 current"><a class="current reference internal" href="#">Kernel modules</a><ul>
<li class="toctree-l2"><a class="reference internal" href="#lab-objectives">Lab objectives</a></li>
<li class="toctree-l2"><a class="reference internal" href="#overview">Overview</a></li>
<li class="toctree-l2"><a class="reference internal" href="#an-example-of-a-kernel-module">An example of a kernel module</a></li>
<li class="toctree-l2"><a class="reference internal" href="#compiling-kernel-modules">Compiling kernel modules</a></li>
<li class="toctree-l2"><a class="reference internal" href="#loading-unloading-a-kernel-module">Loading/unloading a kernel module</a></li>
<li class="toctree-l2"><a class="reference internal" href="#debugging">Debugging</a><ul>
<li class="toctree-l3"><a class="reference internal" href="#objdump">objdump</a></li>
<li class="toctree-l3"><a class="reference internal" href="#addr2line">addr2line</a></li>
<li class="toctree-l3"><a class="reference internal" href="#minicom">minicom</a></li>
<li class="toctree-l3"><a class="reference internal" href="#netconsole">netconsole</a></li>
<li class="toctree-l3"><a class="reference internal" href="#printk-debugging">Printk debugging</a></li>
<li class="toctree-l3"><a class="reference internal" href="#dynamic-debugging">Dynamic debugging</a><ul>
<li class="toctree-l4"><a class="reference internal" href="#dyndbg-options">Dyndbg Options</a></li>
</ul>
</li>
<li class="toctree-l3"><a class="reference internal" href="#kdb-kernel-debugger">KDB: Kernel debugger</a></li>
</ul>
</li>
<li class="toctree-l2"><a class="reference internal" href="#exercises">Exercises</a><ul>
<li class="toctree-l3"><a class="reference internal" href="#intro">0. Intro</a></li>
<li class="toctree-l3"><a class="reference internal" href="#kernel-module">1. Kernel module</a></li>
<li class="toctree-l3"><a class="reference internal" href="#printk">2. Printk</a></li>
<li class="toctree-l3"><a class="reference internal" href="#error">3. Error</a></li>
<li class="toctree-l3"><a class="reference internal" href="#sub-modules">4. Sub-modules</a></li>
<li class="toctree-l3"><a class="reference internal" href="#id3">5. Kernel oops</a></li>
<li class="toctree-l3"><a class="reference internal" href="#module-parameters">6. Module parameters</a></li>
<li class="toctree-l3"><a class="reference internal" href="#proc-info">7. Proc info</a></li>
</ul>
</li>
<li class="toctree-l2"><a class="reference internal" href="#extra-exercises">Extra Exercises</a><ul>
<li class="toctree-l3"><a class="reference internal" href="#kdb">1. KDB</a></li>
<li class="toctree-l3"><a class="reference internal" href="#ps-module">2. PS Module</a></li>
<li class="toctree-l3"><a class="reference internal" href="#memory-info">3. Memory Info</a></li>
<li class="toctree-l3"><a class="reference internal" href="#id5">4. Dynamic Debugging</a></li>
<li class="toctree-l3"><a class="reference internal" href="#dynamic-debugging-during-initialization">5. Dynamic Debugging During Initialization</a></li>
</ul>
</li>
</ul>
</li>
<li class="toctree-l1"><a class="reference internal" href="kernel_api.html">Kernel API</a></li>
<li class="toctree-l1"><a class="reference internal" href="device_drivers.html">Character device drivers</a></li>
<li class="toctree-l1"><a class="reference internal" href="interrupts.html">I/O access and Interrupts</a></li>
<li class="toctree-l1"><a class="reference internal" href="deferred_work.html">Deferred work</a></li>
<li class="toctree-l1"><a class="reference internal" href="block_device_drivers.html">Block Device Drivers</a></li>
<li class="toctree-l1"><a class="reference internal" href="filesystems_part1.html">File system drivers (Part 1)</a></li>
<li class="toctree-l1"><a class="reference internal" href="filesystems_part2.html">File system drivers (Part 2)</a></li>
<li class="toctree-l1"><a class="reference internal" href="networking.html">Networking</a></li>
<li class="toctree-l1"><a class="reference internal" href="memory_mapping.html">Memory mapping</a></li>
<li class="toctree-l1"><a class="reference internal" href="device_model.html">Linux Device Model</a></li>
</ul>
</div>
</div>
</nav>
<section data-toggle="wy-nav-shift" class="wy-nav-content-wrap">
<nav class="wy-nav-top" aria-label="top navigation">
<i data-toggle="wy-nav-top" class="fa fa-bars"></i>
<a href="../index.html">The Linux Kernel</a>
</nav>
<div class="wy-nav-content">
<div class="rst-content">
<div role="navigation" aria-label="breadcrumbs navigation">
<ul class="wy-breadcrumbs">
<li><a href="../index.html">Docs</a> »</li>
<li>Kernel modules</li>
<li class="wy-breadcrumbs-aside">
<a href="../_sources/labs/kernel_modules.rst.txt" rel="nofollow"> View page source</a>
</li>
</ul>
<hr/>
</div>
<div role="main" class="document" itemscope="itemscope" itemtype="http://schema.org/Article">
<div itemprop="articleBody">
<div class="section" id="kernel-modules">
<h1>Kernel modules<a class="headerlink" href="#kernel-modules" title="Permalink to this headline">¶</a></h1>
<div class="section" id="lab-objectives">
<h2>Lab objectives<a class="headerlink" href="#lab-objectives" title="Permalink to this headline">¶</a></h2>
<ul class="simple">
<li>creating simple modules</li>
<li>describing the process of kernel module compilation</li>
<li>presenting how a module can be used with a kernel</li>
<li>simple kernel debugging methods</li>
</ul>
</div>
<div class="section" id="overview">
<h2>Overview<a class="headerlink" href="#overview" title="Permalink to this headline">¶</a></h2>
<p>A monolithic kernel, though faster than a microkernel, has the disadvantage of
lack of modularity and extensibility. On modern monolithic kernels, this has
been solved by using kernel modules. A kernel module (or loadable kernel mode)
is an object file that contains code that can extend the kernel functionality
at runtime (it is loaded as needed); When a kernel module is no longer needed,
it can be unloaded. Most of the device drivers are used in the form of kernel
modules.</p>
<p>For the development of Linux device drivers, it is recommended to download the
kernel sources, configure and compile them and then install the compiled version
on the test /development tool machine.</p>
</div>
<div class="section" id="an-example-of-a-kernel-module">
<h2>An example of a kernel module<a class="headerlink" href="#an-example-of-a-kernel-module" title="Permalink to this headline">¶</a></h2>
<p>Below is a very simple example of a kernel module. When loading into the kernel,
it will generate the message <code class="code docutils literal"><span class="pre">"Hi"</span></code>. When unloading the kernel module, the
<code class="code docutils literal"><span class="pre">"Bye"</span></code> message will be generated.</p>
<div class="highlight-c"><div class="highlight"><pre><span></span><span class="cp">#include</span> <span class="cpf"><linux/kernel.h></span><span class="cp"></span>
<span class="cp">#include</span> <span class="cpf"><linux/init.h></span><span class="cp"></span>
<span class="cp">#include</span> <span class="cpf"><linux/module.h></span><span class="cp"></span>
<span class="n">MODULE_DESCRIPTION</span><span class="p">(</span><span class="s">"My kernel module"</span><span class="p">);</span>
<span class="n">MODULE_AUTHOR</span><span class="p">(</span><span class="s">"Me"</span><span class="p">);</span>
<span class="n">MODULE_LICENSE</span><span class="p">(</span><span class="s">"GPL"</span><span class="p">);</span>
<span class="k">static</span> <span class="kt">int</span> <span class="nf">dummy_init</span><span class="p">(</span><span class="kt">void</span><span class="p">)</span>
<span class="p">{</span>
<span class="n">pr_debug</span><span class="p">(</span><span class="s">"Hi</span><span class="se">\n</span><span class="s">"</span><span class="p">);</span>
<span class="k">return</span> <span class="mi">0</span><span class="p">;</span>
<span class="p">}</span>
<span class="k">static</span> <span class="kt">void</span> <span class="nf">dummy_exit</span><span class="p">(</span><span class="kt">void</span><span class="p">)</span>
<span class="p">{</span>
<span class="n">pr_debug</span><span class="p">(</span><span class="s">"Bye</span><span class="se">\n</span><span class="s">"</span><span class="p">);</span>
<span class="p">}</span>
<span class="n">module_init</span><span class="p">(</span><span class="n">dummy_init</span><span class="p">);</span>
<span class="n">module_exit</span><span class="p">(</span><span class="n">dummy_exit</span><span class="p">);</span>
</pre></div>
</div>
<p>The generated messages will not be displayed on the console but will be saved
in a specially reserved memory area for this, from where they will be extracted
by the logging daemon (syslog). To display kernel messages, you can use the
<strong class="command">dmesg</strong> command or inspect the logs:</p>
<div class="highlight-bash"><div class="highlight"><pre><span></span><span class="c1"># cat /var/log/syslog | tail -2</span>
Feb <span class="m">20</span> <span class="m">13</span>:57:38 asgard kernel: Hi
Feb <span class="m">20</span> <span class="m">13</span>:57:43 asgard kernel: Bye
<span class="c1"># dmesg | tail -2</span>
Hi
Bye
</pre></div>
</div>
</div>
<div class="section" id="compiling-kernel-modules">
<h2>Compiling kernel modules<a class="headerlink" href="#compiling-kernel-modules" title="Permalink to this headline">¶</a></h2>
<p>Compiling a kernel module differs from compiling an user program. First, other
headers should be used. Also, the module should not be linked to libraries.
And, last but not least, the module must be compiled with the same options as
the kernel in which we load the module. For these reasons, there is a standard
compilation method (<code class="code docutils literal"><span class="pre">kbuild</span></code>). This method requires the use of two files:
a <code class="file docutils literal"><span class="pre">Makefile</span></code> and a <code class="file docutils literal"><span class="pre">Kbuild</span></code> file.</p>
<p>Below is an example of a <code class="file docutils literal"><span class="pre">Makefile</span></code>:</p>
<div class="highlight-bash"><div class="highlight"><pre><span></span><span class="nv">KDIR</span> <span class="o">=</span> /lib/modules/<span class="sb">`</span>uname -r<span class="sb">`</span>/build
kbuild:
make -C <span class="k">$(</span>KDIR<span class="k">)</span> <span class="nv">M</span><span class="o">=</span><span class="sb">`</span><span class="nb">pwd</span><span class="sb">`</span>
clean:
make -C <span class="k">$(</span>KDIR<span class="k">)</span> <span class="nv">M</span><span class="o">=</span><span class="sb">`</span><span class="nb">pwd</span><span class="sb">`</span> clean
</pre></div>
</div>
<p>And the example of a <code class="file docutils literal"><span class="pre">Kbuild</span></code> file used to compile a module:</p>
<div class="highlight-bash"><div class="highlight"><pre><span></span><span class="nv">EXTRA_CFLAGS</span> <span class="o">=</span> -Wall -g
obj-m <span class="o">=</span> modul.o
</pre></div>
</div>
<p>As you can see, calling <strong class="command">make</strong> on the <code class="file docutils literal"><span class="pre">Makefile</span></code> file in the
example shown will result in the <strong class="command">make</strong> invocation in the kernel
source directory (<code class="docutils literal"><span class="pre">/lib/modules/`uname</span> <span class="pre">-r`/build</span></code>) and referring to the
current directory (<code class="docutils literal"><span class="pre">M</span> <span class="pre">=</span> <span class="pre">`pwd`</span></code>). This process ultimately leads to reading
the <code class="file docutils literal"><span class="pre">Kbuild</span></code> file from the current directory and compiling the module
as instructed in this file.</p>
<div class="admonition note">
<p class="first admonition-title">Note</p>
<p>For labs we will configure different <strong class="command">KDIR</strong>, according to
the virtual machine specifications:</p>
<div class="last highlight-bash"><div class="highlight"><pre><span></span><span class="nv">KDIR</span> <span class="o">=</span> /home/student/so2/linux
<span class="o">[</span>...<span class="o">]</span>
</pre></div>
</div>
</div>
<p>A <code class="file docutils literal"><span class="pre">Kbuild</span></code> file contains one or more directives for compiling a kernel
module. The easiest example of such a directive is <code class="docutils literal"><span class="pre">obj-m</span> <span class="pre">=</span>
<span class="pre">module.o</span></code>. Following this directive, a kernel module (<code class="code docutils literal"><span class="pre">ko</span></code> - kernel
object) will be created, starting from the <code class="docutils literal"><span class="pre">module.o</span></code> file. <code class="docutils literal"><span class="pre">module.o</span></code> will
be created starting from <code class="docutils literal"><span class="pre">module.c</span></code> or <code class="docutils literal"><span class="pre">module.S</span></code>. All of these files can
be found in the <code class="file docutils literal"><span class="pre">Kbuild</span></code>’s directory.</p>
<p>An example of a <code class="file docutils literal"><span class="pre">Kbuild</span></code> file that uses several sub-modules is shown
below:</p>
<div class="highlight-bash"><div class="highlight"><pre><span></span><span class="nv">EXTRA_CFLAGS</span> <span class="o">=</span> -Wall -g
obj-m <span class="o">=</span> supermodule.o
supermodule-y <span class="o">=</span> module-a.o module-b.o
</pre></div>
</div>
<p>For the example above, the steps to compile are:</p>
<blockquote>
<div><ul class="simple">
<li>compile the <code class="file docutils literal"><span class="pre">module-a.c</span></code> and <code class="file docutils literal"><span class="pre">module-b.c</span></code> sources,
resulting in module-a.o and module-b.o objects</li>
<li><code class="file docutils literal"><span class="pre">module-a.o</span></code> and <code class="file docutils literal"><span class="pre">module-b.o</span></code> will then be linked
in <code class="file docutils literal"><span class="pre">supermodule.o</span></code></li>
<li>from <code class="file docutils literal"><span class="pre">supermodule.o</span></code> will be created <code class="file docutils literal"><span class="pre">supermodule.ko</span></code>
module</li>
</ul>
</div></blockquote>
<p>The suffix of targets in <code class="file docutils literal"><span class="pre">Kbuild</span></code> determines how they are used, as
follows:</p>
<blockquote>
<div><ul class="simple">
<li>M (modules) is a target for loadable kernel modules</li>
<li>Y (yes) represents a target for object files to be compiled and then
linked to a module (<code class="docutils literal"><span class="pre">$(mode_name)-y</span></code>) or within the kernel (<code class="docutils literal"><span class="pre">obj-y</span></code>)</li>
<li>any other target suffix will be ignored by <code class="file docutils literal"><span class="pre">Kbuild</span></code> and will not be
compiled</li>
</ul>
</div></blockquote>
<div class="admonition note">
<p class="first admonition-title">Note</p>
<p class="last">These suffixes are used to easily configure the kernel by running the
<strong class="command">make menuconfig</strong> command or directly editing the
<code class="file docutils literal"><span class="pre">.config</span></code> file. This file sets a series of variables that are
used to determine which features are added to the kernel at build
time. For example, when adding BTRFS support with <strong class="command">make
menuconfig</strong>, add the line <code class="code docutils literal"><span class="pre">CONFIG_BTRFS_FS</span> <span class="pre">=</span> <span class="pre">y</span></code> to the
<code class="file docutils literal"><span class="pre">.config</span></code> file. The BTRFS kbuild contains the line
<code class="docutils literal"><span class="pre">obj-$(CONFIG_BTRFS_FS):=</span> <span class="pre">btrfs.o</span></code>, which becomes <code class="docutils literal"><span class="pre">obj-y:=</span>
<span class="pre">btrfs.o</span></code>. This will compile the <code class="file docutils literal"><span class="pre">btrfs.o</span></code> object and will be
linked to the kernel. Before the variable was set, the line became
<code class="docutils literal"><span class="pre">obj:=btrfs.o</span></code> and so it was ignored, and the kernel was build
without BTRFS support.</p>
</div>
<p>For more details, see the <code class="file docutils literal"><span class="pre">Documentation/kbuild/makefiles.txt</span></code> and
<code class="file docutils literal"><span class="pre">Documentation/kbuild/modules.txt</span></code> files within the kernel sources.</p>
</div>
<div class="section" id="loading-unloading-a-kernel-module">
<h2>Loading/unloading a kernel module<a class="headerlink" href="#loading-unloading-a-kernel-module" title="Permalink to this headline">¶</a></h2>
<p>To load a kernel module, use the <strong class="command">insmod</strong> utility. This utility
receives as a parameter the path to the <code class="file docutils literal"><span class="pre">*.ko</span></code> file in which the module
was compiled and linked. Unloading the module from the kernel is done using
the <strong class="command">rmmod</strong> command, which receives the module name as a parameter.</p>
<div class="highlight-bash"><div class="highlight"><pre><span></span>$ insmod module.ko
$ rmmod module.ko
</pre></div>
</div>
<p>When loading the kernel module, the routine specified as a parameter of the
<code class="docutils literal"><span class="pre">module_init</span></code> macro will be executed. Similarly, when the module is unloaded
the routine specified as a parameter of the <code class="docutils literal"><span class="pre">module_exit</span></code> will be executed.</p>
<p>A complete example of compiling and loading/unloading a kernel module is
presented below:</p>
<div class="highlight-bash"><div class="highlight"><pre><span></span>faust:~/lab-01/modul-lin# ls
Kbuild Makefile modul.c
faust:~/lab-01/modul-lin# make
make -C /lib/modules/<span class="sb">`</span>uname -r<span class="sb">`</span>/build <span class="nv">M</span><span class="o">=</span><span class="sb">`</span><span class="nb">pwd</span><span class="sb">`</span>
make<span class="o">[</span><span class="m">1</span><span class="o">]</span>: Entering directory <span class="sb">`</span>/usr/src/linux-2.6.28.4<span class="s1">'</span>
<span class="s1"> LD /root/lab-01/modul-lin/built-in.o</span>
<span class="s1"> CC [M] /root/lab-01/modul-lin/modul.o</span>
<span class="s1"> Building modules, stage 2.</span>
<span class="s1"> MODPOST 1 modules</span>
<span class="s1"> CC /root/lab-01/modul-lin/modul.mod.o</span>
<span class="s1"> LD [M] /root/lab-01/modul-lin/modul.ko</span>
<span class="s1">make[1]: Leaving directory `/usr/src/linux-2.6.28.4'</span>
faust:~/lab-01/modul-lin# ls
built-in.o Kbuild Makefile modul.c Module.markers
modules.order Module.symvers modul.ko modul.mod.c
modul.mod.o modul.o
faust:~/lab-01/modul-lin# insmod modul.ko
faust:~/lab-01/modul-lin# dmesg <span class="p">|</span> tail -1
Hi
faust:~/lab-01/modul-lin# rmmod modul
faust:~/lab-01/modul-lin# dmesg <span class="p">|</span> tail -2
Hi
Bye
</pre></div>
</div>
<p>Information about modules loaded into the kernel can be found using the
<strong class="command">lsmod</strong> command or by inspecting the <code class="file docutils literal"><span class="pre">/proc/modules</span></code>,
<code class="file docutils literal"><span class="pre">/sys/module</span></code> directories.</p>
</div>
<div class="section" id="debugging">
<h2>Debugging<a class="headerlink" href="#debugging" title="Permalink to this headline">¶</a></h2>
<p>Troubleshooting a kernel module is much more complicated than debugging a
regular program. First, a mistake in a kernel module can lead to blocking the
entire system. Troubleshooting is therefore much slowed down. To avoid reboot,
it is recommended to use a virtual machine (qemu, virtualbox, vmware).</p>
<p>When a module containing bugs is inserted into the kernel, it will eventually
generate a <a class="reference external" href="https://en.wikipedia.org/wiki/Linux_kernel_oops">kernel oops</a>.
A kernel oops is an invalid operation detected by the kernel and can only
be generated by the kernel. For a stable kernel version, it almost certainly
means that the module contains a bug. After the oops appears, the kernel will
continue to work.</p>
<p>Very important to the appearance of a kernel oops is saving the generated
message. As noted above, messages generated by the kernel are saved in logs and
can be displayed with the <strong class="command">dmesg</strong> command. To make sure that no kernel
message is lost, it is recommended to insert/test the kernel directly from the
console, or periodically check the kernel messages. Noteworthy is that an oops
can occur because of a programming error, but also a because of hardware error.</p>
<p>If a fatal error occurs, after which the system can not return to a stable
state, a <a class="reference external" href="https://en.wikipedia.org/wiki/Linux_kernel_panic">kernel panic</a> is
generated.</p>
<p>Look at the kernel module below that contains a bug that generates an oops:</p>
<div class="highlight-c"><div class="highlight"><pre><span></span><span class="cm">/*</span>
<span class="cm"> * Oops generating kernel module</span>
<span class="cm"> */</span>
<span class="cp">#include</span> <span class="cpf"><linux/kernel.h></span><span class="cp"></span>
<span class="cp">#include</span> <span class="cpf"><linux/module.h></span><span class="cp"></span>
<span class="cp">#include</span> <span class="cpf"><linux/init.h></span><span class="cp"></span>
<span class="n">MODULE_DESCRIPTION</span> <span class="p">(</span><span class="s">"Oops"</span><span class="p">);</span>
<span class="n">MODULE_LICENSE</span> <span class="p">(</span><span class="s">"GPL"</span><span class="p">);</span>
<span class="n">MODULE_AUTHOR</span> <span class="p">(</span><span class="s">"PSO"</span><span class="p">);</span>
<span class="cp">#define OP_READ 0</span>
<span class="cp">#define OP_WRITE 1</span>
<span class="cp">#define OP_OOPS OP_WRITE</span>
<span class="k">static</span> <span class="kt">int</span> <span class="nf">my_oops_init</span> <span class="p">(</span><span class="kt">void</span><span class="p">)</span>
<span class="p">{</span>
<span class="kt">int</span> <span class="o">*</span><span class="n">a</span><span class="p">;</span>
<span class="n">a</span> <span class="o">=</span> <span class="p">(</span><span class="kt">int</span> <span class="o">*</span><span class="p">)</span> <span class="mh">0x00001234</span><span class="p">;</span>
<span class="cp">#if OP_OOPS == OP_WRITE</span>
<span class="o">*</span><span class="n">a</span> <span class="o">=</span> <span class="mi">3</span><span class="p">;</span>
<span class="cp">#elif OP_OOPS == OP_READ</span>
<span class="n">printk</span> <span class="p">(</span><span class="n">KERN_ALERT</span> <span class="s">"value = %d</span><span class="se">\n</span><span class="s">"</span><span class="p">,</span> <span class="o">*</span><span class="n">a</span><span class="p">);</span>
<span class="cp">#else</span>
<span class="cp">#error "Unknown op for oops!"</span>
<span class="cp">#endif</span>
<span class="k">return</span> <span class="mi">0</span><span class="p">;</span>
<span class="p">}</span>
<span class="k">static</span> <span class="kt">void</span> <span class="nf">my_oops_exit</span> <span class="p">(</span><span class="kt">void</span><span class="p">)</span>
<span class="p">{</span>
<span class="p">}</span>
<span class="n">module_init</span> <span class="p">(</span><span class="n">my_oops_init</span><span class="p">);</span>
<span class="n">module_exit</span> <span class="p">(</span><span class="n">my_oops_exit</span><span class="p">);</span>
</pre></div>
</div>
<p>Inserting this module into the kernel will generate an oops:</p>
<div class="highlight-bash"><div class="highlight"><pre><span></span>faust:~/lab-01/modul-oops# insmod oops.ko
<span class="o">[</span>...<span class="o">]</span>
faust:~/lab-01/modul-oops# dmesg <span class="p">|</span> tail -32
BUG: unable to handle kernel paging request at <span class="m">00001234</span>
IP: <span class="o">[</span><c89d4005><span class="o">]</span> my_oops_init+0x5/0x20 <span class="o">[</span>oops<span class="o">]</span>
*de <span class="o">=</span> <span class="m">00000000</span>
Oops: <span class="m">0002</span> <span class="o">[</span><span class="c1">#1] PREEMPT DEBUG_PAGEALLOC</span>
last sysfs file: /sys/devices/virtual/net/lo/operstate
Modules linked in: oops<span class="o">(</span>+<span class="o">)</span> netconsole ide_cd_mod pcnet32 crc32 cdrom <span class="o">[</span>last unloaded: modul<span class="o">]</span>
Pid: <span class="m">4157</span>, comm: insmod Not tainted <span class="o">(</span><span class="m">2</span>.6.28.4 <span class="c1">#2) VMware Virtual Platform</span>
EIP: <span class="m">0060</span>:<span class="o">[</span><c89d4005><span class="o">]</span> EFLAGS: <span class="m">00010246</span> CPU: <span class="m">0</span>
EIP is at my_oops_init+0x5/0x20 <span class="o">[</span>oops<span class="o">]</span>
EAX: <span class="m">00000000</span> EBX: fffffffc ECX: c89d4300 EDX: <span class="m">00000001</span>
ESI: c89d4000 EDI: <span class="m">00000000</span> EBP: c5799e24 ESP: c5799e24
DS: 007b ES: 007b FS: <span class="m">0000</span> GS: <span class="m">0033</span> SS: <span class="m">0068</span>
Process insmod <span class="o">(</span>pid: <span class="m">4157</span>, <span class="nv">ti</span><span class="o">=</span>c5799000 <span class="nv">task</span><span class="o">=</span>c665c780 task.ti<span class="o">=</span>c5799000<span class="o">)</span>
Stack:
c5799f8c c010102d c72b51d8 0000000c c5799e58 c01708e4 <span class="m">00000124</span> <span class="m">00000000</span>
c89d4300 c5799e58 c724f448 <span class="m">00000001</span> c89d4300 c5799e60 c0170981 c5799f8c
c014b698 <span class="m">00000000</span> <span class="m">00000000</span> c5799f78 c5799f20 <span class="m">00000500</span> c665cb00 c89d4300
Call Trace:
<span class="o">[</span><c010102d><span class="o">]</span> ? _stext+0x2d/0x170
<span class="o">[</span><c01708e4><span class="o">]</span> ? __vunmap+0xa4/0xf0
<span class="o">[</span><c0170981><span class="o">]</span> ? vfree+0x21/0x30
<span class="o">[</span><c014b698><span class="o">]</span> ? load_module+0x19b8/0x1a40
<span class="o">[</span><c035e965><span class="o">]</span> ? __mutex_unlock_slowpath+0xd5/0x140
<span class="o">[</span><c0140da6><span class="o">]</span> ? trace_hardirqs_on_caller+0x106/0x150
<span class="o">[</span><c014b7aa><span class="o">]</span> ? sys_init_module+0x8a/0x1b0
<span class="o">[</span><c0140da6><span class="o">]</span> ? trace_hardirqs_on_caller+0x106/0x150
<span class="o">[</span><c0240a08><span class="o">]</span> ? trace_hardirqs_on_thunk+0xc/0x10
<span class="o">[</span><c0103407><span class="o">]</span> ? sysenter_do_call+0x12/0x43
Code: <c7> <span class="m">05</span> <span class="m">34</span> <span class="m">12</span> <span class="m">00</span> <span class="m">00</span> <span class="m">03</span> <span class="m">00</span> <span class="m">00</span> <span class="m">00</span> 5d c3 eb 0d <span class="m">90</span> <span class="m">90</span> <span class="m">90</span> <span class="m">90</span> <span class="m">90</span> <span class="m">90</span> <span class="m">90</span> <span class="m">90</span>
EIP: <span class="o">[</span><c89d4005><span class="o">]</span> my_oops_init+0x5/0x20 <span class="o">[</span>oops<span class="o">]</span> SS:ESP <span class="m">0068</span>:c5799e24
---<span class="o">[</span> end trace 2981ce73ae801363 <span class="o">]</span>---
</pre></div>
</div>
<p>Although relatively cryptic, the message provided by the kernel to the
appearance of an oops provides valuable information about the error. First line:</p>
<div class="highlight-bash"><div class="highlight"><pre><span></span>BUG: unable to handle kernel paging request at <span class="m">00001234</span>
EIP: <span class="o">[</span><c89d4005><span class="o">]</span> my_oops_init + 0x5 / 0x20 <span class="o">[</span>oops<span class="o">]</span>
</pre></div>
</div>
<p>Tells us the cause and the address of the instruction that generated the error.
In our case this is an invalid access to memory.</p>
<p>Next line</p>
<blockquote>
<div><code class="docutils literal"><span class="pre">Oops:</span> <span class="pre">0002</span> <span class="pre">[#</span> <span class="pre">1]</span> <span class="pre">PREEMPT</span> <span class="pre">DEBUG_PAGEALLOC</span></code></div></blockquote>
<p>Tells us that it’s the first oops (#1). This is important in the context that
an oops can lead to other oopses. Usually only the first oops is relevant.
Furthermore, the oops code (<code class="docutils literal"><span class="pre">0002</span></code>) provides information about the error type
(see <code class="file docutils literal"><span class="pre">arch/x86/include/asm/traps.h</span></code>):</p>
<blockquote>
<div><ul class="simple">
<li>Bit 0 == 0 means no page found, 1 means protection fault</li>
<li>Bit 1 == 0 means read, 1 means write</li>
<li>Bit 2 == 0 means kernel, 1 means user mode</li>
</ul>
</div></blockquote>
<p>In this case, we have a write access that generated the oops (bit 1 is 1).</p>
<p>Below is a dump of the registers. It decodes the instruction pointer (<code class="docutils literal"><span class="pre">EIP</span></code>)
value and notes that the bug appeared in the <code class="code docutils literal"><span class="pre">my_oops_init</span></code> function with
a 5-byte offset (<code class="docutils literal"><span class="pre">EIP:</span> <span class="pre">[<c89d4005>]</span> <span class="pre">my_oops_init+0x5</span></code>). The message also
shows the stack content and a backtrace of calls until then.</p>
<p>If an invalid read call is generated (<code class="docutils literal"><span class="pre">#define</span> <span class="pre">OP_OOPS</span> <span class="pre">OP_READ</span></code>), the message
will be the same, but the oops code will differ, which would now be <code class="docutils literal"><span class="pre">0000</span></code>:</p>
<div class="highlight-bash"><div class="highlight"><pre><span></span>faust:~/lab-01/modul-oops# dmesg <span class="p">|</span> tail -33
BUG: unable to handle kernel paging request at <span class="m">00001234</span>
IP: <span class="o">[</span><c89c3016><span class="o">]</span> my_oops_init+0x6/0x20 <span class="o">[</span>oops<span class="o">]</span>
*de <span class="o">=</span> <span class="m">00000000</span>
Oops: <span class="m">0000</span> <span class="o">[</span><span class="c1">#1] PREEMPT DEBUG_PAGEALLOC</span>
last sysfs file: /sys/devices/virtual/net/lo/operstate
Modules linked in: oops<span class="o">(</span>+<span class="o">)</span> netconsole pcnet32 crc32 ide_cd_mod cdrom
Pid: <span class="m">2754</span>, comm: insmod Not tainted <span class="o">(</span><span class="m">2</span>.6.28.4 <span class="c1">#2) VMware Virtual Platform</span>
EIP: <span class="m">0060</span>:<span class="o">[</span><c89c3016><span class="o">]</span> EFLAGS: <span class="m">00010292</span> CPU: <span class="m">0</span>
EIP is at my_oops_init+0x6/0x20 <span class="o">[</span>oops<span class="o">]</span>
EAX: <span class="m">00000000</span> EBX: fffffffc ECX: c89c3380 EDX: <span class="m">00000001</span>
ESI: c89c3010 EDI: <span class="m">00000000</span> EBP: c57cbe24 ESP: c57cbe1c
DS: 007b ES: 007b FS: <span class="m">0000</span> GS: <span class="m">0033</span> SS: <span class="m">0068</span>
Process insmod <span class="o">(</span>pid: <span class="m">2754</span>, <span class="nv">ti</span><span class="o">=</span>c57cb000 <span class="nv">task</span><span class="o">=</span>c66ec780 task.ti<span class="o">=</span>c57cb000<span class="o">)</span>
Stack:
c57cbe34 <span class="m">00000282</span> c57cbf8c c010102d c57b9280 0000000c c57cbe58 c01708e4
<span class="m">00000124</span> <span class="m">00000000</span> c89c3380 c57cbe58 c5db1d38 <span class="m">00000001</span> c89c3380 c57cbe60
c0170981 c57cbf8c c014b698 <span class="m">00000000</span> <span class="m">00000000</span> c57cbf78 c57cbf20 <span class="m">00000580</span>
Call Trace:
<span class="o">[</span><c010102d><span class="o">]</span> ? _stext+0x2d/0x170
<span class="o">[</span><c01708e4><span class="o">]</span> ? __vunmap+0xa4/0xf0
<span class="o">[</span><c0170981><span class="o">]</span> ? vfree+0x21/0x30
<span class="o">[</span><c014b698><span class="o">]</span> ? load_module+0x19b8/0x1a40
<span class="o">[</span><c035d083><span class="o">]</span> ? printk+0x0/0x1a
<span class="o">[</span><c035e965><span class="o">]</span> ? __mutex_unlock_slowpath+0xd5/0x140
<span class="o">[</span><c0140da6><span class="o">]</span> ? trace_hardirqs_on_caller+0x106/0x150
<span class="o">[</span><c014b7aa><span class="o">]</span> ? sys_init_module+0x8a/0x1b0
<span class="o">[</span><c0140da6><span class="o">]</span> ? trace_hardirqs_on_caller+0x106/0x150
<span class="o">[</span><c0240a08><span class="o">]</span> ? trace_hardirqs_on_thunk+0xc/0x10
<span class="o">[</span><c0103407><span class="o">]</span> ? sysenter_do_call+0x12/0x43
Code: <a1> <span class="m">34</span> <span class="m">12</span> <span class="m">00</span> <span class="m">00</span> c7 <span class="m">04</span> <span class="m">24</span> <span class="m">54</span> <span class="m">30</span> 9c c8 <span class="m">89</span> <span class="m">44</span> <span class="m">24</span> <span class="m">04</span> e8 <span class="m">58</span> a0 <span class="m">99</span> f7 <span class="m">31</span>
EIP: <span class="o">[</span><c89c3016><span class="o">]</span> my_oops_init+0x6/0x20 <span class="o">[</span>oops<span class="o">]</span> SS:ESP <span class="m">0068</span>:c57cbe1c
---<span class="o">[</span> end trace 45eeb3d6ea8ff1ed <span class="o">]</span>---
</pre></div>
</div>
<div class="section" id="objdump">
<h3>objdump<a class="headerlink" href="#objdump" title="Permalink to this headline">¶</a></h3>
<p>Detailed information about the instruction that generated the oops can be found
using the <strong class="command">objdump</strong> utility. Useful options to use are <strong class="command">-d</strong>
to disassemble the code and <strong class="command">-S</strong> for interleaving C code in assembly
language code. For efficient decoding, however, we need the address where the
kernel module was loaded. This can be found in <code class="file docutils literal"><span class="pre">/proc/modules</span></code>.</p>
<p>Here’s an example of using <strong class="command">objdump</strong> on the above module to identify
the instruction that generated the oops:</p>
<div class="highlight-bash"><div class="highlight"><pre><span></span>faust:~/lab-01/modul-oops# cat /proc/modules
oops <span class="m">1280</span> <span class="m">1</span> - Loading 0xc89d4000
netconsole <span class="m">8352</span> <span class="m">0</span> - Live 0xc89ad000
pcnet32 <span class="m">33412</span> <span class="m">0</span> - Live 0xc895a000
ide_cd_mod <span class="m">34952</span> <span class="m">0</span> - Live 0xc8903000
crc32 <span class="m">4224</span> <span class="m">1</span> pcnet32, Live 0xc888a000
cdrom <span class="m">34848</span> <span class="m">1</span> ide_cd_mod, Live 0xc886d000
faust:~/lab-01/modul-oops# objdump -dS --adjust-vma<span class="o">=</span>0xc89d4000 oops.ko
oops.ko: file format elf32-i386
Disassembly of section .text:
c89d4000 <init_module>:
<span class="c1">#define OP_READ 0</span>
<span class="c1">#define OP_WRITE 1</span>
<span class="c1">#define OP_OOPS OP_WRITE</span>
static int my_oops_init <span class="o">(</span>void<span class="o">)</span>
<span class="o">{</span>
c89d4000: <span class="m">55</span> push %ebp
<span class="c1">#else</span>
<span class="c1">#error "Unknown op for oops!"</span>
<span class="c1">#endif</span>
<span class="k">return</span> <span class="m">0</span><span class="p">;</span>
<span class="o">}</span>
c89d4001: <span class="m">31</span> c0 xor %eax,%eax
<span class="c1">#define OP_READ 0</span>
<span class="c1">#define OP_WRITE 1</span>
<span class="c1">#define OP_OOPS OP_WRITE</span>
static int my_oops_init <span class="o">(</span>void<span class="o">)</span>
<span class="o">{</span>
c89d4003: <span class="m">89</span> e5 mov %esp,%ebp
int *a<span class="p">;</span>
<span class="nv">a</span> <span class="o">=</span> <span class="o">(</span>int *<span class="o">)</span> 0x00001234<span class="p">;</span>
<span class="c1">#if OP_OOPS == OP_WRITE</span>
*a <span class="o">=</span> <span class="m">3</span><span class="p">;</span>
c89d4005: c7 <span class="m">05</span> <span class="m">34</span> <span class="m">12</span> <span class="m">00</span> <span class="m">00</span> <span class="m">03</span> movl <span class="nv">$0</span>x3,0x1234
c89d400c: <span class="m">00</span> <span class="m">00</span> <span class="m">00</span>
<span class="c1">#else</span>
<span class="c1">#error "Unknown op for oops!"</span>
<span class="c1">#endif</span>
<span class="k">return</span> <span class="m">0</span><span class="p">;</span>
<span class="o">}</span>
c89d400f: 5d pop %ebp
c89d4010: c3 ret
c89d4011: eb 0d jmp c89c3020 <cleanup_module>
c89d4013: <span class="m">90</span> nop
c89d4014: <span class="m">90</span> nop
c89d4015: <span class="m">90</span> nop
c89d4016: <span class="m">90</span> nop
c89d4017: <span class="m">90</span> nop
c89d4018: <span class="m">90</span> nop
c89d4019: <span class="m">90</span> nop
c89d401a: <span class="m">90</span> nop
c89d401b: <span class="m">90</span> nop
c89d401c: <span class="m">90</span> nop
c89d401d: <span class="m">90</span> nop
c89d401e: <span class="m">90</span> nop
c89d401f: <span class="m">90</span> nop
c89d4020 <cleanup_module>:
static void my_oops_exit <span class="o">(</span>void<span class="o">)</span>
<span class="o">{</span>
c89d4020: <span class="m">55</span> push %ebp
c89d4021: <span class="m">89</span> e5 mov %esp,%ebp
<span class="o">}</span>
c89d4023: 5d pop %ebp
c89d4024: c3 ret
c89d4025: <span class="m">90</span> nop
c89d4026: <span class="m">90</span> nop
c89d4027: <span class="m">90</span> nop
</pre></div>
</div>
<p>Note that the instruction that generated the oops (<code class="docutils literal"><span class="pre">c89d4005</span></code> identified
earlier) is:</p>
<blockquote>
<div><code class="docutils literal"><span class="pre">C89d4005:</span> <span class="pre">c7</span> <span class="pre">05</span> <span class="pre">34</span> <span class="pre">12</span> <span class="pre">00</span> <span class="pre">00</span> <span class="pre">03</span> <span class="pre">movl</span> <span class="pre">$</span> <span class="pre">0x3,0x1234</span></code></div></blockquote>
<p>That is exactly what was expected - storing value 3 at 0x0001234.</p>
<p>The <code class="file docutils literal"><span class="pre">/proc/modules</span></code> is used to find the address where a kernel module is
loaded. The <strong class="command">--adjust-vma</strong> option allows you to display instructions
relative to <code class="docutils literal"><span class="pre">0xc89d4000</span></code>. The <strong class="command">-l</strong> option displays the number of
each line in the source code interleaved with the assembly language code.</p>
</div>
<div class="section" id="addr2line">
<h3>addr2line<a class="headerlink" href="#addr2line" title="Permalink to this headline">¶</a></h3>
<p>A more simplistic way to find the code that generated an oops is to use the
<strong class="command">addr2line</strong> utility:</p>
<div class="highlight-bash"><div class="highlight"><pre><span></span>faust:~/lab-01/modul-oops# addr2line -e oops.o 0x5
/root/lab-01/modul-oops/oops.c:23
</pre></div>
</div>
<p>Where <code class="docutils literal"><span class="pre">0x5</span></code> is the value of the program counter (<code class="docutils literal"><span class="pre">EIP</span> <span class="pre">=</span> <span class="pre">c89d4005</span></code>) that
generated the oops, minus the base address of the module (<code class="docutils literal"><span class="pre">0xc89c4000</span></code>)
according to <code class="file docutils literal"><span class="pre">/proc/modules</span></code></p>
</div>
<div class="section" id="minicom">
<h3>minicom<a class="headerlink" href="#minicom" title="Permalink to this headline">¶</a></h3>
<p><strong class="command">Minicom</strong> (or other equivalent utilities, eg <strong class="command">picocom</strong>,
<strong class="command">screen</strong>) is a utility that can be used to connect and interact with a
serial port. The serial port is the basic method for analyzing kernel messages
or interacting with an embedded system in the development phase. There are two
more common ways to connect:</p>
<ul class="simple">
<li>a serial port where the device we are going to use is <code class="file docutils literal"><span class="pre">/dev/ttyS0</span></code></li>
<li>a serial USB port (FTDI) in which case the device we are going to use is
<code class="file docutils literal"><span class="pre">/dev/ttyUSB</span></code>.</li>
</ul>
<p>For the virtual machine used in the lab, the device that we need to use is
displayed after the virtual machine starts:</p>
<div class="highlight-bash"><div class="highlight"><pre><span></span>char device redirected to /dev/pts/20 <span class="o">(</span>label virtiocon0<span class="o">)</span>
</pre></div>
</div>
<p>Minicom use:</p>
<div class="highlight-bash"><div class="highlight"><pre><span></span><span class="c1">#for connecting via COM1 and using a speed of 115,200 characters per second</span>
minicom -b <span class="m">115200</span> -D /dev/ttyS0
<span class="c1">#For USB serial port connection</span>
minicom -D /dev/ttyUSB0
<span class="c1">#To connect to the serial port of the virtual machine</span>
minicom -D /dev/pts/20
</pre></div>
</div>
</div>
<div class="section" id="netconsole">
<h3>netconsole<a class="headerlink" href="#netconsole" title="Permalink to this headline">¶</a></h3>
<p><strong class="command">Netconsole</strong> is a utility that allows logging of kernel debugging
messages over the network. This is useful when the disk logging system does not
work or when serial ports are not available or when the terminal does not
respond to commands. <strong class="command">Netconsole</strong> comes in the form of a kernel
module.</p>
<p>To work, it needs the following parameters:</p>
<blockquote>
<div><ul class="simple">
<li>port, IP address, and the source interface name of the debug station</li>
<li>port, MAC address, and IP address of the machine to which the debug
messages will be sent</li>
</ul>
</div></blockquote>
<p>These parameters can be configured when the module is inserted into the kernel,
or even while the module is inserted if it has been compiled with the
<code class="docutils literal"><span class="pre">CONFIG_NETCONSOLE_DYNAMIC</span></code> option.</p>
<p>An example configuration when inserting <strong class="command">netconsole</strong> kernel module is
as follows:</p>
<div class="highlight-bash"><div class="highlight"><pre><span></span>alice:~# modprobe netconsole <span class="nv">netconsole</span><span class="o">=</span><span class="m">6666</span>@192.168.191.130/eth0,6000@192.168.191.1/00:50:56:c0:00:08
</pre></div>
</div>
<p>Thus, the debug messages on the station that has the address
<code class="docutils literal"><span class="pre">192.168.191.130</span></code> will be sent to the <code class="docutils literal"><span class="pre">eth0</span></code> interface, having source port
<code class="docutils literal"><span class="pre">6666</span></code>. The messages will be sent to <code class="docutils literal"><span class="pre">192.168.191.1</span></code> with the MAC address
<code class="docutils literal"><span class="pre">00:50:56:c0:00:08</span></code>, on port <code class="docutils literal"><span class="pre">6000</span></code>.</p>
<p>Messages can be played on the destination station using <strong class="command">netcat</strong>:</p>
<div class="highlight-bash"><div class="highlight"><pre><span></span>bob:~ <span class="c1"># nc -l -p 6000 -u</span>
</pre></div>
</div>
<p>Alternatively, the destination station can configure <strong class="command">syslogd</strong> to
intercept these messages. More information can be found in
<code class="file docutils literal"><span class="pre">Documentation/networking/netconsole.txt</span></code>.</p>
</div>
<div class="section" id="printk-debugging">
<h3>Printk debugging<a class="headerlink" href="#printk-debugging" title="Permalink to this headline">¶</a></h3>
<p><code class="docutils literal"><span class="pre">The</span> <span class="pre">two</span> <span class="pre">oldest</span> <span class="pre">and</span> <span class="pre">most</span> <span class="pre">useful</span> <span class="pre">debugging</span> <span class="pre">aids</span> <span class="pre">are</span> <span class="pre">Your</span> <span class="pre">Brain</span> <span class="pre">and</span> <span class="pre">Printf</span></code>.</p>
<p>For debugging, a primitive way is often used, but it is quite effective:
<code class="code docutils literal"><span class="pre">printk</span></code> debugging. Although a debugger can also be used, it is generally
not very useful: simple bugs (uninitialized variables, memory management
problems, etc.) can be easily localized by control messages and the
kernel-decoded oop message.</p>
<p>For more complex bugs, even a debugger can not help us too much unless the
operating system structure is very well understood. When debugging a kernel
module, there are a lot of unknowns in the equation: multiple contexts (we have
multiple processes and threads running at a time), interruptions, virtual
memory, etc.</p>
<p>You can use <code class="code docutils literal"><span class="pre">printk</span></code> to display kernel messages to user space. It is
similar to <code class="code docutils literal"><span class="pre">printf</span></code>’s functionality; the only difference is that the
transmitted message can be prefixed with a string of <code class="code docutils literal"><span class="pre">"<n>"</span></code>, where
<code class="code docutils literal"><span class="pre">n</span></code> indicates the error level (loglevel) and has values between <code class="docutils literal"><span class="pre">0</span></code> and
<code class="docutils literal"><span class="pre">7</span></code>. Instead of <code class="code docutils literal"><span class="pre">"<n>"</span></code>, the levels can also be coded by symbolic
constants:</p>
<div class="highlight-c"><div class="highlight"><pre><span></span><span class="n">KERN_EMERG</span> <span class="o">-</span> <span class="n">n</span> <span class="o">=</span> <span class="mi">0</span>
<span class="n">KERN_ALERT</span> <span class="o">-</span> <span class="n">n</span> <span class="o">=</span> <span class="mi">1</span>
<span class="n">KERN_CRIT</span> <span class="o">-</span> <span class="n">n</span> <span class="o">=</span> <span class="mi">2</span>
<span class="n">KERN_ERR</span> <span class="o">-</span> <span class="n">n</span> <span class="o">=</span> <span class="mi">3</span>
<span class="n">KERN_WARNING</span> <span class="o">-</span> <span class="n">n</span> <span class="o">=</span> <span class="mi">4</span>
<span class="n">KERN_NOTICE</span> <span class="o">-</span> <span class="n">n</span> <span class="o">=</span> <span class="mi">5</span>
<span class="n">KERN_INFO</span> <span class="o">-</span> <span class="n">n</span> <span class="o">=</span> <span class="mi">6</span>
<span class="n">KERN_DEBUG</span> <span class="o">-</span> <span class="n">n</span> <span class="o">=</span> <span class="mi">7</span>
</pre></div>
</div>
<p>The definitions of all log levels are found in <code class="file docutils literal"><span class="pre">linux/kern_levels.h</span></code>.
Basically, these log levels are used by the system to route messages sent to
various outputs: console, log files in <code class="file docutils literal"><span class="pre">/var/log</span></code> etc.</p>
<div class="admonition note">
<p class="first admonition-title">Note</p>
<p>To display <code class="code docutils literal"><span class="pre">printk</span></code> messages in user space, the <code class="code docutils literal"><span class="pre">printk</span></code>
log level must be of higher priority than <cite>console_loglevel</cite>
variable. The default console log level can be configured from
<code class="file docutils literal"><span class="pre">/proc/sys/kernel/printk</span></code>.</p>
<p>For instance, the command:</p>
<div class="highlight-bash"><div class="highlight"><pre><span></span><span class="nb">echo</span> <span class="m">8</span> > /proc/sys/kernel/printk
</pre></div>
</div>
<p class="last">will enable all the kernel log messages to be displayed in the
console. That is, the logging level has to be strictly less than the
<code class="code docutils literal"><span class="pre">console_loglevel</span></code> variable. For example, if the
<code class="code docutils literal"><span class="pre">console_loglevel</span></code> has a value of <code class="docutils literal"><span class="pre">5</span></code> (specific to
<code class="code docutils literal"><span class="pre">KERN_NOTICE</span></code>), only messages with loglevel stricter than <code class="docutils literal"><span class="pre">5</span></code>
(i.e <code class="code docutils literal"><span class="pre">KERN_EMERG</span></code>, <code class="code docutils literal"><span class="pre">KERN_ALERT</span></code>, <code class="code docutils literal"><span class="pre">KERN_CRIT</span></code>,
<code class="code docutils literal"><span class="pre">KERN_ERR</span></code>, <code class="code docutils literal"><span class="pre">KERN_WARNING</span></code>) will be shown.</p>
</div>
<p>Console-redirected messages can be useful for quickly viewing the effect of
executing the kernel code, but they are no longer so useful if the kernel
encounters an irreparable error and the system freezes. In this case, the logs
of the system must be consulted, as they keep the information between system
restarts. These are found in <code class="file docutils literal"><span class="pre">/var/log</span></code> and are text files, populated by
<code class="code docutils literal"><span class="pre">syslogd</span></code> and <code class="code docutils literal"><span class="pre">klogd</span></code> during the kernel run. <code class="code docutils literal"><span class="pre">syslogd</span></code> and
<code class="code docutils literal"><span class="pre">klogd</span></code> take the information from the virtual file system mounted in
<code class="file docutils literal"><span class="pre">/proc</span></code>. In principle, with <code class="code docutils literal"><span class="pre">syslogd</span></code> and <code class="code docutils literal"><span class="pre">klogd</span></code> turned on,
all messages coming from the kernel will go to <code class="file docutils literal"><span class="pre">/var/log/kern.log</span></code>.</p>
<p>A simpler version for debugging is using the <code class="file docutils literal"><span class="pre">/var/log/debug</span></code> file. It
is populated only with the <code class="code docutils literal"><span class="pre">printk</span></code> messages from the kernel with the
<code class="code docutils literal"><span class="pre">KERN_DEBUG</span></code> log level.</p>
<p>Given that a production kernel (similar to the one we’re probably running with)
contains only release code, our module is among the few that send messages
prefixed with KERN_DEBUG . In this way, we can easily navigate through the
<code class="file docutils literal"><span class="pre">/var/log/debug</span></code> information by finding the messages corresponding to a
debugging session for our module.</p>
<p>Such an example would be the following:</p>
<div class="highlight-bash"><div class="highlight"><pre><span></span><span class="c1"># Clear the debug file of previous information (or possibly a backup)</span>
$ <span class="nb">echo</span> <span class="s2">"New debug session"</span> > /var/log/debug
<span class="c1"># Run the tests</span>
<span class="c1"># If there is no critical error causing a panic kernel, check the output</span>
<span class="c1"># if a critical error occurs and the machine only responds to a restart,</span>
restart the system and check /var/log/debug.
</pre></div>
</div>
<p>The format of the messages must obviously contain all the information of
interest in order to detect the error, but inserting in the code <code class="code docutils literal"><span class="pre">printk</span></code>
to provide detailed information can be as time-consuming as writing the code to
solve the problem. This is usually a trade-off between the completeness of the
debugging messages displayed using <code class="code docutils literal"><span class="pre">printk</span></code> and the time it takes to
insert these messages into the text.</p>
<p>A very simple way, less time-consuming for inserting <code class="code docutils literal"><span class="pre">printk</span></code> and
providing the possibility to analyze the flow of instructions for tests is the
use of the predefined constants <code class="code docutils literal"><span class="pre">__FILE__</span></code>, <code class="code docutils literal"><span class="pre">__LINE__</span></code> and
<code class="code docutils literal"><span class="pre">__func__</span></code>:</p>
<blockquote>
<div><ul class="simple">
<li><code class="docutils literal"><span class="pre">__FILE__</span></code> is replaced by the compiler with the name of the source file
it is currently being compiled.</li>
<li><code class="docutils literal"><span class="pre">__LINE__</span></code> is replaced by the compiler with the line number on which the
current instruction is found in the current source file.</li>
<li><code class="docutils literal"><span class="pre">__func__</span></code> /<code class="docutils literal"><span class="pre">__FUNCTION__</span></code> is replaced by the compiler with the name
of the function in which the current instruction is found.</li>
</ul>
</div></blockquote>
<div class="admonition note">
<p class="first admonition-title">Note</p>
<p class="last"><code class="code docutils literal"><span class="pre">__FILE__</span></code> and <code class="code docutils literal"><span class="pre">__LINE__</span></code> are part of the ANSI C specifications:
:code`__func__` is part of specification C99; <code class="code docutils literal"><span class="pre">__FUNCTION__</span></code> is a GNU
:codeC extension and is not portable; However, since we write code for the
:codeLinux kernel, we can use it without any problems.</p>
</div>
<p>The following macrodefinition can be used in this case:</p>
<div class="highlight-c"><div class="highlight"><pre><span></span><span class="cp">#define PRINT_DEBUG \</span>
<span class="cp"> printk (KERN_DEBUG "[% s]: FUNC:% s: LINE:% d \ n", __FILE__,</span>
<span class="n">__FUNCTION__</span><span class="p">,</span> <span class="n">__LINE__</span><span class="p">)</span>
</pre></div>
</div>
<p>Then, at each point where we want to see if it is “reached” in execution,
insert PRINT_DEBUG; This is a simple and quick way, and can yield by carefully
analyzing the output.</p>
<p>The <strong class="command">dmesg</strong> command is used to view the messages printed with
<code class="code docutils literal"><span class="pre">printk</span></code> but not appearing on the console.</p>
<p>To delete all previous messages from a log file, run:</p>
<div class="highlight-bash"><div class="highlight"><pre><span></span>cat /dev/null > /var/log/debug
</pre></div>
</div>
<p>To delete messages displayed by the <strong class="command">dmesg</strong> command, run:</p>
<div class="highlight-bash"><div class="highlight"><pre><span></span>dmesg -c
</pre></div>
</div>
</div>
<div class="section" id="dynamic-debugging">
<h3>Dynamic debugging<a class="headerlink" href="#dynamic-debugging" title="Permalink to this headline">¶</a></h3>
<p>Dynamic <a class="reference external" href="https://www.kernel.org/doc/html/v4.15/admin-guide/dynamic-debug-howto.html">dyndbg</a>
debugging enables dynamic debugging activation/deactivation.
Unlike <code class="code docutils literal"><span class="pre">printk</span></code>, it offers more advanced <code class="code docutils literal"><span class="pre">printk</span></code> options for the
messages we want to display; it is very useful for complex modules or
troubleshooting subsystems.
This significantly reduces the amount of messages displayed, leaving only
those relevant for the debug context. To enable <code class="docutils literal"><span class="pre">dyndbg</span></code>, the kernel must be
compiled with the <code class="docutils literal"><span class="pre">CONFIG_DYNAMIC_DEBUG</span></code> option. Once configured,
<code class="code docutils literal"><span class="pre">pr_debug()</span></code>, <code class="code docutils literal"><span class="pre">dev_dbg()</span></code> and <code class="code docutils literal"><span class="pre">print_hex_dump_debug()</span></code>,
<code class="code docutils literal"><span class="pre">print_hex_dump_bytes()</span></code> can be dynamically enabled per call.</p>
<p>The <code class="file docutils literal"><span class="pre">/sys/kernel/debug/dynamic_debug/control</span></code> file from the debugfs (where
<code class="file docutils literal"><span class="pre">/sys/kernel/debug</span></code> is the path to which debugfs was mounted) is used to
filter messages or to view existing filters.</p>
<div class="highlight-c"><div class="highlight"><pre><span></span><span class="n">mount</span> <span class="o">-</span><span class="n">t</span> <span class="n">debugfs</span> <span class="n">none</span> <span class="o">/</span><span class="n">debug</span>
</pre></div>
</div>
<p><a class="reference external" href="http://opensourceforu.com/2010/10/debugging-linux-kernel-with-debugfs/">Debugfs</a>
is a simple file system, used as a kernel-space interface and
user-space interface to configure different debug options. Any debug utility
can create and use its own files /folders in debugfs.</p>
<p>For example, to display existing filters in <code class="docutils literal"><span class="pre">dyndbg</span></code>, you will use:</p>
<div class="highlight-bash"><div class="highlight"><pre><span></span>cat /debug/dynamic_debug/control
</pre></div>
</div>
<p>And to enable the debug message from line <code class="docutils literal"><span class="pre">1603</span></code> in the <code class="file docutils literal"><span class="pre">svcsock.c</span></code> file:</p>
<div class="highlight-bash"><div class="highlight"><pre><span></span><span class="nb">echo</span> <span class="s1">'file svcsock.c line 1603 +p'</span> > /debug/dynamic_debug/control
</pre></div>
</div>
<p>The <code class="file docutils literal"><span class="pre">/debug/dynamic_debug/control</span></code> file is not a regular file. It shows
the <code class="docutils literal"><span class="pre">dyndbg</span></code> settings on the filters. Writing in it with an echo will change
these settings (it will not actually make a write). Be aware that the file
contains settings for <code class="docutils literal"><span class="pre">dyndbg</span></code> debugging messages. Do not log in this file.</p>
<div class="section" id="dyndbg-options">
<h4>Dyndbg Options<a class="headerlink" href="#dyndbg-options" title="Permalink to this headline">¶</a></h4>
<ul>
<li><p class="first"><code class="docutils literal"><span class="pre">func</span></code> - just the debug messages from the functions that have the same
name as the one defined in the filter.</p>
<div class="highlight-bash"><div class="highlight"><pre><span></span><span class="nb">echo</span> <span class="s1">'func svc_tcp_accept +p'</span> > /debug/dynamic_debug/control
</pre></div>
</div>
</li>
<li><p class="first"><code class="docutils literal"><span class="pre">file</span></code> - the name of the file(s) for which we want to display the debug
messages. It can be just the source name, but also the absolute path or
kernel-tree path.</p>
<div class="highlight-bash"><div class="highlight"><pre><span></span>file svcsock.c
file kernel/freezer.c
file /usr/src/packages/BUILD/sgi-enhancednfs-1.4/default/net/sunrpc/svcsock.c
</pre></div>
</div>
</li>
<li><p class="first"><code class="docutils literal"><span class="pre">module</span></code> - module name.</p>
<div class="highlight-bash"><div class="highlight"><pre><span></span>Modules sunrpc
</pre></div>
</div>
</li>
<li><p class="first"><code class="docutils literal"><span class="pre">format</span></code> - only messages whose display format contains the specified string.</p>
<blockquote>
<div><div class="highlight-bash"><div class="highlight"><pre><span></span>format <span class="s2">"nfsd: SETATTR"</span>
</pre></div>
</div>
</div></blockquote>
</li>
<li><p class="first">line - the line or lines for which we want to enable debug calls.</p>
<div class="highlight-bash"><div class="highlight"><pre><span></span><span class="c1"># Triggers debug messages between lines 1603 and 1605 in the svcsock.c file</span>
$ <span class="nb">echo</span> <span class="s1">'file svcsock.c line 1603-1605 +p'</span> > /sys/kernel/debug/dynamic_debug/control
<span class="c1"># Enables debug messages from the beginning of the file to line 1605</span>
$ <span class="nb">echo</span> <span class="s1">'file svcsock.c line -1605 +p'</span> > /sys/kernel/debug/dynamic_debug/control
</pre></div>
</div>
</li>
</ul>
<p>In addition to the above options, a series of flags can be added, removed, or set
with operators <code class="docutils literal"><span class="pre">+`,</span> <span class="pre">``-</span></code> or <code class="docutils literal"><span class="pre">=</span></code>:</p>
<blockquote>
<div><ul class="simple">
<li><code class="docutils literal"><span class="pre">p</span></code> activates the pr_debug() .</li>
<li><code class="docutils literal"><span class="pre">f</span></code> includes the name of the function in the printed message.</li>
<li><code class="docutils literal"><span class="pre">I</span></code> includes the line number in the printed message.</li>
<li><code class="docutils literal"><span class="pre">M</span></code> includes the module name in the printed message.</li>
<li><code class="docutils literal"><span class="pre">T</span></code> includes the thread id if it is not called from interrupt context</li>
<li><code class="docutils literal"><span class="pre">_</span></code> no flag is set.</li>
</ul>
</div></blockquote>
</div>
</div>
<div class="section" id="kdb-kernel-debugger">
<h3>KDB: Kernel debugger<a class="headerlink" href="#kdb-kernel-debugger" title="Permalink to this headline">¶</a></h3>
<p>The kernel debugger has proven to be very useful to facilitate the development and
debugging process. One of its main advantages it the possibility to perform live debugging.
This allows us to monitor, in real time, the accesses to memory or even modify the memory
while debugging.
The debugger has been integrated in the mainline kernel starting with version 2.6.26-rci.
KDB is not a <a href="#id1"><span class="problematic" id="id2">*</span></a>source debugger”, but for a complete analysis it can be used in parallel with
gdb and symbol files – see <a class="reference internal" href="introduction.html#gdb-intro"><span class="std std-ref">the GDB debugging section</span></a></p>
<p>To use KDB, you have the following options:</p>
<blockquote>
<div><ul class="simple">
<li>non-usb keyboard + VGA text console</li>