linux-kernel-labs.github.io/tmp/lectures/debugging.html at master · devzero2000/linux-kernel-labs.github.io

655 lines (531 loc) · 32.8 KB
<!DOCTYPE html>
<!--[if IE 8]><html class="no-js lt-ie9" lang="en" > <![endif]-->
<!--[if gt IE 8]><!--> <html class="no-js" lang="en" > <!--<![endif]-->
  <meta charset="utf-8">
  <meta name="viewport" content="width=device-width, initial-scale=1.0">
  <title>Debugging &mdash; The Linux Kernel  documentation</title>
  <script type="text/javascript" src="../_static/js/modernizr.min.js"></script>
      <script type="text/javascript">
          var DOCUMENTATION_OPTIONS = {
              URL_ROOT:'../',
              VERSION:'',
              LANGUAGE:'None',
              COLLAPSE_INDEX:false,
              FILE_SUFFIX:'.html',
              HAS_SOURCE:  true,
              SOURCELINK_SUFFIX: '.txt'
      </script>
        <script type="text/javascript" src="../_static/jquery.js"></script>
        <script type="text/javascript" src="../_static/underscore.js"></script>
        <script type="text/javascript" src="../_static/doctools.js"></script>
        <script type="text/javascript" src="../_static/asciinema-player.js"></script>
    <script type="text/javascript" src="../_static/js/theme.js"></script>
  <link rel="stylesheet" href="../_static/css/theme.css" type="text/css" />
  <link rel="stylesheet" href="../_static/pygments.css" type="text/css" />
  <link rel="stylesheet" href="../_static/asciinema-player.css" type="text/css" />
  <link rel="stylesheet" href="../_static/theme_overrides.css" type="text/css" />
    <link rel="index" title="Index" href="../genindex.html" />
    <link rel="search" title="Search" href="../search.html" />
    <link rel="next" title="Virtual Machine Setup" href="../labs/vm.html" />
    <link rel="prev" title="Symmetric Multi-Processing" href="smp.html" /> 
<body class="wy-body-for-nav">
  <div class="wy-grid-for-nav">
    <nav data-toggle="wy-nav-shift" class="wy-nav-side">
      <div class="wy-side-scroll">
        <div class="wy-side-nav-search" >
            <a href="../index.html" class="icon icon-home"> The Linux Kernel
          </a>
              <div class="version">
              </div>
<div role="search">
  <form id="rtd-search-form" class="wy-form" action="../search.html" method="get">
    <input type="text" name="q" placeholder="Search docs" />
    <input type="hidden" name="check_keywords" value="yes" />
    <input type="hidden" name="area" value="default" />
        </div>
        <div class="wy-menu wy-menu-vertical" data-spy="affix" role="navigation" aria-label="main navigation">
              <p class="caption"><span class="caption-text">Lectures</span></p>
<ul class="current">
<li class="toctree-l1"><a class="reference internal" href="so2.cs.pub.ro.html">Sisteme de operare 2</a></li>
<li class="toctree-l1"><a class="reference internal" href="intro.html">Introduction</a></li>
<li class="toctree-l1"><a class="reference internal" href="syscalls.html">System Calls</a></li>
<li class="toctree-l1"><a class="reference internal" href="interrupts.html">Interrupts</a></li>
<li class="toctree-l1"><a class="reference internal" href="smp.html">Symmetric Multi-Processing</a></li>
<li class="toctree-l1 current"><a class="current reference internal" href="#">Debugging</a><ul>
<li class="toctree-l2"><a class="reference internal" href="#lecture-objectives">Lecture objectives:</a></li>
<li class="toctree-l2"><a class="reference internal" href="#decoding-an-oops-panic">Decoding an oops/panic</a><ul>
<li class="toctree-l3"><a class="reference internal" href="#decoding-an-oops">Decoding an oops</a></li>
<li class="toctree-l3"><a class="reference internal" href="#addr2line">addr2line</a></li>
<li class="toctree-l3"><a class="reference internal" href="#objdump">objdump</a></li>
<li class="toctree-l3"><a class="reference internal" href="#gdb">gdb</a></li>
<li class="toctree-l3"><a class="reference internal" href="#kernel-panic">Kernel panic</a></li>
<li class="toctree-l2"><a class="reference internal" href="#list-debugging">List debugging</a></li>
<li class="toctree-l2"><a class="reference internal" href="#memory-debugging">Memory debugging</a><ul>
<li class="toctree-l3"><a class="reference internal" href="#debug-slab">DEBUG_SLAB</a></li>
<li class="toctree-l3"><a class="reference internal" href="#kasan">KASAN</a></li>
<li class="toctree-l3"><a class="reference internal" href="#comparison-between-memory-debugging-tools">Comparison between memory debugging tools</a></li>
<li class="toctree-l3"><a class="reference internal" href="#kmemcheck">kmemcheck</a></li>
<li class="toctree-l3"><a class="reference internal" href="#debug-pagealloc">DEBUG_PAGEALLOC</a></li>
<li class="toctree-l3"><a class="reference internal" href="#slub-debug">SLUB_DEBUG</a></li>
<li class="toctree-l2"><a class="reference internal" href="#kmemleak">Kmemleak</a></li>
<li class="toctree-l2"><a class="reference internal" href="#lockdep-checker">Lockdep checker</a></li>
<li class="toctree-l2"><a class="reference internal" href="#other-tools">Other tools</a></li>
<li class="toctree-l1"><a class="reference internal" href="../labs/vm.html">Virtual Machine Setup</a></li>
<p class="caption"><span class="caption-text">Labs</span></p>
<li class="toctree-l1"><a class="reference internal" href="../labs/infrastructure.html">Infrastructure</a></li>
<li class="toctree-l1"><a class="reference internal" href="../labs/introduction.html">Introduction</a></li>
<li class="toctree-l1"><a class="reference internal" href="../labs/kernel_modules.html">Kernel modules</a></li>
<li class="toctree-l1"><a class="reference internal" href="../labs/kernel_api.html">Kernel API</a></li>
<li class="toctree-l1"><a class="reference internal" href="../labs/device_drivers.html">Character device drivers</a></li>
<li class="toctree-l1"><a class="reference internal" href="../labs/interrupts.html">I/O access and Interrupts</a></li>
<li class="toctree-l1"><a class="reference internal" href="../labs/deferred_work.html">Deferred work</a></li>
<li class="toctree-l1"><a class="reference internal" href="../labs/block_device_drivers.html">Block Device Drivers</a></li>
<li class="toctree-l1"><a class="reference internal" href="../labs/filesystems_part1.html">File system drivers (Part 1)</a></li>
<li class="toctree-l1"><a class="reference internal" href="../labs/filesystems_part2.html">File system drivers (Part 2)</a></li>
<li class="toctree-l1"><a class="reference internal" href="../labs/networking.html">Networking</a></li>
<li class="toctree-l1"><a class="reference internal" href="../labs/memory_mapping.html">Memory mapping</a></li>
<li class="toctree-l1"><a class="reference internal" href="../labs/device_model.html">Linux Device Model</a></li>
        </div>
      </div>
    <section data-toggle="wy-nav-shift" class="wy-nav-content-wrap">
      <nav class="wy-nav-top" aria-label="top navigation">
          <i data-toggle="wy-nav-top" class="fa fa-bars"></i>
          <a href="../index.html">The Linux Kernel</a>
      </nav>
      <div class="wy-nav-content">
        <div class="rst-content">
<div role="navigation" aria-label="breadcrumbs navigation">
  <ul class="wy-breadcrumbs">
      <li><a href="../index.html">Docs</a> &raquo;</li>
      <li>Debugging</li>
      <li class="wy-breadcrumbs-aside">
            <a href="../_sources/lectures/debugging.rst.txt" rel="nofollow"> View page source</a>
      </li>
          <div role="main" class="document" itemscope="itemscope" itemtype="http://schema.org/Article">
           <div itemprop="articleBody">
  <div class="section" id="debugging">
<h1>Debugging<a class="headerlink" href="#debugging" title="Permalink to this headline">¶</a></h1>
<p><a class="reference external" href="debugging-slides.html">View slides</a></p>
<div class="section" id="lecture-objectives">
<h2>Lecture objectives:<a class="headerlink" href="#lecture-objectives" title="Permalink to this headline">¶</a></h2>
<p>One essential part of Linux kernel development is debugging. In user space we had
the support of the kernel so we could easily stop processes and use gdb to inspect
their behavior. In the kernel, in order to use gdb we need to use hypervisor like
QEMU or JTAG based hardware interfaces which are not always available. The Linux
kernel provides a set of tools and debug options useful for investigating abnormal
behavior.</p>
<p>In this lecture we will learn about:</p>
<ul class="admonition-debugging simple">
<li>decoding an oops/panic</li>
<li>list debugging</li>
<li>memory debugging</li>
<li>locking debugging</li>
<li>profiling</li>
<div class="section" id="decoding-an-oops-panic">
<h2>Decoding an oops/panic<a class="headerlink" href="#decoding-an-oops-panic" title="Permalink to this headline">¶</a></h2>
<p>An oops is an inconsistent state that the kernel detects inside itself.
Upon detecting an oops the Linux kernel kills the offending process,
prints information that can help debug the problem and continues execution
but with limited reliability.</p>
<p>Lets consider the following Linux kernel module:</p>
<div class="admonition-oops-module highlight-c"><div class="highlight"><pre><span></span><span class="k">static</span> <span class="n">noinline</span> <span class="kt">void</span> <span class="nf">do_oops</span><span class="p">(</span><span class="kt">void</span><span class="p">)</span>
<span class="p">{</span>
    <span class="o">*</span><span class="p">(</span><span class="kt">int</span><span class="o">*</span><span class="p">)</span><span class="mh">0x42</span> <span class="o">=</span> <span class="sc">&#39;a&#39;</span><span class="p">;</span>
<span class="p">}</span>
<span class="k">static</span> <span class="kt">int</span> <span class="nf">so2_oops_init</span><span class="p">(</span><span class="kt">void</span><span class="p">)</span>
<span class="p">{</span>
    <span class="n">pr_info</span><span class="p">(</span><span class="s">&quot;oops_init</span><span class="se">\n</span><span class="s">&quot;</span><span class="p">);</span>
    <span class="n">do_oops</span><span class="p">();</span>
    <span class="k">return</span> <span class="mi">0</span><span class="p">;</span>
<span class="p">}</span>
<span class="k">static</span> <span class="kt">void</span> <span class="nf">so2_oops_exit</span><span class="p">(</span><span class="kt">void</span><span class="p">)</span>
<span class="p">{</span>
    <span class="n">pr_info</span><span class="p">(</span><span class="s">&quot;oops exit</span><span class="se">\n</span><span class="s">&quot;</span><span class="p">);</span>
<span class="p">}</span>
<span class="n">module_init</span><span class="p">(</span><span class="n">so2_oops_init</span><span class="p">);</span>
<span class="n">module_exit</span><span class="p">(</span><span class="n">so2_oops_exit</span><span class="p">);</span>
</pre></div>
<p>Notice that ‘’do_oops’’ function tries to write at an invalid memory address. Because the kernel
cannot find a suitable physical page were to write, it kills the insmod task in the context of
which ‘’do_oops’’ runs. Then it prints the following oops message:</p>
<blockquote>
<div><div class="highlight-bash"><div class="highlight"><pre><span></span>root@qemux86:~/skels/debugging/oops# insmod oops.ko
BUG: unable to handle kernel NULL pointer dereference at <span class="m">00000042</span>
IP: do_oops+0x8/0x10 <span class="o">[</span>oops<span class="o">]</span>
*pde <span class="o">=</span> <span class="m">00000000</span>
Oops: <span class="m">0002</span> <span class="o">[</span><span class="c1">#1] SMP</span>
Modules linked in: oops<span class="o">(</span>O+<span class="o">)</span>
CPU: <span class="m">0</span> PID: <span class="m">234</span> Comm: insmod Tainted: G           O     <span class="m">4</span>.15.0+ <span class="c1">#3</span>
Hardware name: QEMU Standard PC <span class="o">(</span>i440FX + PIIX, <span class="m">1996</span><span class="o">)</span>, BIOS Ubuntu-1.8.2-1ubuntu1 <span class="m">04</span>/01/2014
EIP: do_oops+0x8/0x10 <span class="o">[</span>oops<span class="o">]</span>
EFLAGS: <span class="m">00000292</span> CPU: <span class="m">0</span>
EAX: <span class="m">00000061</span> EBX: <span class="m">00000000</span> ECX: c7ed3584 EDX: c7ece8dc
ESI: c716c908 EDI: c8816010 EBP: c7257df0 ESP: c7257df0
DS: 007b ES: 007b FS: 00d8 GS: <span class="m">0033</span> SS: <span class="m">0068</span>
CR0: <span class="m">80050033</span> CR2: <span class="m">00000042</span> CR3: 0785f000 CR4: <span class="m">00000690</span>
Call Trace:
so2_oops_init+0x17/0x20 <span class="o">[</span>oops<span class="o">]</span>
do_one_initcall+0x37/0x170
? cache_alloc_debugcheck_after.isra.19+0x15f/0x2f0
? __might_sleep+0x32/0x90
? trace_hardirqs_on_caller+0x11c/0x1a0
? do_init_module+0x17/0x1c2
? kmem_cache_alloc+0xa4/0x1e0
? do_init_module+0x17/0x1c2
do_init_module+0x46/0x1c2
load_module+0x1f45/0x2380
SyS_init_module+0xe5/0x100
do_int80_syscall_32+0x61/0x190
entry_INT80_32+0x2f/0x2f
EIP: 0x44902cc2
EFLAGS: <span class="m">00000206</span> CPU: <span class="m">0</span>
EAX: ffffffda EBX: 08afb050 ECX: 0000eef4 EDX: 08afb008
ESI: <span class="m">00000000</span> EDI: bf914dbc EBP: <span class="m">00000000</span> ESP: bf914c1c
DS: 007b ES: 007b FS: <span class="m">0000</span> GS: <span class="m">0033</span> SS: 007b
Code: &lt;a3&gt; <span class="m">42</span> <span class="m">00</span> <span class="m">00</span> <span class="m">00</span> 5d c3 <span class="m">90</span> <span class="m">55</span> <span class="m">89</span> e5 <span class="m">83</span> ec <span class="m">04</span> c7 <span class="m">04</span> <span class="m">24</span> <span class="m">24</span> <span class="m">70</span> <span class="m">81</span> c8 e8
EIP: do_oops+0x8/0x10 <span class="o">[</span>oops<span class="o">]</span> SS:ESP: <span class="m">0068</span>:c7257df0
CR2: <span class="m">0000000000000042</span>
---<span class="o">[</span> end trace 011848be72f8bb42 <span class="o">]</span>---
</pre></div>
</div></blockquote>
<p>An oops contains information about the IP which caused the fault, register status, process,
CPU on which the fault happend like below:</p>
<div class="admonition-oops-information highlight-bash"><div class="highlight"><pre><span></span>root@qemux86:~/skels/debugging/oops# insmod oops.ko
BUG: unable to handle kernel NULL pointer dereference at <span class="m">00000042</span>
IP: do_oops+0x8/0x10 <span class="o">[</span>oops<span class="o">]</span>
*pde <span class="o">=</span> <span class="m">00000000</span>
Oops: <span class="m">0002</span> <span class="o">[</span><span class="c1">#1] SMP</span>
Modules linked in: oops<span class="o">(</span>O+<span class="o">)</span>
CPU: <span class="m">0</span> PID: <span class="m">234</span> Comm: insmod Tainted: G           O     <span class="m">4</span>.15.0+ <span class="c1">#3</span>
Hardware name: QEMU Standard PC <span class="o">(</span>i440FX + PIIX, <span class="m">1996</span><span class="o">)</span>, BIOS Ubuntu-1.8.2-1ubuntu1 <span class="m">04</span>/01/2014
EIP: do_oops+0x8/0x10 <span class="o">[</span>oops<span class="o">]</span>
CR0: <span class="m">80050033</span> CR2: <span class="m">00000042</span> CR3: 0785f000 CR4: <span class="m">00000690</span>
EIP: 0x44902cc2
EFLAGS: <span class="m">00000206</span> CPU: <span class="m">0</span>
EAX: ffffffda EBX: 08afb050 ECX: 0000eef4 EDX: 08afb008
ESI: <span class="m">00000000</span> EDI: bf914dbc EBP: <span class="m">00000000</span> ESP: bf914c1c
DS: 007b ES: 007b FS: <span class="m">0000</span> GS: <span class="m">0033</span> SS: 007b
Code: &lt;a3&gt; <span class="m">42</span> <span class="m">00</span> <span class="m">00</span> <span class="m">00</span> 5d c3 <span class="m">90</span> <span class="m">55</span> <span class="m">89</span> e5 <span class="m">83</span> ec <span class="m">04</span> c7 <span class="m">04</span> <span class="m">24</span> <span class="m">24</span> <span class="m">70</span> <span class="m">81</span> c8 e8
</pre></div>
<p>Another important thing that an oops can provide is the stack trace of functions called before
the fault happend:</p>
<div class="admonition-oops-stacktrace highlight-bash"><div class="highlight"><pre><span></span>root@qemux86:~/skels/debugging/oops# insmod oops.ko
BUG: unable to handle kernel NULL pointer dereference at <span class="m">00000042</span>
Call Trace:
so2_oops_init+0x17/0x20 <span class="o">[</span>oops<span class="o">]</span>
do_one_initcall+0x37/0x170
? cache_alloc_debugcheck_after.isra.19+0x15f/0x2f0
? __might_sleep+0x32/0x90
? trace_hardirqs_on_caller+0x11c/0x1a0
? do_init_module+0x17/0x1c2
? kmem_cache_alloc+0xa4/0x1e0
? do_init_module+0x17/0x1c2
do_init_module+0x46/0x1c2
load_module+0x1f45/0x2380
SyS_init_module+0xe5/0x100
do_int80_syscall_32+0x61/0x190
entry_INT80_32+0x2f/0x2f
</pre></div>
<div class="section" id="decoding-an-oops">
<h3>Decoding an oops<a class="headerlink" href="#decoding-an-oops" title="Permalink to this headline">¶</a></h3>
<ul class="admonition-debugging simple">
<li>CONFIG_DEBUG_INFO</li>
<li>addr2line</li>
<li>gdb</li>
<li>objdump -dSr</li>
<div class="section" id="addr2line">
<h3>addr2line<a class="headerlink" href="#addr2line" title="Permalink to this headline">¶</a></h3>
<p><em>addr2line</em> translates addresses into file names and line numbers. Given
an address in an executable it uses the debugging information to figure out
which file name and line number are associated with it.</p>
<p>Modules are loaded at dynamic addresses but are compiled starting with 0 as
a base address. So, in order to find the line number for a given dynamic address
we need to know module’s load address.</p>
<div class="admonition-addr2line highlight-bash"><div class="highlight"><pre><span></span>$ addr2line -e oops.o  0x08
$ skels/debugging/oops/oops.c:5
$ <span class="c1"># 0x08 is the offset of the offending instruction inside the oops.ko module</span>
</pre></div>
<div class="section" id="objdump">
<h3>objdump<a class="headerlink" href="#objdump" title="Permalink to this headline">¶</a></h3>
<p>Similar we can determine the offending line using objdump:</p>
<div class="admonition-objdump highlight-bash"><div class="highlight"><pre><span></span>$ cat /proc/modules
oops <span class="m">20480</span> <span class="m">1</span> - Loading 0xc8816000 <span class="o">(</span>O+<span class="o">)</span>
$ objdump -dS --adjust-vma<span class="o">=</span>0xc8816000 oops.ko
c8816000:       b8 <span class="m">61</span> <span class="m">00</span> <span class="m">00</span> <span class="m">00</span>          mov    <span class="nv">$0</span>x61,%eax
static noinline void do_oops<span class="o">(</span>void<span class="o">)</span>
<span class="o">{</span>
c8816005:       <span class="m">55</span>                      push   %ebp
c8816006:       <span class="m">89</span> e5                   mov    %esp,%ebp
*<span class="o">(</span>int*<span class="o">)</span><span class="nv">0x42</span> <span class="o">=</span> <span class="s1">&#39;a&#39;</span><span class="p">;</span>
c8816008:       a3 <span class="m">42</span> <span class="m">00</span> <span class="m">00</span> <span class="m">00</span>          mov    %eax,0x42
</pre></div>
<div class="section" id="gdb">
<h3>gdb<a class="headerlink" href="#gdb" title="Permalink to this headline">¶</a></h3>
<div class="admonition-gdb highlight-bash"><div class="highlight"><pre><span></span>$ gdb ./vmlinux
<span class="o">(</span>gdb<span class="o">)</span> list *<span class="o">(</span>do_panic+0x8<span class="o">)</span>
0xc1244138 is in do_panic <span class="o">(</span>lib/test_panic.c:8<span class="o">)</span>.
<span class="m">3</span>
<span class="m">4</span> static struct timer_list panic_timer<span class="p">;</span>
<span class="m">5</span>
<span class="m">6</span> static void do_panic<span class="o">(</span>struct timer_list *unused<span class="o">)</span>
<span class="m">7</span> <span class="o">{</span>
<span class="m">8</span>         *<span class="o">(</span>int*<span class="o">)</span><span class="nv">0x42</span> <span class="o">=</span> <span class="s1">&#39;a&#39;</span><span class="p">;</span>
<span class="m">9</span> <span class="o">}</span>
<span class="m">10</span>
<span class="m">11</span>        static int so2_panic_init<span class="o">(</span>void<span class="o">)</span>
</pre></div>
<div class="section" id="kernel-panic">
<h3>Kernel panic<a class="headerlink" href="#kernel-panic" title="Permalink to this headline">¶</a></h3>
<p>A kernel panic is a special type of oops where the kernel cannot continue execution. For example
if the function do_oops from above was called in the interrupt context, the kernel wouldn’t know how to kill
and it will decide that it is better to crash the kernel and stop execution.</p>
<p>Here is a sample code that will generate a kernel panic:</p>
<div class="admonition-kernel-panic highlight-c"><div class="highlight"><pre><span></span><span class="k">static</span> <span class="k">struct</span> <span class="n">timer_list</span> <span class="n">panic_timer</span><span class="p">;</span>
<span class="k">static</span> <span class="kt">void</span> <span class="nf">do_panic</span><span class="p">(</span><span class="k">struct</span> <span class="n">timer_list</span> <span class="o">*</span><span class="n">unused</span><span class="p">)</span>
<span class="p">{</span>
    <span class="o">*</span><span class="p">(</span><span class="kt">int</span><span class="o">*</span><span class="p">)</span><span class="mh">0x42</span> <span class="o">=</span> <span class="sc">&#39;a&#39;</span><span class="p">;</span>
<span class="p">}</span>
<span class="k">static</span> <span class="kt">int</span> <span class="nf">so2_panic_init</span><span class="p">(</span><span class="kt">void</span><span class="p">)</span>
<span class="p">{</span>
    <span class="n">pr_info</span><span class="p">(</span><span class="s">&quot;panic_init</span><span class="se">\n</span><span class="s">&quot;</span><span class="p">);</span>
    <span class="n">timer_setup</span><span class="p">(</span><span class="o">&amp;</span><span class="n">panic_timer</span><span class="p">,</span>  <span class="n">do_panic</span><span class="p">,</span> <span class="mi">0</span><span class="p">);</span>
    <span class="n">mod_timer</span><span class="p">(</span><span class="o">&amp;</span><span class="n">panic_timer</span><span class="p">,</span> <span class="n">jiffies</span> <span class="o">+</span> <span class="mi">2</span> <span class="o">*</span> <span class="n">HZ</span><span class="p">);</span>
    <span class="k">return</span> <span class="mi">0</span><span class="p">;</span>
<span class="p">}</span>
</pre></div>
<p>Loading the module will generate the following kernel panic message:</p>
<div class="highlight-bash"><div class="highlight"><pre><span></span>root@qemux86:~/skels/debugging/panic# insmod panic.ko
panic: loading out-of-tree module taints kernel.
root@qemux86:~/skels/debugging/panic# BUG: unable to handle kernel NULL pointer dereference at <span class="m">00000042</span>
IP: do_panic+0x8/0x10 <span class="o">[</span>panic<span class="o">]</span>
*pde <span class="o">=</span> <span class="m">00000000</span>
Oops: <span class="m">0002</span> <span class="o">[</span><span class="c1">#1] SMP</span>
Modules linked in: panic<span class="o">(</span>O<span class="o">)</span>
CPU: <span class="m">0</span> PID: <span class="m">0</span> Comm: swapper/0 Tainted: G           O     <span class="m">4</span>.15.0+ <span class="c1">#19</span>
Hardware name: QEMU Standard PC <span class="o">(</span>i440FX + PIIX, <span class="m">1996</span><span class="o">)</span>, BIOS Ubuntu-1.8.2-1ubuntu1 <span class="m">04</span>/01/2014
EIP: do_panic+0x8/0x10 <span class="o">[</span>panic<span class="o">]</span>
EFLAGS: <span class="m">00010246</span> CPU: <span class="m">0</span>
EAX: <span class="m">00000061</span> EBX: <span class="m">00000101</span> ECX: 000002d8 EDX: <span class="m">00000000</span>
ESI: c8817000 EDI: c8819200 EBP: c780ff34 ESP: c780ff34
DS: 007b ES: 007b FS: 00d8 GS: <span class="m">0000</span> SS: <span class="m">0068</span>
CR0: <span class="m">80050033</span> CR2: <span class="m">00000042</span> CR3: 0716b000 CR4: <span class="m">00000690</span>
Call Trace:
&lt;SOFTIRQ&gt;
call_timer_fn+0x63/0xf0
? process_timeout+0x10/0x10
run_timer_softirq+0x14f/0x170
? 0xc8817000
? trace_hardirqs_on_caller+0x9b/0x1a0
__do_softirq+0xde/0x1f2
? __irqentry_text_end+0x6/0x6
do_softirq_own_stack+0x57/0x70
&lt;/SOFTIRQ&gt;
irq_exit+0x7d/0x90
smp_apic_timer_interrupt+0x4f/0x90
? trace_hardirqs_off_thunk+0xc/0x1d
apic_timer_interrupt+0x3a/0x40
EIP: default_idle+0xa/0x10
EFLAGS: <span class="m">00000246</span> CPU: <span class="m">0</span>
EAX: c15c97c0 EBX: <span class="m">00000000</span> ECX: <span class="m">00000000</span> EDX: <span class="m">00000001</span>
ESI: <span class="m">00000000</span> EDI: <span class="m">00000000</span> EBP: c15c3f48 ESP: c15c3f48
DS: 007b ES: 007b FS: 00d8 GS: <span class="m">0000</span> SS: <span class="m">0068</span>
arch_cpu_idle+0x9/0x10
default_idle_call+0x19/0x30
do_idle+0x105/0x180
cpu_startup_entry+0x25/0x30
rest_init+0x1e3/0x1f0
start_kernel+0x305/0x30a
i386_start_kernel+0x95/0x99
startup_32_smp+0x15f/0x164
Code: &lt;a3&gt; <span class="m">42</span> <span class="m">00</span> <span class="m">00</span> <span class="m">00</span> 5d c3 <span class="m">90</span> <span class="m">55</span> <span class="m">89</span> e5 <span class="m">83</span> ec <span class="m">08</span> c7 <span class="m">04</span> <span class="m">24</span> <span class="m">24</span> <span class="m">80</span> <span class="m">81</span> c8 e8
EIP: do_panic+0x8/0x10 <span class="o">[</span>panic<span class="o">]</span> SS:ESP: <span class="m">0068</span>:c780ff34
CR2: <span class="m">0000000000000042</span>
---<span class="o">[</span> end trace 77f49f83f2e42f91 <span class="o">]</span>---
Kernel panic - not syncing: Fatal exception in interrupt
Kernel Offset: disabled
---<span class="o">[</span> end Kernel panic - not syncing: Fatal exception in interrupt
</pre></div>
<div class="section" id="list-debugging">
<h2>List debugging<a class="headerlink" href="#list-debugging" title="Permalink to this headline">¶</a></h2>
<p>In order to catch access to uninitialized elements the kernel uses poison
magic values.</p>
<div class="admonition-list-debugging highlight-bash"><div class="highlight"><pre><span></span>static inline void list_del<span class="o">(</span>struct list_head *entry<span class="o">)</span>
<span class="o">{</span>
        __list_del<span class="o">(</span>entry-&gt;prev, entry-&gt;next<span class="o">)</span><span class="p">;</span>
        entry-&gt;next <span class="o">=</span> <span class="o">(</span>struct list_head*<span class="o">)</span>LIST_POISON1<span class="p">;</span>
        entry-&gt;prev <span class="o">=</span> <span class="o">(</span>struct list_head*<span class="o">)</span>LIST_POISON2<span class="p">;</span>
<span class="o">}</span>
BUG: unable to handle kernel NULL pointer dereference at <span class="m">00000100</span>
IP: crush+0x80/0xb0 <span class="o">[</span>list<span class="o">]</span>
</pre></div>
<div class="section" id="memory-debugging">
<h2>Memory debugging<a class="headerlink" href="#memory-debugging" title="Permalink to this headline">¶</a></h2>
<p>There are several tools for memory debugging:</p>
<ul class="admonition-memory-debugging simple">
<li>DEBUG_SLAB</li>
<li>KASAN</li>
<li>kmemcheck</li>
<li>DEBUG_PAGEALLOC</li>
<div class="section" id="debug-slab">
<h3>DEBUG_SLAB<a class="headerlink" href="#debug-slab" title="Permalink to this headline">¶</a></h3>
<ul class="admonition-debug-slab simple">
<li>CONFIG_DEBUG_SLAB</li>
<li>poisoned based memory debuggers</li>
<img alt="../_images/ditaa-3f19098db799e7cd1f7015c72cb556be4489ccda.png" src="../_images/ditaa-3f19098db799e7cd1f7015c72cb556be4489ccda.png" />
<div class="section" id="kasan">
<h3>KASAN<a class="headerlink" href="#kasan" title="Permalink to this headline">¶</a></h3>
<p>KASan is a dynamic memory error detector designed to find use-after-free
and out-of-bounds bugs.</p>
<p>The main idea of KASAN is to use shadow memory to record whether each byte
of memory is safe to access or not, and use compiler’s instrumentation to
check the shadow memory on each memory access.</p>
<p>Address sanitizer uses 1 byte of shadow memory to track 8 bytes of kernel
address space. It uses 0-7 to encode the number of consecutive bytes at
the beginning of the eigh-byte region that are valid.</p>
<p>See Documentation/dev-tools/kasan.rst for more information and have a look
at lib/test_kasan.c for an example of problems that KASan can detect.</p>
<ul class="admonition-kasan simple">
<li>dynamic memory error detector</li>
<li>finds user-after-free or out-of-bound bugs</li>
<li>uses shadow memory to track memory operations</li>
<li>lib/test_kasan.c</li>
<div class="section" id="comparison-between-memory-debugging-tools">
<h3>Comparison between memory debugging tools<a class="headerlink" href="#comparison-between-memory-debugging-tools" title="Permalink to this headline">¶</a></h3>
<div class="section" id="kmemcheck">
<h3>kmemcheck<a class="headerlink" href="#kmemcheck" title="Permalink to this headline">¶</a></h3>
<p>KASan can do almost everything that kmemcheck can. KASan uses compile-time
instrumentation, which makes it significantly faster than kmemcheck.
The only advantage of kmemcheck over KASan is detection of unitialized
memory reads.</p>
<div class="section" id="debug-pagealloc">
<h3>DEBUG_PAGEALLOC<a class="headerlink" href="#debug-pagealloc" title="Permalink to this headline">¶</a></h3>
<p>KASan is slower than DEBUG_PAGEALLOC, but KASan works on sub-page
granularity level, so it able to find more bugs.</p>
<div class="section" id="slub-debug">
<h3>SLUB_DEBUG<a class="headerlink" href="#slub-debug" title="Permalink to this headline">¶</a></h3>
<blockquote>
<div><ul class="simple">
<li>SLUB_DEBUG has lower overhead than KASan.</li>
<li>SLUB_DEBUG in most cases are not able to detect bad reads,
KASan able to detect both reads and writes.</li>
<li>In some cases (e.g. redzone overwritten) SLUB_DEBUG detect
bugs only on allocation/freeing of object. KASan catch
bugs right before it will happen, so we always know exact
place of first bad read/write.</li>
</div></blockquote>
<div class="section" id="kmemleak">
<h2>Kmemleak<a class="headerlink" href="#kmemleak" title="Permalink to this headline">¶</a></h2>
<p>Kmemleak provides a way of detecting kernel memory leaks in a way similar
to a tracing garbage collector. You can find more information here in
Documentation/dev-tools/kmemleak.rst.</p>
<ul class="admonition-kmemleak simple">
<li>CONFIG_DEBUG_KMEMLEAK</li>
<li>mount -t debugfs nodev /sys/kernel/debug</li>
<li>echo scan &gt; /sys/kernel/debug/kmemleak</li>
<li>echo clear &gt; /sys/kernel/debug/kmemleak</li>
<li>cat /sys/kernel/debug/kmemleak</li>
<p>Lets have a look at tools/labs/debugging/leak/leak.c file.</p>
<div class="section" id="lockdep-checker">
<h2>Lockdep checker<a class="headerlink" href="#lockdep-checker" title="Permalink to this headline">¶</a></h2>
<ul class="admonition-lockdep-checker simple">
<li>CONFIG_DEBUG_LOCKDEP</li>
<li>lock inversion</li>
<li>cyclic dependency</li>
<li>incorrect usage of locks</li>
<span class="admonition-perf"></span></div>
<div class="section" id="other-tools">
<h2>Other tools<a class="headerlink" href="#other-tools" title="Permalink to this headline">¶</a></h2>
<ul class="admonition-other-tools simple">
<li>ftrace</li>
<li>kprobes</li>
<li>sparse</li>
<li>coccinelle</li>
<li>checkpatch.pl</li>
<li>printk</li>
<li>dump_stack()</li>
           </div>
          </div>
          <footer>
    <div class="rst-footer-buttons" role="navigation" aria-label="footer navigation">
        <a href="../labs/vm.html" class="btn btn-neutral float-right" title="Virtual Machine Setup" accesskey="n" rel="next">Next <span class="fa fa-arrow-circle-right"></span></a>
        <a href="smp.html" class="btn btn-neutral float-left" title="Symmetric Multi-Processing" accesskey="p" rel="prev"><span class="fa fa-arrow-circle-left"></span> Previous</a>
  <div role="contentinfo">
        &copy; Copyright The kernel development community
  Built with <a href="http://sphinx-doc.org/">Sphinx</a> using a <a href="https://github.com/rtfd/sphinx_rtd_theme">theme</a> provided by <a href="https://readthedocs.org">Read the Docs</a>. 
        </div>
      </div>
    </section>
  <script type="text/javascript">
      jQuery(function () {
          SphinxRtdTheme.Navigation.enable(true);
  </script>
Provide feedback

Saved searches

Use saved searches to filter your results more quickly

FilesExpand file tree

debugging.html

Latest commit

History

debugging.html

File metadata and controls
