debugging_a_minidump.rst (11814B)
1 Debugging A Minidump 2 ==================== 3 4 +--------------------------------------------------------------------+ 5 | This page is an import from MDN and the contents might be outdated | 6 +--------------------------------------------------------------------+ 7 8 The 9 `minidump <http://msdn.microsoft.com/en-us/library/windows/desktop/ms680369%28v=vs.85%29.aspx>`__ 10 file format contains data about a crash on Windows. It is used by 11 `rust-minidump <https://github.com/luser/rust-minidump>`__, 12 `Breakpad <https://wiki.mozilla.org/Breakpad>`__, and also by various 13 Windows debugging tools. Each minidump includes the following data. 14 15 - Details about the exception which led to the crash. 16 - Information about each thread in the process: the address which was 17 executing and the register state at the time the process stopped. 18 - A list of shared libraries loaded into the process at the time of the 19 crash. 20 - The stack memory of each thread. 21 - The memory right around the crashing address. 22 - (Optional) Other memory regions, if requested by the application. 23 - (Optional) Other platform-specific data. 24 25 Accessing minidumps from crash reports 26 -------------------------------------- 27 28 Minidumps are not available to everyone. For details on how to gain 29 access and where to find minidump files for crash reports, consult the 30 :ref:`crash report documentation <Understanding Crash Reports>` 31 32 Using rust-minidump's tooling 33 ----------------------------------- 34 35 Most of our crash-reporting infrastructure is based on rust-minidump. 36 The primary tool for this is the 37 `minidump-stackwalk <https://github.com/luser/rust-minidump/tree/master/minidump-stackwalk>`__ 38 CLI application, which includes extensive user documentation. 39 40 That documentation includes a 41 `dedicated section <https://github.com/luser/rust-minidump/tree/master/minidump-stackwalk#analyzing-firefox-minidumps>`__ 42 on locally analyzing Firefox crashreports and minidumps. 43 44 If you're looking for minidump_dump, it's included as part of 45 minidump-stackwalk. 46 47 Using the MS Visual Studio debugger 48 ----------------------------------- 49 50 #. Set up the debugger to :ref:`use the Mozilla symbol 51 server <Using The Mozilla Symbol Server>` and 52 :ref:`source server <Using The Mozilla Source Server>`. 53 #. Double-click on the minidump file to open it in the debugger. 54 #. When it loads, click the green icon in the visual studio debugger 55 toolbar that looks like a play button. 56 57 For Firefox releases older than Firefox 41, you will also need to 58 install the relevant release of Firefox (for example from 59 `here <https://ftp.mozilla.org/pub/mozilla.org/firefox/releases/>`__), 60 and add the directory it is in (e.g., "C:\Program Files\Mozilla 61 Firefox 3.6 Beta 1\") to the same dialog in which you set up the 62 symbol server (in case the binary location in the minidump is not the 63 same as the one on your machine). Note that you can install the 64 relevant release anywhere. Just make sure to configure the symbol 65 server to the directory where you installed it. For releases from 41 66 onward, the binaries are available on the symbol server. 67 68 If this doesn't work, downloading the exact build and crashreporter 69 symbols full files. These can be found in treeherder / build folder. 70 Load Visual Studio, and go to file -> open -> minidump location. Click 71 on "Run Native", and Visual Studio will ask for the corresponding symbol 72 files. For each .dll you wish to have symbols for, you must go to a 73 console and go to the corresponding directory. E.g. (xul.dll should go 74 to xul.pdf in the crashreporter symbols directory). Each directory will 75 have a .pd\_ file. In a command shell run: "expand /r foo.pd\_". Then 76 point Visual Studio to this directory. 77 78 Then you'll be able to examine: 79 80 +------------------+-------------------------------------------------------------------------+ 81 | stack trace | The debugger shows the stack trace. You can right-click on any frame | 82 | | in the stack, and then choose "go to disassembly" or "go to source". | 83 | | (Choosing "go to disassembly" from the source view might not get you | 84 | | to the right place due to optimizations.) When looking at the | 85 | | source, beware that the debugging information will associate all | 86 | | inlined functions as part of the line into which they were inlined, | 87 | | and compiler (with PGO) inlines *very* aggressively (including | 88 | | inlining virtual functions). You can often figure out where you | 89 | | *really* are by reading the disassembly. | 90 +------------------+-------------------------------------------------------------------------+ 91 | registers | In the Registers tab (Debug->Windows->Registers if you don't have | 92 | | it open), you can look at the registers associated with each stack | 93 | | frame, but only at the current state (i.e., the time of the crash). | 94 | | Registers that Visual Studio can't figure out will be grayed-out and | 95 | | have the value 00000000. | 96 +------------------+-------------------------------------------------------------------------+ 97 | stack memory | You open a window (Memory 1, etc.) that shows contiguous segments of | 98 | | memory using the Debug->Windows->Memory menu item. You can then | 99 | | enter the address of the stack pointer (ESP register) in this window | 100 | | and look at the memory on the stack. (The minidump doesn't have the | 101 | | memory on the heap.) It's a good idea to change the "width" dropdown | 102 | | in the top right corner of the window from its default "Auto" to | 103 | | either "8" or "16" so that the memory display is word-aligned. If | 104 | | you're interested in pointers, which is usually the case, you can | 105 | | right click in this window and change the display to show 4-byte | 106 | | words (so that you don't have to reverse the order due to | 107 | | little-endianness). This view, combined with the disassembly, can | 108 | | often be used to reconstruct information beyond what in shown the | 109 | | function parameters. | 110 +------------------+-------------------------------------------------------------------------+ 111 | local variables | In the Watch 1 (etc.) window (which, if you don't have open, you can | 112 | | iget from Debug->Windows->Watch), you can type an expression | 113 | | (e.g., the name of a local variable) and the debugger will show you | 114 | | its value (although it sometimes gets confused). If you're looking | 115 | | at a pointer to a variable that happens to be on the stack, you can | 116 | | even examine member variables by just typing expressions. If Visual | 117 | | Studio can't figure something out from the minidump, it might show | 118 | | you 00000000 (is this true?). | 119 +------------------+-------------------------------------------------------------------------+ 120 121 Using minidump-2-core on Linux 122 ------------------------------ 123 124 The `Breakpad 125 source <https://chromium.googlesource.com/breakpad/breakpad/+/master/>`__ 126 contains a tool called 127 `minidump-2-core <https://chromium.googlesource.com/breakpad/breakpad/+/master/src/tools/linux/md2core/>`__, 128 which converts Linux minidumps into core files. If you checkout and 129 build Breakpad, the binary will be at 130 ``src/tools/linux/md2core/minidump-2-core``. Running the binary with the 131 path to a Linux minidump will generate a core file on stdout which can 132 then be loaded in gdb as usual. You will need to manually download the 133 matching Firefox binaries, but then you can use the :ref:`GDB Python 134 script <Downloading symbols on Linux / Mac OS X>` to download symbols. 135 136 The ``minidump-2-core`` source does not currently handle processing 137 minidumps from a different CPU architecture than the system it was 138 built for. If you want to use it on an ARM dump, for example, you may 139 need to build the tool for ARM and run it under QEMU. 140 141 Using other tools to inspect minidump data 142 ------------------------------------------ 143 144 Ted has a few tools that can be built against an already-built copy of 145 Breakpad to do more targeted inspection. All of these tools assume you 146 have checked out their source in a directory next to the breakpad 147 checkout, and that you have built Breakpad in an objdir named 148 ``obj-breakpad`` at the same level. 149 150 - `stackwalk-http <https://hg.mozilla.org/users/tmielczarek_mozilla.com/stackwalk-http/>`__ 151 is a version of the breakpad's minidump_stackwalk that can fetch symbols 152 over HTTP, and also has the Mozilla symbol server URL baked in. If you 153 run it like ``stackwalk /path/to/dmp /tmp/syms`` it will print the stack 154 trace and save the symbols it downloaded in ``/tmp/syms``. Note that 155 symbols are only uploaded to the symbol server for nightly and 156 release builds, not per-change builds. 157 - `dump-lookup <https://hg.mozilla.org/users/tmielczarek_mozilla.com/dump-lookup/>`__ 158 takes a minidump and prints values on the stack that are potential 159 return addresses. This is useful when a stack trace looks truncated 160 or otherwise wrong. It needs symbol files to produce useful output, 161 so you will generally want to have run ``stackwalk-http`` to download 162 them first. 163 - `get-minidump-instructions <https://hg.mozilla.org/users/tmielczarek_mozilla.com/get-minidump-instructions/>`__ 164 retrieves and displays the memory range surrounding the faulting 165 instruction pointer from a minidump. You will almost always want to 166 run it with the ``--disassemble`` option, which will make it send the 167 bytes through ``objdump`` to display the disassembled instructions. 168 If you also give it a path to symbols (see ``stackwalk-http`` above) 169 it can download the matching source files from hg.mozilla.org and 170 display source interleaved with the disassembly. 171 - `minidump-modules <http://hg.mozilla.org/users/tmielczarek_mozilla.com/minidump-modules>`__ 172 takes a minidump and prints the list of modules from the crashed 173 process. It will print the full path to each module, whereas the 174 Socorro UI only prints the filename for each module for privacy 175 reasons. It also accepts a -v option to print the debug ID for each 176 module, and a -d option to print relative paths to the symbol files 177 that would be used instead of the module filenames. 178 179 Getting a stack trace from a crashed B2G process 180 ------------------------------------------------ 181 182 #. Get the minidump file in the phone at 183 /data/b2g/mozilla/\*.default/minidump/. You can use `adb 184 pull <http://developer.android.com/tools/help/adb.html>`__ for that. 185 #. Build the debug symbols using the command ./build.sh buildsymbols 186 inside the B2G tree. The symbol files will be generated in 187 $OBJDIR/dist/crashreporter-symbols. 188 #. Build and install 189 `google-breakpad <https://code.google.com/p/google-breakpad/>`__. 190 #. Use the 191 `minidump-stackwalk <https://github.com/luser/rust-minidump/tree/master/minidump-stackwalk>`__ 192 tool to get the stack trace. 193 194 .. code:: bash 195 196 Example: 197 198 $ cd B2G 199 $ adb pull /data/b2g/mozilla/*.default/minidump/*.dmp . 200 $ls *.dmp 201 71788789-197e-d769-67167423-4e7aef32.dmp 202 $ minidump-stackwalk 71788789-197e-d769-67167423-4e7aef32.dmp objdir-debug/dist/crashreporter-symbols/