understanding_crash_reports.rst (15543B)
1 Understanding Crash Reports 2 =========================== 3 4 +--------------------------------------------------------------------+ 5 | This page is an import from MDN and the contents might be outdated | 6 +--------------------------------------------------------------------+ 7 8 If a user experiences a crash they will be prompted to submit a raw 9 crash report, which is generated by Breakpad. The raw crash report is 10 received by `Socorro <https://github.com/mozilla/socorro>`__ which 11 `creates <https://github.com/mozilla/socorro/blob/master/socorro/processor/mozilla_processor_2015.py>`__ 12 a processed crash report. The processed crash report is based on the raw 13 crash report but also has a signature, classifications, and a number of 14 improved fields (e.g. OS, product, version). Many of the fields in both 15 the raw crash report and the processed crash report are viewable and 16 searchable on `crash-stats <https://crash-stats.mozilla.org/>`__. 17 Although there are two distinct crash reports, the raw and the 18 processed, people typically talk about a single "crash report" because 19 crash-stats mostly presents them in a combined way. 20 21 Each crash report contains a wealth of data about the crash 22 circumstances. Despite this, many crash reports lack sufficient data for 23 a developer to understand why the crash occurred. As well as providing a 24 general overview, this page aims to highlight parts of a crash report 25 that may provide non-obvious insights. 26 27 Note that most crash report fields are visible, but a few 28 privacy-sensitive parts of it are only available to users who are logged 29 in and have "minidump access". A relatively small number of users have 30 minidump access, and they are required to follow certain rules. For 31 access, see the `Protected Data Access docs on Crash Stats 32 <https://crash-stats.mozilla.org/documentation/protected_data_access/>`__. 33 34 Each crash report has the following tabs: Details, Metadata, Modules, 35 Raw Dump, Extensions, and (optional) Correlations. 36 37 Details tab 38 ----------- 39 40 The Details tab is the first place to look because it contains the most 41 important pieces of information. 42 43 Primary fields 44 ~~~~~~~~~~~~~~ 45 46 | The first part of the Details tab shows a table containing the most 47 important crash report fields. It includes such things as when the 48 crash occurred, in which product and version, the crash kind, and 49 various details about the OS and configuration of the machine on which 50 the crash occurred. The following screenshot shows some of these 51 fields. 52 | |Example fields in the "Details" tab of a crash report| 53 54 All fields have a tool-tip. For many fields, the tool-tip describes its 55 meaning. For all fields, the tool-tip indicates the key to use when you 56 want to do searches involving this field. (The field name is usually but 57 not always similar to the search key. E.g. the field "Adapter Device ID" 58 has the search key "adapter_device_id".) These descriptions are shown in 59 the `SuperSearchFields 60 API <https://crash-stats.mozilla.org/api/SuperSearchFields/>`__ and can be 61 `modified in super_search_fields.py <https://github.com/mozilla-services/socorro/blob/main/socorro/external/es/super_search_fields.py>`__ 62 or by writing up a `bug in Socorro <https://bugzilla.mozilla.org/enter_bug.cgi?format=__standard__&product=Socorro>`__. 63 64 The fields present in this tab vary depending on the crash kind. Not all 65 fields are always present. 66 67 The "Signature" field is the main identifier or label for a crash report. 68 Rather than considering each crash report in isolation, we want to put 69 crash reports into clusters so we can deal with groups of them at once. 70 An ideal clustering algorithm would put all crash reports with the same 71 root cause into a single cluster, and all crash reports with different 72 root causes into different clusters. The crash signature is our 73 imperfect but still useful attempt at such an algorithm. Most crash 74 signatures are based on the crashing stack trace, but some 75 special-purpose annotations are used to indicate particular kinds of 76 crashes. 77 78 - ``Abort``: A controlled abort, e.g. via ``NS_RUNTIMEABORT``. 79 (Controlled aborts that occur via ``MOZ_CRASH`` or 80 ``MOZ_RELEASE_ASSERT`` currently don't get an ``Abort`` annotation, 81 but they do get a "MOZ_CRASH Reason" field.) 82 - ``OOM | <size>``, where ``<size>`` is one of ``large``, ``small``, 83 ``unknown``: an out-of-memory (OOM) abort. The ``<size>`` annotation 84 is determined by the "OOM Allocation Size" field; if that field is 85 missing ``<size>`` will be ``unknown``. 86 - ``hang``: a hang prior to shutdown. 87 - ``shutdownhang``: a hang during shutdown. 88 - ``IPCError-browser``: a problem involving IPC. If the parent Firefox 89 process detects that the child process has sent broken or 90 unprocessable IPDL data, or is not shutting down in a timely manner, 91 it kills the child process with a crash report. These crashes will 92 now have a signature that indicates why the process was killed, 93 rather than the child stack at the moment. 94 95 When no special-purpose annotation is present and the signature begins 96 with a stack frame, it's usually a vanilla uncontrolled crash. The crash 97 cause can be determined from the "Crash Reason" field. Most commonly 98 it's a bad memory access. In that case, on Windows you can tell from the 99 reason field if the crash occurred while reading, writing or executing 100 memory (e.g. ``EXCEPTION_VIOLATION_ACCESS_READ`` indicates a bad memory 101 read). On Mac and Linux the reason will be SIGSEGV or SIGBUS and you 102 cannot tell from this field what kind of memory access it was. 103 104 See `this 105 file <https://github.com/mozilla-services/socorro/blob/master/socorro/signature/README.rst>`__ 106 for a detailed explanation of the crash report signature generation 107 procedure, and for information on how modify this procedure. 108 109 There are no fields that uniquely identify the user that a crash report 110 came from, but if you want to know if multiple crashes come from a 111 single user the "Install Time" field is a good choice. Use it in 112 conjunction with other fields that don't change, such as those 113 describing the OS or graphics card, for additional confidence. 114 115 For bad memory accesses, the "Crash Address" field can give additional 116 indications what went wrong. 117 118 - 0x0 is probably a null pointer deference[*]. 119 - Small addresses like 0x8 can indicate an object access (e.g. 120 ``this->mFoo``) via a null ``this`` pointer. 121 - Addresses like 0xfffffffffd8 might be stack accesses, depending on 122 the platform[*]. 123 - Addresses like 0x80cdefd3 might be heap accesses, depending on the 124 platform. 125 - Addresses may be poisoned: 0xe4 indicates the address comes from 126 memory that has been allocated by jemalloc but not yet initialized; 127 0xe5 indicates the address comes from memory freed by jemalloc. The 128 JS engine also has multiple poison values defined in 129 ``js/src/jsutil.h``. 130 131 [*] Note that due to the way addressing works on x86-64, if the crash 132 address is 0x0 for a Linux/macOS crash report, or 0xffffffffffffffff for 133 a Windows crash report, it's highly likely that the value is incorrect. 134 (There is a `bug 135 report <https://bugzilla.mozilla.org/show_bug.cgi?id=1493342>`__ open 136 for this problem.) You can sanity-check these crashes by looking at the 137 raw dump or minidump in the Raw Dump tab (see below). 138 139 Note that for non-release builds the "Version" field represents multiple 140 different builds since nightly and beta version numbers are reused for 141 builds created over a series of days until the version number is bumped. 142 (The "Build ID" field can disambiguate.) It's not currently possible to 143 `restrict searches to a given version or 144 later <https://bugzilla.mozilla.org/show_bug.cgi?id=1401517>`__ (using 145 >= with a build ID and a given release channel may work around this). 146 147 Some fields, such as "URL" and "Email Address", are privacy-sensitive 148 and are only visible to users with minidump access. 149 150 The Windows-only "Total Virtual Memory" field indicates if the Firefox 151 build and OS are 32-bit or 64-bit. 152 153 - A value of 2 GiB indicates 32-bit Firefox on 32-bit Windows. 154 - A value of 3 or 4 GiB indicates 32-bit Firefox on 64-bit Windows 155 (a.k.a. "WoW64"). Such a user could switch to 64-bit Firefox. 156 - A value much larger than 4 GiB (e.g. 128 TiB) indicates 64-bit 157 Firefox. (The "Build Architecture" field should be "amd64" in this 158 case.) 159 160 Some crash reports might contain a memory report. This memory report will 161 have been made some time before the crash, at a time when available 162 memory was low. In this case, a number of key measurements from the 163 memory report are shown in the Details tab, each one having a field name 164 starting with "MR:", short for "memory report". The full memory report 165 can be obtained in the Raw Dump tab (see below). 166 167 Bug-related information 168 ~~~~~~~~~~~~~~~~~~~~~~~ 169 170 The second part of the Details tab shows bug-related information, as the 171 following screenshot shows. 172 173 |Information relating to bug reports in the "Details" tab of a crash 174 report| 175 176 The "Report this bug in" links can be used to easily file bug reports. 177 Each one links to a Bugzilla bug report creation page that has various 178 fields pre-filled, such as the crash signature. 179 180 The "Related Bugs" section shows related bug reports, as determined by 181 the crash signature. 182 183 Stack traces 184 ~~~~~~~~~~~~ 185 186 The third part of the Details tab shows the stack trace and thread 187 number of the crashing thread, as the following screenshot shows. 188 189 |Information relating to threads in the "Details" tab of a crash report| 190 191 Each stack frame has a link to the source code, when possible. If a 192 crash is new, the regressing changeset can often be identified by 193 looking for recent changes in the blame annotations for one or more of 194 the top stack frames. Blame annotations are also good for identifying 195 who might know about the code in question. 196 197 Sometimes the highlighted source code is puzzling, e.g. the identified 198 line may not touch memory even though the crash is memory-related. This 199 can be caused by compiler optimizations. It's often better to look at 200 the disassembly (e.g. in a minidump) to understand exactly what code is 201 being executed. 202 203 Stack frame entries take on a variety of forms. 204 205 - The simplest are functions names, such as ``NS_InitXPCOM2``. 206 - Name/address pairs such as ``nss3.dll@0x1eb720`` are within system 207 libraries. 208 - Names such as ``F1398665248_____________________________`` ('F' 209 followed by many numbers then many underscores) are in Flash. 210 - Addresses such as ``@0xe1a850ac`` may indicate an address that wasn't 211 part of any legitimate code. If an address such as this occurs in the 212 first stack frame, the crash may be 213 `exploitable <https://developer.mozilla.org/en-US/docs/Mozilla/Security/Exploitable_crashes>`__. 214 215 Stack traces for other threads can be viewed by clicking on the small 216 "Show other threads" link. 217 218 If the crash report is for a hang, the crashing thread will be the 219 "watchdog" thread, which exists purely to detect hangs; its top stack 220 frame will be something 221 like\ :literal:`mozilla::`anonymous namespace'::RunWatchdog`. In that 222 case you should look at the other threads' stack traces to determine the 223 problem; many of them will be waiting on some kind of response, as shown 224 by a top stack frame containing a function like 225 ``NtWaitForSingleObject`` or ``ZwWaitForMultipleObjects``. 226 227 Metadata tab 228 ------------ 229 230 The Metadata tab is similar to the first part of the Details tab, 231 containing a table with various fields. These are the fields from the 232 raw crash report, ordered alphabetically by field name, but with 233 privacy-sensitive fields shown only to users with minidump access. There 234 is some overlap with the fields shown in the Details tab. 235 236 Modules tab 237 ----------- 238 239 The modules tab shows all the system libraries loaded at the time of the 240 crash, as the following screenshot shows. 241 242 |Table of modules in the "Modules" tab of a crash report| 243 244 On Windows these are mostly DLLs, on Mac they are mostly ``.dylib`` 245 files, and on Linux they are mostly ``.so`` files. 246 247 This information is most useful for Windows crashes, because DLLs loaded 248 by antivirus software or malware often cause Firefox to crash. 249 Correlations between loaded modules and crash signatures can be seen in 250 the "Correlations" tab (see below). 251 252 `This page <https://support.mozilla.org/en-US/kb/helping-crashes>`__ 253 says that files lacking version/debug identifier/debug filename are 254 likely to be malware. 255 256 Raw Dump tab 257 ------------ 258 259 The first part of the Raw Dump tab shows the raw crash report, in JSON 260 format. Once again, privacy-sensitive fields are shown only to users 261 with minidump access. 262 263 |JSON data in the "Raw Dump" tab of a crash report| 264 265 For users with minidump access, the second part of the Raw Dump tab has 266 some links, as the following screenshot shows. 267 268 |Links to downloadable files in the "Raw Dump" tab of a crash report| 269 270 These links are to the following items. 271 272 #. A minidump. Minidumps can be extremely useful in understanding a 273 crash report; see :ref:`this page <Debugging A Minidump>` for an 274 explanation how to use them. 275 #. The aforementioned JSON raw crash report. 276 #. The memory report contained within the crash report. 277 #. The unredacted crash report, which has additional information. 278 279 Extensions tab 280 -------------- 281 282 The Extensions tab shows which extensions are installed and enabled. 283 284 |Table of extensions in the "Extensions" tab of a crash report| 285 286 Usually it just shows an ID rather than the proper extension name. 287 288 Note that several extensions ship by default with Firefox and so will be 289 present in almost all crash reports. (The exact set of default 290 extensions depends on the release channel.) The least obvious of these 291 has an Id of ``{972ce4c6-7e08-4474-a285-3208198ce6fd}``, which is the 292 default Firefox theme. Some (but not all) of the other extensions 293 shipped by default have the following Ids: ``webcompat@mozilla.org``, 294 ``e10srollout@mozilla.org``, ``firefox@getpocket.com``, 295 ``flyweb@mozilla.org``, ``loop@mozilla.org``. 296 297 If an extension only has a hexadecimal identifier, a Google search of 298 that identifier is usually enough to identify the extension's name. 299 300 This information is useful because some crashes are caused by 301 extensions. Correlations between extensions and crash signatures can be 302 seen in the "Correlations" tab (see below). 303 304 Correlations tab 305 ---------------- 306 307 This tab is only shown when crash-stats identifies correlations between 308 a crash and modules or extensions that are present, which happens 309 occasionally. 310 311 See also 312 -------- 313 314 - `A talk about understanding crash 315 reports <https://air.mozilla.org/a-talk-about-understanding-crash-reports/>`__, 316 by David Baron, from March 2016. 317 - :ref:`A guide to searching crash reports` 318 319 .. |Example fields in the "Details" tab of a crash report| image:: https://mdn.mozillademos.org/files/13579/Details1.png 320 .. |Information relating to bug reports in the "Details" tab of a crash report| image:: https://mdn.mozillademos.org/files/13581/Details2.png 321 .. |Information relating to threads in the "Details" tab of a crash report| image:: https://mdn.mozillademos.org/files/13583/Details3.png 322 .. |Table of modules in the "Modules" tab of a crash report| image:: https://mdn.mozillademos.org/files/13593/Modules1.png 323 .. |JSON data in the "Raw Dump" tab of a crash report| image:: https://mdn.mozillademos.org/files/13595/RawDump1.png 324 .. |Links to downloadable files in the "Raw Dump" tab of a crash report| image:: https://mdn.mozillademos.org/files/14047/raw-dump-links.png 325 .. |Table of extensions in the "Extensions" tab of a crash report| image:: https://mdn.mozillademos.org/files/13599/Extensions1.png