local_symbols.rst (3073B)
1 Symbolicating TreeHerder stacks locally 2 ======================================= 3 4 When using tools like the :ref:`Dark Matter Detector (DMD)` or 5 :ref:`refcount logging<Refcount Tracing and Balancing>` to 6 investigate issues occurring on TreeHerder that you can't reproduce locally, you 7 can often end up with unsymbolicated stacks. Fortunately, there is a way to 8 symbolicate these stacks on your own machine. 9 10 These instructions are for a Linux TreeHerder build for MacOS, so they might 11 require some modifications for other combinations of platforms. 12 13 Download ``target.tar.bz2`` and ``target.crashreporter-symbols.zip`` from the 14 Build job. **Note that these files are very large so you'll want to delete 15 them and the extracted files when you are done.** 16 17 These files each contain a large number of files, so I'd recommend creating 18 a directory for each of them. Call these ``<TARGET_DIR>`` and ``<SYMB_DIR>``, 19 and move the prior two files into these two directories, respectively. 20 21 Go to ``<TARGET_DIR>`` and run something like 22 23 .. code-block:: shell 24 25 tar xf target.tar.bz2 26 27 then go to ``<SYMB_DIR>`` and run something like 28 29 .. code-block:: shell 30 31 unzip target.crashreporter-symbols.zip 32 33 You should be able to delete the two original files now. 34 35 Next we need to ensure that the locations of binaries are rewritten from 36 where they are on TreeHerder to where we have them locally. We'll do this by 37 editing ``fix_stacks.py``. This file is located in the ``tools/rb/`` directory of 38 the Firefox source directory. You need to add these two lines to the function 39 ``fixSymbols``, after ``line_str`` is defined and before it is written to 40 ``fix_stacks.stdin``. I've done this right before the definition of 41 ``is_missing_newline``. 42 43 .. code-block:: python 44 45 line_str = line_str.replace("/builds/worker/workspace/build/application/firefox/firefox", 46 "<TARGET_DIR>/firefox/firefox-bin") 47 line_str = line_str.replace("/builds/worker/workspace/build/application/firefox/libxul.so", 48 "<TARGET_DIR>/firefox/libxul.so") 49 50 The initial locations should appear verbatim in the stack you are trying to 51 symbolicate, so double check that they match. Also, ``<TARGET_DIR>`` of course 52 needs to be replaced with the actual local directories where those files are 53 located. Note that the ``firefox`` executable is changed to ``firefox-bin``. 54 I don't know why that is necessary, but only the latter existed for me. 55 56 Finally, we need to make it so that the stack fixer can find the location of 57 the breakpad symbols we downloaded. If you are running ``fix_stacks.py`` via 58 ``dmd.py`` or directly (in a recent version), you can do this by running with the 59 environment variable ``BREAKPAD_SYMBOLS_PATH`` set to the ``<SYMB_DIR>`` from above. 60 If that doesn't work, you'll have to edit ``initFixStacks`` in ``fix_stacks.py`` to 61 set ``breakpadSymsDir`` to ``<SYMB_DIR>``. 62 63 With all of that done, you should now be able to run ``dmd.py`` or ``fix_stacks.py`` 64 to fix the stacks. Note that the stack fixing process can take a minute or two.