mozbuild-files.rst (8046B)
1 .. _mozbuild-files: 2 3 =============== 4 moz.build Files 5 =============== 6 7 ``moz.build`` files are the mechanism by which tree metadata (notably 8 the build configuration) is defined. 9 10 Directories in the tree contain ``moz.build`` files which declare 11 functionality for their respective part of the tree. This includes 12 things such as the list of C++ files to compile, where to find tests, 13 etc. 14 15 ``moz.build`` files are actually Python scripts. However, their 16 execution is governed by special rules. This is explained below. 17 18 moz.build Python Sandbox 19 ======================== 20 21 As mentioned above, ``moz.build`` files are Python scripts. However, 22 they are executed in a special Python *sandbox* that significantly 23 changes and limits the execution environment. The environment is so 24 different, it's doubtful most ``moz.build`` files would execute without 25 error if executed by a vanilla Python interpreter (e.g. ``python 26 moz.build``. 27 28 The following properties make execution of ``moz.build`` files special: 29 30 1. The execution environment exposes a limited subset of Python. 31 2. There is a special set of global symbols and an enforced naming 32 convention of symbols. 33 3. Some symbols are inherited from previously-executed ``moz.build`` 34 files. 35 36 The limited subset of Python is actually an extremely limited subset. 37 Only a few symbols from ``__builtin__`` are exposed. These include 38 ``True``, ``False``, ``None``, ``sorted``, ``int``, and ``set``. Global 39 functions like ``import``, ``print``, and ``open`` aren't available. 40 Without these, ``moz.build`` files can do very little. *This is by design*. 41 42 The execution sandbox treats all ``UPPERCASE`` variables specially. Any 43 ``UPPERCASE`` variable must be known to the sandbox before the script 44 executes. Any attempt to read or write to an unknown ``UPPERCASE`` 45 variable will result in an exception being raised. Furthermore, the 46 types of all ``UPPERCASE`` variables is strictly enforced. Attempts to 47 assign an incompatible type to an ``UPPERCASE`` variable will result in 48 an exception being raised. 49 50 The strictness of behavior with ``UPPERCASE`` variables is a very 51 intentional design decision. By ensuring strict behavior, any operation 52 involving an ``UPPERCASE`` variable is guaranteed to have well-defined 53 side-effects. Previously, when the build configuration was defined in 54 ``Makefiles``, assignments to variables that did nothing would go 55 unnoticed. ``moz.build`` files fix this problem by eliminating the 56 potential for false promises. 57 58 After a ``moz.build`` file has completed execution, only the 59 ``UPPERCASE`` variables are used to retrieve state. 60 61 The set of variables and functions available to the Python sandbox is 62 defined by the :py:mod:`mozbuild.frontend.context` module. The 63 data structures in this module are consumed by the 64 :py:class:`mozbuild.frontend.reader.MozbuildSandbox` class to construct 65 the sandbox. There are tests to ensure that the set of symbols exposed 66 to an empty sandbox are all defined in the ``context`` module. 67 This module also contains documentation for each symbol, so nothing can 68 sneak into the sandbox without being explicitly defined and documented. 69 70 Reading and Traversing moz.build Files 71 ====================================== 72 73 The process for reading ``moz.build`` files roughly consists of: 74 75 1. Start at the root ``moz.build`` (``<topsrcdir>/moz.build``). 76 2. Evaluate the ``moz.build`` file in a new sandbox. 77 3. Emit the main *context* and any *sub-contexts* from the executed 78 sandbox. 79 4. Extract a set of ``moz.build`` files to execute next. 80 5. For each additional ``moz.build`` file, goto #2 and repeat until all 81 referenced files have executed. 82 83 From the perspective of the consumer, the output of reading is a stream 84 of :py:class:`mozbuild.frontend.reader.context.Context` instances. Each 85 ``Context`` defines a particular aspect of data. Consumers iterate over 86 these objects and do something with the data inside. Each object is 87 essentially a dictionary of all the ``UPPERCASE`` variables populated 88 during its execution. 89 90 .. note:: 91 92 Historically, there was only one ``context`` per ``moz.build`` file. 93 As the number of things tracked by ``moz.build`` files grew and more 94 and more complex processing was desired, it was necessary to split these 95 contexts into multiple logical parts. It is now common to emit 96 multiple contexts per ``moz.build`` file. 97 98 Build System Reading Mode 99 ------------------------- 100 101 The traditional mode of evaluation of ``moz.build`` files is what's 102 called *build system traversal mode.* In this mode, the ``CONFIG`` 103 variable in each ``moz.build`` sandbox is populated from data coming 104 from ``config.status``, which is produced by ``configure``. 105 106 During evaluation, ``moz.build`` files often make decisions conditional 107 on the state of the build configuration. e.g. *only compile foo.cpp if 108 feature X is enabled*. 109 110 In this mode, traversal of ``moz.build`` files is governed by variables 111 like ``DIRS`` and ``TEST_DIRS``. For example, to execute a child 112 directory, ``foo``, you would add ``DIRS += ['foo']`` to a ``moz.build`` 113 file and ``foo/moz.build`` would be evaluated. 114 115 .. _mozbuild_fs_reading_mode: 116 117 Filesystem Reading Mode 118 ----------------------- 119 120 There is an alternative reading mode that doesn't involve the build 121 system and doesn't use ``DIRS`` variables to control traversal into 122 child directories. This mode is called *filesystem reading mode*. 123 124 In this reading mode, the ``CONFIG`` variable is a dummy, mostly empty 125 object. Accessing all but a few special variables will return an empty 126 value. This means that nearly all ``if CONFIG['FOO']:`` branches will 127 not be taken. 128 129 Instead of using content from within the evaluated ``moz.build`` 130 file to drive traversal into subsequent ``moz.build`` files, the set 131 of files to evaluate is controlled by the thing doing the reading. 132 133 A single ``moz.build`` file is not guaranteed to be executable in 134 isolation. Instead, we must evaluate all *parent* ``moz.build`` files 135 first. For example, in order to evaluate ``/foo/moz.build``, one must 136 execute ``/moz.build`` and have its state influence the execution of 137 ``/foo/moz.build``. 138 139 Filesystem reading mode is utilized to power the 140 :ref:`mozbuild_files_metadata` feature. 141 142 Technical Details 143 ----------------- 144 145 The code for reading ``moz.build`` files lives in 146 :py:mod:`mozbuild.frontend.reader`. The Python sandboxes evaluation results 147 (:py:class:`mozbuild.frontend.context.Context`) are passed into 148 :py:mod:`mozbuild.frontend.emitter`, which converts them to classes defined 149 in :py:mod:`mozbuild.frontend.data`. Each class in this module defines a 150 domain-specific component of tree metadata. e.g. there will be separate 151 classes that represent a JavaScript file vs a compiled C++ file or test 152 manifests. This means downstream consumers of this data can filter on class 153 types to only consume what they are interested in. 154 155 There is no well-defined mapping between ``moz.build`` file instances 156 and the number of :py:mod:`mozbuild.frontend.data` classes derived from 157 each. Depending on the content of the ``moz.build`` file, there may be 1 158 object derived or 100. 159 160 The purpose of the ``emitter`` layer between low-level sandbox execution 161 and metadata representation is to facilitate a unified normalization and 162 verification step. There are multiple downstream consumers of the 163 ``moz.build``-derived data and many will perform the same actions. This 164 logic can be complicated, so we have a component dedicated to it. 165 166 :py:class:`mozbuild.frontend.reader.BuildReader`` and 167 :py:class:`mozbuild.frontend.reader.TreeMetadataEmitter`` have a 168 stream-based API courtesy of generators. When you hook them up properly, 169 the :py:mod:`mozbuild.frontend.data` classes are emitted before all 170 ``moz.build`` files have been read. This means that downstream errors 171 are raised soon after sandbox execution. 172 173 Lots of the code for evaluating Python sandboxes is applicable to 174 non-Mozilla systems. In theory, it could be extracted into a standalone 175 and generic package. However, until there is a need, there will 176 likely be some tightly coupled bits.