test_manifests.rst (7583B)
1 .. _test_manifests: 2 3 ============== 4 Test Manifests 5 ============== 6 7 Many test suites have their test metadata defined in files called 8 **test manifests**. 9 10 Test manifests are divided into two flavors: :ref:`manifestparser_manifests` 11 and :ref:`reftest_manifests`. 12 13 Naming Convention 14 ================= 15 16 The build system does not enforce file naming for test manifest files. 17 However, the following convention is used. 18 19 mochitest.toml 20 For the *plain* flavor of mochitests. 21 22 chrome.toml 23 For the *chrome* flavor of mochitests. 24 25 browser.toml 26 For the *browser chrome* flavor of mochitests. 27 28 a11y.toml 29 For the *a11y* flavor of mochitests. 30 31 xpcshell.toml 32 For *xpcshell* tests. 33 34 .. _manifestparser_manifests: 35 36 ManifestParser Manifests 37 ========================== 38 39 ManifestParser manifests are essentially toml files that conform to a basic 40 set of assumptions. 41 42 The :doc:`reference documentation </mozbase/manifestparser>` 43 for manifestparser manifests describes the basic format of test manifests. 44 45 In summary, manifests are toml files with section names describing test files:: 46 47 ["test_foo.js"] 48 ["test_bar.js"] 49 50 Keys under sections can hold metadata about each test:: 51 52 ["test_foo.js"] 53 skip-if = ["os == 'win'"] 54 ["test_foo.js"] 55 skip-if = ["os == 'linux' && debug"] 56 ["test_baz.js"] 57 fail-if = [ 58 "os == 'mac'", 59 "os == 'android'", 60 ] 61 62 There is a special **DEFAULT** section whose keys/metadata apply to all 63 sections/tests:: 64 65 [DEFAULT] 66 property = value 67 68 ["test_foo.js"] 69 70 In the above example, **test_foo.js** inherits the metadata **property = value** 71 from the **DEFAULT** section. 72 73 Recognized Metadata 74 ------------------- 75 76 Test manifests can define some common keys/metadata to influence behavior. 77 Those keys are as follows: 78 79 head 80 List of files that will be executed before the test file. (Used in 81 xpcshell tests.) 82 83 tail 84 List of files that will be executed after the test file. (Used in 85 xpcshell tests.) 86 87 support-files 88 List of additional files required to run tests. This is typically 89 defined in the **DEFAULT** section. 90 91 Unlike other file lists, *support-files* supports a globbing mechanism 92 to facilitate pulling in many files with minimal typing. This globbing 93 mechanism is activated if an entry in this value contains a ``*`` 94 character. A single ``*`` will wildcard match all files in a directory. 95 A double ``**`` will descend into child directories. For example, 96 ``data/*`` will match ``data/foo`` but not ``data/subdir/bar`` where 97 ``data/**`` will match ``data/foo`` and ``data/subdir/bar``. 98 99 Support files starting with ``/`` are placed in a root directory, rather 100 than a location determined by the manifest location. For mochitests, 101 this allows for the placement of files at the server root. The source 102 file is selected from the base name (e.g., ``foo`` for ``/path/foo``). 103 Files starting with ``/`` cannot be selected using globbing. 104 105 Some support files are used by tests across multiple directories. In 106 this case, a test depending on a support file from another directory 107 must note that dependency with the path to the required support file 108 in its own **support-files** entry. These use a syntax where paths 109 starting with ``!/`` will indicate the beginning of the path to a 110 shared support file starting from the root of the srcdir. For example, 111 if a manifest at ``dom/base/test/mochitest.toml`` has a support file, 112 ``dom/base/test/server-script.sjs``, and a mochitest in 113 ``dom/workers/test`` depends on that support file, the test manifest 114 at ``dom/workers/test/mochitest.toml`` must include 115 ``!/dom/base/test/server-script.sjs`` in its **support-files** entry. 116 117 generated-files 118 List of files that are generated as part of the build and don't exist in 119 the source tree. 120 121 The build system assumes that each manifest file, test file, and file 122 listed in **head**, **tail**, and **support-files** is static and 123 provided by the source tree (and not automatically generated as part 124 of the build). This variable tells the build system not to make this 125 assumption. 126 127 This variable will likely go away sometime once all generated files are 128 accounted for in the build config. 129 130 If a generated file is not listed in this key, a clobber build will 131 likely fail. 132 133 dupe-manifest 134 Record that this manifest duplicates another manifest. 135 136 The common scenario is two manifest files will include a shared 137 manifest file via the ``["include:file"]`` special section. The build 138 system enforces that each test file is only provided by a single 139 manifest. Having this key present bypasses that check. 140 141 The value of this key is ignored. 142 143 skip-if 144 Skip this test if the specified condition is true. 145 See :ref:`manifest_filter_language`. 146 147 Conditions can be specified on multiple lines, where each line is implicitly 148 joined by a logical OR (``||``). This makes it easier to add comments to 149 distinct failures. For example: 150 151 .. parsed-literal:: 152 153 ["test_foo.js"] 154 skip-if = [ 155 "os == 'mac' && fission", # bug 123 - fails on fission 156 "os == 'windows' && debug", # bug 456 - hits an assertion 157 ] 158 159 fail-if 160 Expect test failure if the specified condition is true. 161 See :ref:`manifest_filter_language`. 162 163 Conditions can be specified on multiple lines (see ``skip-if``). 164 165 run-sequentially 166 Run test sequentially if the specified condition is true. 167 See :ref:`manifest_filter_language`. 168 169 To always run a test sequentially, use ``run-sequentially = ["true"]``. 170 171 scheme 172 Changes the scheme and domain from which the test runs. (Only used in mochitest suites) 173 174 There are two possible values: 175 - ``http`` (default): The test will run from http://mochi.test:8888 176 - ``https``: The test will run from https://example.com:443 177 178 .. _manifest_filter_language: 179 180 Manifest Filter Language 181 ------------------------ 182 183 Some manifest keys accept a special filter syntax as their values. These 184 values are essentially boolean expressions that are evaluated at test 185 execution time. 186 187 The expressions can reference a well-defined set of variables, such as 188 ``os`` and ``debug``. These variables are populated from the 189 ``mozinfo.json`` file. For the full list of available variables, see 190 the :ref:`mozinfo documentation <mozinfo_attributes>`. 191 192 See 193 `the source <https://hg.mozilla.org/mozilla-central/file/default/testing/mozbase/manifestparser/manifestparser/manifestparser.py>`_ for the full documentation of the 194 expression syntax until it is documented here. 195 196 .. rstcheck: ignore-directives=todo 197 .. todo:: 198 199 Document manifest filter language. 200 201 .. _manifest_file_installation: 202 203 File Installation 204 ----------------- 205 206 Files referenced by manifests are automatically installed into the object 207 directory into paths defined in 208 :py:func:`mozbuild.frontend.emitter.TreeMetadataEmitter._process_test_manifest`. 209 210 Relative paths resolving to parent directory (e.g. 211 ``support-files = ../foo.txt`` have special behavior. 212 213 For ``support-files``, the file will be installed to the default destination 214 for that manifest. Only the file's base name is used to construct the final 215 path: directories are irrelevant. Files starting with ``/`` are an exception, 216 these are installed relative to the root of the destination; the base name is 217 instead used to select the file.. 218 219 For all other entry types, the file installation is skipped. 220 221 .. _reftest_manifests: 222 223 Reftest Manifests 224 ================= 225 226 See `MDN <https://developer.mozilla.org/en-US/docs/Creating_reftest-based_unit_tests>`_.