test.rst (6727B)
1 Test Kind 2 ========= 3 4 The ``test`` kind defines both desktop and mobile tests for builds. Each YAML 5 file referenced in ``kind.yml`` defines the full set of tests for the 6 associated suite. 7 8 The process of generating tests goes like this, based on a set of YAML files 9 named in ``kind.yml``: 10 11 * For each build task, determine the related test platforms based on the build 12 platform. For example, a Windows 2010 build might be tested on Windows 7 13 and Windows 10. Each test platform specifies "test sets" indicating which 14 tests to run. This is configured in the file named 15 ``test-platforms.yml``. 16 17 * Each test set is expanded to a list of tests to run. This is configured in 18 the file named by ``test-sets.yml``. A platform may specify several test 19 sets, in which case the union of those sets is used. 20 21 * Each named test is looked up in the file named by ``tests.yml`` to find a 22 test description. This test description indicates what the test does, how 23 it is reported to treeherder, and how to perform the test, all in a 24 platform-independent fashion. 25 26 * Each test description is converted into one or more tasks. This is 27 performed by a sequence of transforms defined in the ``transforms`` key in 28 ``kind.yml``. See :ref:`transforms` for more information. 29 30 * The resulting tasks become a part of the task graph. 31 32 .. important:: 33 34 This process generates *all* test jobs, regardless of tree or try syntax. 35 It is up to a later stages of the task-graph generation (the target set and 36 optimization) to select the tests that will actually be performed. 37 38 Variants 39 -------- 40 41 Sometimes we want to run the same tests under a different Firefox context, 42 usually this means with a pref set. The concept of ``variants`` was invented to 43 handle this use case. A variant is a stanza of configuration that can be merged 44 into each test definition. Variants are defined in the `variants.yml`_ file. 45 See this file for an up to date list of active variants and the pref(s) they 46 set. 47 48 Each variant must conform to the 49 :py:data:`~gecko_taskgraph.transforms.test.variant_description_schema`: 50 51 * **description** (required) - A description explaining what the variant is for. 52 * **component** (required) - The name of the component that owns the variant. It 53 should be formatted as ``PRODUCT``::``COMPONENT``. 54 * **expiration** (required) - The date when the variant will be expired (maximum 55 6 months). 56 * **suffix** (required) - A suffix to apply to the task label and treeherder symbol. 57 * **when** - A `json-e`_ expression that must evaluate to ``true`` for the variant 58 to be applied. The ``task`` definition is passed in as context. 59 * **replace** - A dictionary that will overwrite keys in the task definition. 60 * **merge** - A dictionary that will be merged into the task definition using 61 the :py:func:`~taskgraph.util.templates.merge` function. 62 63 .. note:: 64 65 Exceptions can be requested to have a variant without expiration (using 66 "never") if this is a shipped mode we support. Teams should contact the CI 67 team to discuss this before submitting a patch if they think their variant 68 qualifies. All exceptions will require director approval. 69 70 71 Defining Variants 72 ~~~~~~~~~~~~~~~~~ 73 74 Variants can be defined in the test YAML files using the ``variants`` key. E.g: 75 76 .. code-block:: yaml 77 78 example-suite: 79 variants: 80 - foo 81 - bar 82 83 This will split the task into three. The original task, the task with the 84 config from the variant named 'foo' merged in and the task with the config from 85 the variant named 'bar' merged in. 86 87 88 Composite Variants 89 ~~~~~~~~~~~~~~~~~~ 90 91 Sometimes we want to run tasks with multiple variants enabled at once. This can 92 be achieved with "composite variants". Composite variants are simply two or 93 more variant names joined with the ``+`` sign. Using the previous example, if 94 we wanted to run both the ``foo`` and ``bar`` variants together, we could do: 95 96 .. code-block:: yaml 97 98 example-suite: 99 variants: 100 - foo+bar 101 102 This will first merge or replace the config of ``foo`` into the task, followed 103 by the config of ``bar``. Care should be taken if both variants are replacing 104 the same keys. The last variant's configuration will be the one that gets used. 105 106 107 Expired Variants 108 ~~~~~~~~~~~~~~~~ 109 Ideally, when a variant is not needed anymore, it should be dropped (even if it 110 has not expired). If you need to extend the expiration date, you can submit a 111 patch to modify the expiration date in the `variants.yml`_ file. Variants will 112 not be scheduled to run after the expiration date. 113 114 If an expired variant is not dropped, the triage owner of the component will be 115 notified. If the expired variant persists for an extended period, the autonag 116 bot will escalate to notify the manager and director of the triage owner. Once 117 at that point, we will submit a patch to remove the variant from Taskcluster and 118 manifest conditions pending the triage owner / manager to review. 119 120 Please subscribe to alerts from `firefox-ci <https://groups.google.com/a/mozilla.com/g/firefox-ci>` 121 group in order to be aware of changes to the CI, scheduling, or the policy. 122 123 .. _variants.yml: https://searchfox.org/mozilla-central/source/taskcluster/kinds/test/variants.yml 124 .. _json-e: https://json-e.js.org/ 125 126 127 Setting 128 ------- 129 130 A test ``setting`` is the set of conditions under which a test is running. 131 Aside from the chunk number, a ``setting`` uniquely distinguishes a task from 132 another that is running the same set of tests. There are three types of inputs 133 that make up a ``setting``: 134 135 1. Platform - Bits of information that describe the underlying platform the 136 test is running on. This includes things like the operating system and 137 version, CPU architecture, etc. 138 139 2. Build - Bits of information that describe the build being tested. This 140 includes things like the build type and which build attributes (like 141 ``asan``, ``ccov``, etc) are enabled. 142 143 3. Runtime - Bits of information that describe the configured state of Firefox. 144 This includes things like prefs and environment variables. Note that tasks 145 should only set runtime configuration via the variants system (see 146 `Variants`_). 147 148 Test ``settings`` are available in the ``task.extra.test-setting`` object in 149 all test tasks. They are defined by the 150 :py:func:`~gecko_taskgraph.transforms.test.set_test_setting` transform 151 function. 152 153 The full schema is defined in the 154 :py:data:`~gecko_taskgraph.transforms.test.test_setting_description_schema`. 155 156 Setting Hash 157 ~~~~~~~~~~~~ 158 159 In addition to the three top-level objects, there is also a ``_hash`` key which 160 contains a hash of the rest of the setting object. This is a convenient way for 161 consumers to group or compare tasks that run under the same setting.