index.rst (8273B)
1 How Tos 2 ======= 3 4 All of this equipment is here to help you get your work done more efficiently. 5 However, learning how task-graphs are generated is probably not the work you 6 are interested in doing. This section should help you accomplish some of the 7 more common changes to the task graph with minimal fuss. 8 9 Taskgraph's documentation provides many relevant how-to guides: 10 11 .. note:: 12 13 If you come across references to the ``taskgraph`` command, simply prepend 14 ``./mach`` to the command to make it work in ``mozilla-central``. 15 16 .. toctree:: 17 18 Run Taskgraph Locally <https://taskcluster-taskgraph.readthedocs.io/en/latest/howto/run-locally.html> 19 Debug Taskgraph <https://taskcluster-taskgraph.readthedocs.io/en/latest/howto/debugging.html> 20 Use Fetches <https://taskcluster-taskgraph.readthedocs.io/en/latest/howto/use-fetches.html> 21 Use Docker Images <https://taskcluster-taskgraph.readthedocs.io/en/latest/howto/docker.html> 22 Create Actions <https://taskcluster-taskgraph.readthedocs.io/en/latest/howto/create-actions.html> 23 24 See Taskgraph's `how-to section`_ for even more guides! 25 26 .. _how-to section: https://taskcluster-taskgraph.readthedocs.io/en/latest/howto/index.html 27 28 Common Changes 29 -------------- 30 31 Additionally, here are some tips for common changes you wish to make within 32 ``mozilla-central``. 33 34 Changing Test Characteristics 35 ............................. 36 37 First, find the test description. This will be in 38 ``taskcluster/kinds/*/tests.yml``, for the appropriate kind (consult 39 :ref:`kinds`). You will find a YAML stanza for each test suite, and each 40 stanza defines the test's characteristics. For example, the ``chunks`` 41 property gives the number of chunks to run. This can be specified as a simple 42 integer if all platforms have the same chunk count, or it can be keyed by test 43 platform. For example: 44 45 .. code-block:: yaml 46 47 chunks: 48 by-test-platform: 49 linux64/debug: 10 50 default: 8 51 52 The full set of available properties is in 53 ``taskcluster/gecko_taskgraph/transforms/test/__init__.py``. Some other 54 commonly-modified properties are ``max-run-time`` (useful if tests are being 55 killed for exceeding maxRunTime) and ``treeherder-symbol``. 56 57 .. note:: 58 59 Android tests are also chunked at the mozharness level, so you will need to 60 modify the relevant mozharness config, as well. 61 62 Adding a Test Suite 63 ................... 64 65 To add a new test suite, you will need to know the proper mozharness invocation 66 for that suite, and which kind it fits into (consult :ref:`kinds`). 67 68 Add a new stanza to ``taskcluster/kinds/<kind>/tests.yml``, copying from the other 69 stanzas in that file. The meanings should be clear, but authoritative 70 documentation is in 71 ``taskcluster/gecko_taskgraph/transforms/test/__init__.py`` should you need 72 it. The stanza name is the name by which the test will be referenced in try 73 syntax. 74 75 Add your new test to a test set in ``test-sets.yml`` in the same directory. If 76 the test should only run on a limited set of platforms, you may need to define 77 a new test set and reference that from the appropriate platforms in 78 ``test-platforms.yml``. If you do so, include some helpful comments in 79 ``test-sets.yml`` for the next person. 80 81 Greening Up a New Test 82 ...................... 83 84 When a test is not yet reliably green, configuration for that test should not 85 be landed on integration branches. Of course, you can control where the 86 configuration is landed! For many cases, it is easiest to green up a test in 87 try: push the configuration to run the test to try along with your work to fix 88 the remaining test failures. 89 90 When working with a group, check out a "twig" repository to share among your 91 group, and land the test configuration in that repository. Once the test is 92 green, merge to an integration branch and the test will begin running there as 93 well. 94 95 Adding a New Task 96 ................. 97 98 If you are adding a new task that is not a test suite, there are a number of 99 options. A few questions to consider: 100 101 * Is this a new build platform or variant that will produce an artifact to 102 be run through the usual test suites? 103 104 * Does this task depend on other tasks? Do other tasks depend on it? 105 106 * Is this one of a few related tasks, or will you need to generate a large 107 set of tasks using some programmatic means (for example, chunking)? 108 109 * How is the task actually executed? Mozharness? Mach? 110 111 * What kind of environment does the task require? 112 113 Armed with that information, you can choose among a few options for 114 implementing this new task. Try to choose the simplest solution that will 115 satisfy your near-term needs. Since this is all implemented in-tree, it 116 is not difficult to refactor later when you need more generality. 117 118 Existing Kind 119 ````````````` 120 121 The simplest option is to add your task to an existing kind. This is most 122 practical when the task "makes sense" as part of that kind -- for example, if 123 your task is building an installer for a new platform using mozharness scripts 124 similar to the existing build tasks, it makes most sense to add your task to 125 the ``build`` kind. If you need some additional functionality in the kind, 126 it's OK to modify the implementation as necessary, as long as the modification 127 is complete and useful to the next developer to come along. 128 129 Tasks in the ``build`` kind generate Firefox installers, and the ``test`` kind 130 will add a full set of Firefox tests for each ``build`` task. 131 132 New Kind 133 ```````` 134 135 The next option to consider is adding a new kind. A distinct kind gives you 136 some isolation from other task types, which can be nice if you are adding an 137 experimental kind of task. 138 139 Kinds can range in complexity. The simplest sort of kind uses the transform 140 loader to read a list of jobs from the ``jobs`` key, and applies the standard 141 ``job`` and ``task`` transforms: 142 143 .. code-block:: yaml 144 145 implementation: taskgraph.task.transform:TransformTask 146 transforms: 147 - taskgraph.transforms.job:transforms 148 - taskgraph.transforms.task:transforms 149 jobs: 150 - ..your job description here.. 151 152 Job descriptions are defined and documented in 153 ``taskcluster/gecko_taskgraph/transforms/job/__init__.py``. 154 155 Custom Kind Loader 156 `````````````````` 157 158 If your task depends on other tasks, then the decision of which tasks to create 159 may require some code. For example, the ``test`` kind iterates over 160 the builds in the graph, generating a full set of test tasks for each one. This specific 161 post-build behavior is implemented as a loader defined in ``taskcluster/gecko_taskgraph/loader/test.py``. 162 163 A custom loader is useful when the set of tasks you want to create is not 164 static but based on something else (such as the available builds) or when the 165 dependency relationships for your tasks are complex. 166 167 Custom Transforms 168 ````````````````` 169 170 Most loaders apply a series of ":ref:`transforms`" that start with 171 an initial human-friendly description of a task and end with a task definition 172 suitable for insertion into a Taskcluster queue. 173 174 Custom transforms can be useful to apply defaults, simplifying the YAML files 175 in your kind. They can also apply business logic that is more easily expressed 176 in code than in YAML. 177 178 Transforms need not be one-to-one: a transform can produce zero or more outputs 179 for each input. For example, the test transforms perform chunking by producing 180 an output for each chunk of a given input. 181 182 Ideally those transforms will produce job descriptions, so you can use the 183 existing ``job`` and ``task`` transforms: 184 185 .. code-block:: yaml 186 187 transforms: 188 - taskgraph.transforms.my_stuff:transforms 189 - taskgraph.transforms.job:transforms 190 - taskgraph.transforms.task:transforms 191 192 Try to keep transforms simple, single-purpose and well-documented! 193 194 Custom Run-Using 195 ```````````````` 196 197 If the way your task is executed is unique (so, not a mach command or 198 mozharness invocation), you can add a new implementation of the job 199 description's "run" section. Before you do this, consider that it might be a 200 better investment to modify your task to support invocation via mozharness or 201 mach, instead. If this is not possible, then adding a new file in 202 ``taskcluster/gecko_taskgraph/transforms/jobs`` with a structure similar to its peers 203 will make the new run-using option available for job descriptions.