taskgraph.rst (5752B)
1 Taskgraph Overview 2 ================== 3 4 Taskgraph builds a directed acyclic graph, where each node in the graph 5 represents a task, and each edge represents the dependency relationships 6 between them. 7 8 See Taskgraph's `graph generation documentation`_ for more details. 9 10 Decision Task 11 ------------- 12 13 The decision task is the first task created when a new graph begins. It is 14 responsible for creating the rest of the task graph. 15 16 The decision task for pushes is defined in-tree, in ``.taskcluster.yml``. That 17 task description invokes ``mach taskcluster decision`` with some metadata about 18 the push. That mach command determines the required graph of tasks to submit, 19 then calls the Taskcluster API to create the tasks. 20 21 .. note:: 22 23 ``mach taskgraph decision`` is *not* meant to be invoked by humans. 24 Instead, follow `this guide`_ (prepending ``mach`` to all commands) to 25 invoke Taskgraph locally without submitting anything to Taskcluster. 26 27 .. _kinds: 28 29 Kinds 30 ----- 31 32 Generation starts with "kinds". These are yaml files that denote groupings of 33 tasks that are loosely related to one another. For example, in Gecko there are 34 ``build`` and ``test`` kinds. Each kind has its own directory under 35 `taskcluster/kinds`_ which contains a ``kind.yml`` file. 36 37 For more information on kinds, see Taskgraph's `kind documentation`_. For a 38 list of available kinds in ``mozilla-central``, see the :doc:`kinds reference 39 </taskcluster/kinds>`. 40 41 Loader 42 ------ 43 44 Next, a "loader" is responsible for parsing each ``kind.yml`` file and turning 45 it into an initial set of tasks. Loaders will always parse kinds in an ordering 46 satisfying the ``kind-dependencies`` key. 47 48 See Taskgraph's `loader documentation`_ for more details. 49 50 .. _transforms: 51 52 Transforms 53 ---------- 54 55 After the initial set of tasks are loaded, transformations are applied to each 56 task. Transforms are Python functions that take a generator of tasks as input, 57 and yields a generator of tasks as output. Possibly modifying, adding or removing 58 tasks along the way. 59 60 See Taskgrpah's `transforms documentation`_ for more details on transforms, or 61 the :doc:`transforms section </taskcluster/transforms/index>` for information 62 on some of the transforms available in ``mozilla-central``. 63 64 Target Tasks 65 ------------ 66 67 After transformation is finished, the `target_tasks`_ module filters out any 68 tasks that aren't applicable to the current :doc:`parameter set 69 </taskcluster/parameters>`. 70 71 Optimization 72 ------------ 73 74 After the "target graph" is generated, an optimization process looks to remove 75 or replace unnecessary tasks in the graph. For instance, a task may be removed 76 if the push doesn't modify any of the files that could affect it. 77 78 See Taskgraph's `optimization documentation`_ for more details, or the 79 :doc:`optimization strategies <optimization/index>` available in 80 ``mozilla-central``. 81 82 83 Task Parameterization 84 --------------------- 85 86 A few components of tasks are only known at the very end of the decision task 87 -- just before the ``queue.createTask`` call is made. These are specified 88 using simple parameterized values, as follows: 89 90 ``{"relative-datestamp": "certain number of seconds/hours/days/years"}`` 91 Objects of this form will be replaced with an offset from the current time 92 just before the ``queue.createTask`` call is made. For example, an 93 artifact expiration might be specified as ``{"relative-datestamp": "1 94 year"}``. 95 96 ``{"task-reference": "string containing <dep-name>"}`` 97 The task definition may contain "task references" of this form. These will 98 be replaced during the optimization step, with the appropriate taskId for 99 the named dependency substituted for ``<dep-name>`` in the string. 100 Additionally, `decision` and `self` can be used a dependency names to refer 101 to the decision task, and the task itself. Multiple labels may be 102 substituted in a single string, and ``<<>`` can be used to escape a literal 103 ``<``. 104 105 ``{"artifact-reference": "..<dep-name/artifact/name>.."}`` 106 Similar to a ``task-reference``, but this substitutes a URL to the queue's 107 ``getLatestArtifact`` API method (for which a GET will redirect to the 108 artifact itself). 109 110 .. _taskgraph-graph-config: 111 112 Graph Configuration 113 ------------------- 114 115 There are several configuration settings that are pertain to the entire 116 taskgraph. These are specified in :file:`config.yml` at the root of the 117 taskgraph configuration (typically :file:`taskcluster`). The available 118 settings are documented inline in `taskcluster/gecko_taskgraph/config.py 119 <https://searchfox.org/mozilla-central/source/taskcluster/gecko_taskgraph/config.py>`_. 120 121 .. _taskgraph-trust-domain: 122 123 Action Tasks 124 ------------ 125 126 Action Tasks are tasks which perform an action based on a manual trigger (e.g 127 clicking a button in Treeherder). Actions are how it is possible to retrigger 128 or "Add New Jobs". 129 130 For more information, see Taskgraph's `actions documentation`_. 131 132 .. _graph generation documentation: https://taskcluster-taskgraph.readthedocs.io/en/latest/concepts/task-graphs.html 133 .. _this guide: https://taskcluster-taskgraph.readthedocs.io/en/latest/howto/run-locally.html 134 .. _taskcluster/kinds: https://searchfox.org/mozilla-central/source/taskcluster/kinds 135 .. _kind documentation: https://taskcluster-taskgraph.readthedocs.io/en/latest/concepts/kind.html 136 .. _loader documentation: https://taskcluster-taskgraph.readthedocs.io/en/latest/concepts/loading.html 137 .. _transforms documentation: https://taskcluster-taskgraph.readthedocs.io/en/latest/concepts/transforms.html 138 .. _target_tasks: 139 .. _optimization documentation: https://taskcluster-taskgraph.readthedocs.io/en/latest/concepts/optimization.html 140 .. _actions documentation: https://taskcluster-taskgraph.readthedocs.io/en/latest/howto/create-actions.html