partials.rst (5685B)
1 Partial Update Generation 2 ========================= 3 4 Overview 5 -------- 6 7 Windows, Mac and Linux releases have partial updates, to reduce 8 the file size end-users have to download in order to receive new 9 versions. These are created using a docker image, some Python, 10 ``mbsdiff``, and the tools in ``tools/update-packaging`` 11 12 The task has been called 'Funsize' for quite some time. This might 13 make sense depending on what brands of chocolate bar are available 14 near you. 15 16 How the Task Works 17 ------------------ 18 19 Funsize uses a docker image that's built in-tree, named funsize-update-generator. 20 The image contains some Python to examine the task definition and determine 21 what needs to be done, but it downloads tools like ``mar`` and ``mbsdiff`` 22 from either locations specified in the task definition, or default mozilla-central 23 locations. 24 25 The 'extra' section of the task definition contains most of the payload, under 26 the 'funsize' key. In here is a list of partials that this specific task will 27 generate, and each entry includes the earlier (or 'from') version, and the most 28 recent (or 'to') version, which for most releases will likely be a taskcluster 29 artifact. 30 31 .. code-block:: json 32 33 { 34 "to_mar": "https://tc.net/api/queue/v1/task/EWtBFqVuT-WqG3tGLxWhmA/artifacts/public/build/ach/target.complete.mar", 35 "product": "Firefox", 36 "dest_mar": "target-60.0b8.partial.mar", 37 "locale": "ach", 38 "from_mar": "http://archive.mozilla.org/pub/firefox/candidates/60.0b8-candidates/build1/update/linux-i686/ach/firefox-60.0b8.complete.mar", 39 "update_number": 2, 40 "platform": "linux32", 41 "previousVersion": "60.0b8", 42 "previousBuildNumber": "1", 43 "branch": "mozilla-beta" 44 } 45 46 The 'update number' indicates how many released versions there are between 'to' and the current 'from'. 47 For example, if we are building a partial update for the current nightly from the previous one, the update 48 number will be 1. For the release before that, it will be 2. This lets us use generic output artifact 49 names that we can rename in the later ``beetmover`` tasks. 50 51 Inside the task, for each partial it has been told to generate, it will download, unpack and virus 52 scan the 'from_mar' and 'to_mar', download the tools, and run ``make_incremental_update.sh`` from 53 ``tools/update-packaging``. 54 55 If a scope is given for a set of temporary S3 credentials, the task will use a caching script, 56 to allow reuse of the diffs made for larger files. Some of the larger files are not localised, 57 and this allows us to save a lot of compute time. 58 59 For Releases 60 ------------ 61 62 Partials are made as part of the ``promote`` task group. The previous 63 versions used to create the update are specified in ship-it by 64 Release Management. 65 66 Nightly Partials 67 ---------------- 68 69 Since nightly releases don't appear in ship-it, the partials to create 70 are determined in the decision task. This was controversial, and so here 71 are the assumptions and reasons, so that when an alternative solution is 72 discovered, we can assess it in context: 73 74 1. Balrog is the source of truth for previous nightly releases. 75 2. Re-running a task should produce the same results. 76 3. A task's input and output should be specified in the definition. 77 4. A task transform should avoid external dependencies. This is to 78 increase the number of scenarios in which 'mach taskgraph' works. 79 5. A task graph doesn't explicitly know that it's intended for nightlies, 80 only that specific tasks are only present for nightly. 81 6. The decision task is explicitly told that its target is nightly 82 using the target-tasks-method argument. 83 84 a. From 2 and 3, this means that the partials task itself cannot query 85 balrog for the history, as it may get different results when re-run, 86 and hides the inputs and outputs from the task definition. 87 b. From 4, anything run by 'mach taskgraph' is an inappropriate place 88 to query Balrog, even if it results in a repeatable task graph. 89 c. Since these restrictions don't apply to the decision task, and given 90 6, we can query Balrog in the decision task if the target-tasks-method 91 given contains 'nightly', such as 'nightly_desktop' or 'nightly_linux' 92 93 Using the decision task involves making fewer, larger queries to Balrog, 94 and storing the results for task graph regeneration and later audit. At 95 the moment this data is stored in the ``parameters`` under the label 96 ``release_history``, since the parameters are an existing method for 97 passing data to the task transforms, but a case could be made 98 for adding a separate store, as it's a significantly larger number of 99 records than anything else in the parameters. 100 101 Nightly Partials and Beetmover 102 ------------------------------ 103 104 A release for a specific platform and locale may not have a history of 105 prior releases that can be used to build partial updates. This could be 106 for a variety of reasons, such as a new locale, or a hiatus in nightly 107 releases creating too long a gap in the history. 108 109 This means that the ``partials`` and ``partials-signing`` tasks may have 110 nothing to do for a platform and locale. If this is true, then the tasks 111 are filtered out in the ``transform``. 112 113 This does mean that the downstream task, ``beetmover-repackage`` can not 114 rely on the ``partials-signing`` task existing. It depends on both the 115 ``partials-signing`` and ``repackage-signing`` task, and chooses which 116 to depend on in the transform. 117 118 If there is a history in the ``parameters`` ``release_history`` section 119 then ``beetmover-repackage`` will depend on ``partials-signing``. 120 Otherwise, it will depend on ``repackage-signing``. 121 122 This is not ideal, as it results in unclear logic in the task graph 123 generation. It will be improved.