partner-repacks.rst (11453B)
1 Partner repacks 2 =============== 3 .. _partner repacks: 4 5 We create slightly-modified Firefox releases for some extra audiences 6 7 * EME-free builds, which disable DRM plugins by default 8 * Funnelcake builds, which are used for Mozilla experiments 9 * partner builds, which customize Firefox for external partners 10 11 We use the phrase "partner repacks" to refer to all these builds because they 12 use the same process of repacking regular Firefox releases with additional files. 13 The specific differences depend on the type of build. 14 15 We produce partner repacks for some beta builds, and for release builds, as part of the release 16 automation. We don't produce any files to update these builds as they are handled automatically 17 (see updates_). 18 19 We also produce :ref:`partner attribution` builds, which are Firefox Windows installers with a cohort identifier 20 added. 21 22 Parameters & Scheduling 23 ----------------------- 24 25 Partner repacks have a number of parameters which control how they work: 26 27 * ``release_enable_emefree`` 28 * ``release_enable_partner_repack`` 29 * ``release_partner_config`` 30 * ``release_partner_build_number`` 31 * ``release_partners`` 32 33 We split the repacks into two 'paths', EME-free and everything else, to retain some 34 flexibility over enabling/disabling them separately. This costs us some duplication of the kinds 35 in the repacking stack. The two enable parameters are booleans to turn these two paths 36 on/off. We set them in shipit's `is_partner_enabled() <https://github.com/mozilla-releng/shipit/blob/main/api/src/shipit_api/admin/release.py#L93>`_ when starting a 37 release. They're both true for Firefox betas >= b8 and releases, but otherwise disabled. 38 39 ``release_partner_config`` is a dictionary of configuration data which drives the task generation 40 logic. It's usually looked up during the release promotion action task, using the Github 41 GraphQL API in the `get_partner_config_by_url() 42 <python/taskgraph.util.html#taskgraph.util.partners.get_partner_config_by_url>`_ function, with the 43 url defined in `taskcluster/config.yml <https://searchfox 44 .org/mozilla-release/search?q=regexp%3A^partner+path%3Aconfig.yml&redirect=true>`_. 45 46 ``release_partner_build_number`` is an integer used to create unique upload paths in the firefox 47 candidates directory, while ``release_partners`` is a list of partners that should be 48 repacked (i.e. a subset of the whole config). Both are intended for use when respinning a few partners after 49 the regular Firefox has shipped. More information on that can be found in the 50 `RelEng Docs <https://moz-releng-docs.readthedocs.io/en/latest/procedures/misc-operations/off-cycle-partner-repacks-and-funnelcake.html>`_. 51 52 Most of the machine time for generating partner repacks takes place in the `promote` phase of the 53 automation, or `promote_rc` in the case of X.0 release candidates. The EME-free builds are copied into the 54 Firefox releases directory in the `push` phase, along with the regular bits. 55 56 57 Configuration 58 ------------- 59 60 We need some configuration to know *what* to repack, and *how* to do that. The *what* is defined by 61 default.xml manifests, as used with the `repo <https://gerrit.googlesource.com/git-repo>`_ tool 62 for git. The `default.xml for EME-free <https://github 63 .com/mozilla-partners/mozilla-EME-free-manifest/blob/master/default.xml>`_ illustrates this:: 64 65 <?xml version="1.0" ?> 66 <manifest> 67 <remote fetch="git@github.com:mozilla-partners/" name="mozilla-partners"/> 68 <remote fetch="git@github.com:mozilla/" name="mozilla"/> 69 70 <project name="repack-scripts" path="scripts" remote="mozilla-partners" revision="master"/> 71 <project name="build-tools" path="scripts/tools" remote="mozilla" revision="master"/> 72 <project name="mozilla-EME-free" path="partners/mozilla-EME-free" remote="mozilla-partners" revision="master"/> 73 </manifest> 74 75 The repack-scripts and build-tools repos are found in all manifests, and then there is a list of 76 partner repositories which contain the *how* configuration. Some of these repos are not publicly 77 visible. 78 79 A partner repository may contain multiple configurations inside the ``desktop`` directory. Each 80 subdirectory must contain a ``repack.cfg`` and a ``distribution`` directory, the latter 81 containing the customizations needed. Here's `EME-free's repack.cfg <https://github.com/mozilla-partners/mozilla-EME-free/blob/master/desktop/mozilla-EME-free/repack.cfg>`_:: 82 83 aus="mozilla-EMEfree" 84 dist_id="mozilla-EMEfree" 85 dist_version="1.0" 86 linux-i686=false 87 linux-x86_64=false 88 locales="ach af an ar" # truncated for display here 89 mac=true 90 win32=true 91 win64=true 92 output_dir="%(platform)s-EME-free/%(locale)s" 93 94 Note the list of locales and boolean toggles for enabling platforms. 95 96 All customizations will be placed in the ``distribution`` directory at the root of the Firefox 97 install directory, or in the case of OS X in ``Firefox.app/Contents/Resources/distribution/``. A 98 ``distribution.ini`` file is the minimal requirement, here's an example from `EME-free 99 <https://github.com/mozilla-partners/mozilla-EME-free/blob/master/desktop/mozilla-EME-free/distribution 100 /distribution.ini>`_:: 101 102 # Partner Distribution Configuration File 103 # Author: Mozilla 104 # Date: 2015-03-27 105 106 [Global] 107 id=mozilla-EMEfree 108 version=1.0 109 about=Mozilla Firefox EME-free 110 111 [Preferences] 112 media.eme.enabled=false 113 app.partner.mozilla-EMEfree="mozilla-EMEfree" 114 115 Extensions and other customizations might also be included in repacks. 116 117 118 Repacking process 119 ----------------- 120 121 The stack of tasks to create partner repacks is broadly similar to localised nightlies and 122 regular releases. The basic form is 123 124 * partner repack - insert the customisations into the regular builds 125 * signing - sign the internals which will become the installer (Mac only) 126 * repackage - create the "installer" (Mac and Windows) 127 * chunking dummy - a linux only bridge to ... 128 * repackage signing - sign the "installers" (mainly Windows) 129 * beetmover - move the files to a partner-specific destination 130 * beetmover checksums - possibly beetmove the checksums from previous step 131 132 Some key divergences are: 133 134 * all intermediate artifacts are uploaded with a ``releng/partner`` prefix 135 * we don't insert any binaries on Windows so no need for internal signing 136 * there's no need to create any complete mar files at the repackage step 137 * we only need beetmover checksums for EME-free builds 138 139 140 Partner repack 141 ^^^^^^^^^^^^^^ 142 143 * kinds: ``release-partner-repack`` ``release-eme-free-repack`` 144 * platforms: Typically all (but depends on what's enabled by partner configuration) 145 * upstreams: ``build-signing`` ``l10n-signing`` 146 147 There is one task per platform in this step, calling out to `scripts/desktop_partner_repacks.py 148 <https://hg.mozilla.org/mozilla-central/file/default/testing/mozharness/scripts 149 /desktop_partner_repacks.py>`_ in mozharness to prepare an environment and then perform the repacks. 150 The actual repacking is done by `python/mozrelease/mozrelease/partner_repack.py 151 <https://hg.mozilla.org/mozilla-central/file/default/python/mozrelease/mozrelease/partner_repack.py>`_. 152 153 It takes as input the build-signing and l10n-signing artifacts, which are all zip/tar.gz/tar.xz 154 archives, simplifying the repack process by avoiding dmg and exe. Windows produces ``target.zip`` 155 & ``setup.exe``, Mac is ``target.tar.gz``, Linux is the final product ``target.tar.xz`` 156 (beetmover handles pretty naming as usual). 157 158 Signing 159 ^^^^^^^ 160 161 * kinds: ``release-partner-repack-mac-signing`` ``release-partner-repack-mac-notarization`` 162 * platforms: Mac 163 * upstreams: ``release-partner-repack`` ``release-eme-free-repack`` 164 165 We chunk the single partner repack task out to a signing task with 5 artifacts each. For 166 example, EME-free will become 19 tasks. We collect the target.tar.gz from the 167 upstream, and return a signed target.tar.gz. We use a ``target.dmg`` artifact for 168 nightlies/regular releases, but this is converted to ``target.tar.gz`` by the signing 169 scriptworker before sending it to the signing server, so partners are equivalent. The ``mac-signing`` task 170 signs the binary, and then ``mac-notarization`` submits it to Apple and staples the ticket to it. 171 172 Repackage 173 ^^^^^^^^^ 174 175 * kinds: ``release-partner-repack-repackage`` ``release-eme-free-repack-repackage`` 176 * platforms: Mac & Windows 177 * upstreams: 178 179 * Mac: ``release-partner-signing`` ``release-eme-free-signing`` 180 * Windows: ``release-partner-repack`` ``release-eme-free-repack`` 181 182 Mac has a repackage job for each of the signing tasks. Windows repackages are chunked here to 183 the same granularity as mac. Takes ``target.zip`` & ``setup.exe`` to produce ``target.exe`` on 184 Windows, and ``target.tar.gz`` to produce ``target.dmg`` on Mac. There's no need to produce any 185 complete.mar files here like regular release bits do because we can reuse those. 186 187 Chunking dummy 188 ^^^^^^^^^^^^^^ 189 190 * kinds: ``release-partner-repack-chunking-dummy`` 191 * platforms: Linux 192 * upstreams: ``release-partner-repack`` 193 194 We're need Linux chunked at the next step so this dummy takes care of that for the relatively simple path 195 Linux follows. One task per sub config+locale combination, the same as Windows and Mac. This doesn't need to 196 exist for EME-free because we don't need to create Linux builds there. 197 198 Repackage Signing 199 ^^^^^^^^^^^^^^^^^ 200 201 * kinds: ``release-partner-repack-repackage-signing`` ``release-eme-free-repack-repackage-signing`` 202 * platforms: All 203 * upstreams: 204 205 * Mac & Windows: ``release-partner-repackage`` ``release-eme-free-repackage`` 206 * Linux: ``release-partner-repack-chunking-dummy`` 207 208 This step GPG signs all platforms, and authenticode signs the Windows installer. 209 210 Beetmover 211 ^^^^^^^^^ 212 213 * kinds: ``release-partner-repack-beetmover`` ``release-eme-free-repack-beetmover`` 214 * platforms: All 215 * upstreams: ``release-partner-repack-repackage-signing`` ``release-eme-free-repack-repackage-signing`` 216 217 Moves and renames the artifacts to their public location in the `candidates directory 218 <https://archive.mozilla.org/pub/firefox/candidates/>`_. Each task will 219 have the ``project:releng:beetmover:action:push-to-partner`` and 220 ``project:releng:beetmover:bucket:release`` scopes. There's a separate partner 221 code path in `beetmoverscript 222 <https://github.com/mozilla-releng/scriptworker-scripts/tree/master/beetmoverscript>`_. 223 224 Beetmover checksums 225 ^^^^^^^^^^^^^^^^^^^ 226 227 * kinds: ``release-eme-free-repack-beetmover-checksums`` 228 * platforms: Mac & Windows 229 * upstreams: ``release-eme-free-repack-repackage-beetmover`` 230 231 The EME-free builds should be present in our SHA256SUMS file and friends (`e.g. <https://archive 232 .mozilla.org/pub/firefox/releases/61.0/SHA256SUMS>`_) so we beetmove the target.checksums from 233 the beetmover tasks into the candidates directory. They get picked up by the 234 ``release-generate-checksums`` kind. 235 236 .. _updates: 237 238 Updates 239 ------- 240 241 It's very rare to need to update a partner repack differently from the original 242 release build but we retain that capability. A partner build with distribution name ``foo``, 243 based on a release Firefox build, will query for an update on the ``release-cck-foo`` channel. If 244 the update server `Balrog <http://mozilla-balrog.readthedocs.io/en/latest/>`_ finds no rule for 245 that channel it will fallback to the ``release`` channel. The update files for the regular releases do not 246 modify the ``distribution/`` directory, so the customizations are not modified. 247 248 `Bug 1430254 <https://bugzilla.mozilla.org/show_bug.cgi?id=1430254>`_ is an example of an exception to this 249 logic.