toolchains.rst (13713B)
1 .. _build_toolchains: 2 3 =========================== 4 Creating Toolchain Archives 5 =========================== 6 7 There are various scripts in the repository for producing archives 8 of the build tools (e.g. compilers and linkers) required to build. 9 10 Clang and Rust 11 ============== 12 13 To modify the toolchains used for a particular task, you may need several 14 things: 15 16 1. A `build task`_ 17 18 2. Which uses a toolchain task 19 20 - `clang toolchain`_ 21 - `rust toolchain`_ 22 23 3. Which uses a git fetch 24 25 - `clang fetch`_ 26 - (from-source ``dev`` builds only) `rust fetch`_ 27 28 4. (clang only) Which uses a `config json`_ 29 30 5. Which takes patches_ you may want to apply. 31 32 For the most part, you should be able to accomplish what you want by 33 copying/editing the existing examples in those files. 34 35 .. _build task: https://searchfox.org/mozilla-central/rev/168c45a7acc44e9904cfd4eebcb9eb080e05699c/taskcluster/ci/build/linux.yml#5-45 36 .. _clang toolchain: https://searchfox.org/mozilla-central/rev/168c45a7acc44e9904cfd4eebcb9eb080e05699c/taskcluster/ci/toolchain/clang.yml#51-72 37 .. _rust toolchain: https://searchfox.org/mozilla-central/rev/168c45a7acc44e9904cfd4eebcb9eb080e05699c/taskcluster/ci/toolchain/rust.yml#57-74 38 .. _clang fetch: https://searchfox.org/mozilla-central/rev/168c45a7acc44e9904cfd4eebcb9eb080e05699c/taskcluster/ci/fetch/toolchains.yml#413-418 39 .. _rust fetch: https://searchfox.org/mozilla-central/rev/168c45a7acc44e9904cfd4eebcb9eb080e05699c/taskcluster/ci/fetch/toolchains.yml#434-439 40 .. _config json: https://searchfox.org/mozilla-central/rev/168c45a7acc44e9904cfd4eebcb9eb080e05699c/build/build-clang/clang-linux64.json 41 .. _patches: https://searchfox.org/mozilla-central/rev/168c45a7acc44e9904cfd4eebcb9eb080e05699c/build/build-clang/static-llvm-symbolizer.patch 42 43 Clang 44 ----- 45 46 Building clang is handled by `build-clang.py`_, which uses several resources 47 in the `build-clang`_ directory. Read the `build-clang README`_ for more 48 details. 49 50 Note for local builds: build-clang.py can be run on developer machines but its 51 lengthy multi-stage build process is unnecessary for most local development. The 52 upstream `LLVM Getting Started Guide`_ has instructions on how to build 53 clang more directly. 54 55 .. _build-clang.py: https://searchfox.org/mozilla-central/source/build/build-clang/build-clang.py 56 .. _build-clang README: https://searchfox.org/mozilla-central/source/build/build-clang/README 57 .. _build-clang: https://searchfox.org/mozilla-central/source/build/build-clang/ 58 .. _LLVM Getting Started Guide: https://llvm.org/docs/GettingStarted.html 59 60 Rust 61 ---- 62 63 Rust builds are handled by `repack_rust.py`_. The primary purpose of 64 that script is to download prebuilt tarballs from the Rust project. 65 66 It uses the same basic format as `rustup` for specifying the toolchain 67 (via ``--channel``): 68 69 - request a stable build with ``1.xx.y`` (e.g. ``1.47.0``) 70 - request a beta build with ``beta-yyyy-mm-dd`` (e.g. ``beta-2020-08-26``) 71 - request a nightly build with ``nightly-yyyy-mm-dd`` (e.g. ``nightly-2020-08-26``) 72 - request a build from `Rust's ci`_ with ``bors-$sha`` (e.g. ``bors-796a2a9bbe7614610bd67d4cd0cf0dfff0468778``) 73 - request a from-source build with ``dev`` 74 75 Rust From Source 76 ---------------- 77 78 As of this writing, from-source builds for Rust are a new feature, and not 79 used anywhere by default. The feature was added so that we can test patches 80 to rustc against the tree. Expect things to be a bit hacky and limited. 81 82 Most importantly, building from source requires your toolchain to have a 83 `fetch of the rust tree`_ as well as `clang and binutils toolchains`_. It is also 84 recommended to upgrade the worker-type to e.g. ``b-linux-large``. 85 86 Rust's build dependencies are fairly minimal, and it has a sanity check 87 that should catch any missing or too-old dependencies. See the `Rust README`_ 88 for more details. 89 90 Patches are set via `the --patch flag`_ (passed via ``toolchain/rust.yml``). 91 Patch paths are assumed to be relative to ``/build/build-rust/``, and may be 92 optionally prefixed with ``module-path:`` to specify they apply to that git 93 submodule in the Rust source. e.g. ``--patch src/llvm-project:mypatch.diff`` 94 patches rust's llvm with ``/build/build-rust/mypatch.diff``. There are no 95 currently checked in rust patches to use as an example, but they should be 96 the same format as `the clang ones`_. 97 98 Rust builds are not currently configurable, and uses a `hardcoded config.toml`_, 99 which you may need to edit for your purposes. See Rust's `example config`_ for 100 details/defaults. Note that these options do occasionally change, so be sure 101 you're using options for the version you're targeting. For instance, there was 102 a large change around Rust ~1.48, and the currently checked in config was for 103 1.47, so it may not work properly when building the latest version of Rust. 104 105 Rust builds are currently limited to targeting only the host platform. 106 Although the machinery is in place to request additional targets, the 107 cross-compilation fails for some unknown reason. We have not yet investigated 108 what needs to be done to get this working. 109 110 While Rust generally maintains a clean tree for building ``rustc`` and 111 ``cargo``, other tools like ``rustfmt`` or ``miri`` are allowed to be 112 transiently broken. This means not every commit in the Rust tree will be 113 able to build the `tools we require`_. 114 115 Although ``repack_rust`` considers ``rustfmt`` an optional package, Rust builds 116 do not currently implement this and will fail if ``rustfmt`` is busted. Some 117 attempt was made to work around it, but `more work is needed`_. 118 119 .. _Rust's ci: https://github.com/rust-lang/rust/pull/77875#issuecomment-736092083 120 .. _repack_rust.py: https://searchfox.org/mozilla-central/source/taskcluster/scripts/misc/repack_rust.py 121 .. _fetch of the rust tree: https://searchfox.org/mozilla-central/rev/168c45a7acc44e9904cfd4eebcb9eb080e05699c/taskcluster/ci/toolchain/rust.yml#69-71 122 .. _clang and binutils toolchains: https://searchfox.org/mozilla-central/rev/168c45a7acc44e9904cfd4eebcb9eb080e05699c/taskcluster/ci/toolchain/rust.yml#72-74 123 .. _the --patch flag: https://searchfox.org/mozilla-central/rev/168c45a7acc44e9904cfd4eebcb9eb080e05699c/taskcluster/scripts/misc/repack_rust.py#667-675 124 .. _the clang ones: https://searchfox.org/mozilla-central/rev/168c45a7acc44e9904cfd4eebcb9eb080e05699c/build/build-clang/static-llvm-symbolizer.patch 125 .. _Rust README: https://github.com/rust-lang/rust/#building-on-a-unix-like-system 126 .. _hardcoded config.toml: https://searchfox.org/mozilla-central/rev/168c45a7acc44e9904cfd4eebcb9eb080e05699c/taskcluster/scripts/misc/repack_rust.py#384-421 127 .. _example config: https://github.com/rust-lang/rust/blob/b7ebc6b0c1ba3c27ebb17c0b496ece778ef11e18/config.toml.example 128 .. _tools we require: https://searchfox.org/mozilla-central/rev/168c45a7acc44e9904cfd4eebcb9eb080e05699c/taskcluster/scripts/misc/repack_rust.py#398 129 .. _more work is needed: https://github.com/rust-lang/rust/issues/79249 130 131 Python 132 ------ 133 134 Python is built from source by ``taskcluster/scripts/misc/build-cpython.sh`` on 135 Linux and OSX. We use the upstream installer on Windows, through 136 ``taskcluster/scripts/misc/pack-cpython.sh``. In order to ensure consistency, we 137 use the same version for both approaches. Note however that the Windows installer is 138 not packaged for all patch versions, so there might be a slight delta there. 139 140 Windows 141 ======= 142 143 The ``build/vs/generate_yaml.py`` and ``taskcluster/scripts/misc/get_vs.py`` 144 scripts are used to manage and get Windows toolchains containing Visual 145 Studio executables, SDKs, etc. 146 147 The ``build/vs/generate_yaml.py`` script is used to generate one of the 148 YAML files used in the relevant toolchain task. The exact command line 149 used to generate the file is stored in the header of the YAML file itself. 150 Each YAML file records the necessary downloads from Microsoft servers to 151 install the required Visual Studio components given on the command line. 152 153 The ``taskcluster/scripts/misc/get_vs.py`` script takes a YAML file as 154 input and fills a directory with the corresponding Visual Studio components. 155 156 Both scripts should be run via ``mach python --virtualenv build``. The 157 latter is automatically invoked by the bootstrapping mechanism. 158 159 160 MacOS 161 ===== 162 163 The ``build/macosx/catalog.py`` and ``taskcluster/scripts/misc/unpack-sdk.py`` 164 scripts are used to manage and get macOS SDKs. 165 166 The ``build/macosx/catalog.py`` script is used to explore the Apple 167 software update catalog. Running the script with no argument will show 168 a complete list of "products". You probably don't want that, but rather 169 start with a filter: 170 171 .. code-block:: shell 172 173 $ ./mach python build/macosx/catalog.py --filter SDK 174 061-44071 Beats Updater 1.0 175 071-29699 Command Line Tools for Xcode 12.5 176 001-89745 Command Line Tools for Xcode 12.4 177 071-54303 Command Line Tools for Xcode 12.5 178 002-41708 Command Line Tools for Xcode 13.2 179 002-83793 Command Line Tools for Xcode 13.4 180 012-92431 Command Line Tools for Xcode 14.2 181 032-64167 Command Line Tools for Xcode 14.3 182 183 From there, pick the id of the product you're interested in, and run the 184 script again with that id: 185 186 .. code-block:: shell 187 188 $ ./mach python build/macosx/catalog.py 032-64167 189 com.apple.pkg.CLTools_Executables https://swcdn.apple.com/content/downloads/38/61/032-64167-A_F8LL7XSTW6/k3kg0uip4kxd3qupgy6y8fzp27mnxdpt6y/CLTools_Executables.pkg 190 com.apple.pkg.CLTools_SDK_macOS13 https://swcdn.apple.com/content/downloads/38/61/032-64167-A_F8LL7XSTW6/k3kg0uip4kxd3qupgy6y8fzp27mnxdpt6y/CLTools_macOSNMOS_SDK.pkg 191 com.apple.pkg.CLTools_SDK_macOS12 https://swcdn.apple.com/content/downloads/38/61/032-64167-A_F8LL7XSTW6/k3kg0uip4kxd3qupgy6y8fzp27mnxdpt6y/CLTools_macOSLMOS_SDK.pkg 192 com.apple.pkg.CLTools_macOS_SDK https://swcdn.apple.com/content/downloads/38/61/032-64167-A_F8LL7XSTW6/k3kg0uip4kxd3qupgy6y8fzp27mnxdpt6y/CLTools_macOS_SDK.pkg 193 com.apple.pkg.CLTools_SwiftBackDeploy https://swcdn.apple.com/content/downloads/38/61/032-64167-A_F8LL7XSTW6/k3kg0uip4kxd3qupgy6y8fzp27mnxdpt6y/CLTools_SwiftBackDeploy.pkg 194 195 From there, pick the id of the package you're interested in, and run the 196 script again with a combination of both product and package ids to inspect 197 its content and ensure that's what you're looking for. 198 199 .. code-block:: shell 200 201 $ ./mach python build/macosx/catalog.py 032-64167/com.apple.pkg.CLTools_SDK_macOS13 202 Library 203 Library/Developer 204 Library/Developer/CommandLineTools 205 Library/Developer/CommandLineTools/SDKs 206 Library/Developer/CommandLineTools/SDKs/MacOSX13.sdk 207 Library/Developer/CommandLineTools/SDKs/MacOSX13.3.sdk 208 Library/Developer/CommandLineTools/SDKs/MacOSX13.3.sdk/usr 209 (...) 210 211 Once you have found the SDK you want, you can create or update toolchain tasks 212 in ``taskcluster/kinds/toolchain/macosx-sdk.yml``. 213 214 The ``taskcluster/scripts/misc/unpack-sdk.py`` script takes the url of a SDK 215 package, the sha512 hash for its content, the path to the SDK in the package, 216 and an output directory, and extracts the package in that directory. 217 218 Both scripts should be run via ``mach python``. The latter is automatically 219 invoked by the bootstrapping mechanism. 220 221 On automation, the script will download the file from tooltool instead of the 222 original url, so the file should also be uploaded to tooltool with `internal` 223 visibility. 224 See https://github.com/mozilla-releng/tooltool. 225 226 Firefox for Android with Gradle 227 =============================== 228 229 To build Firefox for Android with Gradle in automation, archives 230 containing both the Gradle executable and a Maven repository 231 comprising the exact build dependencies are produced and uploaded to 232 an internal Mozilla server. The build automation will download, 233 verify, and extract these archive before building. These archives 234 provide a self-contained Gradle and Maven repository so that machines 235 don't need to fetch additional Maven dependencies at build time. 236 (Gradle and the downloaded Maven dependencies can be both 237 redistributed publicly.) 238 239 Archiving the Gradle executable is straight-forward, but archiving a 240 local Maven repository is not. Therefore a toolchain job exists for 241 producing the required archives, `android-gradle-dependencies`. The 242 job runs in a container based on a custom Docker image and spawns a 243 Sonatype Nexus proxying Maven repository process in the background. 244 The job builds Firefox for Android using Gradle and the in-tree Gradle 245 configuration rooted at ``build.gradle``. The spawned proxying Maven 246 repository downloads external dependencies and collects them. After 247 the Gradle build completes, the job archives the Gradle version used 248 to build, and the downloaded Maven repository, and exposes them as 249 Task Cluster artifacts. 250 251 To update the version of Gradle in the archive produced, update 252 ``gradle/wrapper/gradle-wrapper.properties``. Be sure to also update 253 the SHA256 checksum to prevent poisoning the build machines! 254 255 To update the versions of Gradle dependencies used, update 256 ``dependencies`` sections in the in-tree Gradle configuration rooted 257 at ``build.gradle``. Once you are confident your changes build 258 locally, push a fresh build to try. The `android-gradle-dependencies` 259 toolchain should run automatically, fetching your new dependencies and 260 wiring them into the appropriate try build jobs. 261 262 To update the version of Sonatype Nexus, update the `sonatype-nexus` 263 `fetch` task definition. 264 265 To modify the Sonatype Nexus configuration, typically to proxy a new 266 remote Maven repository, modify 267 `taskcluster/scripts/misc/android-gradle-dependencies/nexus.xml`. 268 269 There is also a toolchain job that fetches the Android SDK and related 270 packages. To update the versions of packaged fetched, modify 271 `python/mozboot/mozboot/android-packages.txt` and update the various 272 in-tree versions accordingly.