sccache-dist.rst (10039B)
1 .. _sccache_dist: 2 3 ================================== 4 Distributed sccache (sccache-dist) 5 ================================== 6 7 `sccache <https://github.com/mozilla/sccache>`_ is a ccache-like tool written in 8 Rust by Mozilla and many contributors. 9 10 sccache-dist, its distributed variant, elevates this functionality by enabling 11 the distribution and caching of Rust compilations across multiple machines. 12 Please consider using sccache-dist when you have several machines 13 compiling Firefox on the same network. 14 15 The steps for setting up your machine as an sccache-dist server are detailed below. 16 17 In addition to improved security properties, distributed sccache offers 18 distribution and caching of rust compilation, so it should be an improvement 19 above and beyond what we see with icecc. Build servers run on Linux and 20 distributing builds is currently supported from Linux, macOS, and Windows. 21 22 23 Steps for distributing a build as an sccache-dist client 24 ======================================================== 25 26 Start by following the instructions at https://github.com/mozilla/sccache/blob/master/docs/DistributedQuickstart.md#configure-a-client 27 to configure your sccache distributed client. 28 *NOTE* If you're distributing from Linux a toolchain will be packaged 29 automatically and provided to the build server. If you're distributing from 30 Windows or macOS, start by using the cross-toolchains provided by 31 ``./mach bootstrap`` rather than attempting to use ``icecc-create-env``. 32 sccache 0.2.12 or above is recommended, and the auth section of your config 33 must read:: 34 35 [dist.auth] 36 type = "mozilla" 37 38 * If you're compiling from a macOS client, there are a handful of additional 39 considerations outlined here: 40 https://github.com/mozilla/sccache/blob/master/docs/DistributedQuickstart.md#considerations-when-distributing-from-macos. 41 42 Run ``./mach bootstrap`` to download prebuilt toolchains to 43 ``~/.mozbuild/clang-dist-toolchain.tar.xz`` and 44 ``~/.mozbuild/rustc-dist-toolchain.tar.xz``. This is an example of the paths 45 that should be added to your client config to specify toolchains to build on 46 macOS, located at ``~/Library/Application Support/Mozilla.sccache/config``:: 47 48 [[dist.toolchains]] 49 type = "path_override" 50 compiler_executable = "/path/to/home/.rustup/toolchains/stable-x86_64-apple-darwin/bin/rustc" 51 archive = "/path/to/home/.mozbuild/rustc-dist-toolchain.tar.xz" 52 archive_compiler_executable = "/builds/worker/toolchains/rustc/bin/rustc" 53 54 [[dist.toolchains]] 55 type = "path_override" 56 compiler_executable = "/path/to/home/.mozbuild/clang/bin/clang" 57 archive = "/path/to/home/.mozbuild/clang-dist-toolchain.tar.xz" 58 archive_compiler_executable = "/builds/worker/toolchains/clang/bin/clang" 59 60 [[dist.toolchains]] 61 type = "path_override" 62 compiler_executable = "/path/to/home/.mozbuild/clang/bin/clang++" 63 archive = "/path/to/home/.mozbuild/clang-dist-toolchain.tar.xz" 64 archive_compiler_executable = "/builds/worker/toolchains/clang/bin/clang" 65 66 Note that the version of ``rustc`` found in ``rustc-dist-toolchain.tar.xz`` 67 must match the version of ``rustc`` used locally. The distributed archive 68 will contain the version of ``rustc`` used by automation builds, which may 69 lag behind stable for a few days after Rust releases, which is specified by 70 the task definition in 71 `this file <https://hg.mozilla.org/mozilla-central/file/tip/taskcluster/kinds/toolchain/dist-toolchains.yml>`_. 72 For instance, to specify 1.37.0 rather than the current stable, run 73 ``rustup toolchain add 1.37.0`` and point to 74 ``/path/to/home/.rustup/toolchains/1.37.0-x86_64-apple-darwin/bin/rustc`` in your 75 client config. 76 77 The build system currently requires an explicit target to be passed with 78 ``HOST_CFLAGS`` and ``HOST_CXXFLAGS`` e.g.:: 79 80 export HOST_CFLAGS="--target=x86_64-apple-darwin16.0.0" 81 export HOST_CXXFLAGS="--target=x86_64-apple-darwin16.0.0" 82 83 * Compiling from a Windows client is supported but hasn't seen as much testing 84 as other platforms. The following example mozconfig can be used as a guide:: 85 86 ac_add_options CCACHE="C:/Users/<USER>/.mozbuild/sccache/sccache.exe" 87 88 export CC="C:/Users/<USER>/.mozbuild/clang/bin/clang-cl.exe --driver-mode=cl" 89 export CXX="C:/Users/<USER>/.mozbuild/clang/bin/clang-cl.exe --driver-mode=cl" 90 export HOST_CC="C:/Users/<USER>/.mozbuild/clang/bin/clang-cl.exe --driver-mode=cl" 91 export HOST_CXX="C:/Users/<USER>/.mozbuild/clang/bin/clang-cl.exe --driver-mode=cl" 92 93 The client config should be located at 94 ``~/AppData/Roaming/Mozilla/sccache/config/config``, and as on macOS custom 95 toolchains should be obtained with ``./mach bootstrap`` and specified in the 96 client config, for example:: 97 98 [[dist.toolchains]] 99 type = "path_override" 100 compiler_executable = "C:/Users/<USER>/.mozbuild/clang/bin/clang-cl.exe" 101 archive = "C:/Users/<USER>/.mozbuild/clang-dist-toolchain.tar.xz" 102 archive_compiler_executable = "/builds/worker/toolchains/clang/bin/clang" 103 104 [[dist.toolchains]] 105 type = "path_override" 106 compiler_executable = "C:/Users/<USER>/.rustup/toolchains/stable-x86_64-pc-windows-msvc/bin/rustc.exe" 107 archive = "C:/Users/<USER>/.mozbuild/rustc-dist-toolchain.tar.xz" 108 archive_compiler_executable = "/builds/worker/toolchains/rustc/bin/rustc" 109 110 * Add the following to your mozconfig:: 111 112 ac_add_options CCACHE=/path/to/home/.mozbuild/sccache/sccache 113 114 If you're compiling from a macOS client, you might need some additional configuration:: 115 116 # Set the target flag to Darwin 117 export CFLAGS="--target=x86_64-apple-darwin16.0.0" 118 export CXXFLAGS="--target=x86_64-apple-darwin16.0.0" 119 export HOST_CFLAGS="--target=x86_64-apple-darwin16.0.0" 120 export HOST_CXXFLAGS="--target=x86_64-apple-darwin16.0.0" 121 122 # Specify the macOS SDK to use 123 ac_add_options --with-macos-sdk=/path/to/MacOSX-SDKs/MacOSX13.3.sdk 124 125 You can get the right macOS SDK by downloading an old version of XCode from 126 `developer.apple.com <https://developer.apple.com>`_ and unpacking the SDK 127 from it. 128 129 * When attempting to get your client running, the output of ``sccache -s`` should 130 be consulted to confirm compilations are being distributed. To receive helpful 131 logging from the local daemon in case they aren't, run 132 ``SCCACHE_NO_DAEMON=1 SCCACHE_START_SERVER=1 SCCACHE_LOG=sccache=trace path/to/sccache`` 133 in a terminal window separate from your build prior to building. *NOTE* use 134 ``RUST_LOG`` instead of ``SCCACHE_LOG`` if your build of ``sccache`` does not 135 include `pull request 822 136 <https://github.com/mozilla/sccache/pull/822>`_. (``sccache`` binaries from 137 ``mach bootstrap`` do include this PR.) 138 139 * Run ``./mach build -j<value>`` with an appropriately large ``<value>``. 140 ``sccache --dist-status`` should provide the number of cores available to you 141 (or a message if you're not connected). In the future this will be integrated 142 with the build system to automatically select an appropriate value. 143 144 This should be enough to distribute your build and replace your use of icecc. 145 Bear in mind there may be a few speedbumps, and please ensure your version of 146 sccache is current before investigating further. Please see the common questions 147 section below and ask for help if anything is preventing you from using it over 148 email (dev-builds), on slack in #sccache, or in #build on irc. 149 150 Steps for setting up a server 151 ============================= 152 153 Build servers must run linux and use bubblewrap 0.3.0+ for sandboxing of compile 154 processes. This requires a kernel 4.6 or greater, so Ubuntu 18+, RHEL 8, or 155 similar. 156 157 * Run ``./mach bootstrap`` or 158 ``./mach artifact toolchain --from-build linux64-sccache`` to acquire a recent 159 version of ``sccache-dist``. Please use a ``sccache-dist`` binary acquired in 160 this fashion to ensure compatibility with statically linked dependencies. 161 162 * The instructions at https://github.com/mozilla/sccache/blob/master/docs/DistributedQuickstart.md#configure-a-build-server 163 should contain everything else required to configure and run the server. 164 165 *NOTE* Port 10500 will be used by convention for builders. 166 Please use port 10500 in the ``public_addr`` section of your builder config. 167 168 Extra logging may be helpful when setting up a server. To enable logging, 169 run your server with 170 ``sudo env SCCACHE_LOG=sccache=trace ~/.mozbuild/sccache/sccache-dist server --config ~/.config/sccache/server.conf`` 171 (or similar). *NOTE* ``sudo`` *must* come before setting environment variables 172 for this to work. *NOTE* use ``RUST_LOG`` instead of ``SCCACHE_LOG`` if your 173 build of ``sccache`` does not include `pull request 822 174 <https://github.com/mozilla/sccache/pull/822>`_. (``sccache`` binaries from 175 ``mach bootstrap`` do include this PR.) 176 177 178 Common questions/considerations 179 =============================== 180 181 * My build is still slow: scache-dist can only do so much with parts of the 182 build that aren't able to be parallelized. To start debugging a slow build, 183 ensure the "Successful distributed compilations" line in the output of 184 ``sccache -s`` dominates other counts. For a full build, at least a 2-3x 185 improvement should be observed. 186 187 * My build output is incomprehensible due to a flood of warnings: clang will 188 treat some warnings differently when it's fed preprocessed code in a separate 189 invocation (preprocessing occurs locally with sccache-dist). Adding 190 ``rewrite_includes_only = true`` to the ``dist`` section of your client config 191 will improve this; however, setting this will cause build failures with a 192 commonly deployed version of ``glibc``. This option will default to ``true`` 193 once the fix is more widely available. Details of this fix can be found in 194 `this patch <https://sourceware.org/ml/libc-alpha/2019-11/msg00431.html>`_. 195 196 * My build fails with a message about incompatible versions of rustc between 197 dependent crates: if you're using a custom toolchain check that the version 198 of rustc in your ``rustc-dist-toolchain.tar.xz`` is the same as the version 199 you're running locally.