index.rst (13685B)
1 ================================= 2 Using third-party Python packages 3 ================================= 4 5 Mach and its associated commands have a variety of 3rd-party Python dependencies. Many of these 6 are vendored in ``third_party/python``, while others are installed at runtime via ``pip``. 7 8 The dependencies of Mach itself can be found at ``python/sites/mach.txt``. Mach commands 9 may have additional dependencies which are specified at ``python/sites/<site>.txt``. 10 11 For example, the following Mach command would have its 3rd-party dependencies declared at 12 ``python/sites/foo.txt``. 13 14 .. code:: python 15 16 @Command( 17 "foo-it", 18 virtualenv_name="foo", 19 ) 20 # ... 21 def foo_it_command(): 22 import specific_dependency 23 24 The format of ``<site>.txt`` files are documented further in the 25 :py:class:`~mach.requirements.MachEnvRequirements` class. 26 27 Adding a Python package 28 ======================= 29 30 There's two ways of using 3rd-party Python dependencies: 31 32 * :ref:`pip install the packages <python-pip-install>`. Python dependencies with native code must 33 be installed using ``pip``. This is the recommended technique for adding new Python dependencies. 34 * :ref:`Vendor the source of the Python package in-tree <python-vendor>`. Dependencies of the Mach 35 core logic or of building Firefox itself must be vendored. 36 37 .. note:: 38 39 For dependencies that meet both restrictions (dependency of Mach/build, *and* has 40 native code), see the :ref:`mach-and-build-native-dependencies` section below. 41 42 .. _python-pip-install: 43 44 ``pip install`` the package 45 ~~~~~~~~~~~~~~~~~~~~~~~~~~~ 46 47 To add a ``pip install``-d package dependency, add it to your site's 48 ``python/sites/<site>.txt`` manifest file: 49 50 .. code:: text 51 52 ... 53 pypi:new-package==<version> 54 55 If you'd like to lock dependencies and validate hashes, you can alternatively specify a path 56 to a ``requirements.txt`` file: 57 58 .. code:: text 59 60 ... 61 requirements-txt:path/to/requirements.txt 62 63 The ``requirements.txt`` file can be generated using any tool you like, but it must include 64 hashes for all listed packages. 65 66 .. note:: 67 68 Some tasks are not permitted to use external resources, and for those we can 69 publish packages to an internal PyPI mirror. 70 See `how to upload to internal PyPI <https://wiki.mozilla.org/ReleaseEngineering/How_To/Upload_to_internal_Pypi>`_ 71 for more details. 72 73 .. _python-vendor: 74 75 Vendoring Python packages 76 ~~~~~~~~~~~~~~~~~~~~~~~~~ 77 78 To vendor a Python package run ``./mach vendor python --add 79 <package>~=<major>.<minor>``. This will add your dependency to 80 ``third_party/python/pyproject.toml`` then begin the re-vendoring process 81 for all dependencies. The `pyproject.toml` is used by ``uv`` to create a 82 lockfile (``uv.lock``) that ensures all the dependencies are compatible. 83 This lockfile is then used to generate a ``third_party/python/requirements.txt`` 84 which is then used by ``pip`` to download all dependencies into the 85 ``third_party/python`` directory. 86 87 .. note:: 88 The dependency you are attempting to add may not be compatible with what's 89 already vendored. In this case, the lockfile generation/dependency 90 resolution will fail with an error message along the lines of ``No 91 solution found when resolving dependencies:``. You may be able to get 92 around this by pinning your dependency to a newer or older version. If 93 that doesn't work you can try modifying the pin(s) of the already vendored 94 dependency(ies) that are causing the conflict(s). 95 96 Beware that this is a rather painful process. Changing the version of an 97 already vendored dependency may break functionality somewhere in the codebase. 98 This means that even if you get ``uv`` to make a compatible lockfile, you 99 may have caused a breakage somewhere else that ``uv`` cannot foresee. It is 100 your responsibility to fix anything you break, otherwise your changes will be 101 rejected or backed out if the issue isn't discovered until after landing. 102 103 If you change pins for packages to workaround issues, please add comments in the 104 ``third_party/python/pyproject.toml`` for each necessary pin indicating why it's 105 needed and which dependency(ies) need it. Doing so will make it much easier for 106 the next person that comes along trying to do the same thing. 107 108 After the ``./mach vendor python`` completes successfully, you'll need to add that package 109 and any new transitive dependencies (you'll see them added in ``third_party/python/requirements.txt``) 110 to the associated site's dependency manifest in ``python/sites/<site>.txt``: 111 112 .. code:: text 113 114 ... 115 vendored:third_party/python/new-package 116 vendored:third_party/python/new-package-dependency-foo 117 vendored:third_party/python/new-package-dependency-bar 118 ... 119 120 To remove a vendored package run ``./mach vendor python --remove <package>``. This re-creates the lockfile 121 with that dependency removed (along with any transitive dependencies that aren't shared) and re-vendor 122 everything. 123 124 .. note:: 125 - You can add or remove multiple packages at the same time: ``./mach vendor python --add <package_one> --add <package_two>`` 126 - If desired, you can add/remove dependencies manually in the ``third_party/python/pyproject.toml``. Once you've made your changes, just run ``./mach vendor python`` without the ``--add`` and/or ``--remove`` arguments. 127 128 After the ``./mach vendor python`` completes successfully you'll need to remove the package and transitive 129 dependencies from all the site manifest files (``python/sites/<site>.txt``) that used the removed package(s). 130 131 .. note:: 132 133 The following policy applies to **ALL** vendored packages: 134 135 * Vendored PyPI libraries **MUST NOT** be modified 136 * Vendored libraries **SHOULD** be released copies of libraries available on 137 PyPI. 138 139 * When considering manually vendoring a package, discuss the situation with 140 the ``#build`` team to ensure that other, more maintainable options are exhausted. 141 142 .. note:: 143 144 We require that it is possible to build Firefox using only a checkout of the source, 145 without depending on a package index. This ensures that building Firefox is 146 deterministic and dependable, avoids packages from changing out from under us, 147 and means we’re not affected when 3rd party services are offline. We don't want a 148 DoS against PyPI or a random package maintainer removing an old tarball to delay 149 a Firefox chemspill. Therefore, packages required by Mach core logic or for building 150 Firefox itself must be vendored. 151 152 If the vendored dependencies in the ``third_party/python/pyproject.toml`` are not pinned with 153 ``==``, they can be automatically upgraded. You can upgrade either a single package, 154 or all packages. 155 156 To upgrade an individual unpinned package just run ``./mach vendor python --upgrade-package <package>``. You can also update 157 multiple specific packages at the same time: ``./mach vendor python --upgrade-package <package_one> --upgrade-package <package_two>`` 158 159 To upgrade all unpinned packages just run ``./mach vendor python --upgrade``. 160 161 For both cases the process is essentially the same. ``uv`` is invoked and it will determine if there is/are 162 newer versions available. If there aren't any compatible upgrades available then nothing will be vendored. If 163 there are, then everything will be re-vendored. 164 165 .. note:: 166 If an upgrade adds new transitive dependencies, you will need to add them to the site(s) manifest files 167 (the same as you need to when adding a new package). 168 169 By default ``./mach vendor python`` only fully runs if changes are detected in the ``uv.lock`` file. If you 170 want to force the full vendor to run, just add ``--force``. 171 172 173 If the package contains optional native dependencies, they won't be compiled as 174 part of the vendored package. It is however possible to prefer the pypi version 175 which may contain the native bits, while allowing to fallback to the vendored 176 version: 177 178 .. code:: text 179 180 ... 181 vendored-fallback:pypi-package-name:third_party/python/new-package:explanation 182 ... 183 184 185 .. _mach-and-build-native-dependencies: 186 187 Mach/Build Native 3rd-party Dependencies 188 ======================================== 189 190 There are cases where Firefox is built without being able to ``pip install``, but where 191 native 3rd party Python dependencies enable optional functionality. This can't be solved 192 by vendoring the platform-specific libraries, as then each one would have to be stored 193 multiple times in-tree according to how many platforms we wish to support. 194 195 Instead, this is solved by pre-installing such native packages onto the host system 196 in advance, then having Mach attempt to use such packages directly from the system. 197 This feature is only viable in very specific environments, as the system Python packages 198 have to be compatible with Mach's vendored packages. 199 200 .. note: 201 202 All of these native build-specific dependencies **MUST** be optional requirements 203 as to support the "no strings attached" builds that only use vendored packages. 204 205 To control this behaviour, the ``MACH_BUILD_PYTHON_NATIVE_PACKAGE_SOURCE`` environment 206 variable can be used: 207 208 .. list-table:: ``MACH_BUILD_PYTHON_NATIVE_PACKAGE_SOURCE`` 209 :header-rows: 1 210 211 * - ``MACH_BUILD_PYTHON_NATIVE_PACKAGE_SOURCE`` 212 - Behaviour 213 * - ``"pip"`` 214 - Mach will ``pip install`` all needed dependencies from PyPI at runtime into a Python 215 virtual environment that's reused in future Mach invocations. 216 * - ``"none"`` 217 - Mach will perform the build using only vendored packages. No Python virtual environment 218 will be created for Mach. 219 * - ``"system"`` 220 - Mach will use the host system's Python packages as part of doing the build. This option 221 allows the usage of native Python packages without leaning on a ``pip install`` at 222 build-time. This is generally slower because the system Python packages have to 223 be asserted to be compatible with Mach. Additionally, dependency lockfiles are ignored, 224 so there's higher risk of breakage. Finally, as with ``"none"``, no Python virtualenv 225 environment is created for Mach. 226 * - ``<unset>`` 227 - Same behaviour as ``"pip"`` if ``MOZ_AUTOMATION`` isn't set. Otherwise, uses 228 the same behaviour as ``"none"``. 229 230 There's a couple restrictions here: 231 232 * ``MACH_BUILD_PYTHON_NATIVE_PACKAGE_SOURCE`` only applies to the top-level ``"mach"`` site, 233 the ``"common"`` site and the ``"build"`` site. All other sites will use ``pip install`` at 234 run-time as needed. 235 236 * ``MACH_BUILD_PYTHON_NATIVE_PACKAGE_SOURCE="system"`` is not allowed when using any site other 237 than ``"mach"``, ``"common"`` or ``"build"``, because: 238 239 * As described in :ref:`package-compatibility` below, packages used by Mach are still 240 in scope when commands are run, and 241 * The host system is practically guaranteed to be incompatible with commands' dependency 242 lockfiles. 243 244 The ``MACH_BUILD_PYTHON_NATIVE_PACKAGE_SOURCE`` environment variable fits into the following use 245 cases: 246 247 Mozilla CI Builds 248 ~~~~~~~~~~~~~~~~~ 249 250 We need access to the native packages of ``zstandard`` and ``psutil`` to extract archives and 251 get OS information respectively. Use ``MACH_BUILD_PYTHON_NATIVE_PACKAGE_SOURCE="system"``. 252 253 Mozilla CI non-Build Tasks 254 ~~~~~~~~~~~~~~~~~~~~~~~~~~ 255 256 We generally don't want to create a Mach virtual environment to avoid redundant processing, 257 but it's ok to ``pip install`` for specific command sites as needed, so leave 258 ``MACH_BUILD_PYTHON_NATIVE_PACKAGE_SOURCE`` unset (``MOZ_AUTOMATION`` implies the default 259 behaviour of ``MACH_BUILD_PYTHON_NATIVE_PACKAGE_SOURCE="none"``). 260 261 In cases where native packages *are* needed by Mach, use 262 ``MACH_BUILD_PYTHON_NATIVE_PACKAGE_SOURCE="pip"``. 263 264 Downstream CI Builds 265 ~~~~~~~~~~~~~~~~~~~~ 266 267 Sometimes these builds happen in sandboxed, network-less environments, and usually these builds 268 don't need any of the behaviour enabled by installing native Python dependencies. 269 Use ``MACH_BUILD_PYTHON_NATIVE_PACKAGE_SOURCE="none"``. 270 271 Gentoo Builds 272 ~~~~~~~~~~~~~ 273 274 When installing Firefox via the package manager, Gentoo generally builds it from source rather than 275 distributing a compiled binary artifact. Accordingly, users doing a build of Firefox in this 276 context don't want stray files created in ``~/.mozbuild`` or unnecessary ``pip install`` calls. 277 Use ``MACH_BUILD_PYTHON_NATIVE_PACKAGE_SOURCE="none"``. 278 279 Firefox Developers 280 ~~~~~~~~~~~~~~~~~~ 281 282 Leave ``MACH_BUILD_PYTHON_NATIVE_PACKAGE_SOURCE`` unset so that all Mach commands can be run, 283 Python dependency lockfiles are respected, and optional behaviour is enabled by installing 284 native packages. 285 286 .. _package-compatibility: 287 288 Package compatibility 289 ===================== 290 291 Mach requires that all commands' package requirements be compatible with those of Mach itself. 292 (This is because functions and state created by Mach are still usable from within the commands, and 293 they may still need access to their associated 3rd-party modules). 294 295 However, it is OK for Mach commands to have package requirements which are incompatible with each 296 other. This allows the flexibility for some Mach commands to depend on modern dependencies while 297 other, more mature commands may still only be compatible with a much older version. 298 299 .. note:: 300 301 Only one version of a package may be vendored at any given time. If two Mach commands need to 302 have conflicting packages, then at least one of them must ``pip install`` the package instead 303 of vendoring. 304 305 If a Mach command's dependency conflicts with a vendored package, and that vendored package 306 isn't needed by Mach itself, then that vendored dependency should be moved from 307 ``python/sites/mach.txt`` to its associated environment.