index.rst (5410B)
1 Turning on Firefox tests for a new configuration 2 ================================================ 3 4 You are ready to go with turning on Firefox tests for a new config. Once you 5 get to this stage, you will have seen a try push with all the tests running 6 (many not green) to verify some tests pass and there are enough machines 7 available to run tests. 8 9 For the purpose of this document, assume you are tasked with upgrading Windows 10 10 OS from version 1803 -> 1903. To simplify this we can call this `windows_1903`, 11 and we need to: 12 13 * create meta bug 14 * push to try 15 * run skip-fails 16 * repeat 2 more times 17 * land changes and turn on tests 18 * turn on run only failures 19 20 If you are running this manually or on configs/tests that are not supported with 21 `./mach try --new-test-config`, then please follow the steps `here <manual.html>`__ 22 23 24 Create Meta Bug 25 --------------- 26 27 This is a simple step where you create a meta bug to track the failures associated 28 with the tests you are greening up. If this is a test suite (i.e. `devtools`), it 29 is ok to have a meta bug just for the test suite and the new platform. 30 31 All bugs related to tests skipped or failing will be blocking this meta bug. 32 33 Push to Try Server 34 ------------------ 35 36 Now that you have a configuration setup and machines available via try server, it 37 is time to run try. If you are migrating mochitest or xpcshell, then you can do: 38 39 ``./mach try fuzzy --no-artifact --full --rebuild 10 --new-test-config -q 'test-windows10-64-1903 mochitest-browser-chrome !ccov !ship !browsertime !talos !asan'`` 40 41 This will run many tests (thanks to --full and --rebuild 10), but will give plenty 42 of useful data. 43 44 In the scenario you are migrating tests such as: 45 * performance 46 * web-platform-tests 47 * reftest / crashtest / jsreftest 48 * mochitest-webgl (has a different process for test skipping) 49 * cppunittest / gtest / junit 50 * marionette / firefox-ui / telemetry 51 52 then please follow the steps `here <manual.html>`__ 53 54 If you are migrating to a small machine pool, it is best to avoid `--rebuild 10` and 55 instead do `--rebuild 3`. Likewise please limit your jobs to be the specific test 56 suite and variant. The size of a worker pool is shown at the Workers page of the 57 Taskcluster instance. 58 59 Run skip-fails 60 -------------- 61 62 When the try push is completed it is time to run skip-fails. Skip-fails will 63 look at all the test results and automatically create a set of local changes 64 with skip-if conditions to green up the tests faster. 65 66 ``./mach manifest skip-fails --b bugzilla.mozilla.org -m <meta_bug_id> --turbo "https://treeherder.mozilla.org/jobs?repo=try&revision=<rev>"`` 67 68 Please input the proper `meta_bug_id` and `rev` into the above command. 69 70 The first time running this, you will need to get a `bugzilla api key <https://bugzilla.mozilla.org/userprefs.cgi?tab=apikey>`__. copy 71 this key and add it to your `~/.config/python-bugzilla/bugzillarc` file: 72 73 .. code-block:: none 74 75 cat bugzillarc 76 [DEFAULT] 77 url = https://bugzilla.mozilla.org 78 [bugzilla.mozilla.org] 79 api_key = <key> 80 81 When the command finishes, you will have new bugs created that are blocking the 82 meta bug. In addition you will have many changes to manifests adding skip-if 83 conditions. For tests than fail 40% of the time or for entire manifests that 84 take >20 minutes to run on opt or >40 minutes on debug. 85 86 You will need to create a commit (or `--amend` your previous commit if this is round 2 or 3): 87 88 ``hg commit -m "Bug <meta_bug_id> - Green up tests for <suite> on <platform>"`` 89 90 91 Repeat 2 More Times 92 ------------------- 93 94 In 3 rounds this should be complete and ready to submit for review and turn on 95 the new tests. 96 97 There will be additional failures, those will follow the normal process of 98 intermittents. 99 100 101 Land Changes and Turn on Tests 102 ------------------------------ 103 104 After you have a green test run, it is time to land the patches. There could 105 be changes needed to the taskgraph in order to add the new hardware type and 106 duplicate tests to run on both the old and the new, or create a new variant and 107 denote which tests to run on that variant. 108 109 Using our example of ``windows_1903``, this would be a new worker type that 110 would require these edits: 111 112 * `transforms/tests.py <https://searchfox.org/mozilla-central/source/taskcluster/taskgraph/transforms/tests.py#97>`__ (duplicate windows 10 entries) 113 * `test-platforms.py <https://searchfox.org/mozilla-central/source/taskcluster/kinds/test/test-platforms.yml#229>`__ (copy windows10 debug/opt/shippable/asan entries and make win10_1903) 114 * `test-sets.py <https://searchfox.org/mozilla-central/source/taskcluster/kinds/test/test-sets.yml#293>`__ (ideally you need nothing, otherwise copy ``windows-tests`` and edit the test list) 115 116 In general this should allow you to have tests scheduled with no custom flags 117 in try server and all of these will be scheduled by default on 118 ``mozilla-central``, ``autoland``, and ``release-branches``. 119 120 Turn on Run Only Failures 121 ------------------------- 122 123 Now that we have tests running regularly, the next step is to take all the 124 disabled tests and run them in the special failures job. 125 126 We have a basic framework created, but for every test harness (i.e. xpcshell, 127 mochitest-gpu, browser-chrome, devtools, web-platform-tests, crashtest, etc.), 128 there will need to be a corresponding tier-3 job that is created. 129 130 TODO: point to examples of how to add this after we get our first jobs running.