SKIP-FAILS.txt (16567B)
1 Design Specification for ./mach manifest skip-fails 2 =================================================== 3 4 The first of the mach manifest subcommands is skip-fails. This command 5 can be used to automatically edit manifests to skip tests that are 6 failing as well as file the corresponding bugs for the failures. This 7 is particularly useful when “greening up” a new platform. 8 9 The user documentation for skip-fails is here: 10 https://firefox-source-docs.mozilla.org/mozbase/manifestparser.html#using-mach-manifest-skip-fails 11 12 The skip-fails command knows how to manage failures for TOML, WPT 13 and REFTEST manifests. Additional developer design notes are included here. 14 15 16 Running skip-fails for TOML 17 --------------------------- 18 19 The user documentation for TOML manifests is: 20 https://firefox-source-docs.mozilla.org/mozbase/manifestparser.html 21 https://firefox-source-docs.mozilla.org/testing/tests-for-new-config/index.html#run-skip-fails 22 23 Usage 24 ----- 25 26 $ ./mach manifest skip-fails -h 27 usage: mach [global arguments] manifest skip-fails [command arguments] 28 29 Global Arguments: 30 31 -v, --verbose Print verbose output. 32 --quiet Don't print as much output. 33 -l, --log-file FILENAME 34 Filename to write log data to. 35 --log-interval Prefix log line with interval from last message rather 36 than relative time. Note that this is NOT execution 37 time if there are parallel operations. 38 --no-interactive Automatically selects the default option on any 39 interactive prompts. If the output is not a terminal, 40 then --no-interactive is assumed. 41 --log-no-times Do not prefix log lines with times. By default, mach 42 will prefix each output line with the time since 43 command start. 44 -h, --help Show this help message. 45 --debug-command Start a Python debugger when command is dispatched. 46 --profile-command Capture a Python profile of the mach process as command 47 is dispatched. 48 --settings FILENAME Path to settings file. 49 50 Sub Command Arguments: 51 try_url Treeherder URL for try (please use quotes) 52 -b, --bugzilla BUGZILLA 53 Bugzilla instance [disable] 54 -C, --clear-cache [CLEAR_CACHE] 55 clear cache REVISION (or all) 56 -c, --carryover Set carryover mode (only skip failures for platform 57 matches) 58 -d, --dry-run Determine manifest changes, but do not write them 59 -F, --use-failures USE_FAILURES 60 Use failures from file 61 -f, --save-failures SAVE_FAILURES 62 Save failures to file 63 -I, --implicit-vars Use implicit variables in reftest manifests 64 -i, --task-id TASK_ID 65 Task id to write a condition for instead of all tasks 66 from the push 67 -k, --known-intermittents 68 Set known intermittents mode (only skip failures known 69 intermittents) 70 -M, --max-failures MAX_FAILURES 71 Maximum number of failures to skip (-1 == no limit) 72 -m, --meta-bug-id META_BUG_ID 73 Meta Bug id 74 -n, --new-version NEW_VERSION 75 New version to use for annotations 76 -N, --new-failures Set new failures mode (only add conditions for new 77 failures) 78 -r, --failure-ratio FAILURE_RATIO 79 Ratio of test failures/total to skip [0.4] 80 -R, --replace-tbd Replace Bug TBD in manifests by filing new bugs 81 -s, --turbo Skip all secondary failures 82 -T, --use-tasks USE_TASKS 83 Use tasks from file 84 -t, --save-tasks SAVE_TASKS 85 Save tasks to file 86 -u, --user-agent USER_AGENT 87 User-Agent to use for mozci if queries are forbidden 88 from treeherder 89 -v, --verbose Verbose mode 90 91 92 Design 93 ------ 94 95 1. Classify the failure based on the following: 96 a. If less than 3 runs, classify "unknown" 97 b. If zero failures, classify "success" 98 c. If the failure ratio of failed runs / total runs is less than 40% 99 classify "intermittent" 100 d. Else edit the manifest to skip this failure 101 102 2. Manifest variable information 103 a. Variables used in Python are set: 104 https://searchfox.org/firefox-main/source/testing/mozbase/mozinfo/mozinfo/platforminfo.py 105 106 NOTE: that the task metadata runtimes are turned in to test variants by looking 107 up the "mozinfo" field that corresponds to the "runtime" key in 108 taskcluster/test_configs/variants.yml 109 110 NOTE: special excpetions are these runtimes are mapped to these variants: 111 - no-fission -> !fission 112 - 1proc -> !e10s 113 114 NOTE: multiple variants are stored in the field test_variant concatenated by '+', 115 for example: "socketprocess_networking+!fission" 116 117 b. Variables used in JavaScript are set: 118 https://searchfox.org/firefox-main/source/layout/tools/reftest/manifest.sys.mjs 119 120 c. Use arch instead of processor 121 https://searchfox.org/firefox-main/source/testing/mozbase/mozinfo/mozinfo/mozinfo.py#170 122 123 aarch64 124 arm7 125 x86 126 x86_64 127 128 d. Build types 129 An an optimized build "opt" implies none of these build types: "debug", "ccov", "asan", "tsan". 130 A build type "!debug" implies "opt" 131 132 e. A "platform match" for the purpose of bug carryover means 133 - "os" is not specified in the existing condition or "os" is the same, and 134 - The existing build type is not present and the "os" explictly the same, or 135 - The existing build type is present: 136 + "opt" (or "!debug") in both existing skip-if and proposed skip-if 137 + "!opt" or one of ["debug", "asan", "tsan", "ccov" ] in both 138 (covers the case of "debug" in both) 139 EXCEPT does not match if one is "debug" and the other is 140 in ["asan", "tsan", "ccov"] 141 - Else not a match 142 143 f. If the argument `-n, --new-version NEW_VERSION` is used 144 then the os_version used in any skip-if expression 145 will be NEW_VERSION -- regardless of the os, arch or 146 another actual platform information. This is intended to workaround a bug 147 where task metadata would report "macosx 10.15", yet the worker was actually 148 "macosx 14.70". By setting `--new-version 14.70` then any skip-if 149 expression will be created with `os_version == 14.70` which would then match 150 what mozinfo says on the next try run. 151 152 3. Bugzilla interaction 153 a. The bugzilla instance will be set to (in increasing precedence): 154 i. Default: "bugzilla.allizom.org" 155 ii. From the environment variable BUGZILLA 156 iii. From the command line arg --bugzilla 157 158 b. NOTE: if the BUGZILLA env var or command line arg is "disable" 159 then Bugzilla interaction will be disabled. 160 161 c. Setting --dry-run prevents editing manifests, but will not disable reading Bugzilla. 162 Nevertheless no new bugs will be created nor comments added. 163 Note existing bugs may be mentioned in an info message as appropriate. 164 165 d. If --meta-bug-id is NOT set no bugs will be filed. Any created manifest bugs will 166 require referencing a meta-bug-id. 167 168 e. If there is a skip line for this manifest and the platform matches (see above) then 169 add an info message about carryover 170 do NOT edit any existing bug nor create a new bug 171 If there is an existing manifest comment, then carry it over to the new skip line 172 Else do NOT add a new comment (as a comment did not exist before) 173 Else (no skip lines match the platform) 174 Add a new skip-if line and reference the bug for this manifest + meta-bug-id 175 (create a new bug if needed) 176 177 f. Special case for carryover when a [default] section exists 178 and contains a skip-if with a platform match 179 AND our current failure matches that skip-if line for a real manifest 180 THEN carry over the comment from [default] to the new skip-if line. 181 182 4. Modes for skip-fails 183 a. Normal mode 184 Tests with failures found based on the above heuristics 185 are disabled with skip-if conditions in the appropriate manifests 186 and bugs filed (or updated). 187 188 b. Carry-over mode 189 Only consider adding skip-if conditions which match the 190 platform (see above) of previous conditions. Does not perform any 191 bugzilla changes. Requires meta-bug-id ( future actions will be cached in 192 {topsrcdir}/.skip_fails_cache/meta-bug-{meta_bug_id}/actions.json ) 193 194 c. Known intermittents mode 195 Only consider adding skip-if conditions which have known 196 intermittent bugs (and are not carryover bugs). 197 form (see above) of previous conditions. Does not perform any 198 bugzilla changes. Requires meta-bug-id ( future actions will be cached in 199 {topsrcdir}/.skip_fails_cache/meta-bug-{meta_bug_id}/actions.json ) 200 201 For each failure, if the job_id can be determined then bug_suggestions will 202 be retrieved from 203 https://treeherder.mozilla.org/api/project/{repo}/jobs/{job_id}/bug_suggestions/ 204 each object with "search" field of ^PROCESS-CRASH or ^TEST-UNEXPECTED 205 will be considered if one of the bugs in 206 bugs.open_recent.<i>.summary ends with 'single tracking bug$' 207 then 208 bugs.open_recent.<i>.id has the bug id 209 210 d. New failures mode 211 Will only edit manifest skip-if conditions for new failures 212 (i.e. not carryover nor known intermittents). Will record 213 bug as "TBD". Requires meta-bug-id ( future actions will be cached in 214 {topsrcdir}/.skip_fails_cache/meta-bug-{meta_bug_id}/actions.json ) 215 216 e. Replace TBD mode 217 Will only edit manifest skip-if conditions for new failures 218 by filing new bugs and replacing TBD with actual bug number. 219 Requires meta-bug-id ( new bug actions will be read from 220 {topsrcdir}/.skip_fails_cache/meta-bug-{meta_bug_id}/actions.json ) 221 222 5. Cache for skip-fails 223 At the top of the source tree is a cache directory for skip-fails: 224 ".skip_fails_cache". This directory will be created if it is not present. 225 When running skip-fails if --use-tasks has not been set then 226 the cache directory will be searched for a tasks file corresponding 227 the revision (i.e. ".skip_fails_cache/REVISION/tasks.json"). If 228 that file is present then it is used. Similarly for --use-failures. 229 If --save-tasks has not been set then, by default, the tasks will 230 be saved in the cache directory. Similarly for --save-failures. 231 Any specific cache directory can be cleared with --clear-cache REVISION. 232 All cache directories can be deleted with --clear-cache (no argument). 233 Any cache directories which are over 45 days old will be automatically 234 cleared. 235 236 As design changes for skip-fails / TOML, notes will be added here. 237 238 239 Running skip-fails for WPT 240 --------------------------- 241 242 The user documentation for WPT manifests is: 243 https://firefox-source-docs.mozilla.org/web-platform/index.html 244 245 1. Classify the failure based on the following: 246 a. If less than 3 runs, classify "unknown" 247 b. If zero failures, classify "success" 248 c. If the ratio of failed runs / total runs is less than 40% 249 classify "intermittent" 250 d. Else edit the manifest to skip this failure 251 252 As design changes for skip-fails / WPT, notes will be added here. 253 254 255 Running skip-fails for REFTEST 256 ------------------------------ 257 258 The user documentation for REFTEST manifests is: 259 https://firefox-source-docs.mozilla.org/layout/Reftest.html 260 261 Current Design notes for skip-fails / REFTEST: 262 263 As skip-fails for REFTEST finds a failure it will take the following actions: 264 265 1. Classify the failure based on the following: 266 a. If less than 3 runs, classify "unknown" 267 b. If zero failures, classify "success" 268 c. If the ratio of failed runs / total runs is less than 40% 269 classify "low frequency intermittent" (INTERMITTENT) 270 c. If the ratio of failed runs / total runs is less than 80% 271 classify "high frequency intermittent" (DISABLE_INTERMITTENT) 272 d. If the ratio is <= 100%, classify "failure" (DISABLE_FAILURE) 273 274 2. Propose a new range (differences, pixels) 275 a. For "low frequency intermittent" errors do nothing 276 b. For "high frequency intermittent" set the range 0-Y (i.e. always 277 start at zero to avoid UNEXPECTED PASS) 278 c. For "failure" set the range X-Y (i.e. the range as reported in 279 reftest_errorsummary.log). If the failure is UNEXPECTED PASS 280 then set the range to 0-Y. 281 d. NOTE: The maximum for differences and pixels will be increased 282 by 5% over the reported values in reftest_errorsummary.log 283 e. NOTE: If for whatever reason the range is calculated to be 0-0 then 284 remove the fuzzy-if entirely 285 286 3. Merge with an existing fuzzy-if for this OS platform if the other 287 criteria differ in only one dimension. Current dimensions include: 288 289 OS: gtkWidget, winWidget, cocoaWidget, Android 290 build_type: optimized, isDebugBuild, isCoverageBuild, AddressSanitizer, ThreadSanitizer 291 is64Bit 292 fission 293 useDrawSnapshot 294 swgl 295 <OTHER> 296 297 4. Based on a command line switch choose do use implicit variables (or not). 298 a. The following variables can implicit values: 299 is64Bit: true 300 fission: true 301 useDrawSnapshot: false 302 swgl: false 303 If the value of an implicit variable matches it's default then it may be 304 elided from the conditional expression. 305 If the value of an implicit variable is the oppposite of it's default 306 value then it may be specified in the condition, e.g. !is64Bit 307 When merging an existing condition with a new failure condition where 308 the one dimension of difference is an implicit variable then the 309 condition must include the disjunction of the variable, e.g. (is64Bit&&!is64Bit) 310 That is essential so that the reftest sandbox does not add the implicit 311 variable and cover only one of the two possible values. 312 b. When implicit variables are not used then the following heuristics apply: 313 The variable is64Bit will be elided for all platforms except for winWidget. 314 The variable useDrawSnapshot will be elided for all platforms except gtkWidget. 315 316 5. Position of fuzzy-if conditions 317 In reftest manifest the rightmost applicable condition is used. 318 319 a. First sorting criterion is based on OS in this order: 320 OSES = ["Android", "cocoaWidget", "appleSilicon", "gtkWidget", "winWidget"] 321 This is because appleSilicon is a narrower specification than 322 cocoaWidget and would be ignored if not to the right of cocoaWidget. 323 324 b. The second sorting criterion is the number of dimensions that participate 325 in the expression (including implicits) from least dimensions to 326 greatest dimensions. 327 328 The last fuzzy-if will be placed on the line immediately to the left of the 329 "type" (e.g. == or !=) (or to the left of an optional "HTTP" expression, if present). 330 Non fuzzy-if statements will appear before the fuzzy-if statements. 331 In this way the more generic conditions will only be superceded by very specific 332 conditions. 333 334 335 Design Specification for ./mach manifest clean-skip-fails 336 ========================================================= 337 338 The purpose / rationale for this command is to-be-documented. 339 340 description="Update manifests to remove skip-if conditions for a specific platform. 341 Only works for TOML manifests.", 342 343 The user documentation is: 344 https://firefox-source-docs.mozilla.org/... 345 346 Usage 347 ----- 348 Sub Command Arguments: 349 manifest_search_path Path to the folder containing the manifests to update, 350 or the path to a single manifest 351 -o, --os OS_NAME OS to remove (linux, mac, win) 352 -s, --os_version OS_VERSION 353 Version of the OS to remove (eg: 18.04 for linux) 354 -p, --processor PROCESSOR 355 Type of processor architecture to remove (eg: x86) 356 357 358 Design Specification for ./mach manifest high-freq-skip-fails 359 ========================================================= 360 361 The purpose / rationale for this command is to-be-documented. 362 363 description="Update manifests to skip failing tests", 364 365 The user documentation is: 366 https://firefox-source-docs.mozilla.org/... 367 368 Usage 369 ----- 370 Sub Command Arguments: 371 -f, --failures FAILURES 372 Minimum number of failures for the bug to be skipped 373 -d, --days DAYS Number of days to look for failures since now