index.rst (6377B)
1 Understanding Treeherder Results 2 ================================ 3 4 `Treeherder <https://treeherder.mozilla.org/userguide>`__ serves as the primary dashboard for developers looking to view CI test results from their try pushes. The main purpose of the dashboard is to display all the tasks and their status along with tool to view logs, see failures, add jobs, cancel jobs, or connect to other tools. 5 6 When a test fails, it is important to figure out if this is a regression or an intermittent failure. It is typically to see 3-10% of the jobs fail, although the majority of the failures are intermittent (infrastructure failed, timing is off, test is sort of flaky, etc.). Sometimes a regression occurs but the failure is assumed intermittent. 7 8 There are a few tools to use when a test fails: 9 * Confirm Failure (CF) 10 * Retrigger (R) 11 * NEW annotation (NEW) 12 * MITTEN icon 13 * Bug Suggestions 14 15 The quick answer is: use confirm failure for all failures and when all results are in, whichever tests are orange and do not have a mitten icon are what need further investigation. 16 17 It is best to understand how each of the tools work and use them in combination to be most efficient with your time and to find a regression. 18 19 Confirm Failure (CF) 20 -------------------- 21 This tool will give a strong signal when it is applicable. Confirm failure works by running the failing test path 10x in the same browser. We have found that this is the strongest signal compared to other methods, but there are limitations: 22 * This requires a test failure that is discoverable in the failure line. 23 * This requires the failure to be in a supported test harness (web-platform-test, mochitest*, reftest, xpcshell). 24 * Some exceptions exist around specific hardware (e.g. android reftests) or required machines. 25 * Running this can result in infrastructure failure. 26 * Some specific tests do not work well when run in this method. 27 * This launches a CF task for every failure (up to 10) discovered in the failing task, so often you end up with >1 CF job. 28 29 When this runs, a new task is scheduled and typically only takes a few minutes to complete (a retrigger can take 15-90 minutes depending on the task). If you run confirm failure on a failure and there is a lack of test path, unsupported harness or other well known limitations, a retrigger will automatically be scheduled. 30 31 To launch confirm failure you need to select the task 32 33 .. image:: img/th_select_task.png 34 :width: 300 35 36 37 then click the task action menu 38 39 .. image:: img/th_task_action.png 40 :width: 300 41 42 43 and finally select "Confirm Test Failures" 44 45 .. image:: img/th_confirm_failures.png 46 :width: 300 47 48 49 When the jobs are done, Treeherder will determine if the CF job passes and add a |MITTEN| icon to the original failure if it is intermittent. 50 51 In the future we are planning to run confirm failure automatically on NEW failures only. 52 53 54 Retrigger (R) 55 ------------- 56 When a retrigger happens the entire task is rerun. This sounds straightforward if a test fails once, rerun it and if it passes it is intermittent. In reality that isn't the strongest signal, but sometimes it is the best signal. Here are some limitations: 57 * a retrigger has infrastructure failures 58 * a retrigger has a different set of test failures (maybe not even running your test) 59 * a retrigger runs other tests that could influence the state of the browser/machine causing uncertain results 60 * a retrigger can take a long time (many tasks run >30 minutes) 61 62 Given these limitations, a retrigger is a useful tool for gathering more information, just be prepared to double check the logs and failures in more detail if the task doesn't succeed. 63 64 To launch a Retrigger, you can: 65 select a task: 66 67 .. image:: img/th_select_task.png 68 :width: 300 69 70 71 click the rotating arrow icon in the task action bar: 72 73 .. image:: img/th_retrigger.png 74 :width: 300 75 76 77 78 OR type the 'r' key on your keyboard 79 80 If a task is retriggered and it is green, the original failing task will have a |MITTEN| icon. 81 82 83 NEW annotations (NEW) 84 --------------------- 85 Treeherder keeps a cache of every failure line seen on Autoland and Mozilla-Central for the last 3 weeks. When a new failure line shows up it is flagged in the task failure summary with a NEW tag. The NEW tag is very successful at finding nearly all regressions, the downside is that many of the NEW tags are seen on intermittents (if an intermittent wasn't seen recently, or the failure line is slightly different). 86 87 NEW failures are for all tasks (build, lint, test, etc.) and for all failures (infra failures as well) 88 89 .. image:: img/th_new.png 90 :width: 400 91 92 93 On Try server, the NEW annotations are shown, and can act as a way to quickly filter a large number of failing tasks down to a more manageable number. It is best practice to run confirm failure or retrigger on the failing NEW task. To view only failing tasks, you can: 94 click on the filter icon for the entire page 95 96 .. image:: img/th_filter.png 97 :width: 300 98 99 100 select the field "failure classification" and select the value "new failure not classified" 101 102 .. image:: img/th_filter_classifications.png 103 :width: 300 104 105 106 then click "Add" 107 108 .. image:: img/th_filter_add.png 109 :width: 300 110 111 112 the above will add a `&failure_classification=6` to the url, you can add that manually if you wish. 113 114 115 MITTEN icon 116 ----------- 117 Another feature in Treeherder is the |MITTEN| icon. This is added to an orange job if the job had a retrigger or confirm failure and it is green. This is a great visual shortcut to ignore job failures. 118 119 120 Bug Suggestions 121 --------------- 122 Built into Treeherder originally is the feature when looking at the "Failure Summary" there will be bug suggestions showing you similar bugs that match the failure. 123 124 .. image:: img/th_bug_suggestions.png 125 :width: 300 126 127 128 If there is a `Single Tracking Bug <../sheriffed-intermittents/index.html#single-tracking-bugs>`__, only that will be shown. 129 130 Some caveats to keep in mind: 131 * It is easy to assume if there is a bug that the failure is an intermittent. 132 * this doesn't tell you if you have made an intermittent a permanent failure. 133 * the bug could be for a different configuration (look at failure history and compare platform, build type, test variant to make sure this failure isn't spreading) 134 * the bug could be inactive for months. 135 136 137 138 .. |MITTEN| image:: img/th_mitten.png 139 :width: 30