index.rst (4725B)
1 ################### 2 Performance Testing 3 ################### 4 5 .. toctree:: 6 :maxdepth: 2 7 :hidden: 8 :glob: 9 10 awsy 11 DAMP 12 IndexedDB 13 mach-try-perf 14 mozperftest 15 perf-sheriffing 16 perfcompare 17 performance-infrastructure 18 perftest-in-a-nutshell 19 raptor 20 talos 21 telemetry-alerting 22 23 Performance tests are designed to catch performance regressions before they reach our 24 end users. At this time, there is no unified approach for these types of tests, 25 but `mozperftest </testing/perfdocs/mozperftest.html>`_ aims to provide this in the future. 26 27 For more detailed information about each test suite and project, see their documentation: 28 29 * :doc:`awsy` 30 * :doc:`DAMP` 31 * :doc:`IndexedDB` 32 * :doc:`mach-try-perf` 33 * :doc:`mozperftest` 34 * :doc:`perf-sheriffing` 35 * :doc:`perfcompare` 36 * :doc:`performance-infrastructure` 37 * :doc:`perftest-in-a-nutshell` 38 * :doc:`raptor` 39 * :doc:`talos` 40 * :doc:`telemetry-alerting` 41 42 43 Here are the active PerfTest components/modules and their respective owners: 44 45 * AWFY (Are We Fast Yet) - 46 - Owner: Beatrice A. 47 - Description: A public dashboard comparing Firefox and Chrome performance metrics 48 * AWSY (Are We Slim Yet) 49 - Owner: Alexandru F. 50 - Co-owner: Andrej 51 - Description: Project that tracks memory usage across builds 52 * Raptor 53 - Owner: Sparky 54 - Co-owner: Kash 55 - Description: Test harness that uses Browsertime (based on webdriver) as the underlying engine to run performance tests 56 * CondProf (Conditioned profiles) 57 - Owner: Sparky 58 - Co-owner: Jmaher 59 - Description: Provides tooling to build, and obtain profiles that are preconditioned in some way. 60 * Infrastructure 61 - Owner: Sparky 62 - Co-owners: Kash, Andrej 63 - Description: All things involving: TaskCluster, Youtube Playback, Bitbar, Mobile Configs, etc... 64 * Mozperftest 65 - Owner: Sparky 66 - Co-owners: Kash, Andrej 67 - Description: Testing framework used to run performance tests 68 * Mozperftest Tools 69 - Owner: Sparky 70 - Co-owner: Kash, Andrej 71 - Description: Various tools used by performance testing team 72 * Mozproxy 73 - Owner: Kash 74 - Co-owner: Sparky 75 - Description: An http proxy used to run tests against third-party websites in a reliable and reproducible way 76 * PerfCompare 77 - Owner: Carla S. 78 - Co-owner: Beatrice A. 79 - Description: Performance comparison tool used to compare performance of different commits within a repository 80 * PerfDocs 81 - Owner: Sparky 82 - Co-owner: Kash, Andrej 83 - Description: Automatically generated performance test engineering documentation 84 * PerfHerder 85 - Owner: Beatrice A. 86 - Co-owner: Andra A. 87 - Description: The framework used by the performance sheriffs to find performance regressions and for storing, and visualizing our performance data. 88 * Performance Sheriffing 89 - Owner: Andra A. 90 - Co-owner: Beatrice A. 91 - Description: Performance sheriffs are responsible for finding commits that cause performance regressions and getting fixes from devs or backing out the changes 92 * Talos 93 - Owner: Sparky 94 - Co-owner: Kash, Andrej 95 - Description: Testing framework used to run Firefox-specific performance tests 96 * WebPageTest 97 - Owner: Andrej 98 - Co-owner: Sparky 99 - Description: A test running in the mozperftest framework used as a third party performance benchmark 100 101 You can additionally reach out to our team on 102 the `#perftest <https://matrix.to/#/#perftest:mozilla.org>`_ channel on matrix 103 104 For more information about the performance testing team, 105 `visit the wiki page <https://wiki.mozilla.org/TestEngineering/Performance>`_. 106 107 108 Critical Performance Tests 109 -------------------------- 110 111 Some of our performance tests are marked as critical tests. These are ones where we do not want to regress their metric under any circumstances unless the regression is explicitly approved by appropriate parties. When one of these tests regress, the regressor patch is backed out immediately by sheriffs after it has been found and confirmed. 112 113 The following list contains all of our existing critical tests: 114 115 * `Speedometer 3 on Windows 11 <https://treeherder.mozilla.org/perfherder/graphs?highlightAlerts=1&highlightChangelogData=1&highlightCommonAlerts=0&replicates=0&series=autoland,5257591,1,13&timerange=2592000>`_ 116 * `NewsSite Applink Startup on Android A55 <https://treeherder.mozilla.org/perfherder/graphs?highlightAlerts=1&highlightChangelogData=1&highlightCommonAlerts=0&replicates=0&series=autoland,5310509,1,15&timerange=2592000>`_