developing.rst (5669B)
1 Developing in mozperftest 2 ========================= 3 4 Architecture overview 5 --------------------- 6 7 `mozperftest` implements a mach command that is a thin wrapper on the 8 top of `runner.py`, which allows us to run the tool without having to go through 9 a mach call. Command arguments are prepared in `argparser.py` and then made 10 available for the runner. 11 12 The runner creates a `MachEnvironment` instance (see `environment.py`) and a 13 `Metadata` instance (see `metadata.py`). These two objects are shared during the 14 whole test and used to share data across all parts. 15 16 The runner then calls `MachEnvironment.run`, which is in charge of running the test. 17 The `MachEnvironment` instance runs a sequence of **layers**. 18 19 Layers are classes responsible for one single aspect of a performance test. They 20 are organized in three categories: 21 22 - **system**: anything that sets up and tears down some resources or services 23 on the system. Existing system layers: **android**, **proxy** 24 - **test**: layers that are in charge of running a test to collect metrics. 25 Existing test layers: **browsertime** and **androidlog** 26 - **metrics**: all layers that process the metrics to turn them into usable 27 metrics. Existing system layers: **perfherder** and **console** 28 29 The MachEnvironment instance collects a series of layers for each category and 30 runs them sequentially. 31 32 The goal of this organization is to allow adding new performance tests runners 33 that will be based on a specific combination of layers. To avoid messy code, 34 we need to make sure that each layer represents a single aspect of the process 35 and that is completely independent of other layers (besides sharing the data 36 through the common environment.) 37 38 For instance, we could use `perftest` to run a C++ benchmark by implementing a 39 new **test** layer. 40 41 42 Layer 43 ----- 44 45 A layer is a class that inherits from `mozperftest.layers.Layer` and implements 46 a few methods and class variables. 47 48 List of methods and variables: 49 50 - `name`: name of the layer (class variable, mandatory) 51 - `activated`: boolean to activate by default the layer (class variable, False) 52 - `user_exception`: will trigger the `on_exception` hook when an exception occurs 53 - `arguments`: dict containing arguments. Each argument is following 54 the `argparser` standard 55 - `run(self, metadata)`: called to execute the layer 56 - `setup(self)`: called when the layer is about to be executed 57 - `teardown(self)`: called when the layer is exiting 58 59 Example:: 60 61 class EmailSender(Layer): 62 """Sends an email with the results. 63 """ 64 name = "email" 65 activated = False 66 67 arguments = { 68 "recipient": { 69 "type": str, 70 "default": "tarek@mozilla.com", 71 "help": "Recipient", 72 }, 73 } 74 75 def setup(self): 76 self.server = smtplib.SMTP(smtp_server,port) 77 78 def teardown(self): 79 self.server.quit() 80 81 def __call__(self, metadata): 82 self.server.send_email(self.get_arg("recipient"), metadata.results()) 83 84 85 It can then be added to one of the top functions that are used to create a list 86 of layers for each category: 87 88 - **mozperftest.metrics.pick_metrics** for the metrics category 89 - **mozperftest.system.pick_system** for the system category 90 - **mozperftest.test.pick_browser** for the test category 91 92 And also added in each `get_layers` function in each of those categories. 93 The `get_layers` functions are invoked when building the argument parser. 94 95 In our example, adding the `EmailSender` layer will add two new options: 96 97 - **--email** a flag to activate the layer 98 - **--email-recipient** 99 100 101 Important layers 102 ---------------- 103 104 **mozperftest** can be used to run performance tests against browsers using the 105 **browsertime** test layer. It leverages the `browsertime.js 106 <https://www.sitespeed.io/documentation/browsertime/>`_ framework and provides 107 a full integration into Mozilla's build and CI systems. 108 109 Browsertime uses the Selenium Webdriver client to drive the browser, and 110 provides some metrics to measure performance during a user journey. 111 112 113 Coding style 114 ------------ 115 116 For the coding style, we want to: 117 118 - Follow `PEP 257 <https://www.python.org/dev/peps/pep-0257/>`_ for docstrings 119 - Avoid complexity as much as possible 120 - Use modern Python 3 code (for instance `pathlib` instead of `os.path`) 121 - Avoid dependencies on Mozilla build projects and frameworks as much as possible 122 (mozharness, mozbuild, etc), or make sure they are isolated and documented 123 124 125 Landing patches 126 --------------- 127 128 .. warning:: 129 130 It is mandatory for each patch to have a test. Any change without a test 131 will be rejected. 132 133 Before landing a patch for mozperftest, make sure you run `perftest-test`:: 134 135 % ./mach perftest-test 136 => black [OK] 137 => flake8 [OK] 138 => remove old coverage data [OK] 139 => running tests [OK] 140 => coverage 141 Name Stmts Miss Cover Missing 142 ------------------------------------------------------------------------------------------ 143 mozperftest/metrics/notebook/analyzer.py 29 20 31% 26-36, 39-42, 45-51 144 ... 145 mozperftest/system/proxy.py 37 0 100% 146 ------------------------------------------------------------------------------------------ 147 TOTAL 1614 240 85% 148 149 [OK] 150 151 The command will run `black`, `flake8` and also make sure that the test coverage has not regressed. 152 153 You can use the `-s` option to bypass flake8/black to speed up your workflow, but make 154 sure you do a full tests run. You can also pass the name of one single test module.