design.rst (5164B)
1 wptrunner Design 2 ================ 3 4 The design of wptrunner is intended to meet the following 5 requirements: 6 7 * Possible to run tests from W3C web-platform-tests. 8 9 * Tests should be run as fast as possible. In particular it should 10 not be necessary to restart the browser between tests, or similar. 11 12 * As far as possible, the tests should run in a "normal" browser and 13 browsing context. In particular many tests assume that they are 14 running in a top-level browsing context, so we must avoid the use 15 of an ``iframe`` test container. 16 17 * It must be possible to deal with all kinds of behaviour of the 18 browser under test, for example, crashing, hanging, etc. 19 20 * It should be possible to add support for new platforms and browsers 21 with minimal code changes. 22 23 * It must be possible to run tests in parallel to further improve 24 performance. 25 26 * Test output must be in a machine readable form. 27 28 Architecture 29 ------------ 30 31 In order to meet the above requirements, wptrunner is designed to 32 push as much of the test scheduling as possible into the harness. This 33 allows the harness to monitor the state of the browser and perform 34 appropriate action if it gets into an unwanted state e.g. kill the 35 browser if it appears to be hung. 36 37 The harness will typically communicate with the browser via some remote 38 control protocol such as WebDriver. However for browsers where no such 39 protocol is supported, other implementation strategies are possible, 40 typically at the expense of speed. 41 42 The overall architecture of wptrunner is shown in the diagram below: 43 44 .. image:: architecture.svg 45 46 .. currentmodule:: wptrunner 47 48 The main entry point to the code is :py:func:`~wptrunner.run_tests` in 49 ``wptrunner.py``. This is responsible for setting up the test 50 environment, loading the list of tests to be executed, and invoking 51 the remainder of the code to actually execute some tests. 52 53 The test environment is encapsulated in the 54 :py:class:`~environment.TestEnvironment` class. This defers to code in 55 ``web-platform-tests`` which actually starts the required servers to 56 run the tests. 57 58 The set of tests to run is defined by the 59 :py:class:`~testloader.TestLoader`. This is constructed with a 60 :py:class:`~testloader.TestFilter` (not shown), which takes any filter arguments 61 from the command line to restrict the set of tests that will be 62 run. The :py:class:`~testloader.TestLoader` reads both the ``web-platform-tests`` 63 JSON manifest and the expectation data stored in ini files and 64 produces a :py:class:`multiprocessing.Queue` of tests to run, and 65 their expected results. 66 67 Actually running the tests happens through the 68 :py:class:`~testrunner.ManagerGroup` object. This takes the :py:class:`~multiprocessing.Queue` of 69 tests to be run and starts a :py:class:`~testrunner.TestRunnerManager` for each 70 instance of the browser under test that will be started. These 71 :py:class:`~testrunner.TestRunnerManager` instances are each started in their own 72 thread. 73 74 A :py:class:`~testrunner.TestRunnerManager` coordinates starting the product under 75 test, and outputting results from the test. In the case that the test 76 has timed out or the browser has crashed, it has to restart the 77 browser to ensure the test run can continue. The functionality for 78 initialising the browser under test, and probing its state 79 (e.g. whether the process is still alive) is implemented through a 80 :py:class:`~browsers.base.Browser` object. An implementation of this class must be 81 provided for each product that is supported. 82 83 The functionality for actually running the tests is provided by a 84 :py:class:`~testrunner.TestRunner` object. :py:class:`~testrunner.TestRunner` instances are 85 run in their own child process created with the 86 :py:mod:`multiprocessing` module. This allows them to run concurrently 87 and to be killed and restarted as required. Communication between the 88 :py:class:`~testrunner.TestRunnerManager` and the :py:class:`~testrunner.TestRunner` is 89 provided by a pair of queues, one for sending messages in each 90 direction. In particular test results are sent from the 91 :py:class:`~testrunner.TestRunner` to the :py:class:`~testrunner.TestRunnerManager` using one 92 of these queues. 93 94 The :py:class:`~testrunner.TestRunner` object is generic in that the same 95 :py:class:`~testrunner.TestRunner` is used regardless of the product under 96 test. However the details of how to run the test may vary greatly with 97 the product since different products support different remote control 98 protocols (or none at all). These protocol-specific parts are placed 99 in the :py:class:`~executors.base.TestExecutor` object. There is typically a different 100 :py:class:`~executors.base.TestExecutor` class for each combination of control protocol 101 and test type. The :py:class:`~testrunner.TestRunner` is responsible for pulling 102 each test off the :py:class:`multiprocessing.Queue` of tests and passing it down to 103 the :py:class:`~executors.base.TestExecutor`. 104 105 The executor often requires access to details of the particular 106 browser instance that it is testing so that it knows e.g. which port 107 to connect to to send commands to the browser. These details are 108 encapsulated in the :py:class:`~browsers.base.ExecutorBrowser` class.