IndexedDB.rst (4142B)
1 ============================= 2 IndexedDB Performance Testing 3 ============================= 4 5 How to run tests on CI: 6 ----------------------- 7 * Windows: ``mach try perf --full -q "test-windows10-64-shippable-qr/opt-browsertime-indexeddb"`` 8 * Linux: ``mach try perf --full -q "test-linux1804-64-shippable-qr/opt-browsertime-indexeddb"`` 9 * Mac: ``mach try perf --full -q "test-macosx1015-64-shippable-qr/opt-browsertime-indexeddb"`` 10 * All but 32-bit jobs: ``mach try perf --chrome --safari --full -q 'shippable-browsertime-indexeddb !32'`` 11 * In general: 12 13 * Open test selection interface with ``mach try perf --full`` 14 * Filter out the preferred tests by typing letters which are expected to be part of the test job name string (as in the -q argument above) 15 * Note down the string used as a filter for rerunning the job (or rerun it with ``mach try again --list-tasks`` and ``mach try again --index``) 16 17 How to run tests locally with the profiler? 18 ------------------------------------------- 19 * Build the browser with release or release with debug symbols flags (not in debug mode) 20 * Use ``mach raptor --browsertime -t $(test_name) --gecko-profile --post-startup-delay=1000`` where test name, such as ``addMarN`` is one of the items listed in ``testing/raptor/raptor/tests/custom/browsertime-indexeddb.ini`` 21 * After the test is complete, the generated profile is opened with the default browser. 22 * The generated profile file path is listed also in the command line output. 23 * For best symbolication results, it may help to 24 25 * run the same browser build that was used for the tests with ``./mach run`` 26 * navigate to "profiler.firefox.com" 27 * use the "Load a profile from file" button 28 29 How to compare performance to a different browser? 30 -------------------------------------------------- 31 * The test outputs a ``time_duration`` value for all supported browsers 32 * Using Chrome as an example, 33 34 * ``mach raptor --browsertime -t $(test_name) --post-startup-delay=1000 --app=chrome -b "/c/Program Files/Google/Chrome/Application/chrome.exe"`` 35 * where test name, such as ``addMarN`` is one of the items listed in ``testing/raptor/raptor/tests/custom/browsertime-indexeddb.ini`` 36 * browser executable path after the ``-b`` argument varies locally 37 * in some cases, a test driver argument such as ``--browsertime-chromedriver`` may be required 38 39 How to add more tests? 40 ---------------------- 41 * For the test boilerplate, copy and rename an old test script such as ``testing/raptor/browsertime/indexeddb_write.js`` under the ``testing/raptor/browsertime/`` directory 42 * Modify the test case script argument of ``commands.js.run`` / Selenium's ``executeAsyncScript`` 43 44 * Test parameters can be passed to such script with syntax ``${variable_name}`` where ``variable_name`` represents the parameter in the context of ``executeAsyncScript`` or ``commands.js.run``. 45 * Use quotes to capture a string value, for example ``"${variable_name}"`` 46 * TIP: Debugging the test case could be simpler by serving it locally without the boilerplate 47 48 * Add ``[test_name]`` section to file ``testing/raptor/raptor/tests/custom/browsertime-indexeddb.ini`` where ``test_name`` **must be 10 characters or less** in order to be a valid ``Treeherder`` test name 49 50 * Under the ``[test_name]`` section, specify the test script name as a value of ``test_script=`` 51 * Under the ``[test_name]`` section, specify the test parameters as a sequence of ``--browsertime.key=value`` arguments as a value of ``browsertime_args =`` 52 * Under the ``[test_name]`` section, override any other values as needed 53 54 * Add test as a subtest to run for Desktop ``taskcluster/kinds/browsertime/desktop.yml`` (maybe also for mobile) 55 * Add test documentation to ``testing/raptor/raptor/perfdocs/config.yml`` 56 57 * Generated files: 58 59 * Run ``./mach lint --warnings --outgoing --fix`` to regenerate the documentation and task files, and warn about omissions 60 * Running ``./mach lint -l perfdocs --fix .`` may also be needed 61 62 * Testing: 63 64 * Test the new test by running it with the profiler 65 * Test the new test by running it with a different browser 66 * Test the new test by triggering it on CI