test.rst (4641B)
1 Running Automated JavaScript Tests 2 ================================== 3 4 SpiderMonkey uses two separate test suites. 5 6 The `Test262 test suite <https://github.com/tc39/test262>`__ is the implementation conformance test suite for ECMA-262, the language specification for JavaScript. All JavaScript engines use Test262 to ensure that they implement JavaScript correctly. Test262 is run using ``mach jstests``. 7 8 In addition to Test262, SpiderMonkey also has a large collection of internal tests. These tests are run using ``mach jit-test``. 9 10 Both sets of tests can be run from a normal build of the JS shell. 11 12 Running Test262 locally 13 ~~~~~~~~~~~~~~~~~~~~~~~ 14 15 The jstests shell harness is in ``js/src/tests/jstests.py``. It can be invoked using 16 17 .. code:: bash 18 19 ./mach jstests 20 21 Note that mach will generally find the JS shell itself; the --shell argument can be used to specify the location manually. All other flags will be passed along to the harness. 22 23 Test262 includes a lot of tests. When working on a specific feature, it is often useful to specify a subset of tests: 24 25 .. code:: bash 26 27 ./mach jstests TEST_PATH_SUBSTRING [TEST_PATH_SUBSTRING_2 ...] 28 29 This runs all tests whose paths contain any TEST_PATH_SUBSTRING. To exclude tests, you can use the ``--exclude=EXCLUDED`` option. Other useful options include: 30 31 - ``-s`` (``--show-cmd``): Show the exact command line used to run each test. 32 - ``-o`` (``--show-output``): Print each test's output. 33 - ``--args SHELL_ARGS``: Extra arguments to pass to the JS shell. 34 - ``--rr`` Run tests under the `rr <https://rr-project.org/>`__ record-and-replay debugger. 35 36 For a complete list of options, use: 37 38 .. code:: bash 39 40 ./mach jstests -- -h 41 42 The Test262 tests can also be run in the browser, using: 43 44 .. code:: bash 45 46 ./mach jstestbrowser 47 48 To run a specific test, you can use the ``--filter=PATTERN`` option, where PATTERN is a RegExp pattern that is tested against ``file:///{PATH_TO_OBJ_DIR}/dist/test-stage/jsreftest/tests/jsreftest.html?test={RELATIVE_PATH_TO_TEST_FROM_js/src/tests}``. 49 50 For historical reasons, ``jstests`` also includes a SpiderMonkey specific suite of additional language feature tests in ``js/src/tests/non262``. 51 52 Running jit-tests locally 53 ~~~~~~~~~~~~~~~~~~~~~~~~~ 54 55 The jit-test shell harness is in ``js/src/jit-test/jit_test.py``. It can be invoked using 56 57 .. code:: bash 58 59 ./mach jit-test 60 61 Basic usage of ``mach jit-test`` is similar to ``mach jstests``. A subset of tests can be specified: 62 63 .. code:: bash 64 65 ./mach jit-test TEST_PATH_SUBSTRING [TEST_PATH_SUBSTRING_2 ...] 66 67 The ``--jitflags`` option allows you to test the JS executable with different flags. The flags are used to control which features are run, such as which JITs are enabled and how quickly they will kick in. The valid options are named bundles like 'all', 'interp', 'none', etc. See the full list in the ``-h`` / ``--help`` output. 68 69 Another helpful option specific to ``jit-tests`` is ``-R <filename>`` (or ``--retest <filename>``). The first time this is run, it runs the entire test suite and writes a list of tests that failed to the given filename. The next time it is run, it will run only the tests in the given filename, and then will write the list of tests that failed when it is done. Thus, you can keep retesting as you fix bugs, and only the tests that failed the last time will run, speeding things up a bit. 70 71 Other useful options include: 72 73 - ``-s`` (``--show-cmd``): Show the exact command line used to run each test. 74 - ``-o`` (``--show-output``): Print each test's output. 75 - ``--args SHELL_ARGS``: Extra arguments to pass to the JS shell. 76 - ``--debug-rr`` Run a test under the `rr <https://rr-project.org/>`__ record-and-replay debugger. 77 - ``--cgc`` Run a test with frequent compacting GCs (equivalent to ``SM(cgc)``) 78 79 Adding new jit-tests 80 ~~~~~~~~~~~~~~~~~~~~ 81 82 Creating new tests for jit-tests is easy. Just add a new JS file in an appropriate directory under ``js/src/jit-test/tests``. (By default, tests should go in ``test/basic``.) The test harness will automatically find the test and run it. The test is considered to pass if the exit code of the JS shell is zero (i.e., JS didn't crash and there were no JS errors). Use the ``assertEq`` function to verify values in your test. 83 84 There are some advanced options for tests. See the README (in ``js/src/jit-tests``) for details. 85 86 Running jstests on Treeherder 87 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 88 89 On Treeherder, jstests run in the browser are shown as ``R(J)`` (search for ``jsreftest`` in ``mach try fuzzy``). SpiderMonkey shell jobs are shown as ``SM(...)``; most of them include JS shell runs of both jstests and jit-tests.