Wasm Spec Tests

This directory contains scripts and configs to manage the in-tree import of the wasm spec testsuite.

Format of tests

Spec tests are given in test/core of the spec repository as .wast files. A .wast file is a superset of the .wat format with commands for running tests.

The spec interpreter can natively consume .wast files to run the tests, but we cannot run them directly ourselves. To workaround this, we have a tool which can convert .wast files to .js that can be run efficiently in our jit-test harness.

Running tests

Tests are imported to jit-test to be run in the JS shell.

To run under jit-test:

cd js/src
./jit-test.py path_to_build/dist/bin/js wasm/spec/

Test importing

There are many proposals in addition to the canonical spec. Each proposal is a fork of the canonical spec and may modify any amount of files.

This causes a challenge for any engine implementing multiple proposals, as any proposal may conflict (and often will) with each other. This makes it infeasible to merge all spec and proposal repos together.

Testing each proposal separately in full isn't an attractive option either, as most tests are unchanged and that would cause a significant amount of wasted computation time.

For this reason, we generate a set of separate test-suites that are 'pruned' to obtain a minimal set of tests. The tool works by merging each proposal with the proposal it is based off of and removing tests that have not changed.

Configuration

The import tool relies on config.toml for the list of proposals to import, and config-lock.toml for a list of commits to use for each proposal.

The lock file makes test importing deterministic and controllable. This is useful as proposals often make inconvenient and breaking changes.

Operation

# From the root of the repo, navigate into js/src/jit-test/etc/wasm/
# Add, remove, or modify proposals
vim generate-spectests/config.toml
# Remove locks for any proposals you wish to pull the latest changes on
vim generate-spectests/config-lock.toml
# Import the tests
make update
# View the tests that were imported
git status
# Run the imported tests and note failures
./jit-test.py dist/bin/js wasm/spec/
# Exclude test failures
vim generate-spectests/config.toml
# Re-import the tests to exclude failing tests
make update
# Commit the changes
git commit

Debugging import failures

Proposals can often have conflicts with their upstream proposals. This is okay, and the test importer will fallback to building tests on an unmerged tree.

This will likely result in extra tests being imported due to spurious differences between the proposal and upstream, but generally is okay.

The import tool uses RUST_LOG to output debug information. Makefile automatically uses RUST_LOG=info. Change that to RUST_LOG=debug to get verbose output of all the commands run.