tor-browser

The Tor Browser
git clone https://git.dasho.dev/tor-browser.git
Log | Files | Refs | README | LICENSE

index.md (16308B)

TPS

Table of Contents

The majority of this document is targeting the Firefox developer that would like to understand TPS and/or write TPS tests. If you'd like to simply run TPS, see the How to run TPS section.

What is TPS

TPS is a test automation framework for Firefox Sync. TPS takes in test configuration as JavaScript files defined in a configuration file and runs all tests in sequence.

Each TPS test is made of one or more phases that each have their own assertions and run in sequence. If any phase fails, then the test fails and the remaining phases are not run. However, other tests will still run.

Why TPS exists

TPS runs against the real Mozilla Accounts server and Sync servers and thus is a good way to test Firefox Sync end-to-end without mocking the servers.

Architecture

High level Diagram

The following diagram describes the flow when running TPS.

flowchart TD
TPS[TPS Runner]-->T[Run Test Suite]
T-->T1[Run Next Single Test]
T1-->S[Setup]
S-->P[Run Next Phase]
P-->LP[Launch Test Profile]
LP-->Q{Phase Success?}
Q-->|No| CL[Clean up]
Q-->|Yes| Q2{Any more phases?}
Q2-->|Yes|P
Q2-->|No| CL
CL-->CR[Collect Results]
CR-->Q3{Any more tests?}
Q3-->|Yes|T1
Q3-->|No| D[Done]

Single Test Sequence Diagram

The following sequence diagram describes the involved entities executing a single TPS test

sequenceDiagram
actor U as User
participant TPS as TPS Runner
participant F as Firefox Profile
participant TE as TPS Extension
U->>TPS: Start test
TPS->>TPS: Parse Test file
loop Every Phase
TPS->>F: Launch Phase w/prefs to install TPS extension
F->>TE: Install Extension
TE->>F: Read prefs to get test file
F->>TE: Test file and TPS Runner configuration
loop Every instruction
TE->>F: Execute test instructions and assertions
F->>TE: Result
end
TE->>F: Phase Done
F->>TPS: Done
TPS->>F: Read logs to get test results
end
TPS->>U: Print test results

Phases

Each Phase is mapped to a Firefox profile, phases may re-use profiles if they are mapped to them, see the phases object for details on the mapping. All phases are signed in to the same Mozilla Account.

Example

For example, a simple test that defines two phases, one that uploads bookmarks and another that downloads them can be described as follows, see the section on Test Format for details on the format of the tests.

EnableEngines(["bookmarks"]); // Enables the bookmark engine across all profiles(phases) that this test runs

/*
 * The list of phases mapped to their corresponding profiles.  The object
 * here must be in JSON format as it will get parsed by the Python
 * testrunner. It is parsed by the YAML package, so it relatively flexible.
 */
var phases = {
  phase1: "profile1",
  phase2: "profile2",
};

// the initial list of bookmarks to add to the browser
var bookmarksInitial = {
  menu: [
    { folder: "foldera" },
    { folder: "folderb" },
  ],

  "menu/foldera": [{ uri: "http://www.cnn.com", title: "CNN" }], // One bookmark in menu/foldera pointing to CNN
  "menu/folderb": [{ uri: "http://www.apple.com", title: "Apple"}], // One bookmark in menu/folderb pointing to apple
};

// Add bookmarks to profile1 and sync.
Phase("phase1", [
  [Bookmarks.add, bookmarksInitial],
  [Bookmarks.verify, bookmarksInitial],
  [Sync],
  [Bookmarks.verify, bookmarksInitial],
]);


// Sync, then verify that bookmarks added by phase 1 are present.
Phase("phase2", [
  [Sync],
  [Bookmarks.verify, bookmarksInitial],
]);

The TPS Extension

When the Firefox profile representing the phase loads, it first installs an extension. The extension is what executes the tests by instructing Firefox and reading from Firefox to assert that Sync is working properly.

The test files execute in the extension, and the extension defines all the functions that the test files may use. For instance, in the above example the Phase function is defined here

Test Format

Test group configuration

The tests are referenced by a json file that references all the tests TPS will run. By default TPS will run the configuration in `services/sync/tests/tps/all_tests.json`. The test group configuration is a json object with one key named tests that is itself a json object. The tests object has a key for every test to run, and the key should be the name of the test file. The value for each test file is the empty object {} if the test should run, or {"disabled": "Bug <BUG NUMBER>"} where <BUG NUMBER> is a Bugzilla bug number referencing why the test was disabled.

Test Files

The phases object

Test Files are JavaScript files that will be run by the TPS extension once the Firefox profile is loaded. However, before that is done the TPS framework will load the first object defined in test file as yaml. In other words, for the following example:

var phases = {
    phase1: "profile1",
    phase2: "profile2",
    phase3: "profile1", // phase 1 and 3 reuse the same profile, "profile1"
}

// ... rest of the test file

The inside of the curly brackets will be parsed as yaml to identify the profiles that TPS will run and map each phase to a profile. In the example above, phase1 and phase3 reuse the same profile, but phase2 uses it's own profile. The rest of the file is regular JavaScript that will be loaded in the TPS Extension.

Test File Capabilities

The TPS Extension exports a set of functions and objects that test files may use. See the `tps.sys.mjs` for up-to-date details, the following is the list of export capabilities as of April 10th, 2024:

Enabling Sync Engines

To enable sync engines, which you should do before executing a test that depends on an engine, use the EnableEngines function. The function takes an array of strings representing the engine names. Eg:

EnableEngines(["bookmarks", "history"]);
Start a phase

Phases are run on their assigned profiles profiles in the order they are declared in the phases object. To define what a phase does use the Phase function that takes in a phase name as the first argument - this should be the same name defined in the [phases object](#the-phases-object) - and a 2D array of actions as a second argument. Each action array in the 2D is run in sequence. The inner array defines what the action is, and any arguments to that action. For example:

Phase("phase1", [[Sync], [Bookmarks.add, bookmarkInitial]]);

Note that the example assumes that the phases object and bookmarkInitial are defined in test file.

In the above example:

- [Sync] as its first array which means that the phase will first Sync - [Bookmarks.add, bookmarkInitial] as its second member which means that after the Sync, a Bookmarks.add operation will run, and it will add the bookmarkInitial.

Actions

Actions are run inside phases as arguments to the `Phase` function.

General Actions

There are few general actions that aren't tied to a data type:

Data Type specific Actions

Some actions are supported for individual data types. Those actions are triggered by using <DataType>.<Action>. The following is a list of all possible <DataType> values: For more details on which actions are supported for which data type, see the following sections.

Bookmarks Actions

Example in test_existing_bookmarks.js

Addresses Actions

Example in test_addresses.js

CreditCard Actions

Example in test_creditcards.js

Addons Actions

Example in test_addon_reconciling.js

Formdata Actions

Example in test_formdata.js

History Actions

Example in test_history.js

Passwords Actions

Example in test_passwords.js

Prefs Actions

Example in test_prefs.js

Tabs Actions

Example in test_tabs.js

Windows Actions

Example in test_privbrw_tabs.js

ExtStorage Actions

Example in test_extstorage.js

Writing new TPS tests

To write new TPS tests follow the following instructions:

  1. verify: Verifies that the value of the given key is the given value, fails the phase otherwise.
  2. verify: Verifies that the value of the given key is the given value, fails the phase otherwise.
  3. verify: Verifies that the value of the given key is the given value, fails the phase otherwise.
  4. verify: Verifies that the value of the given key is the given value, fails the phase otherwise.

How to run TPS

Installation

TPS requires several packages to operate properly. To install TPS and required packages, use the create_venv.py script provided in the testing/tps directory:

python3 create_venv.py /path/to/create/virtualenv

This script will create a virtalenv and install TPS into it.

You must then activate the virtualenv by executing:

source /path/to/virtualenv/Scripts/activate

/path/to/virtualenv/Scripts/activate.bat

TPS can then be run by executing:

runtps --binary=/path/to/firefox

Note: You can run the tps tests in headless mode by using MOZ_HEADLESS=1. This will make

your computer somewhat useable while the tests are running.

When you are done with TPS, you can deactivate the virtualenv by executing deactivate

Configuration

To edit the TPS configuration, do not edit config/config.json.in in the tree. Instead, edit config.json inside your virtualenv; it will be located at the top level of where you specified the virtualenv be created - eg, for the example above, it will be /path/to/create/virtualenv/config.json

Setting Up Test Accounts

Mozilla Accounts

To create a test account for using the Mozilla Account authentication perform the following steps:

Note: Currently, the TPS tests rely on how restmail returns the verification code

You should use restmail or something very similar.

Gmail and other providers might give a The request was blocked for security reasons

  1. (win):

- Replace %account_prefix% with your own test name

  1. (win):
  2. (win):
  3. (win):
  4. (win):

Now you will be able to use this account for TPS. Note that these steps can be done in either a test profile or in a private browsing window - you might want to avoid doing that in a "real" profile that's already connected to Sync.