junit.rst (20351B)
1 .. -*- Mode: rst; fill-column: 80; -*- 2 3 ==================== 4 Junit Test Framework 5 ==================== 6 7 GeckoView has `a lot 8 <https://searchfox.org/mozilla-central/rev/36904ac58d2528fc59f640db57cc9429103368d3/mobile/android/geckoview/src/androidTest/java/org/mozilla/geckoview/test/rule/GeckoSessionTestRule.java>`_ 9 of `custom 10 <https://searchfox.org/mozilla-central/source/mobile/android/geckoview/src/androidTest/assets/web_extensions/test-support>`_ 11 code that is used to run junit tests. This document is an overview of what this 12 code does and how it works. 13 14 .. contents:: Table of Contents 15 :depth: 2 16 :local: 17 18 Introduction 19 ============ 20 21 `GeckoView <https://geckoview.dev>`_ is an Android Library that can be used to 22 embed Gecko, the Web Engine behind Firefox, in applications. It is the 23 foundation for Firefox on Android, and it is intended to be used to build Web 24 Browsers, but can also be used to build other types of apps that need to 25 display Web content. 26 27 GeckoView itself has no UI elements besides the Web View and uses Java 28 interfaces called "delegates" to let embedders (i.e. apps that use GeckoView) 29 implement UI behavior. 30 31 For example, when a Web page's JavaScript code calls ``alert('Hello')`` the 32 embedder will receive a call to the `onAlertPrompt 33 <https://mozilla.github.io/geckoview/javadoc/mozilla-central/org/mozilla/geckoview/GeckoSession.PromptDelegate.html#onAlertPrompt-org.mozilla.geckoview.GeckoSession-org.mozilla.geckoview.GeckoSession.PromptDelegate.AlertPrompt->`_ 34 method of the `PromptDelegate 35 <https://mozilla.github.io/geckoview/javadoc/mozilla-central/org/mozilla/geckoview/GeckoSession.PromptDelegate.html>`_ 36 interface with all the information needed to display the prompt. 37 38 As most delegate methods deal with UI elements, GeckoView will execute them on 39 the UI thread for the embedder's convenience. 40 41 GeckoResult 42 ----------- 43 44 One thing that is important to understand for what follows is `GeckoResult 45 <https://mozilla.github.io/geckoview/javadoc/mozilla-central/org/mozilla/geckoview/GeckoResult.html>`_. 46 ``GeckoResult`` is a promise-like object that is used throughout the GeckoView 47 API, it allows embedders to asynchronously respond to delegate calls and 48 GeckoView to return results asynchronously. This is especially important for 49 GeckoView as it never provides synchronous access to Gecko as a design 50 principle. 51 52 For example, when installing a WebExtension in GeckoView, the resulting 53 `WebExtension 54 <https://mozilla.github.io/geckoview/javadoc/mozilla-central/org/mozilla/geckoview/WebExtension.html>`_ 55 object is returned in a ``GeckoResult``, which is completed when the extension 56 is fully installed: 57 58 .. code:: java 59 60 public GeckoResult<WebExtension> install(...) 61 62 To simplify memory safety, ``GeckoResult`` will always `execute callbacks 63 <https://searchfox.org/mozilla-central/rev/36904ac58d2528fc59f640db57cc9429103368d3/mobile/android/geckoview/src/main/java/org/mozilla/geckoview/GeckoResult.java#740-744>`_ 64 in the same thread where it was created, turning asynchronous code into 65 single-threaded javascript-style code. This is currently `implemented 66 <https://searchfox.org/mozilla-central/rev/36904ac58d2528fc59f640db57cc9429103368d3/mobile/android/geckoview/src/main/java/org/mozilla/geckoview/GeckoResult.java#285>`_ 67 using the Android Looper for the thread, which restricts ``GeckoResult`` to 68 threads that have a looper, like the Android UI thread. 69 70 Testing overview 71 ---------------- 72 73 Given that GeckoView is effectively a translation layer between Gecko and the 74 embedder, it's mostly tested through integration tests. The vast majority of 75 the GeckoView tests are of the form: 76 77 - Load simple test web page 78 - Interact with the web page through a privileged JavaScript test API 79 - Verify that the right delegates are called with the right inputs 80 81 and most of the test framework is built around making sure that these 82 interactions are easy to write and verify. 83 84 Tests in GeckoView can be run using the ``mach`` interface, which is used by 85 most Gecko tests. E.g. to run the `loadUnknownHost 86 <https://searchfox.org/mozilla-central/rev/36904ac58d2528fc59f640db57cc9429103368d3/mobile/android/geckoview/src/androidTest/java/org/mozilla/geckoview/test/NavigationDelegateTest.kt#186-196>`_ 87 test in ``NavigationDelegateTest`` you would type on your terminal: 88 89 .. code:: shell 90 91 ./mach geckoview-junit org.mozilla.geckoview.test.NavigationDelegateTest#loadUnknownHost 92 93 Another way to run GeckoView tests is through the `Android Studio IDE 94 <https://developer.android.com/studio>`_. By running tests this way, however, 95 some parts of the test framework are not initialized, and thus some tests 96 behave differently or fail, as will be explained later. 97 98 Testing envelope 99 ---------------- 100 101 Being a library, GeckoView has a natural, stable, testing envelope, namely the 102 GeckoView API. The vast majority of GeckoView tests only use 103 publicly-accessible APIs to verify the behavior of the API. 104 105 Whenever the API is not enough to properly test behavior, the testing framework 106 offers targeted "privileged" testing APIs. 107 108 Using a restricted, stable testing envelope has proven over the years to be an 109 effective way of writing consistent tests that don't break upon refactoring. 110 111 Testing Environment 112 ------------------- 113 114 When run through ``mach``, the GeckoView junit tests run in a similar 115 environment as mochitests (a type of Web regression tests used in Gecko). They 116 have access to the mochitest web server at `example.com`, and inherit most of 117 the testing prefs and profile. 118 119 Note the environment will not be the same as mochitests when the test is run 120 through Android Studio, the prefs will be inherited from the default GeckoView 121 prefs (i.e. the same prefs that would be enabled in a consumer's build of 122 GeckoView) and the mochitest web server will not be available. 123 124 Tests account for this using the `isAutomation 125 <https://searchfox.org/mozilla-central/rev/95d8478112eecdd0ee249a941788e03f47df240b/mobile/android/geckoview/src/androidTest/java/org/mozilla/geckoview/test/util/Environment.java#36-38>`_ 126 check, which essentially checks whether the test is running under ``mach`` or 127 via Android Studio. 128 129 Unlike most other junit tests in the wild, GeckoView tests run in the UI 130 thread. This is done so that the GeckoResult objects are created on the right 131 thread. Without this, every test would most likely include a lot of blocks that 132 run code in the UI thread, adding significant boilerplate. 133 134 Running tests on the UI thread is achieved by registering a custom ``TestRule`` 135 called `GeckoSessionTestRule 136 <https://searchfox.org/mozilla-central/rev/36904ac58d2528fc59f640db57cc9429103368d3/mobile/android/geckoview/src/androidTest/java/org/mozilla/geckoview/test/NavigationDelegateTest.kt#186-196>`_, 137 which, among other things, `overrides the evaluate 138 <https://searchfox.org/mozilla-central/rev/95d8478112eecdd0ee249a941788e03f47df240b/mobile/android/geckoview/src/androidTest/java/org/mozilla/geckoview/test/rule/GeckoSessionTestRule.java#1307,1312>`_ 139 method and wraps everything into a ``instrumentation.runOnMainSync`` call. 140 141 Verifying delegates 142 =================== 143 144 As mentioned earlier, verifying that a delegate call happens is one of the most 145 common assertions that a GeckoView test makes. To facilitate that, 146 ``GeckoSessionTestRule`` offers several ``delegate*`` utilities like: 147 148 .. code:: java 149 150 sessionRule.delegateUntilTestEnd(...) 151 sessionRule.delegateDuringNextWait(...) 152 sessionRule.waitUntilCalled(...) 153 sessionRule.forCallbacksDuringWait(...) 154 155 These all take an arbitrary delegate object (which may include multiple 156 delegate implementations) and handle installing and cleaning up the delegate as 157 needed. 158 159 Another set of facilities that ``GeckoSessionTestRule`` offers allow tests to 160 synchronously ``wait*`` for events, e.g. 161 162 .. code:: java 163 164 sessionRule.waitForJS(...) 165 sessionRule.waitForResult(...) 166 sessionRule.waitForPageStop(...) 167 168 These facilities work together with the ``delegate*`` facilities by marking the 169 ``NextWait`` or the ``DuringWait`` events. 170 171 As an example, a test could load a page using ``session.loadUri``, wait until 172 the page has finished loading using ``waitForPageStop`` and then verify that 173 the expected delegate was called using ``forCallbacksDuringWait``. 174 175 Note that the ``DuringWait`` here always refers to the last time a ``wait*`` 176 method was called and finished executing. 177 178 The next sections will go into how this works and how it's implemented. 179 180 Tracking delegate calls 181 ----------------------- 182 183 One thing you might have noticed in the above section is that 184 ``forCallbacksDuringWait`` moves "backward" in time by replaying the delegates 185 called that happened while the wait was being executed. 186 ``GeckoSessionTestRule`` achieves this by `injecting a proxy object 187 <https://searchfox.org/mozilla-central/rev/95d8478112eecdd0ee249a941788e03f47df240b/mobile/android/geckoview/src/androidTest/java/org/mozilla/geckoview/test/rule/GeckoSessionTestRule.java#1137>`_ 188 into every delegate, and `proxying every call 189 <https://searchfox.org/mozilla-central/rev/95d8478112eecdd0ee249a941788e03f47df240b/mobile/android/geckoview/src/androidTest/java/org/mozilla/geckoview/test/rule/GeckoSessionTestRule.java#1091-1092>`_ 190 to the current delegate according to the ``delegate`` test calls. 191 192 The proxy delegate `is built 193 <https://searchfox.org/mozilla-central/rev/95d8478112eecdd0ee249a941788e03f47df240b/mobile/android/geckoview/src/androidTest/java/org/mozilla/geckoview/test/rule/GeckoSessionTestRule.java#1105-1106>`_ 194 using the Java reflection's ``Proxy.newProxyInstance`` method and receives `a 195 callback 196 <https://searchfox.org/mozilla-central/rev/95d8478112eecdd0ee249a941788e03f47df240b/mobile/android/geckoview/src/androidTest/java/org/mozilla/geckoview/test/rule/GeckoSessionTestRule.java#1030-1031>`_ 197 every time a method on the delegate is being executed. 198 199 ``GeckoSessionTestRule`` maintains a list of `"default" delegates 200 <https://searchfox.org/mozilla-central/rev/95d8478112eecdd0ee249a941788e03f47df240b/mobile/android/geckoview/src/androidTest/java/org/mozilla/geckoview/test/rule/GeckoSessionTestRule.java#743-752>`_ 201 used in GeckoView, and will `use reflection 202 <https://searchfox.org/mozilla-central/rev/95d8478112eecdd0ee249a941788e03f47df240b/mobile/android/geckoview/src/androidTest/java/org/mozilla/geckoview/test/rule/GeckoSessionTestRule.java#585>`_ 203 to match the object passed into the ``delegate*`` calls to the proxy delegates. 204 205 For example, when calling 206 207 .. code:: java 208 209 sessionRule.delegateUntilTestEnd(object : NavigationDelegate, ProgressDelegate {}) 210 211 ``GeckoSessionTestRule`` will know to redirect all ``NavigationDelegate`` and 212 ``ProgressDelegate`` calls to the object passed in ``delegateUntilTestEnd``. 213 214 Replaying delegate calls 215 ------------------------ 216 217 Some delegate methods require output data to be passed in by the embedder, and 218 this requires extra care when going "backward in time" by replaying the 219 delegate's call. 220 221 For example, whenever a page loads, GeckoView will call 222 ``GeckoResult<AllowOrDeny> onLoadRequest(...)`` to know if the load can 223 continue or not. When replaying delegates, however, we don't know what the 224 value of ``onLoadRequest`` will be (or if the test is going to install a 225 delegate for it, either!). 226 227 What ``GeckoSessionTestRule`` does, instead, is to `return the default value 228 <https://searchfox.org/mozilla-central/rev/95d8478112eecdd0ee249a941788e03f47df240b/mobile/android/geckoview/src/androidTest/java/org/mozilla/geckoview/test/rule/GeckoSessionTestRule.java#1092>`_ 229 for the delegate method, and ignore the replayed delegate method return value. 230 This can be a little confusing for test writers, for example this code `will 231 not` stop the page from loading: 232 233 .. code:: java 234 235 session.loadUri("https://www.mozilla.org") 236 sessionRule.waitForPageStop() 237 sessionRule.forCallbacksDuringWait(object : NavigationDelegate { 238 override fun onLoadRequest(session: GeckoSession, request: LoadRequest) : 239 GeckoResult<AllowOrDeny>? { 240 // this value is ignored 241 return GeckoResult.deny() 242 } 243 }) 244 245 as the page has already loaded by the time the ``forCallbacksDuringWait`` call is 246 executed. 247 248 Tracking Waits 249 -------------- 250 251 To track when a ``wait`` occurs and to know when to replay delegate calls, 252 ``GeckoSessionTestRule`` `stores 253 <https://searchfox.org/mozilla-central/rev/95d8478112eecdd0ee249a941788e03f47df240b/mobile/android/geckoview/src/androidTest/java/org/mozilla/geckoview/test/rule/GeckoSessionTestRule.java#1075>`_ 254 the list of delegate calls in a ``List<CallRecord>`` object, where 255 ``CallRecord`` is a class that has enough information to replay a delegate 256 call. The test rule will track the `start and end index 257 <https://searchfox.org/mozilla-central/rev/95d8478112eecdd0ee249a941788e03f47df240b/mobile/android/geckoview/src/androidTest/java/org/mozilla/geckoview/test/rule/GeckoSessionTestRule.java#1619>`_ 258 of the last wait's delegate calls and `replay it 259 <https://searchfox.org/mozilla-central/rev/95d8478112eecdd0ee249a941788e03f47df240b/mobile/android/geckoview/src/androidTest/java/org/mozilla/geckoview/test/rule/GeckoSessionTestRule.java#1697-1724>`_ 260 when ``forCallbacksDuringWait`` is called. 261 262 To wait until a delegate call happens, the test rule will first `examine 263 <https://searchfox.org/mozilla-central/rev/95d8478112eecdd0ee249a941788e03f47df240b/mobile/android/geckoview/src/androidTest/java/org/mozilla/geckoview/test/rule/GeckoSessionTestRule.java#1585>`_ 264 the already executed delegate calls using the call record list described above. 265 If none of the calls match, then it will `wait for new calls 266 <https://searchfox.org/mozilla-central/rev/95d8478112eecdd0ee249a941788e03f47df240b/mobile/android/geckoview/src/androidTest/java/org/mozilla/geckoview/test/rule/GeckoSessionTestRule.java#1589>`_ 267 to happen, using ``UiThreadUtils.waitForCondition``. 268 269 ``waitForCondition`` is also used to implement other type of ``wait*`` methods 270 like ``waitForResult``, which waits until a ``GeckoResult`` is executed. 271 272 ``waitForCondition`` runs on the UI thread, and it synchronously waits for an 273 event to occur. The events it waits for normally execute on the UI thread as 274 well, so it `injects itself 275 <https://searchfox.org/mozilla-central/rev/95d8478112eecdd0ee249a941788e03f47df240b/mobile/android/geckoview/src/androidTest/java/org/mozilla/geckoview/test/util/UiThreadUtils.java#145,153>`_ 276 in the Android event loop, checking for the condition after every event has 277 executed. If no more events remain in the queue, `it posts a delayed 100ms 278 <https://searchfox.org/mozilla-central/rev/95d8478112eecdd0ee249a941788e03f47df240b/mobile/android/geckoview/src/androidTest/java/org/mozilla/geckoview/test/util/UiThreadUtils.java#136-141>`_ 279 task to avoid clogging the event loop. 280 281 Executing Javascript 282 ==================== 283 284 As you might have noticed from an earlier section, the test rule allows tests 285 to run arbitrary JavaScript code using ``waitForJS``. The GeckoView API, 286 however, doesn't offer such an API. 287 288 The way ``waitForJS`` and ``evaluateJS`` are implemented will be the focus of 289 this section. 290 291 How embedders run javascript 292 ---------------------------- 293 294 The only supported way of accessing a web page for embedders is to `write a 295 built-in WebExtension 296 <https://firefox-source-docs.mozilla.org/mobile/android/geckoview/consumer/web-extensions.html>`_ 297 and install it. This was done intentionally to avoid having to rewrite a lot of 298 the Web-Content-related APIs that the WebExtension API offers. 299 300 GeckoView extends the WebExtension API to allow embedders to communicate to the 301 extension by `overloading 302 <https://searchfox.org/mozilla-central/rev/95d8478112eecdd0ee249a941788e03f47df240b/mobile/android/modules/geckoview/GeckoViewWebExtension.jsm#221>`_ 303 the native messaging API (which is not normally implemented on mobile). 304 Embedders can register themselves as a `native app 305 <https://mozilla.github.io/geckoview/javadoc/mozilla-central/org/mozilla/geckoview/WebExtension.MessageDelegate.html>`_ 306 and the built-in extension will be able to `exchange messages 307 <https://mozilla.github.io/geckoview/javadoc/mozilla-central/org/mozilla/geckoview/WebExtension.Port.html#postMessage-org.json.JSONObject->`_ 308 and `open ports 309 <https://mozilla.github.io/geckoview/javadoc/mozilla-central/org/mozilla/geckoview/WebExtension.MessageDelegate.html#onConnect-org.mozilla.geckoview.WebExtension.Port->`_ 310 with the embedder. 311 312 This is still a controversial topic among smaller embedders, especially solo 313 developers, and we have discussed internally the possibility to expose a 314 simpler API to run one-off javascript snippets, similar to what Chromium's 315 WebView offers, but nothing has been developed so far. 316 317 The test runner extension 318 ------------------------- 319 320 To run arbitrary javascript in GeckoView, the test runner installs a `support 321 extension 322 <https://searchfox.org/mozilla-central/source/mobile/android/geckoview/src/androidTest/assets/web_extensions/test-support>`_. 323 324 The test framework then `establishes 325 <https://searchfox.org/mozilla-central/rev/95d8478112eecdd0ee249a941788e03f47df240b/mobile/android/geckoview/src/androidTest/java/org/mozilla/geckoview/test/rule/GeckoSessionTestRule.java#1827>`_ 326 a port for the background script, used to run code in the main process, and a 327 port for every window, to be able to run javascript on test web pages. 328 329 When ``evaluateJS`` is called, the test framework will send `a message 330 <https://searchfox.org/mozilla-central/rev/95d8478112eecdd0ee249a941788e03f47df240b/mobile/android/geckoview/src/androidTest/java/org/mozilla/geckoview/test/rule/GeckoSessionTestRule.java#1912>`_ 331 to the extension which then `calls eval 332 <https://searchfox.org/mozilla-central/rev/95d8478112eecdd0ee249a941788e03f47df240b/mobile/android/geckoview/src/androidTest/assets/web_extensions/test-support/test-support.js#21>`_ 333 on it and returns the `JSON`-stringified version of the result `back 334 <https://searchfox.org/mozilla-central/rev/95d8478112eecdd0ee249a941788e03f47df240b/mobile/android/geckoview/src/androidTest/java/org/mozilla/geckoview/test/rule/GeckoSessionTestRule.java#1952-1956>`_ 335 to the test framework. 336 337 The test framework also supports promises with `evaluatePromiseJS 338 <https://searchfox.org/mozilla-central/rev/95d8478112eecdd0ee249a941788e03f47df240b/mobile/android/geckoview/src/androidTest/java/org/mozilla/geckoview/test/rule/GeckoSessionTestRule.java#1888>`_. 339 It works similarly to ``evaluateJS`` but instead of returning the stringified 340 value, it `sets 341 <https://searchfox.org/mozilla-central/rev/95d8478112eecdd0ee249a941788e03f47df240b/mobile/android/geckoview/src/androidTest/java/org/mozilla/geckoview/test/rule/GeckoSessionTestRule.java#1879>`_ 342 the return value of the ``eval`` call into the ``this`` object, keyed by a 343 randomly-generated UUID. 344 345 .. code:: java 346 347 this[uuid] = eval(...) 348 349 ``evaluatePromiseJS`` then returns an ``ExtensionPromise`` Java object which 350 has a ``getValue`` method on it, which will essentially execute `await 351 this[uuid] 352 <https://searchfox.org/mozilla-central/rev/95d8478112eecdd0ee249a941788e03f47df240b/mobile/android/geckoview/src/androidTest/java/org/mozilla/geckoview/test/rule/GeckoSessionTestRule.java#1883-1885>`_ 353 to get the value from the promise when needed. 354 355 Beyond executing javascript 356 --------------------------- 357 358 A natural way of breaking the boundaries of the GeckoView API is to run a 359 so-called "experiment extension". Experiment extensions have access to the full 360 Gecko front-end, which is written in JavaScript, and don't have limits on what 361 they can do. Experiment extensions are essentially what old add-ons used to be 362 in Firefox, very powerful and very dangerous. 363 364 The test runner uses experiments to offer `privileged APIs 365 <https://searchfox.org/mozilla-central/rev/95d8478112eecdd0ee249a941788e03f47df240b/mobile/android/geckoview/src/androidTest/assets/web_extensions/test-support/test-api.js>`_ 366 to tests like ``setPref`` or ``getLinkColor`` (which is not normally available 367 to websites for privacy concerns). 368 369 Each privileged API is exposed as an `ordinary Java API 370 <https://searchfox.org/mozilla-central/rev/95d8478112eecdd0ee249a941788e03f47df240b/mobile/android/geckoview/src/androidTest/java/org/mozilla/geckoview/test/rule/GeckoSessionTestRule.java#2101>`_ 371 and the test framework doesn't offer a way to run arbitrary chrome code to 372 discourage developers from relying too much on implementation-dependent 373 privileged code.