perfdocs.rst (5433B)
1 PerfDocs 2 ======== 3 4 `PerfDocs`_ is a tool that checks to make sure all performance tests are documented in tree. 5 6 At the moment, it is only used for this documentation verification, but in the future it will also auto-generate documentation from these descriptions that will be displayed in the source-docs documentation page (rather than the wiki, which is where they currently reside). 7 8 Run Locally 9 ----------- 10 11 The mozlint integration of PerfDocs can be run using mach: 12 13 .. parsed-literal:: 14 15 $ mach lint --linter perfdocs . 16 17 Documentation can be regenerated for performance tests by including the ``--fix`` flag: 18 19 .. parsed-literal:: 20 21 $ mach lint --linter perfdocs . --fix 22 23 24 Configuration 25 ------------- 26 27 There are no configuration options available for this linter. It scans the full source tree under ``testing``, looking for folders named ``perfdocs``, validates their content, and regenerates the documentation (if ``--fix`` is provided). This has been implemented for all performance testing harnesses, and the documentation generated gets displayed in :ref:`Performance Testing`. 28 29 In the ``perfdocs`` folders, there needs to be an ``index.rst`` file and it needs to contain the string ``{documentation}`` in some location in the file which is where the test documentation will be placed. The folders must also have a ``config.yml`` file following this schema: 30 31 .. code-block:: python 32 33 CONFIG_SCHEMA = { 34 "definitions": { 35 "metrics_schema": { 36 "metric_name": { 37 "type": "object", 38 "properties": { 39 "aliases": {"type": "array", "items": {"type": "string"}}, 40 "description": {"type": "string"}, 41 "matcher": {"type": "string"}, 42 }, 43 "required": ["description", "aliases"], 44 }, 45 }, 46 }, 47 "type": "object", 48 "properties": { 49 "name": {"type": "string"}, 50 "manifest": {"type": "string"}, 51 "static-only": {"type": "boolean"}, 52 "metrics": {"$ref": "#/definitions/metrics_schema"}, 53 "suites": { 54 "type": "object", 55 "properties": { 56 "suite_name": { 57 "type": "object", 58 "properties": { 59 "tests": { 60 "type": "object", 61 "properties": { 62 "test_name": {"type": "string"}, 63 }, 64 }, 65 "description": {"type": "string"}, 66 "owner": {"type": "string"}, 67 }, 68 "required": ["description"], 69 } 70 }, 71 }, 72 }, 73 "required": ["name", "manifest", "static-only", "suites"], 74 } 75 76 Here is an example of a configuration file for the Raptor framework: 77 78 .. parsed-literal:: 79 80 name: raptor 81 manifest: testing/raptor/raptor/raptor.toml 82 suites: 83 desktop: 84 description: "Desktop tests." 85 tests: 86 raptor-tp6: "Raptor TP6 tests." 87 mobile: 88 description: "Mobile tests" 89 benchmarks: 90 description: "Benchmark tests." 91 tests: 92 wasm: "All wasm tests." 93 94 Metrics that produce alerts can also be documented like so: 95 96 .. parsed-literal:: 97 98 name: raptor 99 manifest: testing/raptor/raptor/raptor.toml 100 metrics: 101 "First Paint": 102 description: "The description of the metric." 103 aliases: 104 - fcp 105 - anAliasForFCP 106 # Optional regex to match the metrics found in the tests with this 107 # documented metric 108 matcher: f.* 109 suites: 110 desktop: 111 description: "Desktop tests." 112 tests: 113 raptor-tp6: "Raptor TP6 tests." 114 mobile: 115 description: "Mobile tests" 116 benchmarks: 117 description: "Benchmark tests." 118 tests: 119 wasm: "All wasm tests." 120 121 The documented metrics must exist in the tests for the suite. If they are not, then validation will fail. The same is true if a metric in a test is not documented. Also, if ``metrics`` are defined, then a ``metrics.rst`` file is expected to be found in the ``perfdocs`` folder for the given suite. It must contain the string ``{metrics_documentation}`` where the documentation should be added. The ``metrics.rst`` is renamed ``{suite-name}-metrics.rst`` in the generated folder, so if it needs to be linked to in the ``index.rst`` file, it should contain a ``{metrics_rst_name}`` string for where the link should be added - it's expected to be found in a toctree section. 122 123 Note that there needs to be a FrameworkGatherer implemented for the framework being documented since each of them may have different ways of parsing test manifests for the tests. See `RaptorGatherer <https://searchfox.org/mozilla-central/source/tools/lint/perfdocs/framework_gatherers.py>`_ for an example gatherer that was implemented for Raptor. 124 125 Sources 126 ------- 127 128 * `Configuration <https://searchfox.org/mozilla-central/source/tools/lint/perfdocs.yml>`__ 129 * `Source <https://searchfox.org/mozilla-central/source/tools/lint/perfdocs>`__