kata-containers/tests/metrics/cmd/checkmetrics

checkmetrics

Overview

The checkmetrics tool is used to check the metrics results files in JSON format. Results files are checked against configs stored in a TOML file that contains baseline expectations for the results.

checkmetrics checks for a matching results file for each entry in the TOML file with an appropriate json file extension. Failure to find a matching file is classified as a failure for that individual TOML entry.

checkmetrics continues to process all entries in the TOML file and prints its final results in a summary table to stdout.

checkmetrics exits with a failure code if any of the TOML entries did not complete successfully.

JSON file format

JSON results files only need to be valid JSON, and contain some form of numeric results that can be extracted into a string or list of numeric results using the jq JSON query tool.

baseline TOML layout

The baseline TOML file is composed of one [[metric]] section per result that is processed. Each section contains a number of parameters, some optional:

|name	       |   type	  |    description                                   | 
|----------------------------------------------------------------------------
|`name`	       |   string	  |    Filename containing results (minus .json ext.)|
|`type`	       |   string	  |    json (optional, json is the default)          |
|`description` |   string	  |    Description of test (optional)                |
|`checkvar`    |   string	  |    jq query string to extract results from JSON  |
|`checktype`   |   string	  |    Property to check ("mean", "max" etc.)        |
|`minval`      |   float	  |    Minimum value the checked property should be  |
|`maxval`      |   float	  |    Maximum value the checked property should be  |
|`midval`      |   float	  |    Middle value used for percentage range check  |
|`minpercent`  |   float	  |    Minimum percentage from midval check boundary |
|`maxpercent`  |   float	  |    Maximum percentage from midval check boundary |

Supported file types

At this time only JSON formatted results files are supported.

Supported checktypes

The following checktypes are supported. All are tested to fall within the bounds set by the minval and maxval. That is:

minval <= Result <= maxval

|check	|description                                              |
|-----------------------------------------------------------------|
|mean	|the mean of all the results extracted by the jq query    |
|min	|the minimum (smallest) result                            |
|max	|the maximum (largest) result                             |
|sd	|the standard deviation of the results                    |
|cov	the coefficient of variation (relative standard deviation)|

Options

checkmetrics takes a number of options. Some are mandatory.

TOML base file path (mandatory)

--basefile value    path to baseline TOML metrics file

Debug mode

--debug             enable debug output in the log

Log file path

--log value         set the log file path

Metrics results directory path (mandatory)

--metricsdir value  directory containing results files

Percentage presentation mode

--percentage        present results as percentage differences

Help

--help, -h          show help

Version

--version, -v       print the version

Output

The checkmetrics tool outputs a summary table after processing all metrics sections, and returns a non-zero return code if any of the metrics checks fail.

Example output:

Report Summary:
+-----+----------------------+-----------+-----------+-----------+-------+-----------+-----------+------+------+-----+
| P/F |         NAME         |    FLR    |   MEAN    |   CEIL    |  GAP  |    MIN    |    MAX    | RNG  | COV  | ITS |
+-----+----------------------+-----------+-----------+-----------+-------+-----------+-----------+------+------+-----+
| F   | boot-times           |      0.50 |      1.36 |      0.70 | 40.0% |      1.34 |      1.38 | 2.7% | 1.3% |   2 |
| F   | memory-footprint     | 100000.00 | 284570.56 | 110000.00 | 10.0% | 284570.56 | 284570.56 | 0.0% | 0.0% |   1 |
| P   | memory-footprint-ksm | 100000.00 | 101770.22 | 110000.00 | 10.0% | 101770.22 | 101770.22 | 0.0% | 0.0% |   1 |
+-----+----------------------+-----------+-----------+-----------+-------+-----------+-----------+------+------+-----+
Fails: 2, Passes 1

Example percentage mode output:

Report Summary:
+-----+----------------------+-------+--------+--------+-------+--------+--------+------+------+-----+
| P/F |         NAME         |  FLR  |  MEAN  |  CEIL  |  GAP  |  MIN   |  MAX   | RNG  | COV  | ITS |
+-----+----------------------+-------+--------+--------+-------+--------+--------+------+------+-----+
| *F* | boot-times           | 83.3% | 226.8% | 116.7% | 33.3% | 223.8% | 229.8% | 2.7% | 1.3% |   2 |
| *F* | memory-footprint     | 95.2% | 271.0% | 104.8% | 9.5%  | 271.0% | 271.0% | 0.0% | 0.0% |   1 |
| P   | memory-footprint-ksm | 92.7% | 99.3%  | 107.3% | 14.6% | 99.3%  | 99.3%  | 0.0% | 0.0% |   1 |
+-----+----------------------+-------+--------+--------+-------+--------+--------+------+------+-----+
Fails: 2, Passes 1

Output Columns

|name	|description                                                    |
|-----------------------------------------------------------------------|
|P/F	|Pass/Fail                                                      |
|NAME	|Name of the test/check                                         |
|FLR	|Floor - the minval to check against                            |
|MEAN	|The mean of the results                                        |
|CEIL	|Ceiling - the maxval to check against                          |
|GAP	|The range (gap) between the minval and maxval, as a % of minval|
|MIN	|The minimum result in the data set                             |
|MAX	|The maximum result in the data set                             |
|RNG	|The % range (spread) between the min and max result, WRT min   |
|COV	|The coefficient of variation of the results                    |
|ITS	|The number of results (iterations)                             |

Example invocation

For example, to invoke the checkmetrics tool, enter the following:

BASEFILE=`pwd`/../../metrics/baseline/baseline.toml
METRICSDIR=`pwd`/../../metrics/results

$ ./checkmetrics --basefile ${BASEFILE} --metricsdir ${METRICSDIR}