mirror of
https://github.com/kata-containers/kata-containers.git
synced 2025-08-20 17:04:02 +00:00
415578cf3b
This PR adds link to the unreference docs in the cmd path to make them more discoverable. Signed-off-by: Gabriela Cervantes <gabriela.cervantes.tellez@intel.com>
172 lines
6.3 KiB
Markdown
172 lines
6.3 KiB
Markdown
# `checkmetrics`
|
|
|
|
## Overview
|
|
|
|
The `checkmetrics` tool is used to check the metrics results files in
|
|
JSON format. Results files are checked against configs stored in a TOML
|
|
file that contains baseline expectations for the results.
|
|
|
|
`checkmetrics` checks for a matching results file for each entry in the
|
|
TOML file with an appropriate `json` file extension. Failure to find a matching
|
|
file is classified as a failure for that individual TOML entry.
|
|
|
|
`checkmetrics` continues to process all entries in the TOML file and prints its
|
|
final results in a summary table to stdout.
|
|
|
|
`checkmetrics` exits with a failure code if any of the TOML entries did not
|
|
complete successfully.
|
|
|
|
### JSON file format
|
|
|
|
JSON results files only need to be valid JSON, and contain some form of numeric results
|
|
that can be extracted into a string or list of numeric results using the jq JSON query tool.
|
|
|
|
## baseline TOML layout
|
|
|
|
The baseline TOML file is composed of one `[[metric]]` section per result that is processed.
|
|
Each section contains a number of parameters, some optional:
|
|
|
|
```
|
|
|name | type | description |
|
|
|----------------------------------------------------------------------------
|
|
|`name` | string | Filename containing results (minus .json ext.)|
|
|
|`type` | string | json (optional, json is the default) |
|
|
|`description` | string | Description of test (optional) |
|
|
|`checkvar` | string | jq query string to extract results from JSON |
|
|
|`checktype` | string | Property to check ("mean", "max" etc.) |
|
|
|`minval` | float | Minimum value the checked property should be |
|
|
|`maxval` | float | Maximum value the checked property should be |
|
|
|`midval` | float | Middle value used for percentage range check |
|
|
|`minpercent` | float | Minimum percentage from midval check boundary |
|
|
|`maxpercent` | float | Maximum percentage from midval check boundary |
|
|
```
|
|
|
|
### Supported file types
|
|
|
|
At this time only JSON formatted results files are supported.
|
|
|
|
### Supported `checktypes`
|
|
|
|
The following `checktypes` are supported. All are tested to fall within the bounds set by the `minval`
|
|
and `maxval`. That is:
|
|
|
|
> `minval <= Result <= maxval`
|
|
|
|
```
|
|
|check |description |
|
|
|-----------------------------------------------------------------|
|
|
|mean |the mean of all the results extracted by the jq query |
|
|
|min |the minimum (smallest) result |
|
|
|max |the maximum (largest) result |
|
|
|sd |the standard deviation of the results |
|
|
|cov the coefficient of variation (relative standard deviation)|
|
|
```
|
|
|
|
## Options
|
|
|
|
`checkmetrics` takes a number of options. Some are mandatory.
|
|
|
|
### TOML base file path (mandatory)
|
|
|
|
```
|
|
--basefile value path to baseline TOML metrics file
|
|
```
|
|
|
|
### Debug mode
|
|
|
|
```
|
|
--debug enable debug output in the log
|
|
```
|
|
|
|
### Log file path
|
|
|
|
```
|
|
--log value set the log file path
|
|
```
|
|
|
|
### Metrics results directory path (mandatory)
|
|
|
|
```
|
|
--metricsdir value directory containing results files
|
|
```
|
|
|
|
### Percentage presentation mode
|
|
|
|
```
|
|
--percentage present results as percentage differences
|
|
```
|
|
|
|
### Help
|
|
|
|
```
|
|
--help, -h show help
|
|
```
|
|
|
|
### Version
|
|
|
|
```
|
|
--version, -v print the version
|
|
```
|
|
|
|
## Output
|
|
|
|
The `checkmetrics` tool outputs a summary table after processing all metrics sections, and returns
|
|
a non-zero return code if any of the metrics checks fail.
|
|
|
|
Example output:
|
|
|
|
```
|
|
Report Summary:
|
|
+-----+----------------------+-----------+-----------+-----------+-------+-----------+-----------+------+------+-----+
|
|
| P/F | NAME | FLR | MEAN | CEIL | GAP | MIN | MAX | RNG | COV | ITS |
|
|
+-----+----------------------+-----------+-----------+-----------+-------+-----------+-----------+------+------+-----+
|
|
| F | boot-times | 0.50 | 1.36 | 0.70 | 40.0% | 1.34 | 1.38 | 2.7% | 1.3% | 2 |
|
|
| F | memory-footprint | 100000.00 | 284570.56 | 110000.00 | 10.0% | 284570.56 | 284570.56 | 0.0% | 0.0% | 1 |
|
|
| P | memory-footprint-ksm | 100000.00 | 101770.22 | 110000.00 | 10.0% | 101770.22 | 101770.22 | 0.0% | 0.0% | 1 |
|
|
+-----+----------------------+-----------+-----------+-----------+-------+-----------+-----------+------+------+-----+
|
|
Fails: 2, Passes 1
|
|
```
|
|
|
|
Example percentage mode output:
|
|
|
|
```
|
|
Report Summary:
|
|
+-----+----------------------+-------+--------+--------+-------+--------+--------+------+------+-----+
|
|
| P/F | NAME | FLR | MEAN | CEIL | GAP | MIN | MAX | RNG | COV | ITS |
|
|
+-----+----------------------+-------+--------+--------+-------+--------+--------+------+------+-----+
|
|
| *F* | boot-times | 83.3% | 226.8% | 116.7% | 33.3% | 223.8% | 229.8% | 2.7% | 1.3% | 2 |
|
|
| *F* | memory-footprint | 95.2% | 271.0% | 104.8% | 9.5% | 271.0% | 271.0% | 0.0% | 0.0% | 1 |
|
|
| P | memory-footprint-ksm | 92.7% | 99.3% | 107.3% | 14.6% | 99.3% | 99.3% | 0.0% | 0.0% | 1 |
|
|
+-----+----------------------+-------+--------+--------+-------+--------+--------+------+------+-----+
|
|
Fails: 2, Passes 1
|
|
```
|
|
|
|
### Output Columns
|
|
|
|
```
|
|
|name |description |
|
|
|-----------------------------------------------------------------------|
|
|
|P/F |Pass/Fail |
|
|
|NAME |Name of the test/check |
|
|
|FLR |Floor - the minval to check against |
|
|
|MEAN |The mean of the results |
|
|
|CEIL |Ceiling - the maxval to check against |
|
|
|GAP |The range (gap) between the minval and maxval, as a % of minval|
|
|
|MIN |The minimum result in the data set |
|
|
|MAX |The maximum result in the data set |
|
|
|RNG |The % range (spread) between the min and max result, WRT min |
|
|
|COV |The coefficient of variation of the results |
|
|
|ITS |The number of results (iterations) |
|
|
```
|
|
|
|
## Example invocation
|
|
|
|
For example, to invoke the `checkmetrics` tool, enter the following:
|
|
|
|
```
|
|
BASEFILE=`pwd`/../../metrics/baseline/baseline.toml
|
|
METRICSDIR=`pwd`/../../metrics/results
|
|
|
|
$ ./checkmetrics --basefile ${BASEFILE} --metricsdir ${METRICSDIR}
|
|
```
|