diff --git a/tests/metrics/cmd/checkmetrics/README.md b/tests/metrics/cmd/checkmetrics/README.md new file mode 100644 index 0000000000..fd705100e5 --- /dev/null +++ b/tests/metrics/cmd/checkmetrics/README.md @@ -0,0 +1,171 @@ +# `checkmetrics` + +## Overview + +The checkmetrics tool is used to check the metrics results files in +JSON format. Results files are checked against configs stored in a TOML +file that contains baseline expectations for the results. + +`checkmetrics` checks for a matching results file for each entry in the +TOML file with an appropriate json file extension. Failure to find a matching +file is classified as a failure for that individual TOML entry. + +`checkmetrics` continues to process all entries in the TOML file and prints its +final results in a summary table to stdout. + +`checkmetrics` exits with a failure code if any of the TOML entries did not +complete successfully. + +### JSON file format + +JSON results files only need to be valid JSON, and contain some form of numeric results +that can be extracted into a string or list of numeric results using the jq JSON query tool. + +## baseline TOML layout + +The baseline TOML file is composed of one `[[metric]]` section per result that is processed. +Each section contains a number of parameters, some optional: + +``` +|name | type | description | +|---------------------------------------------------------------------------- +|name | string | Filename containing results (minus .json ext.)| +|type | string | json (optional, json is the default) | +|description | string | Description of test (optional) | +|checkvar | string | jq query string to extract results from JSON | +|checktype | string | Property to check ("mean", "max" etc.) | +|minval | float | Minimum value the checked property should be | +|maxval | float | Maximum value the checked property should be | +|midval | float | Middle value used for percentage range check | +|minpercent | float | Minimum percentage from midval check boundary | +|maxpercent | float | Maximum percentage from midval check boundary | +``` + +### Supported file types + +At this time only JSON formatted results files are supported. + +### Supported checktypes + +The following checktypes are supported. All are tested to fall within the bounds set by the minval +and maxval. That is: + +> `minval <= Result <= maxval` + +``` +|check |description | +|-----------------------------------------------------------------| +|mean |the mean of all the results extracted by the jq query | +|min |the minimum (smallest) result | +|max |the maximum (largest) result | +|sd |the standard deviation of the results | +|cov the coefficient of variation (relative standard deviation)| +``` + +## Options + +`checkmetrics` takes a number of options. Some are mandatory. + +### TOML base file path (mandatory) + +``` +--basefile value path to baseline TOML metrics file +``` + +### Debug mode + +``` +--debug enable debug output in the log +``` + +### Log file path + +``` +--log value set the log file path +``` + +### Metrics results directory path (mandatory) + +``` +--metricsdir value directory containing results files +``` + +### Percentage presentation mode + +``` +--percentage present results as percentage differences +``` + +### Help + +``` +--help, -h show help +``` + +### Version + +``` +--version, -v print the version +``` + +## Output + +The checkmetrics tool outputs a summary table after processing all metrics sections, and returns +a non-zero return code if any of the metrics checks fail. + +Example output: + +``` +Report Summary: ++-----+----------------------+-----------+-----------+-----------+-------+-----------+-----------+------+------+-----+ +| P/F | NAME | FLR | MEAN | CEIL | GAP | MIN | MAX | RNG | COV | ITS | ++-----+----------------------+-----------+-----------+-----------+-------+-----------+-----------+------+------+-----+ +| F | boot-times | 0.50 | 1.36 | 0.70 | 40.0% | 1.34 | 1.38 | 2.7% | 1.3% | 2 | +| F | memory-footprint | 100000.00 | 284570.56 | 110000.00 | 10.0% | 284570.56 | 284570.56 | 0.0% | 0.0% | 1 | +| P | memory-footprint-ksm | 100000.00 | 101770.22 | 110000.00 | 10.0% | 101770.22 | 101770.22 | 0.0% | 0.0% | 1 | ++-----+----------------------+-----------+-----------+-----------+-------+-----------+-----------+------+------+-----+ +Fails: 2, Passes 1 +``` + +Example percentage mode output: + +``` +Report Summary: ++-----+----------------------+-------+--------+--------+-------+--------+--------+------+------+-----+ +| P/F | NAME | FLR | MEAN | CEIL | GAP | MIN | MAX | RNG | COV | ITS | ++-----+----------------------+-------+--------+--------+-------+--------+--------+------+------+-----+ +| *F* | boot-times | 83.3% | 226.8% | 116.7% | 33.3% | 223.8% | 229.8% | 2.7% | 1.3% | 2 | +| *F* | memory-footprint | 95.2% | 271.0% | 104.8% | 9.5% | 271.0% | 271.0% | 0.0% | 0.0% | 1 | +| P | memory-footprint-ksm | 92.7% | 99.3% | 107.3% | 14.6% | 99.3% | 99.3% | 0.0% | 0.0% | 1 | ++-----+----------------------+-------+--------+--------+-------+--------+--------+------+------+-----+ +Fails: 2, Passes 1 +``` + +### Output Columns + +``` +|name |description | +|-----------------------------------------------------------------------| +|P/F |Pass/Fail | +|NAME |Name of the test/check | +|FLR |Floor - the minval to check against | +|MEAN |The mean of the results | +|CEIL |Ceiling - the maxval to check against | +|GAP |The range (gap) between the minval and maxval, as a % of minval| +|MIN |The minimum result in the data set | +|MAX |The maximum result in the data set | +|RNG |The % range (spread) between the min and max result, WRT min | +|COV |The coefficient of variation of the results | +|ITS |The number of results (iterations) | +``` + +## Example invocation + +For example, to invoke the checkmetrics tool, enter the following: + +``` +BASEFILE=`pwd`/../../metrics/baseline/baseline.toml +METRICSDIR=`pwd`/../../metrics/results + +$ ./checkmetrics --basefile ${BASEFILE} --metricsdir ${METRICSDIR} +```