Merge pull request #7173 from GabyCT/topic/addcheckm

checkmetrics: Add checkmetrics makefile and documentation

tests/metrics/README.md

@@ -192,3 +192,7 @@ happen if `JSON_URL` is set. The settings are not mutually exclusive, and both c
 set if necessary.
 
 `JSON_TX_ONELINE` applies to both types of transmission.
+
+## `checkmetrics`
+
+`checkmetrics` is a CLI tool to check a metrics CI results file. For further reference see the [`checkmetrics`](cmd/checkmetrics).

tests/metrics/cmd/checkmetrics/Makefile

@@ -0,0 +1,19 @@
+# Copyright (c) 2023 Intel Corporation
+#
+# SPDX-License-Identifier: Apache-2.0
+
+TARGET     := checkmetrics
+PREFIX     := /usr
+BINDIR     := $(PREFIX)/bin
+DESTTARGET := $(BINDIR)/$(TARGET)
+
+all:
+	go build -ldflags "-X main.sysBaseFile=$(DESTBASE)" -o $(TARGET)
+
+install:
+	install -D $(TARGET) $(DESTTARGET)
+
+clean:
+	rm -f $(DESTTARGET)
+
+.PHONY: install clean

tests/metrics/cmd/checkmetrics/README.md

@@ -0,0 +1,171 @@
+# `checkmetrics`
+
+## Overview
+
+The `checkmetrics` tool is used to check the metrics results files in 
+JSON format. Results files are checked against configs stored in a TOML
+file that contains baseline expectations for the results.
+
+`checkmetrics` checks for a matching results file for each entry in the
+TOML file with an appropriate `json` file extension. Failure to find a matching
+file is classified as a failure for that individual TOML entry.
+
+`checkmetrics` continues to process all entries in the TOML file and prints its
+final results in a summary table to stdout.
+
+`checkmetrics` exits with a failure code if any of the TOML entries did not
+complete successfully.
+
+### JSON file format
+
+JSON results files only need to be valid JSON, and contain some form of numeric results
+that can be extracted into a string or list of numeric results using the jq JSON query tool.
+
+## baseline TOML layout
+
+The baseline TOML file is composed of one `[[metric]]` section per result that is processed.
+Each section contains a number of parameters, some optional:
+
+```
+|name	       |   type	  |    description                                   | 
+|----------------------------------------------------------------------------
+|`name`	       |   string	  |    Filename containing results (minus .json ext.)|
+|`type`	       |   string	  |    json (optional, json is the default)          |
+|`description` |   string	  |    Description of test (optional)                |
+|`checkvar`    |   string	  |    jq query string to extract results from JSON  |
+|`checktype`   |   string	  |    Property to check ("mean", "max" etc.)        |
+|`minval`      |   float	  |    Minimum value the checked property should be  |
+|`maxval`      |   float	  |    Maximum value the checked property should be  |
+|`midval`      |   float	  |    Middle value used for percentage range check  |
+|`minpercent`  |   float	  |    Minimum percentage from midval check boundary |
+|`maxpercent`  |   float	  |    Maximum percentage from midval check boundary |
+```
+
+### Supported file types
+
+At this time only JSON formatted results files are supported.
+
+### Supported `checktypes`
+
+The following `checktypes` are supported. All are tested to fall within the bounds set by the `minval`
+and `maxval`. That is:
+
+> `minval <= Result <= maxval`
+
+```
+|check	|description                                              |
+|-----------------------------------------------------------------|
+|mean	|the mean of all the results extracted by the jq query    |
+|min	|the minimum (smallest) result                            |
+|max	|the maximum (largest) result                             |
+|sd	|the standard deviation of the results                    |
+|cov	the coefficient of variation (relative standard deviation)|
+```
+
+## Options
+
+`checkmetrics` takes a number of options. Some are mandatory.
+
+### TOML base file path (mandatory)
+
+```
+--basefile value    path to baseline TOML metrics file
+```
+
+### Debug mode
+
+```
+--debug             enable debug output in the log
+```
+
+### Log file path
+
+```
+--log value         set the log file path
+```
+
+### Metrics results directory path (mandatory)
+
+```
+--metricsdir value  directory containing results files
+```
+
+### Percentage presentation mode
+
+```
+--percentage        present results as percentage differences
+```
+
+### Help
+
+```
+--help, -h          show help
+```
+
+### Version
+
+```
+--version, -v       print the version
+```
+
+## Output
+
+The `checkmetrics` tool outputs a summary table after processing all metrics sections, and returns
+a non-zero return code if any of the metrics checks fail.
+
+Example output:
+
+```
+Report Summary:
++-----+----------------------+-----------+-----------+-----------+-------+-----------+-----------+------+------+-----+
+| P/F |         NAME         |    FLR    |   MEAN    |   CEIL    |  GAP  |    MIN    |    MAX    | RNG  | COV  | ITS |
++-----+----------------------+-----------+-----------+-----------+-------+-----------+-----------+------+------+-----+
+| F   | boot-times           |      0.50 |      1.36 |      0.70 | 40.0% |      1.34 |      1.38 | 2.7% | 1.3% |   2 |
+| F   | memory-footprint     | 100000.00 | 284570.56 | 110000.00 | 10.0% | 284570.56 | 284570.56 | 0.0% | 0.0% |   1 |
+| P   | memory-footprint-ksm | 100000.00 | 101770.22 | 110000.00 | 10.0% | 101770.22 | 101770.22 | 0.0% | 0.0% |   1 |
++-----+----------------------+-----------+-----------+-----------+-------+-----------+-----------+------+------+-----+
+Fails: 2, Passes 1
+```
+
+Example percentage mode output:
+
+```
+Report Summary:
++-----+----------------------+-------+--------+--------+-------+--------+--------+------+------+-----+
+| P/F |         NAME         |  FLR  |  MEAN  |  CEIL  |  GAP  |  MIN   |  MAX   | RNG  | COV  | ITS |
++-----+----------------------+-------+--------+--------+-------+--------+--------+------+------+-----+
+| *F* | boot-times           | 83.3% | 226.8% | 116.7% | 33.3% | 223.8% | 229.8% | 2.7% | 1.3% |   2 |
+| *F* | memory-footprint     | 95.2% | 271.0% | 104.8% | 9.5%  | 271.0% | 271.0% | 0.0% | 0.0% |   1 |
+| P   | memory-footprint-ksm | 92.7% | 99.3%  | 107.3% | 14.6% | 99.3%  | 99.3%  | 0.0% | 0.0% |   1 |
++-----+----------------------+-------+--------+--------+-------+--------+--------+------+------+-----+
+Fails: 2, Passes 1
+```
+
+### Output Columns
+
+```
+|name	|description                                                    |
+|-----------------------------------------------------------------------|
+|P/F	|Pass/Fail                                                      |
+|NAME	|Name of the test/check                                         |
+|FLR	|Floor - the minval to check against                            |
+|MEAN	|The mean of the results                                        |
+|CEIL	|Ceiling - the maxval to check against                          |
+|GAP	|The range (gap) between the minval and maxval, as a % of minval|
+|MIN	|The minimum result in the data set                             |
+|MAX	|The maximum result in the data set                             |
+|RNG	|The % range (spread) between the min and max result, WRT min   |
+|COV	|The coefficient of variation of the results                    |
+|ITS	|The number of results (iterations)                             |
+```
+
+## Example invocation
+
+For example, to invoke the `checkmetrics` tool, enter the following:
+
+```
+BASEFILE=`pwd`/../../metrics/baseline/baseline.toml
+METRICSDIR=`pwd`/../../metrics/results
+
+$ ./checkmetrics --basefile ${BASEFILE} --metricsdir ${METRICSDIR}
+```