From 2fac2b72fed8b1d72a2205d5f62debec13efd2bb Mon Sep 17 00:00:00 2001 From: Gabriela Cervantes Date: Mon, 26 Jun 2023 16:31:55 +0000 Subject: [PATCH 1/3] checkmetrics: Add checkmetrics makefile This PR adds checkmetrics makefile which is used to process the metrics json results files. Fixes #7172 Signed-off-by: Gabriela Cervantes --- tests/metrics/cmd/checkmetrics/Makefile | 19 +++++++++++++++++++ 1 file changed, 19 insertions(+) create mode 100644 tests/metrics/cmd/checkmetrics/Makefile diff --git a/tests/metrics/cmd/checkmetrics/Makefile b/tests/metrics/cmd/checkmetrics/Makefile new file mode 100644 index 0000000000..656b739a5d --- /dev/null +++ b/tests/metrics/cmd/checkmetrics/Makefile @@ -0,0 +1,19 @@ +# Copyright (c) 2023 Intel Corporation +# +# SPDX-License-Identifier: Apache-2.0 + +TARGET := checkmetrics +PREFIX := /usr +BINDIR := $(PREFIX)/bin +DESTTARGET := $(BINDIR)/$(TARGET) + +all: + go build -ldflags "-X main.sysBaseFile=$(DESTBASE)" -o $(TARGET) + +install: + install -D $(TARGET) $(DESTTARGET) + +clean: + rm -f $(DESTTARGET) + +.PHONY: install clean From aa7946de4762c596b721313571371c8a2006e809 Mon Sep 17 00:00:00 2001 From: Gabriela Cervantes Date: Mon, 26 Jun 2023 17:02:45 +0000 Subject: [PATCH 2/3] checkmetrics: Add general checkmetrics documentation This PR adds the general checkmetrics documentation for kata metrics tests. Signed-off-by: Gabriela Cervantes --- tests/metrics/cmd/checkmetrics/README.md | 171 +++++++++++++++++++++++ 1 file changed, 171 insertions(+) create mode 100644 tests/metrics/cmd/checkmetrics/README.md diff --git a/tests/metrics/cmd/checkmetrics/README.md b/tests/metrics/cmd/checkmetrics/README.md new file mode 100644 index 0000000000..fd705100e5 --- /dev/null +++ b/tests/metrics/cmd/checkmetrics/README.md @@ -0,0 +1,171 @@ +# `checkmetrics` + +## Overview + +The checkmetrics tool is used to check the metrics results files in +JSON format. Results files are checked against configs stored in a TOML +file that contains baseline expectations for the results. + +`checkmetrics` checks for a matching results file for each entry in the +TOML file with an appropriate json file extension. Failure to find a matching +file is classified as a failure for that individual TOML entry. + +`checkmetrics` continues to process all entries in the TOML file and prints its +final results in a summary table to stdout. + +`checkmetrics` exits with a failure code if any of the TOML entries did not +complete successfully. + +### JSON file format + +JSON results files only need to be valid JSON, and contain some form of numeric results +that can be extracted into a string or list of numeric results using the jq JSON query tool. + +## baseline TOML layout + +The baseline TOML file is composed of one `[[metric]]` section per result that is processed. +Each section contains a number of parameters, some optional: + +``` +|name | type | description | +|---------------------------------------------------------------------------- +|name | string | Filename containing results (minus .json ext.)| +|type | string | json (optional, json is the default) | +|description | string | Description of test (optional) | +|checkvar | string | jq query string to extract results from JSON | +|checktype | string | Property to check ("mean", "max" etc.) | +|minval | float | Minimum value the checked property should be | +|maxval | float | Maximum value the checked property should be | +|midval | float | Middle value used for percentage range check | +|minpercent | float | Minimum percentage from midval check boundary | +|maxpercent | float | Maximum percentage from midval check boundary | +``` + +### Supported file types + +At this time only JSON formatted results files are supported. + +### Supported checktypes + +The following checktypes are supported. All are tested to fall within the bounds set by the minval +and maxval. That is: + +> `minval <= Result <= maxval` + +``` +|check |description | +|-----------------------------------------------------------------| +|mean |the mean of all the results extracted by the jq query | +|min |the minimum (smallest) result | +|max |the maximum (largest) result | +|sd |the standard deviation of the results | +|cov the coefficient of variation (relative standard deviation)| +``` + +## Options + +`checkmetrics` takes a number of options. Some are mandatory. + +### TOML base file path (mandatory) + +``` +--basefile value path to baseline TOML metrics file +``` + +### Debug mode + +``` +--debug enable debug output in the log +``` + +### Log file path + +``` +--log value set the log file path +``` + +### Metrics results directory path (mandatory) + +``` +--metricsdir value directory containing results files +``` + +### Percentage presentation mode + +``` +--percentage present results as percentage differences +``` + +### Help + +``` +--help, -h show help +``` + +### Version + +``` +--version, -v print the version +``` + +## Output + +The checkmetrics tool outputs a summary table after processing all metrics sections, and returns +a non-zero return code if any of the metrics checks fail. + +Example output: + +``` +Report Summary: ++-----+----------------------+-----------+-----------+-----------+-------+-----------+-----------+------+------+-----+ +| P/F | NAME | FLR | MEAN | CEIL | GAP | MIN | MAX | RNG | COV | ITS | ++-----+----------------------+-----------+-----------+-----------+-------+-----------+-----------+------+------+-----+ +| F | boot-times | 0.50 | 1.36 | 0.70 | 40.0% | 1.34 | 1.38 | 2.7% | 1.3% | 2 | +| F | memory-footprint | 100000.00 | 284570.56 | 110000.00 | 10.0% | 284570.56 | 284570.56 | 0.0% | 0.0% | 1 | +| P | memory-footprint-ksm | 100000.00 | 101770.22 | 110000.00 | 10.0% | 101770.22 | 101770.22 | 0.0% | 0.0% | 1 | ++-----+----------------------+-----------+-----------+-----------+-------+-----------+-----------+------+------+-----+ +Fails: 2, Passes 1 +``` + +Example percentage mode output: + +``` +Report Summary: ++-----+----------------------+-------+--------+--------+-------+--------+--------+------+------+-----+ +| P/F | NAME | FLR | MEAN | CEIL | GAP | MIN | MAX | RNG | COV | ITS | ++-----+----------------------+-------+--------+--------+-------+--------+--------+------+------+-----+ +| *F* | boot-times | 83.3% | 226.8% | 116.7% | 33.3% | 223.8% | 229.8% | 2.7% | 1.3% | 2 | +| *F* | memory-footprint | 95.2% | 271.0% | 104.8% | 9.5% | 271.0% | 271.0% | 0.0% | 0.0% | 1 | +| P | memory-footprint-ksm | 92.7% | 99.3% | 107.3% | 14.6% | 99.3% | 99.3% | 0.0% | 0.0% | 1 | ++-----+----------------------+-------+--------+--------+-------+--------+--------+------+------+-----+ +Fails: 2, Passes 1 +``` + +### Output Columns + +``` +|name |description | +|-----------------------------------------------------------------------| +|P/F |Pass/Fail | +|NAME |Name of the test/check | +|FLR |Floor - the minval to check against | +|MEAN |The mean of the results | +|CEIL |Ceiling - the maxval to check against | +|GAP |The range (gap) between the minval and maxval, as a % of minval| +|MIN |The minimum result in the data set | +|MAX |The maximum result in the data set | +|RNG |The % range (spread) between the min and max result, WRT min | +|COV |The coefficient of variation of the results | +|ITS |The number of results (iterations) | +``` + +## Example invocation + +For example, to invoke the checkmetrics tool, enter the following: + +``` +BASEFILE=`pwd`/../../metrics/baseline/baseline.toml +METRICSDIR=`pwd`/../../metrics/results + +$ ./checkmetrics --basefile ${BASEFILE} --metricsdir ${METRICSDIR} +``` From 415578cf3b86b66af910beff1e137f23a257dfc2 Mon Sep 17 00:00:00 2001 From: Gabriela Cervantes Date: Mon, 26 Jun 2023 17:49:11 +0000 Subject: [PATCH 3/3] docs: Add general README This PR adds link to the unreference docs in the cmd path to make them more discoverable. Signed-off-by: Gabriela Cervantes --- tests/metrics/README.md | 4 +++ tests/metrics/cmd/checkmetrics/README.md | 36 ++++++++++++------------ 2 files changed, 22 insertions(+), 18 deletions(-) diff --git a/tests/metrics/README.md b/tests/metrics/README.md index 4e2d118700..09091c9999 100644 --- a/tests/metrics/README.md +++ b/tests/metrics/README.md @@ -190,3 +190,7 @@ happen if `JSON_URL` is set. The settings are not mutually exclusive, and both c set if necessary. `JSON_TX_ONELINE` applies to both types of transmission. + +## `checkmetrics` + +`checkmetrics` is a CLI tool to check a metrics CI results file. For further reference see the [`checkmetrics`](cmd/checkmetrics). diff --git a/tests/metrics/cmd/checkmetrics/README.md b/tests/metrics/cmd/checkmetrics/README.md index fd705100e5..5010c9f396 100644 --- a/tests/metrics/cmd/checkmetrics/README.md +++ b/tests/metrics/cmd/checkmetrics/README.md @@ -2,12 +2,12 @@ ## Overview -The checkmetrics tool is used to check the metrics results files in +The `checkmetrics` tool is used to check the metrics results files in JSON format. Results files are checked against configs stored in a TOML file that contains baseline expectations for the results. `checkmetrics` checks for a matching results file for each entry in the -TOML file with an appropriate json file extension. Failure to find a matching +TOML file with an appropriate `json` file extension. Failure to find a matching file is classified as a failure for that individual TOML entry. `checkmetrics` continues to process all entries in the TOML file and prints its @@ -27,28 +27,28 @@ The baseline TOML file is composed of one `[[metric]]` section per result that i Each section contains a number of parameters, some optional: ``` -|name | type | description | +|name | type | description | |---------------------------------------------------------------------------- -|name | string | Filename containing results (minus .json ext.)| -|type | string | json (optional, json is the default) | -|description | string | Description of test (optional) | -|checkvar | string | jq query string to extract results from JSON | -|checktype | string | Property to check ("mean", "max" etc.) | -|minval | float | Minimum value the checked property should be | -|maxval | float | Maximum value the checked property should be | -|midval | float | Middle value used for percentage range check | -|minpercent | float | Minimum percentage from midval check boundary | -|maxpercent | float | Maximum percentage from midval check boundary | +|`name` | string | Filename containing results (minus .json ext.)| +|`type` | string | json (optional, json is the default) | +|`description` | string | Description of test (optional) | +|`checkvar` | string | jq query string to extract results from JSON | +|`checktype` | string | Property to check ("mean", "max" etc.) | +|`minval` | float | Minimum value the checked property should be | +|`maxval` | float | Maximum value the checked property should be | +|`midval` | float | Middle value used for percentage range check | +|`minpercent` | float | Minimum percentage from midval check boundary | +|`maxpercent` | float | Maximum percentage from midval check boundary | ``` ### Supported file types At this time only JSON formatted results files are supported. -### Supported checktypes +### Supported `checktypes` -The following checktypes are supported. All are tested to fall within the bounds set by the minval -and maxval. That is: +The following `checktypes` are supported. All are tested to fall within the bounds set by the `minval` +and `maxval`. That is: > `minval <= Result <= maxval` @@ -110,7 +110,7 @@ and maxval. That is: ## Output -The checkmetrics tool outputs a summary table after processing all metrics sections, and returns +The `checkmetrics` tool outputs a summary table after processing all metrics sections, and returns a non-zero return code if any of the metrics checks fail. Example output: @@ -161,7 +161,7 @@ Fails: 2, Passes 1 ## Example invocation -For example, to invoke the checkmetrics tool, enter the following: +For example, to invoke the `checkmetrics` tool, enter the following: ``` BASEFILE=`pwd`/../../metrics/baseline/baseline.toml