doc: Add required jobs info

Add information about what required jobs are and
our initial guidelines for how jobs are eligible for being
made required, or non-required

Signed-off-by: stevenhorsman <steven@uk.ibm.com>

ci/README.md

@@ -41,7 +41,7 @@ responsible for ensuring that:
 
 ### Jobs that require a maintainer's approval to run
 
-These are the required tests, and our so-called "CI".  These require a
+There are some tests, and our so-called "CI".  These require a
 maintainer's approval to run as parts of those jobs will be running on "paid
 runners", which are currently using Azure infrastructure.
 
@@ -77,7 +77,7 @@ them to merely debug issues.
 
 In the previous section we've mentioned using different runners, now in this section we'll go through each type of runner used.
 
-- Cost free runners:  Those are the runners provided by Github itself, and
+- Cost free runners:  Those are the runners provided by GitHub itself, and
   those are fairly small machines with virtualization capabilities enabled.
 - Azure small instances: Those are runners which have virtualization
   capabilities enabled, 2 CPUs, and 8GB of RAM.  These runners have a "-smaller"
@@ -138,6 +138,63 @@ Following those examples, the community advice during the review, and even
 asking the community directly on Slack are the best ways to get your test
 accepted.
 
+## Required tests
+
+In our CI we have two categories of jobs - required and non-required:
+- Required jobs need to all pass for a PR to be merged normally and
+should cover all the core features on Kata Containers that we want to
+ensure don't have regressions.
+- The non-required jobs are for unstable tests, or for features that
+are experimental and not-fully supported. We'd like those tests to also
+pass on all PRs ideally, but don't block merging if they don't as it's
+not necessarily an indication of the PR code causing regressions.
+
+### Transitioning between required and non-required status
+
+Required jobs that fail block merging of PRs, so we want to ensure that
+jobs are stable and maintained before we make them required.
+
+The [Kata Containers CI Dashboard](https://kata-containers.github.io/)
+is a useful resource to check when collecting evidence of job stability.
+At time of writing it reports the last ten days of Kata CI nightly test
+results for each job. This isn't perfect as it doesn't currently capture
+results on PRs, but is a good guideline for stability.
+
+> [!NOTE]
+> Below are general guidelines about jobs being marked as
+> required/non-required, but they are subject to change and the Kata
+> Architecture Committee may overrule these guidelines at their
+> discretion.
+
+#### Initial marking as required
+
+For new jobs, or jobs that haven't been marked as required recently,
+the criteria to be initially marked as required is ten days
+of passing tests, with no relevant PR failures reported in that time.
+Required jobs also need one or more nominated maintainers that are
+responsible for the stability of their jobs.
+
+> [!NOTE]
+> We don't currently have a good place to record the job maintainers, but
+> once we have this, the intention is to show it on the CI Dashboard so
+> people can find the contact easily.
+
+#### Expectation of required job maintainers
+
+Due to the nature of the Kata Containers community having contributors
+spread around the world, required jobs being blocked due to infrastructure,
+or test issues can have a big impact on work. As such, the expectation is
+that when a problem with a required job is noticed/reported, the maintainers
+have one working day to acknowledge the issue, perform an initial
+investigation and then either fix it, or get it marked as non-required
+whilst the investigation and/or fix it done.
+
+### Re-marking of required status
+
+Once a job has been removed from the required list, it requires two
+consecutive successful nightly test runs before being made required
+again.
+
 ## Running tests
 
 ### Running the tests as part of the CI