docs: add release notes for v0.9.0

Add new features, limitations, and expand the hw support section.

Signed-off-by: Tobin Feldman-Fitzthum <tobin@ibm.com>

.github/ISSUE_TEMPLATE/release-check-list.md

@@ -51,6 +51,8 @@ Releases of most subprojects are now decoupled from releases of the CoCo project
 
 ## The Steps
 
+Note: It may be useful when doing these steps to refer to a previous example. The v0.9.0-alpha1 release applied [these changes](https://github.com/confidential-containers/operator/pull/388/files). After following steps 1-5 below, you should end up with a similar set of changes.
+
 ### Determine release builds
 
 Identify/create the bundles that we will release for Kata and enclave-cc.
@@ -70,29 +72,34 @@ Identify/create the bundles that we will release for Kata and enclave-cc.
     If you absolutely cannot use a Kata release,
     you can consider releasing one of these bundles.
 
+- [ ] 3. :eyes: **Create a peer pods release**
+
+    Create a peer pods release based on the Kata release, by following the [documented flow](https://github.com/confidential-containers/cloud-api-adaptor/blob/main/docs/Release-Process.md).
+
 ### Test Release with Operator
 
-- [ ] 3. :eyes: **Check operator pre-installation and open PR if needed**
+- [ ] 4. :eyes: **Check operator pre-installation and open PR if needed**
     
     The operator uses a pre-install container to setup the node.
     Check that the container matches the dependencies used in Kata
     and that the operator pulls the most recent version of the container.
 
     * Check that the version of the `nydus-snapshotter` used by Kata matches the one used by the operator
-        * Compare `nydus-snapshotter` version in Kata [versions.yaml](https://github.com/kata-containers/kata-containers/blob/main/versions.yaml#L325) with the [Makefile](https://github.com/confidential-containers/operator/blob/main/install/pre-install-payload/Makefile#L4) for the operator pre-install container.
+        * Compare the `nydus-snapshotter` version in Kata [versions.yaml](https://github.com/kata-containers/kata-containers/blob/main/versions.yaml) (search for `nydus-snapshotter` and check its `version` field) with the [Makefile](https://github.com/confidential-containers/operator/blob/main/install/pre-install-payload/Makefile) (check the `NYDUS_SNAPSHOTTER_VERSION` value) for the operator pre-install container.
             * **If they do not match, stop and open a PR now. In the PR, update the operator's Makefile to match the version used in kata. After the PR is merged, continue.**
 
-- [ ] 4. :wrench: **Open a PR to the operator to update the release artifacts**
+- [ ] 5. :wrench: **Open a PR to the operator to update the release artifacts**
 
-    Update the operator to use the payloads identified in steps 1, 2, and 3.
+    Update the operator to use the payloads identified in steps 1, 2, 3, and 4.
 
     Make sure that the operator pulls the most recent version of the pre-install container
-        * Find the last commit in the [pre-install directory](https://github.com/confidential-containers/operator/tree/main/install/pre-install-payload)
-            * As a sanity check, the sha hash of the last commit in that pre-install directory will correspond to a pre-install image in quay, i.e. a reqs-payload image [here](quay.io/confidential-containers/reqs-payload).
-            * Make sure that the commit matches the preInstall / postUninstall image specified for [enclave-cc CRD](https://github.com/confidential-containers/operator/blob/main/config/samples/enclave-cc/base/ccruntime-enclave-cc.yaml) and [ccruntime CRD](https://github.com/confidential-containers/operator/blob/main/config/samples/ccruntime/default/kustomization.yaml)
-                * If these do not match (for instance if you changed the snapshotter in step 3), update the operator so that they do match.
 
-    There are a number of places where the payloads are referenced. Make sure to update all of the following to the tag matching the latest commit hash from steps 1 and 2:
+    * Find the last commit in the [pre-install directory](https://github.com/confidential-containers/operator/tree/main/install/pre-install-payload)
+        * As a sanity check, the sha hash of the last commit in that pre-install directory will correspond to a pre-install image in quay, i.e. a reqs-payload image [here](https://quay.io/confidential-containers/reqs-payload).
+    * Make sure that the commit matches the preInstall / postUninstall image specified for [enclave-cc CRD](https://github.com/confidential-containers/operator/blob/main/config/samples/enclave-cc/base/ccruntime-enclave-cc.yaml) and [ccruntime CRD](https://github.com/confidential-containers/operator/blob/main/config/samples/ccruntime/default/kustomization.yaml)
+        * If these do not match (for instance if you changed the snapshotter in step 4), update the operator so that they do match.
+
+    There are a number of places where the payloads are referenced. Make sure to update all of the following to the tag matching the latest commit hash from steps 1, 2, and 3:
     * Enclave CC:
       * [sim](https://github.com/confidential-containers/operator/blob/main/config/samples/enclave-cc/sim/kustomization.yaml)
       * [hw](https://github.com/confidential-containers/operator/blob/main/config/samples/enclave-cc/hw/kustomization.yaml)
@@ -103,17 +110,17 @@ Identify/create the bundles that we will release for Kata and enclave-cc.
       * [peer-pods](https://github.com/confidential-containers/operator/blob/main/config/samples/ccruntime/peer-pods/kustomization.yaml)
           Note that we need the quay.io/confidential-containers/runtime-payload-ci registry and kata-containers-latest tag
 
-    **Also, update the [operator version](https://github.com/confidential-containers/operator/blob/main/config/release/kustomization.yaml#L7)**
+    **Also, update the [operator version](https://github.com/confidential-containers/operator/blob/main/config/release/kustomization.yaml) (update the `newTag` value)**
 
 ### Final Touches
 
-- [ ] 5. :trophy: **Cut an operator release using the GitHub release tool**
+- [ ] 6. :trophy: **Cut an operator release using the GitHub release tool**
 
-- [ ] 6. :green_book: **Make sure to update the [release notes](https://github.com/confidential-containers/confidential-containers/tree/main/releases) and tag/release the confidential-containers repo using the GitHub release tool.**
+- [ ] 7. :green_book: **Make sure to update the [release notes](https://github.com/confidential-containers/confidential-containers/tree/main/releases) and tag/release the confidential-containers repo using the GitHub release tool.**
 
-- [ ] 7. :hammer: **Poke Wainer Moschetta (@wainersm) to update the release to the OperatorHub. Find the documented flow [here](https://github.com/confidential-containers/operator/blob/main/docs/OPERATOR_HUB.md).**
+- [ ] 8. :hammer: **Poke Wainer Moschetta (@wainersm) to update the release to the OperatorHub. Find the documented flow [here](https://github.com/confidential-containers/operator/blob/main/docs/OPERATOR_HUB.md).**
 
 ### Post-release
 
-- [ ] 8. :wrench: **Open a PR to the operator to go back to latest payloads after release**
-    After the release, the operator's payloads need to go back to what they were (e.g. using "latest" instead of a specific commit sha). As an example, step 4 for the v0.9.0-alpha0 release applied [these changes](https://github.com/confidential-containers/operator/pull/368/files), and for this step, you should use `git revert` to undo such changes you made during the release.
+- [ ] 9. :wrench: **Open a PR to the operator to go back to latest payloads after release**
+    After the release, the operator's payloads need to go back to what they were (e.g. using "latest" instead of a specific commit sha). As an example, the v0.9.0-alpha1 release applied [these changes](https://github.com/confidential-containers/operator/pull/389/files). You should use `git revert -s` for this.

releases/v0.9.0.md

@@ -0,0 +1,94 @@
+# Release Notes for v0.9.0
+
+Release Date: July 26th, 2024
+
+This release is based on [3.7.0](https://github.com/kata-containers/kata-containers/releases/tag/3.7.0) of Kata Containers
+and [v0.9.1](https://github.com/confidential-containers/enclave-cc/releases/tag/v0.9.1) of enclave-cc.
+
+This is the first non-alpha release of Confidential Containers to be based on the main branch of Kata Containers.
+This release does not have complete parity with releases based on CCv0, but it supports most features.
+See the limitations section for more details.
+
+
+Please see the [quickstart guide](../quickstart.md) for details on how to try out Confidential
+Containers.
+
+Please refer to our [Acronyms](https://github.com/confidential-containers/documentation/wiki/Acronyms)
+and [Glossary](https://github.com/confidential-containers/documentation/wiki/Glossary) pages for
+definitions of the acronyms used in this document.
+
+## What's new
+
+* Attestation is supported on SEV-SNP and IBM SE
+* Encrypted container images are supported
+* Authenticated registries are supported
+* Pods with init containers can be run wth Nydus
+* Sealed secrets (as environment variables) are supported
+
+## Hardware Support
+
+Attestation is supported and tested on three platforms: Intel TDX, AMD SEV-SNP, and IBM SE.
+Not all feature have been tested on every platform, but those based on attestation
+are expected to work on the platforms above.
+
+Make sure your host platform is compatible with the hypervisor and guest kernel
+provisioned by coco.
+
+This release has been tested on the following stacks:
+
+### AMD SEV-SNP
+
+* Processor: AMD EPYC 7413
+* Kernel: [6.8.0-rc5-next-20240221-snp-host-cc2568386](https://github.com/confidential-containers/linux/tree/amd-snp-host-202402240000)
+* OS: Ubuntu 22.04.4 LTS
+* k8s: v1.24.0 (Kubeadm)
+* Kustomize: v4.5.4
+
+### Intel TDX
+
+* Kernel: [6.8.0-1004-intel](https://git.launchpad.net/~kobuk-team/ubuntu/+source/linux-intel/tree/?h=noble-main-next)
+* OS: Ubuntu 24.04 LTS
+* k8s: v1.30.2 (Kubeadm)
+* Kustomize: v5.0.4-0.20230601165947-6ce0bf390ce3
+
+### Secure Execution on IBM zSystems (s390x) running LinuxONE
+
+* Hardware: IBM Z16 LPAR
+* Kernel: 5.15.0-113-generic
+* OS: Ubuntu 22.04.1 LTS
+* k8s: v1.28.4 (k3s)
+* Kustomize: v5.3.0
+
+## Limitations
+
+The following are known limitations of this release:
+
+* Signed images are not yet supported.
+* Secure storage is not yet supported.
+* SEV(-ES) does not support attestation.
+* Sealed secrets only supports secrets in environment variables.
+* Credentials for authenticated registries are exposed to the host.
+* Not all features are tested on all platforms.
+* Nydus snapshotter support is not mature.
+  * Nydus snapshotter sometimes fails to pull an image.
+  * Host pulling with Nydus snapshotter is not yet enabled.
+  * Nydus snapshotter is not supported with enclave-cc.
+* Pulling container images inside guest may have negative performance implications including greater resource usage and slower startup.
+* `crio` support is still evolving.
+* Platform support is rapidly changing
+* SELinux is not supported on the host and must be set to permissive if in use.
+* Complete integration with Kubernetes is still in progress.
+  * Existing APIs do not fully support the CoCo security and threat model. [More info](https://github.com/confidential-containers/community/issues/53)
+  * Some commands accessing confidential data, such as `kubectl exec`, may either fail to work, or incorrectly expose information to the host
+* The CoCo community aspires to adopting open source security best practices, but not all practices are adopted yet.
+  * We track our status with the OpenSSF Best Practices Badge, which remained at 75% at the time of this release.
+  * Community has adopted a security reporting protocol, but application and documentation of static and dynamic analysis still needed.
+* Container metadata such as environment variables are not measured.
+* The Kata Agent allows the host to call several dangerous endpoints
+    * Kata Agent does not validate mount requests. A malicious host might be able to mount a shared filesystem into the PodVM.
+    * Policy can be used to block endpoints, but it is not yet tied to the hardware evidence.
+
+## CVE Fixes
+
+None
+