Release: v0.10.0 release notes

Signed-off-by: Chris Porter <cporterbox@gmail.com>

.github/ISSUE_TEMPLATE/cncf-onboarding.md

@@ -1,15 +0,0 @@
----
-name: CNCF onboarding
-about: CNCF onboarding issue tracker
-title: "[CNCF]"
-labels: cncf-onboarding
-assignees: ''
-
----
-
-### Parts of the [CNCF onboarding issue](https://github.com/cncf/toc/issues/799) tracked by this issue
-
-*This is the list of all bullet points from the [CNCF onboarding issue](https://github.com/cncf/toc/issues/799) that this issue will track:*
-
-* [ ] *Bullet point foo*
-* [ ] *Bullet point bar*

.github/ISSUE_TEMPLATE/config.yml

@@ -0,0 +1,7 @@
+contact_links:
+  - name: CNCF Slack for Community discussions and Q&A
+    url: https://slack.cncf.io/
+    about: |
+      Join `#confidential-containers` channel by first getting an invitation for the
+      CNCF Slack workspace. Our channel is the best place for getting questions
+      answered and to chat with the community and project maintainers.

.github/ISSUE_TEMPLATE/project-creation.md

@@ -1,20 +0,0 @@
----
-name: New Project Creation
-about: Request for creating a new Confidential Containers project and associated repos
-title: "[New Project]"
-labels: repo-creation
-assignees: ''
-
----
-
-## Project Description
-
-*Describe the project: Goals, high level architecture* 
-
-## Alignment with Confidential Containers
-
-*Why this should become a Confidential Containers project?*
-
-## GitHub repositories needed
-
-*Name of the repos to be created to support this new project*

.github/ISSUE_TEMPLATE/release-check-list.md

@@ -8,111 +8,119 @@ assignees: ''
 
 # v<TARGET_RELEASE>
 
-## Code freeze
+## Overview
 
-- [ ] 1. Update Enclave CC to use the latest commit from image-rs
+The release process mainly follows from this dependency graph.
 
-    * https://github.com/confidential-containers/enclave-cc/blob/main/src/enclave-agent/Cargo.toml
-      * Change the revision
-      * Run `cargo update -p image-rs`
-        Note that you can point to your own fork here, so you don't actually do changes in the other projects
-        before making sure this step works as expected.
+```mermaid
+flowchart LR
+    Trustee --> Versions.yaml
+    Guest-Components --> Versions.yaml
+    Kata --> kustomization.yaml
+    Guest-Components .-> Client-tool
+    Guest-Components --> enclave-agent
+    enclave-cc --> kustomization.yaml
+    Guest-Components --> versions.yaml
+    Trustee --> versions.yaml
+    Kata --> versions.yaml
 
-- [ ] 2. Update Kata Containers to use the latest commit from image-rs, attestation-agent and td-shim
+    subgraph Kata
+        Versions.yaml
+    end
+    subgraph Guest-Components
+    end
+    subgraph Trustee
+        Client-tool
+    end
+    subgraph enclave-cc
+        enclave-agent
+    end
+    subgraph Operator
+        kustomization.yaml
+        reqs-deploy
+    end
+    subgraph cloud-api-adaptor
+        versions.yaml
+    end
+```
 
-    * image-rs
-      * https://github.com/kata-containers/kata-containers/blob/CCv0/src/agent/Cargo.toml
-      * Change the revision
-      * Run `cargo update -p image-rs`
-          Note that you can point to your own fork here, so you don't actually do changes in the other projects
-          before making sure this step works as expected.
-    * attestation-agent and td-shim
-      * https://github.com/kata-containers/kata-containers/blob/CCv0/versions.yaml
-        * Change the version
+Starting with v0.9.0 the release process no longer involves centralized dependency management.
+In other words, when doing a CoCo release, we don't push the most recent versions of the subprojects
+into Kata and enclave-cc. Instead, dependencies should be updated during the normal process of development.
+Releases of most subprojects are now decoupled from releases of the CoCo project.
 
-- [ ] 3. Wait for kata-runtime-payload-ci to be successfully built
-    * After the previous PR is merged wait for the kata-runtime-payload-ci (https://github.com/kata-containers/kata-containers/actions/workflows/cc-payload-after-push.yaml) has completed, so the latest kata-runtime-payload-ci contains the changes
+## The Steps
 
-- [ ] 4. Check if there are new changes in the pre install payload script
+Note: It may be useful when doing these steps to refer to a previous example. The v0.9.0-alpha1 release applied [these changes](https://github.com/confidential-containers/operator/pull/388/files). After following steps 1-5 below, you should end up with a similar set of changes.
 
-    * https://github.com/confidential-containers/operator/tree/main/install/pre-install-payload
-      * The last commit there must match what's in the following files as preInstall / postUninstall image
-        * Enclave CC: https://github.com/confidential-containers/operator/blob/main/config/samples/enclave-cc/base/ccruntime-enclave-cc.yaml
-        * Kata Containers:
-              Note that for Kata Containers, we're looking for the newTag, below the quay.io/confidential-containers/container-engine-for-cc-payload image
-          * default: https://github.com/confidential-containers/operator/blob/main/config/samples/ccruntime/default/kustomization.yaml
+### Determine release builds
 
-- [ ] 5. Ensure the Operator is using the latest CI builds and that the Operator tests are passsing
+Identify/create the bundles that we will release for Kata and enclave-cc.
 
+- [ ] 1. :eyes: **Create enclave-cc release**
+
+    Enclave-cc does not have regular releases apart from CoCo, so we need to make one.
+    Make sure that the CI [is green](https://github.com/confidential-containers/operator/actions/workflows/enclave-cc-cicd.yaml) and then use the Github release tool to create a tag and release.
+    This should create a bundle [here](https://quay.io/repository/confidential-containers/runtime-payload?tab=tags).
+
+- [ ] 2. :eyes: **Find Kata release version**
+
+    The release will be based on an existing Kata containers bundle.
+    You should use a release of Kata containers.
+    Release bundles can be found [here](https://quay.io/repository/kata-containers/kata-deploy?tab=tags).
+    There is also a bundle built for [each commit](https://quay.io/repository/kata-containers/kata-deploy-ci?tab=tags).
+    If you absolutely cannot use a Kata release,
+    you can consider releasing one of these bundles.
+
+- [ ] 3. :eyes: **Create a peer pods release**
+
+    Create a peer pods release based on the Kata release, by following the [documented flow](https://github.com/confidential-containers/cloud-api-adaptor/blob/main/docs/Release-Process.md).
+
+### Test Release with Operator
+
+- [ ] 4. :eyes: **Check operator pre-installation and open PR if needed**
+    
+    The operator uses a pre-install container to setup the node.
+    Check that the container matches the dependencies used in Kata
+    and that the operator pulls the most recent version of the container.
+
+    * Check that the version of the `nydus-snapshotter` used by Kata matches the one used by the operator
+        * Compare the `nydus-snapshotter` version in Kata [versions.yaml](https://github.com/kata-containers/kata-containers/blob/main/versions.yaml) (search for `nydus-snapshotter` and check its `version` field) with the [Makefile](https://github.com/confidential-containers/operator/blob/main/install/pre-install-payload/Makefile) (check the `NYDUS_SNAPSHOTTER_VERSION` value) for the operator pre-install container.
+            * **If they do not match, stop and open a PR now. In the PR, update the operator's Makefile to match the version used in kata. After the PR is merged, continue.**
+
+- [ ] 5. :wrench: **Open a PR to the operator to update the release artifacts**
+
+    Update the operator to use the payloads identified in steps 1, 2, 3, and 4.
+
+    Make sure that the operator pulls the most recent version of the pre-install container
+
+    * Find the last commit in the [pre-install directory](https://github.com/confidential-containers/operator/tree/main/install/pre-install-payload)
+        * As a sanity check, the sha hash of the last commit in that pre-install directory will correspond to a pre-install image in quay, i.e. a reqs-payload image [here](https://quay.io/confidential-containers/reqs-payload).
+    * Make sure that the commit matches the preInstall / postUninstall image specified for [enclave-cc CRD](https://github.com/confidential-containers/operator/blob/main/config/samples/enclave-cc/base/ccruntime-enclave-cc.yaml) and [ccruntime CRD](https://github.com/confidential-containers/operator/blob/main/config/samples/ccruntime/default/kustomization.yaml)
+        * If these do not match (for instance if you changed the snapshotter in step 4), update the operator so that they do match.
+
+    There are a number of places where the payloads are referenced. Make sure to update all of the following to the tag matching the latest commit hash from steps 1, 2, and 3:
     * Enclave CC:
-      * SIM: https://github.com/confidential-containers/operator/blob/main/config/samples/enclave-cc/sim/kustomization.yaml
-      * HW: https://github.com/confidential-containers/operator/blob/main/config/samples/enclave-cc/base/ccruntime-enclave-cc.yaml
-      * Note that we need the quay.io/confidential-containers/runtime-payload-ci registry and enclave-cc-{SIM,HW}-latest tags
+      * [sim](https://github.com/confidential-containers/operator/blob/main/config/samples/enclave-cc/sim/kustomization.yaml)
+      * [hw](https://github.com/confidential-containers/operator/blob/main/config/samples/enclave-cc/hw/kustomization.yaml)
+      * [base](https://github.com/confidential-containers/operator/blob/main/config/samples/enclave-cc/base/ccruntime-enclave-cc.yaml)
     * Kata Containers:
-      * default: https://github.com/confidential-containers/operator/blob/main/config/samples/ccruntime/default/kustomization.yaml
-      * peer-pods: https://github.com/confidential-containers/operator/blob/main/config/samples/ccruntime/peer-pods/kustomization.yaml
+      * [default](https://github.com/confidential-containers/operator/blob/main/config/samples/ccruntime/default/kustomization.yaml)
+      * [s390x](https://github.com/confidential-containers/operator/blob/main/config/samples/ccruntime/s390x/kustomization.yaml)
+      * [peer-pods](https://github.com/confidential-containers/operator/blob/main/config/samples/ccruntime/peer-pods/kustomization.yaml)
           Note that we need the quay.io/confidential-containers/runtime-payload-ci registry and kata-containers-latest tag
 
-- [ ] 6. Update peer-pods with latest commits of kata-containers and attestation-agent and test it, following the [release candidate testing process](https://github.com/confidential-containers/cloud-api-adaptor/blob/main/docs/Release-Process.md#release-candidate-testing)
-    
-- [ ] 7. Cut an attestation-service v<TARGET_RELEASE> and make images for AS and RVPS, if changes happened in the project.
+    **Also, update the [operator version](https://github.com/confidential-containers/operator/blob/main/config/release/kustomization.yaml) (update the `newTag` value)**
 
-       * https://github.com/confidential-containers/attestation-service
-       * Cut a release (AS/RVPS images will be automatically built triggered by release)
+### Final Touches
 
-- [ ] 8. Cut a guest-components v<TARGET_RELEASE> release
+- [ ] 6. :trophy: **Cut an operator release using the GitHub release tool**
 
-- [ ] 9. Cut a td-shim v<TARGET_RELEASE> release, if changes happened in the project
+- [ ] 7. :green_book: **Make sure to update the [release notes](https://github.com/confidential-containers/confidential-containers/tree/main/releases) and tag/release the confidential-containers repo using the GitHub release tool.**
 
-- [ ] 10. Update kbs to use the tagged attestation-service and guest-components, cut a release and make image
+- [ ] 8. :hammer: **Poke Wainer Moschetta (@wainersm) to update the release to the OperatorHub. Find the documented flow [here](https://github.com/confidential-containers/operator/blob/main/docs/OPERATOR_HUB.md).**
 
-       * https://github.com/confidential-containers/kbs/blob/main/src/api/Cargo.toml
-         * Change the revision for the `as-types` and `attestation-service` crates (both use `v<TARGET_RELEASE>`) and update the lock file
-       * https://github.com/confidential-containers/kbs/blob/main/tools/client/Cargo.toml
-         * Change the revision for the `as-types` and `kbs_protocol` crates (both use `v<TARGET_RELEASE>`)
-       * Cut a release
-         * kbs image will be automatically built triggered by release, so ensure that the [release workflow](https://github.com/confidential-containers/kbs/actions/workflows/release.yaml) ran successfully
+### Post-release
 
-- [ ] 11. Update Enclave CC to use the released version of image-rs
-
-    * redo step 3, but now using v<TARGET_RELEASE>
-
-- [ ] 12. Update Kata Containers to the latest released version of:
-
-    * image-rs (redo step 4, but now using the v<TARGET_RELEASE>)
-    * attestation-agent (redo step 5, but now using the v<TARGET_RELEASE>)
-    * td-shim (redo step 6, but now using the v<TARGET_RELEASE>)
-
-- [ ] 13. Update the operator to use the images generated from the latest commit of both Kata Containers and Enclave CC
-
-    * redo step 8, but now targetting the latest payload image generated for Kata Containers and Enclave CC
-
-- [ ] 14. Make sure all the operator tests are passing
-
-- [ ] 15. Cut an Enclave CC release
-
-- [ ] 16. Add a new Kata Containers tag
-
-- [ ] 17. Wait for release kata-runtime-payload to be successfully built
-    * After the Kata tag is created wait for (https://github.com/kata-containers/kata-containers/actions/workflows/cc-payload.yaml) to be successfully completed, so the latest commit kata-runtime-payload for the release is created
-
-- [ ] 18. Update peer pods to use the release versions and then cut a release following the [documented flow](https://github.com/confidential-containers/cloud-api-adaptor/blob/main/docs/Release-Process.md#cutting-releases)
-
-## Release
-
-
-- [ ] 19. Update the operator to use the release tags coming from Enclave CC and Kata Containers
-
-    * redo step 8, but now targeting the latest release of the payload image generated for Kata Containers eand Enclave CC
-
-- [ ] 20. Update the Operator version
-
-    * https://github.com/confidential-containers/operator/blob/main/config/release/kustomization.yaml#L7
-
-- [ ] 21. Cut an operator release
-
-- [ ] 22. Make sure to update the release notes and tag the confidential-containers repository
-
-    * https://github.com/confidential-containers/documentation/tree/main/releases/v<TARGET_RELEASE>.md
-
-- [ ] 23. Poke Wainer Moschetta (@wainersm) to update the release to the OperatorHub
+- [ ] 9. :wrench: **Open a PR to the operator to go back to latest payloads after release**
+    After the release, the operator's payloads need to go back to what they were (e.g. using "latest" instead of a specific commit sha). As an example, the v0.9.0-alpha1 release applied [these changes](https://github.com/confidential-containers/operator/pull/389/files). You should use `git revert -s` for this.

ADOPTERS.md

@@ -0,0 +1,35 @@
+# Confidential Containers Adopters
+
+This page contains a list of organizations/companies and projects/products who use confidential containers as adopters in different usage levels (development, beta, production, GA etc...) 
+
+**NOTE:** For adding your organization/company and project/product to this table (alphabetical order) fork the repository and open a PR with the required change. 
+See list of adopter types at the bottom of this page.
+
+## Adopters
+
+| Organization/Company                                              | Project/Product                                               | Usage level              | Adopter type                     | Details                                                                   |
+|-------------------------------------------------------------------|---------------------------------------------------------------|--------------------------|----------------------------------|---------------------------------------------------------------------------|
+| [Edgeless Systems](https://www.edgeless.systems/)                 | [Contrast](https://github.com/edgelesssys/contrast)           | Beta                     | Service Provider / Consultancy   | Contrast runs confidential container deployments on Kubernetes at scale.                                |
+| [IBM](https://www.ibm.com/z)                                    | [IBM LinuxONE](https://www.ibm.com/linuxone)                  | Beta                     | Service Provider                 | Confidential Containers with Red Hat OpenShift Container Platform and IBM® Secure Execution for Linux (see [details](https://www.ibm.com/blog/confidential-containers-with-red-hat-openshift-container-platform-and-ibm-secure-execution-for-linux/))  |
+|NanhuLab|Trusted Big Data Sharing System |Beta |Service Provider |The system uses confidential containers to ensure that data users can utilize the data without being able to view the raw data.(No official website yet. For details: yzc@nanhulab.ac.cn) |
+|TBD| | | | |
+|TBD| | | | |
+
+## Adopter types 
+
+See CNCF [definition of an adopter](https://github.com/cncf/toc/blob/main/FAQ.md#what-is-the-definition-of-an-adopter) <br>
+
+Any single company can fall under several categories at once in which case it should enumerate all that apply and only be listed once
+- **End-User** (CNCF member) - Companies and organizations who are End-User members of the CNCF
+- **Another project** - an open source project that leverages a CNCF project as part of their solution, integrates with for compatibility and interoperability,
+  or is used in the supply chain of another project
+- **end users** (non CNCF member)- companies and organizations that are not CNCF End-User members that use the project and cloud native technologies internally, or build upon
+  a cloud native open source project but do not sell the cloud native project externally as a service offering (those are Service Providers). This group is identified in the written
+  form by the convention end user, uncapitalized and unhyphenated
+- **Service Provider** - a Service Provider is an organization that repackages an open source project as a core component of a service offering, sells cloud native services externally.
+  A Service Provider’s customers are considered transitive adopters and should be excluded from identification within the ADOPTERS.md file.
+  Examples of Service Providers (and not end users) include cloud providers (e.g., Alibaba Cloud, AWS, Google Cloud, Microsoft Azure), some infrastructure software vendors,
+  and telecom operators (e.g., AT&T, China Mobile)
+- **Consultancy** - an entity whose purpose is to assist other organizations in developing a solution leveraging cloud native technology. They may be embedded in the end user team and
+  is responsible for the execution of the service. Service Providers may also provide consultancy services, they may also package cloud native technologies for reuse
+  as part of their offerings. These function as proxies for an end user

README.md

@@ -2,6 +2,8 @@
 
 # Confidential Containers
 
+[![CII Best Practices](https://bestpractices.coreinfrastructure.org/projects/5719/badge)](https://bestpractices.coreinfrastructure.org/projects/5719)
+
 ## Welcome to confidential-containers
 
 Confidential Containers is an open source community working to leverage 
@@ -42,3 +44,4 @@ delivering Confidential Computing for guest applications or data inside the TEE
 
 ## License
 [![FOSSA Status](https://app.fossa.com/api/projects/git%2Bgithub.com%2Fconfidential-containers%2Fcommunity.svg?type=large)](https://app.fossa.com/projects/git%2Bgithub.com%2Fconfidential-containers%2Fcommunity?ref=badge_large)
+

governance.md

@@ -75,14 +75,14 @@ Further, as leaders in the community, the SC members will make themselves famili
 
 The current members of the SC are:
 
-* Larry Dewey (@larrydewey) - AMD
+* Larry Dewey (@larrydewey) and Ryan Savino (@ryansavino) - AMD
 * Jiang Liu (@jiangliu) and Jia Zhang (@jiazhang0) - Alibaba
 * James Magowan (@magowan)  and Tobin Feldman-Fitzthum (@fitzthum) - IBM
-* Peter Zhu (@peterzcst) and Fabiano Fidêncio (@fidencio) - Intel
+* Peter Zhu (@peterzcst) and Mikko Ylinen (@mythi) - Intel
 * Pradipta Banerjee (@bpradipt)  and Ariel Adam (@ariel-adam) - Red Hat
 * Samuel Ortiz (@sameo) - Rivos
 * Zvonko Kaiser (@zvonkok) - NVIDIA
-* Vincent Batts (@vbatts) and Ananya Garg (@angarg05) - Microsoft
+* Vincent Batts (@vbatts) and Dan Mihai (@danmihai1) - Microsoft
 
 ### Emeritus Members
 

guides/enclave-cc.md

@@ -5,6 +5,17 @@ This guide assumes that you already have a Kubernetes cluster
 and have deployed the operator as described in the **Installation**
 section of the [quickstart guide](../quickstart.md).
 
+## Configuring Kubernetes cluster when using SGX hardware mode build
+
+Additional setup steps when using the hardware SGX mode are needed:
+
+1. The cluster needs to have [Intel Software Guard Extensions (SGX) device plugin for Kubernetes](
+https://intel.github.io/intel-device-plugins-for-kubernetes/cmd/sgx_plugin/README.html#prerequisites) running.
+1. The cluster needs to have [Intel DCAP aesmd](
+https://github.com/intel/SGXDataCenterAttestationPrimitives) running on every SGX node and the nodes must be registered.
+
+**Note** kind/minikube based clusters are not recommended when using hardware mode SGX.
+
 ## Configuring enclave-cc custom resource to use a different KBC
 
 **Note** Before configuring KBC, please refer to the
@@ -66,7 +77,7 @@ spec:
   containers:
   - image: ghcr.io/confidential-containers/test-container-enclave-cc:encrypted
     name: hello-world
-    workingDir: "/run/rune/boot_instance/"
+    workingDir: "/run/rune/occlum_instance/"
     resources:
       limits:
         sgx.intel.com/epc: 600Mi
@@ -74,7 +85,7 @@ spec:
     - name: OCCLUM_RELEASE_ENCLAVE
       value: "1"
     command:
-    - /run/rune/boot_instance/build/bin/occlum-run
+    - /run/rune/occlum_instance/build/bin/occlum-run
     - /bin/hello_world
   runtimeClassName: enclave-cc
 
@@ -117,6 +128,9 @@ Hello world!
 
 ```
 
+**NOTE** When running in the hardware SGX mode, the logging is disabled
+by default.
+
 We can also verify the host does not have the image for others to use:
 ```sh
 crictl -r unix:///run/containerd/containerd.sock image ls | grep helloworld_enc

guides/ibm-se.md

@@ -0,0 +1,128 @@
+# IBM Secure Execution Guide
+
+This document explains how to install and run a confidential container on an IBM Secure 
+Execution-enabled Z machine (s390x). A secure image is an encrypted Linux image comprising a kernel image,
+an initial RAM file system (initrd) image, and a file specifying kernel parameters (parmfile).
+It is an essential component for running a confidential container. The public key used for
+encryption is associated with a private key managed by a trusted firmware called
+[ultravisor](https://www.ibm.com/docs/en/linux-on-systems?topic=execution-components).
+
+This means that a secure image is machine-specific, resulting in its absence from a released
+payload image in `ccruntime`. To use it, you need to build a secure image with your own public
+key and create a payload image bundled with it. The following sections elaborate on how to
+accomplish this step-by-step.
+
+## Prerequisites
+
+Kindly review the [section](https://github.com/confidential-containers/confidential-containers/blob/main/quickstart.md#prerequisites) titled identically in the `QuickStart`.
+
+- `kustomize`: Kubernetes native configuration management tool which can be installed simply by:
+
+```
+$ mkdir -p $GOPATH/src/github.com/confidential-containers
+$ cd $GOPATH/src/github.com/confidential-containers
+$ git clone https://github.com/confidential-containers/operator.git && cd operator
+$ make kustomize
+$ export PATH=$PATH:$(pwd)/bin
+```
+
+Or simply follow the official [documentation](https://kubectl.docs.kubernetes.io/installation/kustomize/) based on your environment.
+
+## Build a Payload Image via kata-deploy
+
+If you have a local container registry running at `localhost:5000`, refer to the
+[document](https://github.com/kata-containers/kata-containers/blob/main/docs/how-to/how-to-run-kata-containers-with-SE-VMs.md#using-kata-deploy-with-confidential-containers-operator)
+on Kata Containers for details on building a payload image.
+
+## Install Operator
+
+Let us install an operator with:
+
+```
+$ cd $GOPATH/src/github.com/confidential-containers/operator
+$ export IMG=localhost:5000/cc-operator
+$ make docker-build && make docker-push
+$ make install && make deploy
+namespace/confidential-containers-system created
+customresourcedefinition.apiextensions.k8s.io/ccruntimes.confidentialcontainers.org created
+serviceaccount/cc-operator-controller-manager created
+role.rbac.authorization.k8s.io/cc-operator-leader-election-role created
+clusterrole.rbac.authorization.k8s.io/cc-operator-manager-role created
+clusterrole.rbac.authorization.k8s.io/cc-operator-metrics-reader created
+clusterrole.rbac.authorization.k8s.io/cc-operator-proxy-role created
+rolebinding.rbac.authorization.k8s.io/cc-operator-leader-election-rolebinding created
+clusterrolebinding.rbac.authorization.k8s.io/cc-operator-manager-rolebinding created
+clusterrolebinding.rbac.authorization.k8s.io/cc-operator-proxy-rolebinding created
+configmap/cc-operator-manager-config created
+service/cc-operator-controller-manager-metrics-service created
+deployment.apps/cc-operator-controller-manager created
+$ kubectl get pods -n confidential-containers-system --watch
+NAME                                              READY   STATUS    RESTARTS   AGE
+cc-operator-controller-manager-6d6b78b7f5-m9chj   2/2     Running   0          61s
+$ # Press ctrl C to stop watching
+```
+
+## Install Custom Resource (CR)
+
+You can install `ccruntime` with an existing overlay directory named
+[`s390x`](https://github.com/confidential-containers/operator/tree/main/config/samples/ccruntime/s390x), by replacing the image name and tag
+for a payload image with the ones you pushed to the local registry
+(e.g. `localhost:5000/build-kata-deploy:latest`):
+
+```
+$ cd $GOPATH/src/github.com/confidential-containers/operator/config/samples/ccruntime/s390x
+$ kustomize edit set image quay.io/kata-containers/kata-deploy=localhost:5000/build-kata-deploy:latest
+$ kubectl create -k .
+ccruntime.confidentialcontainers.org/ccruntime-sample-s390x created
+$ kubectl get pods -n confidential-containers-system --watch
+NAME                                              READY   STATUS    RESTARTS   AGE
+cc-operator-controller-manager-6d6b78b7f5-m9chj   2/2     Running   0          11m
+cc-operator-daemon-install-srmc4                  1/1     Running   0          2m13s
+cc-operator-pre-install-daemon-t9r2h              1/1     Running   0          3m6s
+$ # To verify if a payload image is pulled from the updated location
+$ kubectl get pods -oyaml -n confidential-containers-system -l 'name=cc-operator-daemon-install' | grep image:
+    image: localhost:5000/build-kata-deploy:test
+    image: localhost:5000/build-kata-deploy:test
+```
+
+You have to wait until a set of runtime classes is deployed like:
+
+```
+$ kubectl get runtimeclass
+NAME           HANDLER        AGE
+kata           kata-qemu      60s
+kata-qemu      kata-qemu      61s
+kata-qemu-se   kata-qemu-se   61s
+```
+
+## Verify the Installation
+
+To verify the installation, use the following runtime class: `kata-qemu-se`:
+
+```
+$ cat <<EOF | kubectl apply -f -
+apiVersion: v1
+kind: Pod
+metadata:
+  name: nginx-kata
+spec:
+  runtimeClassName: kata-qemu-se
+  containers:
+  - name: nginx
+    image: nginx
+EOF
+pod/nginx-kata created
+$ kubectl get pods
+NAME         READY   STATUS    RESTARTS   AGE
+nginx-kata   1/1     Running   0          15s
+```
+
+## Uninstall Resources
+
+You can uninstall confidential containers by removing the resources in reverse order:
+
+```
+$ cd $GOPATH/src/github.com/confidential-containers/operator
+$ kubectl delete -k config/samples/ccruntime/s390x
+$ make undeploy
+```

guides/nontee_demo.md

@@ -1,92 +0,0 @@
-# Encrypted Container Images without Hardware Support
-
-Without Confidential Computing hardware, there is no way to securely provision
-the keys for an encrypted image. Nonetheless, in this demo we describe how to
-test encrypted images support with the non-tee `kata`/`kata-qemu` runtimeclass.
-
-## Creating a CoCo workload using a pre-existing encrypted image
-
-We will now proceed to download and run a sample encrypted container image using the CoCo building blocks.
-
-A demo container image is provided at [docker.io/katadocker/ccv0-ssh](https://hub.docker.com/r/katadocker/ccv0-ssh).
-It is encrypted with [Attestation Agent](https://github.com/confidential-containers/attestation-agent)'s [offline file system key broker](https://github.com/confidential-containers/attestation-agent/tree/64c12fbecfe90ba974d5fe4896bf997308df298d/src/kbc_modules/offline_fs_kbc) and [`aa-offline_fs_kbc-keys.json`](https://github.com/confidential-containers/documentation/blob/main/demos/ssh-demo/aa-offline_fs_kbc-keys.json) as its key file.
-
-We have prepared a sample CoCo operator custom resource that is based on the standard `ccruntime.yaml`, but in addition has the the decryption keys and configuration required to decrypt this sample container image.
-> **Note** All pods started with this sample resource will be able to decrypt the sample container and all keys shown are for demo purposes only and should not be used in production.
-
- To test out creating a workload from the sample encrypted container image, we can take the following steps:
-
-### Swap out the standard custom resource for our sample
-
-Support for multiple custom resources in not available in the current release. Consequently, if a custom resource already exists, then you'll need to remove it first before deploying a new one. We can remove the standard custom resource with:
-```sh
-kubectl delete -k github.com/confidential-containers/operator/config/samples/ccruntime/<CCRUNTIME_OVERLAY>?ref=<RELEASE_VERSION>
-```
-and in it's place install the modified version with the sample container's decryption key:
-```sh
-kubectl apply -k github.com/confidential-containers/operator/config/samples/ccruntime/ssh-demo?ref=<RELEASE_VERSION>
-```
-Wait until each pod has the STATUS of Running.
-```sh
-kubectl get pods -n confidential-containers-system --watch
-```
-### Test creating a workload from the sample encrypted image
-
-Create a new Kubernetes deployment that uses the `docker.io/katadocker/ccv0-ssh` container image with:
-```sh
-cat << EOF > ccv0-ssh-demo.yaml
-kind: Service
-apiVersion: v1
-metadata:
-  name: ccv0-ssh
-spec:
-  selector:
-    app: ccv0-ssh
-  ports:
-  - port: 22
----
-kind: Deployment
-apiVersion: apps/v1
-metadata:
-  name: ccv0-ssh
-spec:
-  selector:
-    matchLabels:
-      app: ccv0-ssh
-  template:
-    metadata:
-      labels:
-        app: ccv0-ssh
-      annotations:
-        io.containerd.cri.runtime-handler: kata
-    spec:
-      runtimeClassName: kata
-      containers:
-      - name: ccv0-ssh
-        image: docker.io/katadocker/ccv0-ssh
-        imagePullPolicy: Always
-EOF
-```
-
-Apply this with:
-```sh
-kubectl apply -f ccv0-ssh-demo.yaml
-```
-and wait for the pod to start. This process should show that we are able to pull the encrypted image, and using the decryption key configured in the CoCo sample guest image, decrypt the container image and create a workload using it.
-
-The demo image has an SSH host key embedded in it, which is protected by it's encryption, but we can download the sample private key and use this to ssh into the container to validate it hasn't been tampered with.
-
-Download the SSH key with:
-```sh
-curl -Lo ccv0-ssh https://raw.githubusercontent.com/confidential-containers/documentation/main/demos/ssh-demo/ccv0-ssh
-```
-Ensure that the permissions are set correctly with:
-```sh
-chmod 600 ccv0-ssh
-```
-
-We can then use the key to ssh into the container:
-```sh
-$ ssh -i ccv0-ssh root@$(kubectl get service ccv0-ssh -o jsonpath="{.spec.clusterIP}")
-```
-You will be prompted about whether the host key fingerprint is correct. This fingerprint should match the one specified in the container image: `wK7uOpqpYQczcgV00fGCh+X97sJL3f6G1Ku4rvlwtR0.`

quickstart.md

@@ -31,7 +31,7 @@ To run the operator you must have an existing Kubernetes cluster that meets the
 - Ensure a minimum of 8GB RAM and 4 vCPU for the Kubernetes cluster node
 - Only containerd runtime based Kubernetes clusters are supported with the current CoCo release
 - The minimum Kubernetes version should be 1.24
-- Ensure at least one Kubernetes node in the cluster is having the label `node-role.kubernetes.io/worker=`
+- Ensure at least one Kubernetes node in the cluster is having the label `node.kubernetes.io/worker=`
 - Ensure SELinux is disabled or not enforced (https://github.com/confidential-containers/operator/issues/115)
 
 For more details on the operator, including the custom resources managed by the operator, refer to the operator [docs](https://github.com/confidential-containers/operator).
@@ -243,9 +243,6 @@ Secrets are provisioned to the guest in conjunction with an attestation, which i
 The rest of this guide focuses on setting up more substantial encrypted/signed workloads using attestation
 and confidential hardware.
 
-See [this guide](./guides/nontee_demo.md) if you would like to deploy an example encrypted image without
-confidential hardware.
-
 CoCo has a modular attestation interface and there are a few options for attestation.
 CoCo provides a generic Key Broker Service (KBS) that the rest of this guide will be focused on.
 The SEV runtime class uses `simple-kbs`, which is described in the [SEV guide](./guides/sev.md).
@@ -272,12 +269,12 @@ A tenant-side CoCo Key Broker System cluster includes:
 - Reference Value Provicer Service (RVPS): Provides reference values for AS.
 - CoCo Keyprovider: Component to encrypt the images following ocicrypt spec.
 
-To quick start the KBS cluster, a `docker-compose` yaml is provided to launch.
+To quick start the KBS cluster, a `docker compose` yaml is provided to launch.
 
 ```shell
 # Clone KBS git repository
 git clone https://github.com/confidential-containers/kbs.git
-cd kbs
+cd kbs/kbs
 export KBS_DIR_PATH=$(pwd)
 
 # Generate a user auth key pair
@@ -285,10 +282,10 @@ openssl genpkey -algorithm ed25519 > config/private.key
 openssl pkey -in config/private.key -pubout -out config/public.pub
 
 # Start KBS cluster
-docker-compose up -d
+docker compose up -d
 ```
 
-If configuration of KBS cluster is required, edit the following config files and restart the KBS cluster with `docker-compose`:
+If configuration of KBS cluster is required, edit the following config files and restart the KBS cluster with `docker compose`:
 
 - `$KBS_DIR_PATH/config/kbs-config.json`: configuration for Key Broker Service.
 - `$KBS_DIR_PATH/config/as-config.json`: configuration for Attestation Service.
@@ -337,7 +334,7 @@ Be sure to replace `[SCHEME]` with registry scheme type like `docker`, replace `
 
 [cosign](https://github.com/sigstore/cosign) is required to sign the container image. Follow the instructions here to install `cosign`:
 
-[cosign installation](https://docs.sigstore.dev/cosign/installation/)
+[cosign installation](https://docs.sigstore.dev/cosign/system_config/installation/)
 
 Generate a cosign key pair and register the public key to KBS storage:
 
@@ -407,7 +404,7 @@ EOT
 
 Be sure to replace `[REGISTRY_URL]` with the desired registry URL of the encrypted image generated in previous step, replace `[RUNTIME_CLASS]` with kata runtime class for CC HW.
 
-Then configure `/opt/confidential-containers/share/defaults/kata-containers/configuration-<RUNTIME_CLASS_SUFFIX>.toml` to add `agent.aa_kbc_params=cc_kbc::<KBS_URI>` to kernal parameters. Here `RUNTIME_CLASS_SUFFIX` is something like `qemu-tdx`, `KBS_URI` is the address of Key Broker Service in KBS cluster like `http://123.123.123.123:8080`.
+Then configure `/opt/confidential-containers/share/defaults/kata-containers/configuration-<RUNTIME_CLASS_SUFFIX>.toml` to add `agent.aa_kbc_params=cc_kbc::<KBS_URI>` to kernel parameters. Here `RUNTIME_CLASS_SUFFIX` is something like `qemu-tdx`, `KBS_URI` is the address of Key Broker Service in KBS cluster like `http://123.123.123.123:8080`.
 
 Deploy encrypted image as a workload:
 

releases/v0.10.0.md

@@ -0,0 +1,92 @@
+# Release Notes for v0.10.0
+
+Release Date: September 27th, 2024
+
+This release is based on [3.9.0](https://github.com/kata-containers/kata-containers/releases/tag/3.9.0) of Kata Containers
+and [v0.10.0](https://github.com/confidential-containers/enclave-cc/releases/tag/v0.10.0) of enclave-cc.
+
+This is the first release of Confidential Containers which has feature parity with CCv0.
+
+Please see the [quickstart guide](../quickstart.md) for details on how to try out Confidential
+Containers.
+
+Please refer to our [Acronyms](https://github.com/confidential-containers/documentation/wiki/Acronyms)
+and [Glossary](https://github.com/confidential-containers/documentation/wiki/Glossary) pages for
+definitions of the acronyms used in this document.
+
+## What's new
+
+* Support to pull and verify `cosign`-signed images
+* Trusted image storage on guest
+* Support Intel Tiber Trust Services as the verifier with Trustee for both Kata bare metal and peer-pods deployments
+* Init-data support for peer pods
+* Image-rs support for whiteouts and for layers with hard-link filenames over 100 characters
+* enclave-cc updated to Ubuntu 22.04 based runtime instance
+
+## Hardware Support
+
+Attestation is supported and tested on three platforms: Intel TDX, AMD SEV-SNP, and IBM SE.
+Not all feature have been tested on every platform, but those based on attestation
+are expected to work on the platforms above.
+
+Make sure your host platform is compatible with the hypervisor and guest kernel
+provisioned by coco.
+
+This release has been tested on the following stacks:
+
+### AMD SEV-SNP
+
+* Processor: AMD EPYC 7413
+* Kernel: [6.8.0-rc5-next-20240221-snp-host-cc2568386](https://github.com/confidential-containers/linux/tree/amd-snp-host-202402240000)
+* OS: Ubuntu 22.04.4 LTS
+* k8s: v1.30.1 (Kubeadm)
+* Kustomize: v4.5.4
+
+### Intel TDX
+
+* Kernel: [6.8.0-1004-intel](https://git.launchpad.net/~kobuk-team/ubuntu/+source/linux-intel/tree/?h=noble-main-next)
+* OS: Ubuntu 24.04 LTS
+* k8s: v1.30.2 (Kubeadm)
+* Kustomize: v5.0.4-0.20230601165947-6ce0bf390ce3
+
+### Secure Execution on IBM zSystems (s390x) running LinuxONE
+
+* Hardware: IBM Z16 LPAR
+* Kernel: 5.15.0-113-generic
+* OS: Ubuntu 22.04.1 LTS
+* k8s: v1.28.4 (k3s)
+* Kustomize: v5.3.0
+
+## Limitations
+
+The following are known limitations of this release:
+
+* SEV(-ES) does not support attestation.
+* Sealed secrets only supports secrets in environment variables.
+* Credentials for authenticated registries are exposed to the host.
+* Not all features are tested on all platforms.
+* Nydus snapshotter support is not mature.
+  * Nydus snapshotter sometimes fails to pull an image.
+  * Host pulling with Nydus snapshotter is not yet enabled.
+  * Nydus snapshotter is not supported with enclave-cc.
+* Pulling container images inside guest may have negative performance implications including greater resource usage and slower startup.
+* `crio` support is still evolving.
+* Platform support is rapidly changing
+* SELinux is not supported on the host and must be set to permissive if in use.
+* Complete integration with Kubernetes is still in progress.
+  * Existing APIs do not fully support the CoCo security and threat model. [More info](https://github.com/confidential-containers/community/issues/53)
+  * Some commands accessing confidential data, such as `kubectl exec`, may either fail to work, or incorrectly expose information to the host
+* The CoCo community aspires to adopting open source security best practices, but not all practices are adopted yet.
+  * We track our status with the OpenSSF Best Practices Badge, which remained at 75% at the time of this release.
+  * Community has adopted a security reporting protocol. The status of this is:
+    * The operator now uses CodeQL for static scans, and it will be added for all other Go-based repositories in the next release.
+    * Dependencies are now better handled with automatic updates using dependabot.
+    * Static scan for Rust-based repos will be "N/A".
+* Container metadata such as environment variables are not measured.
+* The Kata Agent allows the host to call several dangerous endpoints
+    * Kata Agent does not validate mount requests. A malicious host might be able to mount a shared filesystem into the PodVM.
+    * Policy can be used to block endpoints, but it is not yet tied to the hardware evidence.
+
+## CVE Fixes
+
+None

releases/v0.9.0-alpha0.md

@@ -0,0 +1,79 @@
+# Release Notes for v0.9.0-alpha
+
+Release Date: May 2nd, 2024
+
+This is our first release based on Kata Containers main, but it is an alpha release that supports
+only a subset of features.
+
+This release is based on [3.4.0](https://github.com/kata-containers/kata-containers/releases/tag/3.4.0) of Kata Containers
+and [v0.9.0](https://github.com/confidential-containers/enclave-cc/releases/tag/v0.9.0) of enclave-cc.
+
+This release supports pulling images inside of the guest with some caveats but it does
+not support pulling encrypted or signed images inside the guest.
+This release supports attestation and includes the guest components in the rootfs,
+but it does not support any TEE platform.
+Peer pods is also not supported.
+
+**This release was created mainly for development purposes. For a full feature set,
+consider returning to v0.8.0 or using the next release.**
+
+Please see the [quickstart guide](../quickstart.md) for details on how to try out Confidential
+Containers.
+
+Please refer to our [Acronyms](https://github.com/confidential-containers/documentation/wiki/Acronyms)
+and [Glossary](https://github.com/confidential-containers/documentation/wiki/Glossary) pages for a
+definition of the acronyms used in this document.
+
+## What's new
+
+* This release is built from the main branch of Kata Containers.
+* Non-tee attestation is now based on a sample attester and verifier rather than on `offline_fs_kbc`.
+    * Resources can be dynamically delivered in confidential environments.
+    * Trustee is integrated into the Kata Containers CI.
+* All platforms now share one confidential rootfs.
+* All platforms share one confidential guest kernel.
+* Image request timeout is configurable to facilitate pulling large images.
+* Attestation Agent now supports generic `configfs-tsm` ABI for collecting evidence.
+* Enclave-cc moves to unified LibOS bundle for secure rootfs key handling and to the latest Occlum v0.30.1 release that adds SGX EDMM support for dynamically adjusting the enclave size.
+* Adoption of a project-wide security reporting protocol
+
+## Hardware Support
+
+This release does not officially support any hardware platforms.
+It is mainly intended for testing in non-tee environments.
+Future releases will return to previous levels of support.
+
+## Limitations
+
+The following are known limitations of this release:
+
+* Nydus snapshotter support is not mature.
+  * Nydus snapshot sometimes conflicts with existing node configuration.
+  * You may need to remove existing container images/snapshots before installing Nydus snapshotter.
+  * Nydus snapshotter may not support pulling one image with multiple runtime handler annotations even across different pods.
+    * These limitations can apply to the pause image when filesystem passthrough is not enabled.
+  * Host pulling with Nydus snapshotter is not yet enabled.
+  * Nydus snapshotter is not supported with enclave-cc.
+* Pulling container images inside guest may have negative performance implications including greater resource usage and slower startup.
+* `crio` support is still evolving.
+* Platform support is rapidly changing
+* SELinux is not supported on the host and must be set to permissive if in use.
+* The format of encrypted container images is still subject to change
+  * The [oci-crypt](https://github.com/containers/ocicrypt) container image format itself may still change
+  * The tools to generate images are not in their final form
+  * The image format itself is subject to change in upcoming releases
+  * Not all image repositories support encrypted container images.
+* Complete integration with Kubernetes is still in progress.
+  * OpenShift support is not yet complete.
+  * Existing APIs do not fully support the CoCo security and threat model. [More info](https://github.com/confidential-containers/community/issues/53)
+  * Some commands accessing confidential data, such as `kubectl exec`, may either fail to work, or incorrectly expose information to the host
+* The CoCo community aspires to adopting open source security best practices, but not all practices are adopted yet.
+  * We track our status with the OpenSSF Best Practices Badge, which improved to 75% at the time of this release.
+  * Community has adopted a security reporting protocol, but application and documentation of static and dynamic analysis still needed.
+* Container metadata such as environment variables are not measured.
+* Kata Agent does not validate mount requests. A malicious host might be able to mount a shared filesystem into the PodVM.
+
+## CVE Fixes
+
+None
+

releases/v0.9.0-alpha1.md

@@ -0,0 +1,68 @@
+# Release Notes for v0.9.0-alpha1
+
+Release Date: June 21st, 2024
+
+This release is based on [3.6.0](https://github.com/kata-containers/kata-containers/releases/tag/3.4.0) of Kata Containers
+and [v0.9.1](https://github.com/confidential-containers/enclave-cc/releases/tag/v0.9.0) of enclave-cc.
+
+This is an alpha release, with some limitations, but it builds significantly on the previous release.
+For a full feature set, please use v0.8.0 or the upcoming v0.9.0.
+For details of what has been added and what does not yet work,
+see the following sections.
+
+Please see the [quickstart guide](../quickstart.md) for details on how to try out Confidential
+Containers.
+
+Please refer to our [Acronyms](https://github.com/confidential-containers/documentation/wiki/Acronyms)
+and [Glossary](https://github.com/confidential-containers/documentation/wiki/Glossary) pages for
+definitions of the acronyms used in this document.
+
+## What's new
+
+* Cloud API Adaptor (peer pods) is included in the release and based on Kata main.
+* Image pulling inside the guest via the nydus snapshotter is more stable.
+* Platform support has been partially restored.
+* Confidential runtime classes ship with filesystem sharing disabled by default.
+* `kata-qemu-coco-dev` runtime class introduced for nontee development.
+* Several improvements to guest policy
+* `guest_components_procs` option introduced to control which guest components are started.
+
+## Hardware Support
+
+This release tentatively supports multiple TEE platforms, although
+not all features are enabled for all platforms and test coverage
+is limited.
+
+## Limitations
+
+The following are known limitations of this release:
+
+* Encrypted images and signed images are not yet supported.
+* Sealed secrets and encrypted volumes are not yet supported.
+* SEV(-ES) and SEV-SNP do not support attestation.
+* Authenticated registries are not supported.
+* Nydus snapshotter support is not mature.
+  * Nydus snapshotter sometimes fails to pull an image.
+  * Nydus snapshotter cannot handle pods with init containers.
+  * Host pulling with Nydus snapshotter is not yet enabled.
+  * Nydus snapshotter is not supported with enclave-cc.
+* Pulling container images inside guest may have negative performance implications including greater resource usage and slower startup.
+* `crio` support is still evolving.
+* Platform support is rapidly changing
+* SELinux is not supported on the host and must be set to permissive if in use.
+* Complete integration with Kubernetes is still in progress.
+  * OpenShift support is not yet complete.
+  * Existing APIs do not fully support the CoCo security and threat model. [More info](https://github.com/confidential-containers/community/issues/53)
+  * Some commands accessing confidential data, such as `kubectl exec`, may either fail to work, or incorrectly expose information to the host
+* The CoCo community aspires to adopting open source security best practices, but not all practices are adopted yet.
+  * We track our status with the OpenSSF Best Practices Badge, which remained at 75% at the time of this release.
+  * Community has adopted a security reporting protocol, but application and documentation of static and dynamic analysis still needed.
+* Container metadata such as environment variables are not measured.
+* The Kata Agent allows the host to call several dangerous endpoints
+    * Kata Agent does not validate mount requests. A malicious host might be able to mount a shared filesystem into the PodVM.
+    * Policy can be used to block endpoints, but it is not yet tied to the hardware evidence.
+
+## CVE Fixes
+
+None
+

releases/v0.9.0.md

@@ -0,0 +1,94 @@
+# Release Notes for v0.9.0
+
+Release Date: July 26th, 2024
+
+This release is based on [3.7.0](https://github.com/kata-containers/kata-containers/releases/tag/3.7.0) of Kata Containers
+and [v0.9.1](https://github.com/confidential-containers/enclave-cc/releases/tag/v0.9.1) of enclave-cc.
+
+This is the first non-alpha release of Confidential Containers to be based on the main branch of Kata Containers.
+This release does not have complete parity with releases based on CCv0, but it supports most features.
+See the limitations section for more details.
+
+
+Please see the [quickstart guide](../quickstart.md) for details on how to try out Confidential
+Containers.
+
+Please refer to our [Acronyms](https://github.com/confidential-containers/documentation/wiki/Acronyms)
+and [Glossary](https://github.com/confidential-containers/documentation/wiki/Glossary) pages for
+definitions of the acronyms used in this document.
+
+## What's new
+
+* Attestation is supported on SEV-SNP and IBM SE
+* Encrypted container images are supported
+* Authenticated registries are supported
+* Pods with init containers can be run wth Nydus
+* Sealed secrets (as environment variables) are supported
+
+## Hardware Support
+
+Attestation is supported and tested on three platforms: Intel TDX, AMD SEV-SNP, and IBM SE.
+Not all feature have been tested on every platform, but those based on attestation
+are expected to work on the platforms above.
+
+Make sure your host platform is compatible with the hypervisor and guest kernel
+provisioned by coco.
+
+This release has been tested on the following stacks:
+
+### AMD SEV-SNP
+
+* Processor: AMD EPYC 7413
+* Kernel: [6.8.0-rc5-next-20240221-snp-host-cc2568386](https://github.com/confidential-containers/linux/tree/amd-snp-host-202402240000)
+* OS: Ubuntu 22.04.4 LTS
+* k8s: v1.24.0 (Kubeadm)
+* Kustomize: v4.5.4
+
+### Intel TDX
+
+* Kernel: [6.8.0-1004-intel](https://git.launchpad.net/~kobuk-team/ubuntu/+source/linux-intel/tree/?h=noble-main-next)
+* OS: Ubuntu 24.04 LTS
+* k8s: v1.30.2 (Kubeadm)
+* Kustomize: v5.0.4-0.20230601165947-6ce0bf390ce3
+
+### Secure Execution on IBM zSystems (s390x) running LinuxONE
+
+* Hardware: IBM Z16 LPAR
+* Kernel: 5.15.0-113-generic
+* OS: Ubuntu 22.04.1 LTS
+* k8s: v1.28.4 (k3s)
+* Kustomize: v5.3.0
+
+## Limitations
+
+The following are known limitations of this release:
+
+* Signed images are not yet supported.
+* Secure storage is not yet supported.
+* SEV(-ES) does not support attestation.
+* Sealed secrets only supports secrets in environment variables.
+* Credentials for authenticated registries are exposed to the host.
+* Not all features are tested on all platforms.
+* Nydus snapshotter support is not mature.
+  * Nydus snapshotter sometimes fails to pull an image.
+  * Host pulling with Nydus snapshotter is not yet enabled.
+  * Nydus snapshotter is not supported with enclave-cc.
+* Pulling container images inside guest may have negative performance implications including greater resource usage and slower startup.
+* `crio` support is still evolving.
+* Platform support is rapidly changing
+* SELinux is not supported on the host and must be set to permissive if in use.
+* Complete integration with Kubernetes is still in progress.
+  * Existing APIs do not fully support the CoCo security and threat model. [More info](https://github.com/confidential-containers/community/issues/53)
+  * Some commands accessing confidential data, such as `kubectl exec`, may either fail to work, or incorrectly expose information to the host
+* The CoCo community aspires to adopting open source security best practices, but not all practices are adopted yet.
+  * We track our status with the OpenSSF Best Practices Badge, which remained at 75% at the time of this release.
+  * Community has adopted a security reporting protocol, but application and documentation of static and dynamic analysis still needed.
+* Container metadata such as environment variables are not measured.
+* The Kata Agent allows the host to call several dangerous endpoints
+    * Kata Agent does not validate mount requests. A malicious host might be able to mount a shared filesystem into the PodVM.
+    * Policy can be used to block endpoints, but it is not yet tied to the hardware evidence.
+
+## CVE Fixes
+
+None
+

trust_model.md

@@ -37,7 +37,7 @@ Further documentation will highlight specific [threat vectors](./threats_overvie
 considering risk,
 impact, mitigation etc as the project progresses. The Security Assurance section, Page 31, of
 Cloud Native Computing Foundation (CNCF)
-[Cloud Native Security Paper](https://github.com/cncf/tag-security/blob/main/security-whitepaper/v2/CNCF_cloud-native-security-whitepaper-May2022-v2.pdf)
+[Cloud Native Security Paper](https://github.com/cncf/tag-security/blob/3e57e7c472f7053c693292281419ab926155fe2d/community/resources/security-whitepaper/v2/CNCF_cloud-native-security-whitepaper-May2022-v2.pdf)
  will guide this more detailed threat vector effort.
 
 ### Related Prior Effort
@@ -51,7 +51,7 @@ For example:
   "[A Technical Analysis of Confidential Computing](https://confidentialcomputing.io/wp-content/uploads/sites/10/2023/03/CCC-A-Technical-Analysis-of-Confidential-Computing-v1.3_unlocked.pdf)"
   section 5 of which defines the threat model for confidential computing.
 - CNCF Security Technical Advisory Group published
-  "[Cloud Native Security Whitepaper](https://github.com/cncf/tag-security/blob/main/security-whitepaper/v2/CNCF_cloud-native-security-whitepaper-May2022-v2.pdf)"
+  "[Cloud Native Security Whitepaper](https://github.com/cncf/tag-security/blob/3e57e7c472f7053c693292281419ab926155fe2d/community/resources/security-whitepaper/v2/CNCF_cloud-native-security-whitepaper-May2022-v2.pdf)"
 - Kubernetes provides documentation :
   "[Overview of Cloud Native Security](https://kubernetes.io/docs/concepts/security/overview/)"
 - Open Web Application Security Project -

trust_model_personas.md

@@ -5,7 +5,7 @@ Otherwise referred to as actors or agents, these are individuals or groups capab
 carrying out a particular threat.
 In identifying personas we consider :
 - The Runtime Environment, Figure 5, Page 19 of CNCF
-  [Cloud Native Security Paper](https://github.com/cncf/tag-security/blob/main/security-whitepaper/v2/CNCF_cloud-native-security-whitepaper-May2022-v2.pdf).
+  [Cloud Native Security Paper](https://github.com/cncf/tag-security/blob/3e57e7c472f7053c693292281419ab926155fe2d/community/resources/security-whitepaper/v2/CNCF_cloud-native-security-whitepaper-May2022-v2.pdf).
   This highlights three layers, Cloud/Environment, Workload Orchestration, Application.
 - The Kubernetes
   [Overview of Cloud Native Security](https://kubernetes.io/docs/concepts/security/overview/)
@@ -19,7 +19,7 @@ In identifying personas we consider :
 
 In considering personas we recognise that a trust boundary exists between each persona and we
 explore how the least privilege principle (as described on Page 40 of
-  [Cloud Native Security Paper](https://github.com/cncf/tag-security/blob/main/security-whitepaper/v2/CNCF_cloud-native-security-whitepaper-May2022-v2.pdf)
+  [Cloud Native Security Paper](https://github.com/cncf/tag-security/blob/3e57e7c472f7053c693292281419ab926155fe2d/community/resources/security-whitepaper/v2/CNCF_cloud-native-security-whitepaper-May2022-v2.pdf)
 ) should apply to any actions which cross these boundaries.
 
  Confidential containers can provide enhancements to ensure that the expected code/containers
@@ -136,7 +136,7 @@ the images they need but also support the verification method they require.  A k
 relationship is the Workload Provider applying Supply Chain
 Security practices (as
 described on Page 42 of
-[Cloud Native Security Paper](https://github.com/cncf/tag-security/blob/main/security-whitepaper/v2/CNCF_cloud-native-security-whitepaper-May2022-v2.pdf)
+[Cloud Native Security Paper](https://github.com/cncf/tag-security/blob/3e57e7c472f7053c693292281419ab926155fe2d/community/resources/security-whitepaper/v2/CNCF_cloud-native-security-whitepaper-May2022-v2.pdf)
 ) when considering Container
 Image Providers. So the Container Image Provider must support the Workload Providers
 ability to provide assurance to the Data Owner regarding integrity of the code.