packaging: fix image build script

Relative paths are error prone. Fix error.

Signed-off-by: Eric Ernst <eric.g.ernst@gmail.com>

.github/ISSUE_TEMPLATE.md

@@ -1,17 +0,0 @@
-# Description of problem
-
-(replace this text with the list of steps you followed)
-
-# Expected result
-
-(replace this text with an explanation of what you thought would happen)
-
-# Actual result
-
-(replace this text with details of what actually happened)
-
----
-
-(replace this text with the output of the `kata-collect-data.sh` script, after
-you have reviewed its content to ensure it does not contain any private
-information).

.github/workflows/PR-wip-checks.yaml

@@ -0,0 +1,21 @@
+name: Pull request WIP checks
+on:
+  pull_request:
+    types:
+      - opened
+      - synchronize
+      - reopened
+      - edited
+      - labeled
+      - unlabeled
+
+jobs:
+  pr_wip_check:
+    runs-on: ubuntu-latest
+    name: WIP Check
+    steps:
+    - name: WIP Check
+      uses: tim-actions/wip-check@1c2a1ca6c110026b3e2297bb2ef39e1747b5a755
+      with:
+        labels: '["do-not-merge", "wip", "rfc"]'
+        keywords: '["WIP", "wip", "RFC", "rfc", "dnm", "DNM", "do-not-merge"]'

.github/workflows/add-issues-to-project.yaml

@@ -0,0 +1,55 @@
+# Copyright (c) 2020 Intel Corporation
+#
+# SPDX-License-Identifier: Apache-2.0
+#
+
+name: Add newly created issues to the backlog project
+
+on:
+  issues:
+    types:
+      - opened
+      - reopened
+
+jobs:
+  add-new-issues-to-backlog:
+    runs-on: ubuntu-latest
+    steps:
+      - name: Install hub
+        run: |
+          HUB_ARCH="amd64"
+          HUB_VER=$(curl -sL "https://api.github.com/repos/github/hub/releases/latest" |\
+            jq -r .tag_name | sed 's/^v//')
+          curl -sL \
+            "https://github.com/github/hub/releases/download/v${HUB_VER}/hub-linux-${HUB_ARCH}-${HUB_VER}.tgz" |\
+          tar xz --strip-components=2 --wildcards '*/bin/hub' && \
+          sudo install hub /usr/local/bin
+
+      - name: Install hub extension script
+        run: |
+          # Clone into a temporary directory to avoid overwriting
+          # any existing github directory.
+          pushd $(mktemp -d) &>/dev/null
+          git clone --single-branch --depth 1 "https://github.com/kata-containers/.github" && cd .github/scripts
+          sudo install hub-util.sh /usr/local/bin
+          popd &>/dev/null
+
+      - name: Checkout code to allow hub to communicate with the project
+        uses: actions/checkout@v2
+
+      - name: Add issue to issue backlog
+        env:
+          GITHUB_TOKEN: ${{ secrets.KATA_GITHUB_ACTIONS_TOKEN }}
+        run: |
+          issue=${{ github.event.issue.number }}
+
+          project_name="Issue backlog"
+          project_type="org"
+          project_column="To do"
+
+          hub-util.sh \
+            add-issue \
+            "$issue" \
+            "$project_name" \
+            "$project_type" \
+            "$project_column"

.github/workflows/commit-message-check.yaml

@@ -0,0 +1,91 @@
+name: Commit Message Check
+on:
+  pull_request:
+    types:
+      - opened
+      - reopened
+      - synchronize
+
+env:
+  error_msg: |+
+    See the document below for help on formatting commits for the project.
+
+    https://github.com/kata-containers/community/blob/master/CONTRIBUTING.md#patch-forma
+
+jobs:
+  commit-message-check:
+    runs-on: ubuntu-latest
+    name: Commit Message Check
+    steps:
+    - name: Get PR Commits
+      id: 'get-pr-commits'
+      uses: tim-actions/get-pr-commits@v1.0.0
+      with:
+        token: ${{ secrets.GITHUB_TOKEN }}
+
+    - name: DCO Check
+      uses: tim-actions/dco@2fd0504dc0d27b33f542867c300c60840c6dcb20
+      with:
+        commits: ${{ steps.get-pr-commits.outputs.commits }}
+
+    - name: Commit Body Missing Check
+      if: ${{ success() || failure() }}
+      uses: tim-actions/commit-body-check@v1.0.2
+      with:
+        commits: ${{ steps.get-pr-commits.outputs.commits }}
+
+    - name: Check Subject Line Length
+      if: ${{ success() || failure() }}
+      uses: tim-actions/commit-message-checker-with-regex@v0.3.1
+      with:
+        commits: ${{ steps.get-pr-commits.outputs.commits }}
+        pattern: '^.{0,75}(\n.*)*$'
+        error: 'Subject too long (max 75)'
+        post_error: ${{ env.error_msg }}
+
+    - name: Check Body Line Length
+      if: ${{ success() || failure() }}
+      uses: tim-actions/commit-message-checker-with-regex@v0.3.1
+      with:
+        commits: ${{ steps.get-pr-commits.outputs.commits }}
+        # Notes:
+        #
+        # - The subject line is not enforced here (see other check), but has
+        #   to be specified at the start of the regex as the action is passed
+        #   the entire commit message.
+        #
+        # - Body lines *can* be longer than the maximum if they start
+        #   with a non-alphabetic character.
+        #
+        #   This allows stack traces, log files snippets, emails, long URLs,
+        #   etc to be specified. Some of these naturally "work" as they start
+        #   with numeric timestamps or addresses. Emails can but quoted using
+        #   the normal ">" character, markdown bullets ("-", "*") are also
+        #   useful for lists of URLs, but it is always possible to override
+        #   the check by simply space indenting the content you need to add.
+        #
+        # - A SoB comment can be any length (as it is unreasonable to penalise
+        #   people with long names/email addresses :)
+        pattern: '^.+(\n([a-zA-Z].{0,149}|[^a-zA-Z\n].*|Signed-off-by:.*|))+$'
+        error: 'Body line too long (max 72)'
+        post_error: ${{ env.error_msg }}
+
+    - name: Check Fixes
+      if: ${{ success() || failure() }}
+      uses: tim-actions/commit-message-checker-with-regex@v0.3.1
+      with:
+        commits: ${{ steps.get-pr-commits.outputs.commits }}
+        pattern: '\s*Fixes\s*:?\s*(#\d+|github\.com\/kata-containers\/[a-z-.]*#\d+)|^\s*release\s*:'
+        flags: 'i'
+        error: 'No "Fixes" found'
+        post_error: ${{ env.error_msg }}
+        one_pass_all_pass: 'true'
+
+    - name: Check Subsystem
+      if: ${{ success() || failure() }}
+      uses: tim-actions/commit-message-checker-with-regex@v0.3.1
+      with:
+        commits: ${{ steps.get-pr-commits.outputs.commits }}
+        pattern: '^[\s\t]*[^:\s\t]+[\s\t]*:'
+        error: 'Failed to find subsystem in subject'
+        post_error: ${{ env.error_msg }}

.github/workflows/generate-local-artifact-tarball.sh

@@ -0,0 +1,34 @@
+#!/bin/bash
+# Copyright (c) 2019 Intel Corporation
+# Copyright (c) 2020 Ant Group
+#
+# SPDX-License-Identifier: Apache-2.0
+#
+
+set -o errexit
+set -o pipefail
+
+
+main() {
+    artifact_stage=${1:-}
+    artifact=$(echo  ${artifact_stage} | sed -n -e 's/^install_//p' | sed -r 's/_/-/g')
+    if [ -z "${artifact}" ]; then
+        "Scripts needs artifact name to build"
+        exit 1
+    fi
+
+    tag=$(echo $GITHUB_REF | cut -d/ -f3-)
+    pushd $GITHUB_WORKSPACE/tools/packaging
+    git checkout $tag
+    ./scripts/gen_versions_txt.sh $tag
+    popd
+
+    pushd $GITHUB_WORKSPACE/tools/packaging/release
+    source ./kata-deploy-binaries.sh
+    ${artifact_stage} $tag
+    popd
+
+    mv $GITHUB_WORKSPACE/tools/packaging/release/kata-static-${artifact}.tar.gz .
+}
+
+main $@

.github/workflows/kata-deploy-test.yaml

@@ -0,0 +1,53 @@
+on: issue_comment
+name: test-kata-deploy
+jobs:
+  check_comments:
+    runs-on: ubuntu-latest
+    steps:
+      - name: Check for Command
+        id: command
+        uses: kata-containers/slash-command-action@v1
+        with:
+          repo-token: ${{ secrets.GITHUB_TOKEN }}
+          command: "test-kata-deploy"
+          reaction: "true"
+          reaction-type: "eyes"
+          allow-edits: "false"
+          permission-level: admin
+      - name: verify command arg is kata-deploy
+        run: |
+           echo "The command was '${{ steps.command.outputs.command-name }}' with arguments '${{ steps.command.outputs.command-arguments }}'"
+  create-and-test-container:
+    needs: check_comments
+    runs-on: ubuntu-latest
+    steps:
+      - name: get-PR-ref
+        id: get-PR-ref
+        run: |
+            ref=$(cat $GITHUB_EVENT_PATH | jq -r '.issue.pull_request.url' | sed  's#^.*\/pulls#refs\/pull#' | sed 's#$#\/merge#')
+            echo "reference for PR: " ${ref}
+            echo "##[set-output name=pr-ref;]${ref}"
+      - uses: actions/checkout@v2-beta
+        with:
+          ref: ${{ steps.get-PR-ref.outputs.pr-ref }}
+      - name: build-container-image
+        id: build-container-image
+        run: |
+            PR_SHA=$(git log --format=format:%H -n1)
+            VERSION=$(curl https://raw.githubusercontent.com/kata-containers/kata-containers/2.0-dev/VERSION)
+            ARTIFACT_URL="https://github.com/kata-containers/kata-containers/releases/download/${VERSION}/kata-static-${VERSION}-x86_64.tar.xz"
+            wget "${ARTIFACT_URL}" -O ./kata-deploy/kata-static.tar.xz
+            docker build --build-arg KATA_ARTIFACTS=kata-static.tar.xz -t katadocker/kata-deploy-ci:${PR_SHA} ./kata-deploy
+            docker login -u ${{ secrets.DOCKER_USERNAME }} -p ${{ secrets.DOCKER_PASSWORD }}
+            docker push katadocker/kata-deploy-ci:$PR_SHA
+            echo "##[set-output name=pr-sha;]${PR_SHA}"
+      - name: test-kata-deploy-ci-in-aks
+        uses: ./kata-deploy/action
+        with:
+          packaging-sha: ${{ steps.build-container-image.outputs.pr-sha }}
+        env:
+          PKG_SHA: ${{ steps.build-container-image.outputs.pr-sha }}
+          AZ_APPID: ${{ secrets.AZ_APPID }}
+          AZ_PASSWORD: ${{ secrets.AZ_PASSWORD }}
+          AZ_SUBSCRIPTION_ID: ${{ secrets.AZ_SUBSCRIPTION_ID }}
+          AZ_TENANT_ID: ${{ secrets.AZ_TENANT_ID }}

.github/workflows/main.yaml

@@ -2,7 +2,7 @@ name: Publish release tarball
 on: 
   push: 
     tags:
-     - '*'
+     - '1.*'
 
 jobs:
   get-artifact-list:
@@ -10,12 +10,11 @@ jobs:
     steps:
       - name: get the list
         run: |
-         git clone https://github.com/kata-containers/packaging
-         pushd packaging
+         pushd $GITHUB_WORKSPACE
          tag=$(echo $GITHUB_REF | cut -d/ -f3-)
          git checkout $tag
          popd
-         ./packaging/artifact-list.sh > artifact-list.txt
+         $GITHUB_WORKSPACE/tools/packaging/artifact-list.sh > artifact-list.txt
       - name: save-artifact-list
         uses: actions/upload-artifact@master
         with:

.github/workflows/move-issues-to-in-progress.yaml

@@ -0,0 +1,78 @@
+# Copyright (c) 2020 Intel Corporation
+#
+# SPDX-License-Identifier: Apache-2.0
+#
+
+name: Move issues to "In progress" in backlog project when referenced by a PR
+
+on:
+  pull_request_target:
+    types:
+      - opened
+      - reopened
+
+jobs:
+  move-linked-issues-to-in-progress:
+    runs-on: ubuntu-latest
+    steps:
+      - name: Install hub
+        run: |
+          HUB_ARCH="amd64"
+          HUB_VER=$(curl -sL "https://api.github.com/repos/github/hub/releases/latest" |\
+            jq -r .tag_name | sed 's/^v//')
+          curl -sL \
+            "https://github.com/github/hub/releases/download/v${HUB_VER}/hub-linux-${HUB_ARCH}-${HUB_VER}.tgz" |\
+          tar xz --strip-components=2 --wildcards '*/bin/hub' && \
+          sudo install hub /usr/local/bin
+
+      - name: Install hub extension script
+        run: |
+          # Clone into a temporary directory to avoid overwriting
+          # any existing github directory.
+          pushd $(mktemp -d) &>/dev/null
+          git clone --single-branch --depth 1 "https://github.com/kata-containers/.github" && cd .github/scripts
+          sudo install hub-util.sh /usr/local/bin
+          popd &>/dev/null
+
+      - name: Checkout code to allow hub to communicate with the project
+        uses: actions/checkout@v2
+
+      - name: Move issue to "In progress"
+        env:
+          GITHUB_TOKEN: ${{ secrets.KATA_GITHUB_ACTIONS_TOKEN }}
+        run: |
+          pr=${{ github.event.pull_request.number }}
+
+          linked_issue_urls=$(hub-util.sh \
+            list-issues-for-pr "$pr" |\
+            grep -v "^\#"  |\
+            cut -d';' -f3 || true)
+
+          # PR doesn't have any linked issues
+          # (it should, but maybe a new user forgot to add a "Fixes: #XXX" commit).
+          [ -z "$linked_issue_urls" ] && {
+            echo "::error::No linked issues for PR $pr"
+            exit 1
+          }
+
+          project_name="Issue backlog"
+          project_type="org"
+          project_column="In progress"
+
+          for issue_url in $(echo "$linked_issue_urls")
+          do
+            issue=$(echo "$issue_url"| awk -F\/ '{print $NF}' || true)
+
+            [ -z "$issue" ] && {
+              echo "::error::Cannot determine issue number from $issue_url for PR $pr"
+              exit 1
+            }
+
+            # Move the issue to the correct column on the project board
+            hub-util.sh \
+              move-issue \
+              "$issue" \
+              "$project_name" \
+              "$project_type" \
+              "$project_column"
+          done

.github/workflows/release.yaml

@@ -0,0 +1,321 @@
+name: Publish Kata 2.x release artifacts
+on:
+  push:
+    tags:
+     - '2.*'
+
+jobs:
+  get-artifact-list:
+    runs-on: ubuntu-latest
+    steps:
+      - uses: actions/checkout@v2
+      - name: get the list
+        run: |
+         pushd $GITHUB_WORKSPACE
+         tag=$(echo $GITHUB_REF | cut -d/ -f3-)
+         git checkout $tag
+         popd
+         $GITHUB_WORKSPACE/tools/packaging/artifact-list.sh > artifact-list.txt
+      - name: save-artifact-list
+        uses: actions/upload-artifact@v2
+        with:
+          name: artifact-list
+          path: artifact-list.txt
+
+  build-kernel:
+    runs-on: ubuntu-16.04
+    needs: get-artifact-list
+    env:
+      buildstr: "install_kernel"
+    steps:
+      - uses: actions/checkout@v2
+      - name: get-artifact-list
+        uses: actions/download-artifact@v2
+        with:
+          name: artifact-list
+      - run: |
+         sudo apt-get update && sudo apt install -y flex bison libelf-dev bc iptables
+      - name: build-kernel
+        run: |
+         if grep -q $buildstr artifact-list.txt; then
+           $GITHUB_WORKSPACE/.github/workflows/generate-local-artifact-tarball.sh $buildstr
+           echo ::set-env name=artifact-built::true
+         else
+           echo ::set-env name=artifact-built::false
+         fi
+      - name: store-artifacts
+        if: env.artifact-built == 'true'
+        uses: actions/upload-artifact@v2
+        with:
+          name: kata-artifacts
+          path: kata-static-kernel.tar.gz
+
+  build-experimental-kernel:
+    runs-on: ubuntu-16.04
+    needs: get-artifact-list
+    env:
+      buildstr: "install_experimental_kernel"
+    steps:
+      - uses: actions/checkout@v2
+      - name: get-artifact-list
+        uses: actions/download-artifact@v2
+        with:
+          name: artifact-list
+      - run: |
+         sudo apt-get update && sudo apt install -y flex bison libelf-dev bc iptables
+      - name: build-experimental-kernel
+        run: |
+         if grep -q $buildstr artifact-list.txt; then
+           $GITHUB_WORKSPACE/.github/workflows/generate-local-artifact-tarball.sh $buildstr
+           echo ::set-env name=artifact-built::true
+         else
+           echo ::set-env name=artifact-built::false
+         fi
+      - name: store-artifacts
+        if: env.artifact-built == 'true'
+        uses: actions/upload-artifact@v2
+        with:
+          name: kata-artifacts
+          path: kata-static-experimental-kernel.tar.gz
+
+  build-qemu:
+    runs-on: ubuntu-16.04
+    needs: get-artifact-list
+    env:
+      buildstr: "install_qemu"
+    steps:
+      - uses: actions/checkout@v2
+      - name: get-artifact-list
+        uses: actions/download-artifact@v2
+        with:
+          name: artifact-list
+      - name: build-qemu
+        run: |
+         if grep -q $buildstr artifact-list.txt; then
+           $GITHUB_WORKSPACE/.github/workflows/generate-local-artifact-tarball.sh $buildstr
+           echo ::set-env name=artifact-built::true
+         else
+           echo ::set-env name=artifact-built::false
+         fi
+      - name: store-artifacts
+        if: env.artifact-built == 'true'
+        uses: actions/upload-artifact@v2
+        with:
+          name: kata-artifacts
+          path: kata-static-qemu.tar.gz
+
+  build-qemu-virtiofsd:
+    runs-on: ubuntu-16.04
+    needs: get-artifact-list
+    env:
+      buildstr: "install_qemu_virtiofsd"
+    steps:
+      - uses: actions/checkout@v2
+      - name: get-artifact-list
+        uses: actions/download-artifact@v2
+        with:
+          name: artifact-list
+      - name: build-qemu-virtiofsd
+        run: |
+         if grep -q $buildstr artifact-list.txt; then
+           $GITHUB_WORKSPACE/.github/workflows/generate-local-artifact-tarball.sh $buildstr
+           echo ::set-env name=artifact-built::true
+         else
+           echo ::set-env name=artifact-built::false
+         fi
+      - name: store-artifacts
+        if: env.artifact-built == 'true'
+        uses: actions/upload-artifact@v2
+        with:
+          name: kata-artifacts
+          path: kata-static-qemu-virtiofsd.tar.gz
+
+  build-image:
+    runs-on: ubuntu-16.04
+    needs: get-artifact-list
+    env:
+      buildstr: "install_image"
+    steps:
+      - uses: actions/checkout@v2
+      - name: get-artifact-list
+        uses: actions/download-artifact@v2
+        with:
+          name: artifact-list
+      - name: build-image
+        run: |
+         if grep -q $buildstr artifact-list.txt; then
+           $GITHUB_WORKSPACE/.github/workflows/generate-local-artifact-tarball.sh $buildstr
+           echo ::set-env name=artifact-built::true
+         else
+           echo ::set-env name=artifact-built::false
+         fi
+      - name: store-artifacts
+        if: env.artifact-built == 'true'
+        uses: actions/upload-artifact@v2
+        with:
+          name: kata-artifacts
+          path: kata-static-image.tar.gz
+
+  build-firecracker:
+    runs-on: ubuntu-16.04
+    needs: get-artifact-list
+    env:
+      buildstr: "install_firecracker"
+    steps:
+      - uses: actions/checkout@v2
+      - name: get-artifact-list
+        uses: actions/download-artifact@v2
+        with:
+          name: artifact-list
+      - name: build-firecracker
+        run: |
+         if grep -q $buildstr artifact-list.txt; then
+           $GITHUB_WORKSPACE/.github/workflows/generate-local-artifact-tarball.sh $buildstr
+           echo ::set-env name=artifact-built::true
+         else
+           echo ::set-env name=artifact-built::false
+         fi
+      - name: store-artifacts
+        if: env.artifact-built == 'true'
+        uses: actions/upload-artifact@v2
+        with:
+          name: kata-artifacts
+          path: kata-static-firecracker.tar.gz
+
+
+  build-clh:
+    runs-on: ubuntu-16.04
+    needs: get-artifact-list
+    env:
+      buildstr: "install_clh"
+    steps:
+      - uses: actions/checkout@v2
+      - name: get-artifact-list
+        uses: actions/download-artifact@v2
+        with:
+          name: artifact-list
+      - name: build-clh
+        run: |
+         if grep -q $buildstr artifact-list.txt; then
+           $GITHUB_WORKSPACE/.github/workflows/generate-local-artifact-tarball.sh $buildstr
+           echo ::set-env name=artifact-built::true
+         else
+           echo ::set-env name=artifact-built::false
+         fi
+      - name: store-artifacts
+        if: env.artifact-built == 'true'
+        uses: actions/upload-artifact@v2
+        with:
+          name: kata-artifacts
+          path: kata-static-clh.tar.gz
+
+  build-kata-components:
+    runs-on: ubuntu-16.04
+    needs: get-artifact-list
+    env:
+      buildstr: "install_kata_components"
+    steps:
+      - uses: actions/checkout@v2
+      - name: get-artifact-list
+        uses: actions/download-artifact@v2
+        with:
+          name: artifact-list
+      - name: build-kata-components
+        run: |
+         if grep -q $buildstr artifact-list.txt; then
+           $GITHUB_WORKSPACE/.github/workflows/generate-local-artifact-tarball.sh $buildstr
+           echo ::set-env name=artifact-built::true
+         else
+           echo ::set-env name=artifact-built::false
+         fi
+      - name: store-artifacts
+        if: env.artifact-built == 'true'
+        uses: actions/upload-artifact@v2
+        with:
+          name: kata-artifacts
+          path: kata-static-kata-components.tar.gz
+
+  gather-artifacts:
+    runs-on: ubuntu-16.04
+    needs: [build-experimental-kernel, build-kernel, build-qemu, build-qemu-virtiofsd, build-image, build-firecracker, build-kata-components, build-clh]
+    steps:
+      - uses: actions/checkout@v2
+      - name: get-artifacts
+        uses: actions/download-artifact@v2
+        with:
+          name: kata-artifacts
+          path: kata-artifacts
+      - name: colate-artifacts
+        run: |
+          $GITHUB_WORKSPACE/.github/workflows/gather-artifacts.sh
+      - name: store-artifacts
+        uses: actions/upload-artifact@v2
+        with:
+          name: release-candidate
+          path: kata-static.tar.xz
+
+  kata-deploy:
+    needs: gather-artifacts
+    runs-on: ubuntu-latest
+    steps:
+      - uses: actions/checkout@v2
+      - name: get-artifacts
+        uses: actions/download-artifact@v2
+        with:
+          name: release-candidate
+      - name: build-and-push-kata-deploy-ci
+        id: build-and-push-kata-deploy-ci
+        run: |
+          tag=$(echo $GITHUB_REF | cut -d/ -f3-)
+          pushd $GITHUB_WORKSPACE
+          git checkout $tag
+          pkg_sha=$(git rev-parse HEAD)
+          popd
+          mv kata-static.tar.xz $GITHUB_WORKSPACE/tools/packaging/kata-deploy/kata-static.tar.xz
+          docker build --build-arg KATA_ARTIFACTS=kata-static.tar.xz -t katadocker/kata-deploy-ci:$pkg_sha $GITHUB_WORKSPACE/tools/packaging/kata-deploy
+          docker login -u ${{ secrets.DOCKER_USERNAME }} -p ${{ secrets.DOCKER_PASSWORD }}
+          docker push katadocker/kata-deploy-ci:$pkg_sha
+
+          echo "##[set-output name=PKG_SHA;]${pkg_sha}"
+          echo ::set-env name=TAG::$tag
+          mkdir -p packaging/kata-deploy
+          ln -s $GITHUB_WORKSPACE/tools/packaging/kata-deploy/action packaging/kata-deploy/action
+      - name: test-kata-deploy-ci-in-aks
+        uses: ./packaging/kata-deploy/action
+        with:
+          packaging-sha: ${{steps.build-and-push-kata-deploy-ci.outputs.PKG_SHA}}
+        env:
+          PKG_SHA: ${{steps.build-and-push-kata-deploy-ci.outputs.PKG_SHA}}
+          AZ_APPID: ${{ secrets.AZ_APPID }}
+          AZ_PASSWORD: ${{ secrets.AZ_PASSWORD }}
+          AZ_SUBSCRIPTION_ID: ${{ secrets.AZ_SUBSCRIPTION_ID }}
+          AZ_TENANT_ID: ${{ secrets.AZ_TENANT_ID }}
+      - name: push-tarball
+        run: |
+          # tag the container image we created and push to DockerHub
+          tag=$(echo $GITHUB_REF | cut -d/ -f3-)
+          docker tag katadocker/kata-deploy-ci:${{steps.build-and-push-kata-deploy-ci.outputs.PKG_SHA}} katadocker/kata-deploy:${tag}
+          docker push katadocker/kata-deploy:${tag}
+
+  upload-static-tarball:
+    needs: kata-deploy
+    runs-on: ubuntu-latest
+    steps:
+      - uses: actions/checkout@v2
+      - name: download-artifacts
+        uses: actions/download-artifact@v2
+        with:
+          name: release-candidate
+      - name: install hub
+        run: |
+          HUB_VER=$(curl -s "https://api.github.com/repos/github/hub/releases/latest" | jq -r .tag_name | sed 's/^v//')
+          wget -q -O- https://github.com/github/hub/releases/download/v$HUB_VER/hub-linux-amd64-$HUB_VER.tgz | \
+          tar xz --strip-components=2 --wildcards '*/bin/hub' && sudo mv hub /usr/local/bin/hub
+      - name: push static tarball to github
+        run: |
+          tag=$(echo $GITHUB_REF | cut -d/ -f3-)
+          tarball="kata-static-$tag-x86_64.tar.xz"
+          mv kata-static.tar.xz "$GITHUB_WORKSPACE/${tarball}"
+          pushd $GITHUB_WORKSPACE
+          echo "uploading asset '${tarball}' for tag: ${tag}"
+          GITHUB_TOKEN=${{ secrets.GIT_UPLOAD_TOKEN }} hub release edit -m "" -a "${tarball}" "${tag}"

.github/workflows/require-pr-porting-labels.yaml

@@ -0,0 +1,51 @@
+# Copyright (c) 2020 Intel Corporation
+#
+# SPDX-License-Identifier: Apache-2.0
+#
+
+name: Ensure PR has required porting labels
+
+on:
+  pull_request_target:
+    types:
+      - opened
+      - reopened
+      - labeled
+      - unlabeled
+
+jobs:
+  check-pr-porting-labels:
+    runs-on: ubuntu-latest
+    steps:
+      - name: Install hub
+        run: |
+          HUB_ARCH="amd64"
+          HUB_VER=$(curl -sL "https://api.github.com/repos/github/hub/releases/latest" |\
+            jq -r .tag_name | sed 's/^v//')
+          curl -sL \
+            "https://github.com/github/hub/releases/download/v${HUB_VER}/hub-linux-${HUB_ARCH}-${HUB_VER}.tgz" |\
+          tar xz --strip-components=2 --wildcards '*/bin/hub' && \
+          sudo install hub /usr/local/bin
+
+      - name: Checkout code to allow hub to communicate with the project
+        uses: actions/checkout@v2
+        with:
+          token: ${{ secrets.KATA_GITHUB_ACTIONS_TOKEN }}
+
+      - name: Install porting checker script
+        run: |
+          # Clone into a temporary directory to avoid overwriting
+          # any existing github directory.
+          pushd $(mktemp -d) &>/dev/null
+          git clone --single-branch --depth 1 "https://github.com/kata-containers/.github" && cd .github/scripts
+          sudo install pr-porting-checks.sh /usr/local/bin
+          popd &>/dev/null
+
+      - name: Stop PR being merged unless it has a correct set of porting labels
+        env:
+          GITHUB_TOKEN: ${{ secrets.KATA_GITHUB_ACTIONS_TOKEN }}
+        run: |
+          pr=${{ github.event.number }}
+          repo=${{ github.repository }}
+
+          pr-porting-checks.sh "$pr" "$repo"

.gitignore

@@ -1,2 +1,7 @@
 **/*.bk
+**/*.orig
+**/*.rej
 **/target
+**/.vscode
+src/agent/src/version.rs
+src/agent/kata-agent.service

.travis.yml

@@ -5,27 +5,43 @@
 
 dist: bionic
 os: linux
-language: go
-go: 1.14.4
-env: target_branch=$TRAVIS_BRANCH
+
+# set cache directories manually, because
+# we are using a non-standard directory struct
+# cargo root is in srs/agent
+#
+# If needed, caches can be cleared
+# by ways documented in
+# https://docs.travis-ci.com/user/caching#clearing-caches
+language: rust
+rust:
+  - 1.44.1
+cache:
+  cargo: true
+  directories:
+    - src/agent/target
 
 before_install:
   - git remote set-branches --add origin "${TRAVIS_BRANCH}"
   - git fetch
+  - export RUST_BACKTRACE=1
+  - export target_branch=$TRAVIS_BRANCH
   - "ci/setup.sh"
 
 # we use install to run check agent
 # so that it is easy to skip for non-amd64 platform
 install:
-  - "ci/install_rust.sh"
   - export PATH=$PATH:"$HOME/.cargo/bin"
   - export RUST_AGENT=yes
+  - rustup target add x86_64-unknown-linux-musl
+  - sudo ln -sf /usr/bin/g++ /bin/musl-g++
+  - rustup component add rustfmt
   - make -C ${TRAVIS_BUILD_DIR}/src/agent
   - make -C ${TRAVIS_BUILD_DIR}/src/agent check
+  - sudo -E PATH=$PATH make -C ${TRAVIS_BUILD_DIR}/src/agent check
 
 before_script:
   - "ci/install_go.sh"
-  - "ci/install_vc.sh"
   - make -C ${TRAVIS_BUILD_DIR}/src/runtime
   - make -C ${TRAVIS_BUILD_DIR}/src/runtime test
   - sudo -E PATH=$PATH GOPATH=$GOPATH make -C ${TRAVIS_BUILD_DIR}/src/runtime test
@@ -40,6 +56,7 @@ jobs:
   - name: ppc64le test
     os: linux-ppc64le
     install: skip
+    script: skip
   allow_failures:
     - name: ppc64le test
   fast_finish: true

CODEOWNERS

@@ -1,4 +1,4 @@
-# Copyright 2019 Intel Corporation.
+# Copyright (c) 2019 Intel Corporation
 #
 # SPDX-License-Identifier: Apache-2.0
 #
@@ -10,4 +10,3 @@
 # used. See https://help.github.com/articles/about-code-owners/
 
 *.md    @kata-containers/documentation
-

Makefile

@@ -3,21 +3,30 @@
 # SPDX-License-Identifier: Apache-2.0
 #
 
-default: runtime agent
+# List of available components
+COMPONENTS =
 
-runtime:
-	make -C src/runtime
+COMPONENTS += agent
+COMPONENTS += runtime
+COMPONENTS += trace-forwarder
 
-agent:
-	make -C src/agent
+# List of available tools
+TOOLS =
 
-test-runtime:
-	make -C src/runtime test
+TOOLS += agent-ctl
 
-test-agent:
-	make -C src/agent check
+STANDARD_TARGETS = build check clean install test
 
-test: test-runtime test-agent
+include utils.mk
+
+all: build
+
+# Create the rules
+$(eval $(call create_all_rules,$(COMPONENTS),$(TOOLS),$(STANDARD_TARGETS)))
+
+# Non-standard rules
 
 generate-protocols:
 	make -C src/agent generate-protocols
+
+.PHONY: all default

README.md

@@ -8,9 +8,8 @@
         * [Kata Containers-developed components](#kata-containers-developed-components)
             * [Agent](#agent)
             * [KSM throttler](#ksm-throttler)
-            * [Proxy](#proxy)
             * [Runtime](#runtime)
-            * [Shim](#shim)
+            * [Trace forwarder](#trace-forwarder)
         * [Additional](#additional)
             * [Hypervisor](#hypervisor)
             * [Kernel](#kernel)
@@ -75,26 +74,12 @@ The [`kata-ksm-throttler`](https://github.com/kata-containers/ksm-throttler)
 is an optional utility that monitors containers and deduplicates memory to
 maximize container density on a host.
 
-##### Proxy
-
-The [`kata-proxy`](https://github.com/kata-containers/proxy) is a process that
-runs on the host and co-ordinates access to the agent running inside the
-virtual machine.
-
 ##### Runtime
 
 The [`kata-runtime`](src/runtime/README.md) is usually
 invoked by a container manager and provides high-level verbs to manage
 containers.
 
-##### Shim
-
-The [`kata-shim`](https://github.com/kata-containers/shim) is a process that
-runs on the host. It acts as though it is the workload (which actually runs
-inside the virtual machine). This shim is required to be compliant with the
-expectations of the [OCI runtime
-specification](https://github.com/opencontainers/runtime-spec).
-
 ##### Trace forwarder
 
 The [`kata-trace-forwarder`](src/trace-forwarder) is a component only used

VERSION

@@ -1 +1 @@
-2.0.0-alpha2
+2.0.0-rc1

ci/go-no-os-exit.sh

@@ -0,0 +1,30 @@
+#!/bin/bash
+# Copyright (c) 2018 Intel Corporation
+#
+# SPDX-License-Identifier: Apache-2.0
+#
+# Check there are no os.Exit() calls creeping into the code
+# We don't use that exit path in the Kata codebase.
+
+# Allow the path to check to be over-ridden.
+# Default to the current directory.
+go_packages=${1:-.}
+
+echo "Checking for no os.Exit() calls for package [${go_packages}]"
+
+candidates=`go list -f '{{.Dir}}/*.go' $go_packages`
+for f in $candidates; do
+	filename=`basename $f`
+	# skip all go test files
+	[[ $filename == *_test.go ]] && continue
+	# skip exit.go where, the only file we should call os.Exit() from.
+	[[ $filename == "exit.go" ]] && continue
+	files="$f $files"
+done
+
+[ -z "$files" ] && echo "No files to check, skipping" && exit 0
+
+if egrep -n '\<os\.Exit\>' $files; then
+	echo "Direct calls to os.Exit() are forbidden, please use exit() so atexit() works"
+	exit 1
+fi

ci/install_musl.sh

@@ -0,0 +1,23 @@
+#!/bin/bash
+# Copyright (c) 2020 Ant Group
+#
+# SPDX-License-Identifier: Apache-2.0
+#
+
+set -e
+
+install_aarch64_musl() {
+	local arch=$(uname -m)
+	if [ "${arch}" == "aarch64" ]; then
+		local musl_tar="${arch}-linux-musl-native.tgz"
+		local musl_dir="${arch}-linux-musl-native"
+		pushd /tmp
+		curl -sLO https://musl.cc/${musl_tar}
+		tar -zxf ${musl_tar}
+		mkdir -p /usr/local/musl/
+		cp -r ${musl_dir}/* /usr/local/musl/
+		popd
+	fi
+}
+
+install_aarch64_musl

ci/install_yq.sh

@@ -56,12 +56,13 @@ function install_yq() {
 		die "Please install curl"
 	fi
 
-	local yq_version=2.3.0
+	local yq_version=3.1.0
 
-	local yq_url="https://${yq_pkg}/releases/download/${yq_version}/yq_${goos}_${goarch}"
-	curl -o "${yq_path}" -LSsf ${yq_url}
+	## NOTE: ${var,,} => gives lowercase value of var
+	local yq_url="https://${yq_pkg}/releases/download/${yq_version}/yq_${goos,,}_${goarch}"
+	curl -o "${yq_path}" -LSsf "${yq_url}"
 	[ $? -ne 0 ] && die "Download ${yq_url} failed"
-	chmod +x ${yq_path}
+	chmod +x "${yq_path}"
 
 	if ! command -v "${yq_path}" >/dev/null; then
 		die "Cannot not get ${yq_path} executable"
@@ -69,4 +70,3 @@ function install_yq() {
 }
 
 install_yq
-

ci/lib.sh

@@ -5,22 +5,26 @@
 
 export tests_repo="${tests_repo:-github.com/kata-containers/tests}"
 export tests_repo_dir="$GOPATH/src/$tests_repo"
+export branch="${branch:-2.0-dev}"
 
 clone_tests_repo()
 {
-	# KATA_CI_NO_NETWORK is (has to be) ignored if there is
-	# no existing clone.
-	if [ -d "$tests_repo_dir" -a -n "$KATA_CI_NO_NETWORK" ]
+	if [ -d "$tests_repo_dir" -a -n "$CI" ]
 	then
 		return
 	fi
 
 	go get -d -u "$tests_repo" || true
+
+	pushd "${tests_repo_dir}" && git checkout "${branch}" && popd
 }
 
 run_static_checks()
 {
 	clone_tests_repo
+	# Make sure we have the targeting branch
+	git remote set-branches --add origin "${branch}"
+	git fetch -a
 	bash "$tests_repo_dir/.ci/static-checks.sh" "github.com/kata-containers/kata-containers"
 }
 

docs/CODE_OF_CONDUCT.md

@@ -1,3 +0,0 @@
-## Kata Containers Documentation Code of Conduct
-
-Kata Containers follows the [OpenStack Foundation Code of Conduct](https://www.openstack.org/legal/community-code-of-conduct/).

docs/CONTRIBUTING.md

@@ -1,5 +0,0 @@
-# Contributing
-
-## This repo is part of [Kata Containers](https://katacontainers.io)
-
-For details on how to contribute to the Kata Containers project, please see the main [contributing document](https://github.com/kata-containers/community/blob/master/CONTRIBUTING.md).

docs/Developer-Guide.md

@@ -13,8 +13,6 @@
         * [journald rate limiting](#journald-rate-limiting)
             * [`systemd-journald` suppressing messages](#systemd-journald-suppressing-messages)
             * [Disabling `systemd-journald` rate limiting](#disabling-systemd-journald-rate-limiting)
-* [Build and install Kata proxy](#build-and-install-kata-proxy)
-* [Build and install Kata shim](#build-and-install-kata-shim)
 * [Create and install rootfs and initrd image](#create-and-install-rootfs-and-initrd-image)
     * [Build a custom Kata agent - OPTIONAL](#build-a-custom-kata-agent---optional)
     * [Get the osbuilder](#get-the-osbuilder)
@@ -31,26 +29,24 @@
 * [Install a hypervisor](#install-a-hypervisor)
     * [Build a custom QEMU](#build-a-custom-qemu)
         * [Build a custom QEMU for aarch64/arm64 - REQUIRED](#build-a-custom-qemu-for-aarch64arm64---required)
-* [Run Kata Containers with Docker](#run-kata-containers-with-docker)
-    * [Update the Docker systemd unit file](#update-the-docker-systemd-unit-file)
-    * [Create a container using Kata](#create-a-container-using-kata)
+* [Run Kata Containers with Containerd](#run-kata-containers-with-containerd)
 * [Run Kata Containers with Kubernetes](#run-kata-containers-with-kubernetes)
 * [Troubleshoot Kata Containers](#troubleshoot-kata-containers)
 * [Appendices](#appendices)
     * [Checking Docker default runtime](#checking-docker-default-runtime)
     * [Set up a debug console](#set-up-a-debug-console)
-        * [Create a custom image containing a shell](#create-a-custom-image-containing-a-shell)
-        * [Create a debug systemd service](#create-a-debug-systemd-service)
-        * [Build the debug image](#build-the-debug-image)
-        * [Configure runtime for custom debug image](#configure-runtime-for-custom-debug-image)
-        * [Ensure debug options are valid](#ensure-debug-options-are-valid)
-        * [Create a container](#create-a-container)
-        * [Connect to the virtual machine using the debug console](#connect-to-the-virtual-machine-using-the-debug-console)
-        * [Obtain details of the image](#obtain-details-of-the-image)
+      * [Simple debug console setup](#simple-debug-console-setup)
+          * [Enable agent debug console](#enable-agent-debug-console)
+          * [Start `kata-monitor`](#start-kata-monitor)
+          * [Connect to debug console](#connect-to-debug-console)
+      * [Traditional debug console setup](#traditional-debug-console-setup)
+          * [Create a custom image containing a shell](#create-a-custom-image-containing-a-shell)
+          * [Create a debug systemd service](#create-a-debug-systemd-service)
+          * [Build the debug image](#build-the-debug-image)
+          * [Configure runtime for custom debug image](#configure-runtime-for-custom-debug-image)
+          * [Create a container](#create-a-container)
+          * [Connect to the virtual machine using the debug console](#connect-to-the-virtual-machine-using-the-debug-console)
     * [Capturing kernel boot logs](#capturing-kernel-boot-logs)
-    * [Running standalone](#running-standalone)
-        * [Create an OCI bundle](#create-an-oci-bundle)
-        * [Launch the runtime to create a container](#launch-the-runtime-to-create-a-container)
 
 # Warning
 
@@ -67,7 +63,7 @@ The recommended way to create a development environment is to first
 to create a working system.
 
 The installation guide instructions will install all required Kata Containers
-components, plus Docker*, the hypervisor, and the Kata Containers image and
+components, plus *Docker*, the hypervisor, and the Kata Containers image and
 guest kernel.
 
 # Requirements to build individual components
@@ -77,7 +73,7 @@ You need to install the following to build Kata Containers components:
 - [golang](https://golang.org/dl)
 
   To view the versions of go known to work, see the `golang` entry in the
-  [versions database](https://github.com/kata-containers/runtime/blob/master/versions.yaml).
+  [versions database](../versions.yaml).
 
 - `make`.
 - `gcc` (required for building the shim and runtime).
@@ -85,14 +81,14 @@ You need to install the following to build Kata Containers components:
 # Build and install the Kata Containers runtime
 
 ```
-$ go get -d -u github.com/kata-containers/runtime
-$ cd $GOPATH/src/github.com/kata-containers/runtime
+$ go get -d -u github.com/kata-containers/kata-containers
+$ cd $GOPATH/src/github.com/kata-containers/kata-containers/src/runtime
 $ make && sudo -E PATH=$PATH make install
 ```
 
 The build will create the following:
 
-- runtime binary: `/usr/local/bin/kata-runtime`
+- runtime binary: `/usr/local/bin/kata-runtime` and `/usr/local/bin/containerd-shim-kata-v2`
 - configuration file: `/usr/share/defaults/kata-containers/configuration.toml`
 
 # Check hardware requirements
@@ -243,20 +239,6 @@ Restart `systemd-journald` for the changes to take effect:
 $ sudo systemctl restart systemd-journald
 ```
 
-# Build and install Kata proxy
-
-```
-$ go get -d -u github.com/kata-containers/proxy
-$ cd $GOPATH/src/github.com/kata-containers/proxy && make && sudo make install
-```
-
-# Build and install Kata shim
-
-```
-$ go get -d -u github.com/kata-containers/shim
-$ cd $GOPATH/src/github.com/kata-containers/shim && make && sudo make install
-```
-
 # Create and install rootfs and initrd image
 
 ## Build a custom Kata agent - OPTIONAL
@@ -266,14 +248,15 @@ $ cd $GOPATH/src/github.com/kata-containers/shim && make && sudo make install
 > - You should only do this step if you are testing with the latest version of the agent.
 
 ```
-$ go get -d -u github.com/kata-containers/agent
-$ cd $GOPATH/src/github.com/kata-containers/agent && make
+$ go get -d -u github.com/kata-containers/kata-containers
+$ cd $GOPATH/src/github.com/kata-containers/kata-containers/src/agent && make
 ```
 
 ## Get the osbuilder
 
 ```
-$ go get -d -u github.com/kata-containers/osbuilder
+$ go get -d -u github.com/kata-containers/kata-containers
+$ cd $GOPATH/src/github.com/kata-containers/kata-containers/tools/osbuilder
 ```
 
 ## Create a rootfs image
@@ -284,16 +267,16 @@ able to run the `rootfs.sh` script with `USE_DOCKER=true` as expected in
 the following example.
 
 ```
-$ export ROOTFS_DIR=${GOPATH}/src/github.com/kata-containers/osbuilder/rootfs-builder/rootfs
+$ export ROOTFS_DIR=${GOPATH}/src/github.com/kata-containers/kata-containers/tools/osbuilder/rootfs-builder/rootfs
 $ sudo rm -rf ${ROOTFS_DIR}
-$ cd $GOPATH/src/github.com/kata-containers/osbuilder/rootfs-builder
+$ cd $GOPATH/src/github.com/kata-containers/kata-containers/tools/osbuilder/rootfs-builder
 $ script -fec 'sudo -E GOPATH=$GOPATH USE_DOCKER=true SECCOMP=no ./rootfs.sh ${distro}'
 ```
 You MUST choose one of `alpine`, `centos`, `clearlinux`, `debian`, `euleros`, `fedora`, `suse`, and `ubuntu` for `${distro}`. By default `seccomp` packages are not included in the rootfs image. Set `SECCOMP` to `yes` to include them.
 
 > **Note:**
 >
-> - Check the [compatibility matrix](https://github.com/kata-containers/osbuilder#platform-distro-compatibility-matrix) before creating rootfs.
+> - Check the [compatibility matrix](../tools/osbuilder/README.md#platform-distro-compatibility-matrix) before creating rootfs.
 > - You must ensure that the *default Docker runtime* is `runc` to make use of
 >   the `USE_DOCKER` variable. If that is not the case, remove the variable
 >   from the previous command. See [Checking Docker default runtime](#checking-docker-default-runtime).
@@ -313,7 +296,7 @@ $ sudo install -o root -g root -m 0440 ../../agent/kata-containers.target ${ROOT
 ### Build a rootfs image
 
 ```
-$ cd $GOPATH/src/github.com/kata-containers/osbuilder/image-builder
+$ cd $GOPATH/src/github.com/kata-containers/kata-containers/tools/osbuilder/image-builder
 $ script -fec 'sudo -E USE_DOCKER=true ./image_builder.sh ${ROOTFS_DIR}'
 ```
 
@@ -340,9 +323,9 @@ $ (cd /usr/share/kata-containers && sudo ln -sf "$image" kata-containers.img)
 ## Create an initrd image - OPTIONAL
 ### Create a local rootfs for initrd image
 ```
-$ export ROOTFS_DIR="${GOPATH}/src/github.com/kata-containers/osbuilder/rootfs-builder/rootfs"
+$ export ROOTFS_DIR="${GOPATH}/src/github.com/kata-containers/kata-containers/tools/osbuilder/rootfs-builder/rootfs"
 $ sudo rm -rf ${ROOTFS_DIR}
-$ cd $GOPATH/src/github.com/kata-containers/osbuilder/rootfs-builder
+$ cd $GOPATH/src/github.com/kata-containers/kata-containers/tools/osbuilder/rootfs-builder
 $ script -fec 'sudo -E GOPATH=$GOPATH AGENT_INIT=yes USE_DOCKER=true SECCOMP=no ./rootfs.sh ${distro}'
 ```
 `AGENT_INIT` controls if the guest image uses the Kata agent as the guest `init` process. When you create an initrd image,
@@ -352,7 +335,7 @@ You MUST choose one of `alpine`, `centos`, `clearlinux`, `euleros`, and `fedora`
 
 > **Note:**
 >
-> - Check the [compatibility matrix](https://github.com/kata-containers/osbuilder#platform-distro-compatibility-matrix) before creating rootfs.
+> - Check the [compatibility matrix](../tools/osbuilder/README.md#platform-distro-compatibility-matrix) before creating rootfs.
 
 Optionally, add your custom agent binary to the rootfs with the following:
 ```
@@ -362,7 +345,7 @@ $ sudo install -o root -g root -m 0550 -T ../../agent/kata-agent ${ROOTFS_DIR}/s
 ### Build an initrd image
 
 ```
-$ cd $GOPATH/src/github.com/kata-containers/osbuilder/initrd-builder
+$ cd $GOPATH/src/github.com/kata-containers/kata-containers/tools/osbuilder/initrd-builder
 $ script -fec 'sudo -E AGENT_INIT=yes USE_DOCKER=true ./initrd_builder.sh ${ROOTFS_DIR}'
 ```
 
@@ -378,11 +361,11 @@ $ (cd /usr/share/kata-containers && sudo ln -sf "$image" kata-containers-initrd.
 
 # Install guest kernel images
 
-You can build and install the guest kernel image as shown [here](https://github.com/kata-containers/packaging/tree/master/kernel#build-kata-containers-kernel).
+You can build and install the guest kernel image as shown [here](../tools/packaging/kernel/README.md#build-kata-containers-kernel).
 
 # Install a hypervisor
 
-When setting up Kata using a [packaged installation method](https://github.com/kata-containers/documentation/tree/master/install#installing-on-a-linux-system), the `qemu-lite` hypervisor is installed automatically. For other installation methods, you will need to manually install a suitable hypervisor.
+When setting up Kata using a [packaged installation method](install/README.md#installing-on-a-linux-system), the `qemu-lite` hypervisor is installed automatically. For other installation methods, you will need to manually install a suitable hypervisor.
 
 ## Build a custom QEMU
 
@@ -390,7 +373,7 @@ Your QEMU directory need to be prepared with source code. Alternatively, you can
 
 ```
 $ go get -d github.com/kata-containers/qemu
-$ qemu_branch=$(grep qemu-lite- ${GOPATH}/src/github.com/kata-containers/runtime/versions.yaml | cut -d '"' -f2)
+$ qemu_branch=$(grep qemu-lite- ${GOPATH}/src/github.com/kata-containers/kata-containers/versions.yaml | cut -d '"' -f2)
 $ cd ${GOPATH}/src/github.com/kata-containers/qemu
 $ git checkout -b $qemu_branch remotes/origin/$qemu_branch
 $ your_qemu_directory=${GOPATH}/src/github.com/kata-containers/qemu
@@ -399,9 +382,9 @@ $ your_qemu_directory=${GOPATH}/src/github.com/kata-containers/qemu
 To build a version of QEMU using the same options as the default `qemu-lite` version , you could use the `configure-hypervisor.sh` script:
 
 ```
-$ go get -d github.com/kata-containers/packaging
+$ go get -d github.com/kata-containers/kata-containers/tools/packaging
 $ cd $your_qemu_directory
-$ ${GOPATH}/src/github.com/kata-containers/packaging/scripts/configure-hypervisor.sh qemu > kata.cfg
+$ ${GOPATH}/src/github.com/kata-containers/kata-containers/tools/packaging/scripts/configure-hypervisor.sh qemu > kata.cfg
 $ eval ./configure "$(cat kata.cfg)"
 $ make -j $(nproc)
 $ sudo -E make install
@@ -420,58 +403,30 @@ $ go get -d github.com/kata-containers/tests
 $ script -fec 'sudo -E ${GOPATH}/src/github.com/kata-containers/tests/.ci/install_qemu.sh'
 ```
 
-# Run Kata Containers with Docker
-
-## Update the Docker systemd unit file
-
-```
-$ dockerUnit=$(systemctl show -p FragmentPath docker.service | cut -d "=" -f 2)
-$ unitFile=${dockerUnit:-/etc/systemd/system/docker.service.d/kata-containers.conf}
-$ test -e "$unitFile" || { sudo mkdir -p "$(dirname $unitFile)"; echo -e "[Service]\nType=simple\nExecStart=\nExecStart=/usr/bin/dockerd -D --default-runtime runc" | sudo tee "$unitFile"; }
-$ grep -q "kata-runtime=" $unitFile || sudo sed -i 's!^\(ExecStart=[^$].*$\)!\1 --add-runtime kata-runtime=/usr/local/bin/kata-runtime!g' "$unitFile"
-$ sudo systemctl daemon-reload
-$ sudo systemctl restart docker
-```
-
-## Create a container using Kata
-
-```
-$ sudo docker run -ti --runtime kata-runtime busybox sh
-```
+# Run Kata Containers with Containerd
+Refer to the [How to use Kata Containers and Containerd](how-to/containerd-kata.md) how-to guide.
 
 # Run Kata Containers with Kubernetes
-Refer to to the [Run Kata Containers with Kubernetes](how-to/run-kata-with-k8s.md) how-to guide.
+Refer to the [Run Kata Containers with Kubernetes](how-to/run-kata-with-k8s.md) how-to guide.
 
 # Troubleshoot Kata Containers
 
 If you are unable to create a Kata Container first ensure you have
 [enabled full debug](#enable-full-debug)
 before attempting to create a container. Then run the
-[`kata-collect-data.sh`](https://github.com/kata-containers/runtime/blob/master/data/kata-collect-data.sh.in)
+[`kata-collect-data.sh`](../src/runtime/data/kata-collect-data.sh.in)
 script and paste its output directly into a
 [GitHub issue](https://github.com/kata-containers/kata-containers/issues/new).
 
 > **Note:**
 >
 > The `kata-collect-data.sh` script is built from the
-> [runtime](https://github.com/kata-containers/runtime) repository.
+> [runtime](../src/runtime) repository.
 
 To perform analysis on Kata logs, use the
 [`kata-log-parser`](https://github.com/kata-containers/tests/tree/master/cmd/log-parser)
 tool, which can convert the logs into formats (e.g. JSON, TOML, XML, and YAML).
 
-To obtain a full backtrace for the agent, proxy, runtime, or shim send the
-`SIGUSR1` signal to the process ID of the component. The component will send a
-backtrace to the system log on the host system and continue to run without
-interruption.
-
-For example, to obtain a backtrace for `kata-proxy`:
-
-```
-$ sudo kill -USR1 $kata_proxy_pid
-$ sudo journalctl -t kata-proxy
-```
-
 See [Set up a debug console](#set-up-a-debug-console).
 
 # Appendices
@@ -481,9 +436,56 @@ See [Set up a debug console](#set-up-a-debug-console).
 ```
 $ sudo docker info 2>/dev/null | grep -i "default runtime" | cut -d: -f2- | grep -q runc  && echo "SUCCESS" || echo "ERROR: Incorrect default Docker runtime"
 ```
-
 ## Set up a debug console
 
+Kata containers provides two ways to connect to the guest. One is using traditional login service, which needs additional works. In contrast the simple debug console is easy to setup.
+
+### Simple debug console setup
+
+Kata Containers 2.0 supports a shell simulated *console* for quick debug purpose. This approach uses VSOCK to
+connect to the shell running inside the guest which the agent starts. This method only requires the guest image to
+contain either `/bin/sh` or `/bin/bash`.
+
+#### Enable agent debug console
+
+Enable debug_console_enabled in the `configuration.toml` configuration file:
+
+```
+[agent.kata]
+debug_console_enabled = true
+```
+
+This will pass `agent.debug_console agent.debug_console_vport=1026` to agent as kernel parameters, and sandboxes created using this parameters will start a shell in guest if new connection is accept from VSOCK.
+
+#### Start `kata-monitor`
+
+The `kata-runtime exec` command needs `kata-monitor` to get the sandbox's `vsock` address to connect to, first start `kata-monitor`.
+
+```
+$ sudo kata-monitor
+```
+
+`kata-monitor` will serve at `localhost:8090` by default.
+
+
+#### Connect to debug console
+
+Command `kata-runtime exec` is used to connect to the debug console.
+
+```
+$ kata-runtime exec 1a9ab65be63b8b03dfd0c75036d27f0ed09eab38abb45337fea83acd3cd7bacd
+bash-4.2# id
+uid=0(root) gid=0(root) groups=0(root)
+bash-4.2# pwd
+/
+bash-4.2# exit
+exit
+```
+
+If you want to access guest OS through a traditional way, see [Traditional debug console setup)](#traditional-debug-console-setup).
+
+### Traditional debug console setup
+
 By default you cannot login to a virtual machine, since this can be sensitive
 from a security perspective. Also, allowing logins would require additional
 packages in the rootfs, which would increase the size of the image used to
@@ -494,12 +496,12 @@ the following steps (using rootfs or initrd image).
 
 > **Note:** The following debug console instructions assume a systemd-based guest
 > O/S image. This means you must create a rootfs for a distro that supports systemd.
-> Currently, all distros supported by [osbuilder](https://github.com/kata-containers/osbuilder) support systemd
+> Currently, all distros supported by [osbuilder](../tools/osbuilder) support systemd
 > except for Alpine Linux.
 >
 > Look for `INIT_PROCESS=systemd` in the `config.sh` osbuilder rootfs config file
 > to verify an osbuilder distro supports systemd for the distro you want to build rootfs for.
-> For an example, see the [Clear Linux config.sh file](https://github.com/kata-containers/osbuilder/blob/master/rootfs-builder/clearlinux/config.sh).
+> For an example, see the [Clear Linux config.sh file](../tools/osbuilder/rootfs-builder/clearlinux/config.sh).
 >
 > For a non-systemd-based distro, create an equivalent system
 > service using that distro’s init system syntax. Alternatively, you can build a distro
@@ -507,9 +509,9 @@ the following steps (using rootfs or initrd image).
 > additional packages in the rootfs and add “agent.debug_console” to kernel parameters in the runtime
 > config file. This tells the Kata agent to launch the console directly.
 >
-> Once these steps are taken you can connect to the virtual machine using the [debug console](https://github.com/kata-containers/documentation/blob/master/Developer-Guide.md#connect-to-the-virtual-machine-using-the-debug-console).
+> Once these steps are taken you can connect to the virtual machine using the [debug console](Developer-Guide.md#connect-to-the-virtual-machine-using-the-debug-console).
 
-### Create a custom image containing a shell
+#### Create a custom image containing a shell
 
 To login to a virtual machine, you must
 [create a custom rootfs](#create-a-rootfs-image) or [custom initrd](#create-an-initrd-image---optional)
@@ -519,12 +521,12 @@ an additional `coreutils` package.
 For example using CentOS:
 
 ```
-$ cd $GOPATH/src/github.com/kata-containers/osbuilder/rootfs-builder
-$ export ROOTFS_DIR=${GOPATH}/src/github.com/kata-containers/osbuilder/rootfs-builder/rootfs
+$ cd $GOPATH/src/github.com/kata-containers/kata-containers/tools/osbuilder/rootfs-builder
+$ export ROOTFS_DIR=${GOPATH}/src/github.com/kata-containers/kata-containers/tools/osbuilder/rootfs-builder/rootfs
 $ script -fec 'sudo -E GOPATH=$GOPATH USE_DOCKER=true EXTRA_PKGS="bash coreutils" ./rootfs.sh centos'
 ```
 
-### Create a debug systemd service
+#### Create a debug systemd service
 
 Create the service file that starts the shell in the rootfs directory:
 
@@ -553,12 +555,12 @@ Add a dependency to start the debug console:
 $ sudo sed -i '$a Requires=kata-debug.service' ${ROOTFS_DIR}/lib/systemd/system/kata-containers.target
 ```
 
-### Build the debug image
+#### Build the debug image
 
 Follow the instructions in the [Build a rootfs image](#build-a-rootfs-image)
 section when using rootfs, or when using initrd, complete the steps in the [Build an initrd image](#build-an-initrd-image) section.
 
-### Configure runtime for custom debug image
+#### Configure runtime for custom debug image
 
 Install the image:
 
@@ -571,7 +573,7 @@ $ sudo install -o root -g root -m 0640 kata-containers.img "/usr/share/kata-cont
 ```
 
 Next, modify the `image=` values in the `[hypervisor.qemu]` section of the
-[configuration file](https://github.com/kata-containers/runtime#configuration)
+[configuration file](../src/runtime/README.md#configuration)
 to specify the full path to the image name specified in the previous code
 section. Alternatively, recreate the symbolic link so it points to
 the new debug image:
@@ -583,31 +585,18 @@ $ (cd /usr/share/kata-containers && sudo ln -sf "$name" kata-containers.img)
 **Note**: You should take care to undo this change after you finish debugging
 to avoid all subsequently created containers from using the debug image.
 
-### Ensure debug options are valid
+#### Create a container
 
-For the debug console to work, you **must** ensure that proxy debug is
-**disabled** in the configuration file. If proxy debug is enabled, you will
-not see any output when you connect to the virtual machine:
+Create a container as normal. For example using `crictl`:
 
 ```
-$ sudo mkdir -p /etc/kata-containers/
-$ sudo install -o root -g root -m 0640 /usr/share/defaults/kata-containers/configuration.toml /etc/kata-containers
-$ sudo awk '{if (/^\[proxy\.kata\]/) {got=1}; if (got == 1 && /^.*enable_debug/) {print "#enable_debug = true"; got=0; next; } else {print}}' /etc/kata-containers/configuration.toml > /tmp/configuration.toml
-$ sudo install -o root -g root -m 0640 /tmp/configuration.toml /etc/kata-containers/
+$ sudo crictl run -r kata container.yaml pod.yaml
 ```
 
-### Create a container
-
-Create a container as normal. For example using Docker:
+#### Connect to the virtual machine using the debug console
 
 ```
-$ sudo docker run -ti busybox sh
-```
-
-### Connect to the virtual machine using the debug console
-
-```
-$ id=$(sudo docker ps -q --no-trunc)
+$ id=$(sudo crictl pods --no-trunc -q)
 $ console="/var/run/vc/vm/${id}/console.sock"
 $ sudo socat "stdin,raw,echo=0,escape=0x11" "unix-connect:${console}"
 ```
@@ -617,10 +606,10 @@ $ sudo socat "stdin,raw,echo=0,escape=0x11" "unix-connect:${console}"
 To disconnect from the virtual machine, type `CONTROL+q` (hold down the
 `CONTROL` key and press `q`).
 
-### Obtain details of the image
+## Obtain details of the image
 
 If the image is created using
-[osbuilder](https://github.com/kata-containers/osbuilder), the following YAML
+[osbuilder](../tools/osbuilder), the following YAML
 file exists and contains details of the image and how it was created:
 
 ```
@@ -637,54 +626,22 @@ command inside the container to view the kernel boot logs.
 If however you are unable to `exec` into the container, you can enable some debug
 options to have the kernel boot messages logged into the system journal.
 
-Which debug options you enable depends on if you are using the hypervisor `vsock` mode
-or not, as defined by the `use_vsock` setting in the `[hypervisor.qemu]` section of
-the configuration file. The following details the settings:
-
-- For `use_vsock = false`:
-    - Set `enable_debug = true` in both the `[hypervisor.qemu]` and `[proxy.kata]` sections
-- For `use_vsock = true`:
-    - Set `enable_debug = true` in both the `[hypervisor.qemu]` and `[shim.kata]` sections
+- Set `enable_debug = true` in the `[hypervisor.qemu]` and `[runtime]` sections
 
 For generic information on enabling debug in the configuration file, see the
 [Enable full debug](#enable-full-debug) section.
 
-The kernel boot messages will appear in the `kata-proxy` or `kata-shim` log appropriately,
+The kernel boot messages will appear in the `containerd` or `CRI-O` log appropriately,
 such as:
 
 ```bash
-$ sudo journalctl -t kata-proxy
+$ sudo journalctl -t containerd
 -- Logs begin at Thu 2020-02-13 16:20:40 UTC, end at Thu 2020-02-13 16:30:23 UTC. --
 ...
-Feb 13 16:20:56 minikube kata-proxy[17371]: time="2020-02-13T16:20:56.608714324Z" level=info msg="[    1.418768] brd: module loaded\n" name=kata-proxy pid=17371 sandbox=a13ffb2b9b5a66f7787bdae9a427fa954a4d21ec4031d0179eee2573986a8a6e source=agent
-Feb 13 16:20:56 minikube kata-proxy[17371]: time="2020-02-13T16:20:56.628493231Z" level=info msg="[    1.438612] loop: module loaded\n" name=kata-proxy pid=17371 sandbox=a13ffb2b9b5a66f7787bdae9a427fa954a4d21ec4031d0179eee2573986a8a6e source=agent
-Feb 13 16:20:56 minikube kata-proxy[17371]: time="2020-02-13T16:20:56.67707956Z" level=info msg="[    1.487165]  pmem0: p1\n" name=kata-proxy pid=17371 sandbox=a13ffb2b9b5a66f7787bdae9a427fa954a4d21ec4031d0179eee2573986a8a6e source=agent
+time="2020-09-15T14:56:23.095113803+08:00" level=debug msg="reading guest console" console-protocol=unix console-url=/run/vc/vm/ab9f633385d4987828d342e47554fc6442445b32039023eeddaa971c1bb56791/console.sock pid=107642 sandbox=ab9f633385d4987828d342e47554fc6442445b32039023eeddaa971c1bb56791 source=virtcontainers subsystem=sandbox vmconsole="[    0.395399] brd: module loaded"
+time="2020-09-15T14:56:23.102633107+08:00" level=debug msg="reading guest console" console-protocol=unix console-url=/run/vc/vm/ab9f633385d4987828d342e47554fc6442445b32039023eeddaa971c1bb56791/console.sock pid=107642 sandbox=ab9f633385d4987828d342e47554fc6442445b32039023eeddaa971c1bb56791 source=virtcontainers subsystem=sandbox vmconsole="[    0.402845] random: fast init done"
+time="2020-09-15T14:56:23.103125469+08:00" level=debug msg="reading guest console" console-protocol=unix console-url=/run/vc/vm/ab9f633385d4987828d342e47554fc6442445b32039023eeddaa971c1bb56791/console.sock pid=107642 sandbox=ab9f633385d4987828d342e47554fc6442445b32039023eeddaa971c1bb56791 source=virtcontainers subsystem=sandbox vmconsole="[    0.403544] random: crng init done"
+time="2020-09-15T14:56:23.105268162+08:00" level=debug msg="reading guest console" console-protocol=unix console-url=/run/vc/vm/ab9f633385d4987828d342e47554fc6442445b32039023eeddaa971c1bb56791/console.sock pid=107642 sandbox=ab9f633385d4987828d342e47554fc6442445b32039023eeddaa971c1bb56791 source=virtcontainers subsystem=sandbox vmconsole="[    0.405599] loop: module loaded"
+time="2020-09-15T14:56:23.121121598+08:00" level=debug msg="reading guest console" console-protocol=unix console-url=/run/vc/vm/ab9f633385d4987828d342e47554fc6442445b32039023eeddaa971c1bb56791/console.sock pid=107642 sandbox=ab9f633385d4987828d342e47554fc6442445b32039023eeddaa971c1bb56791 source=virtcontainers subsystem=sandbox vmconsole="[    0.421324] memmap_init_zone_device initialised 32768 pages in 12ms"
 ...
 ```
-
-## Running standalone
-
-It is possible to start the runtime without a container manager. This is
-mostly useful for testing and debugging purposes.
-
-### Create an OCI bundle
-
-To build an
-[OCI bundle](https://github.com/opencontainers/runtime-spec/blob/master/bundle.md),
-required by the runtime:
-
-```
-$ bundle="/tmp/bundle"
-$ rootfs="$bundle/rootfs"
-$ mkdir -p "$rootfs" && (cd "$bundle" && kata-runtime spec)
-$ sudo docker export $(sudo docker create busybox) | tar -C "$rootfs" -xvf -
-```
-
-### Launch the runtime to create a container
-
-Run the runtime standalone by providing it with the path to the
-previously-created [OCI bundle](#create-an-oci-bundle):
-
-```
-$ sudo kata-runtime --log=/dev/stdout run --bundle "$bundle" foo
-```

docs/Limitations.md

@@ -39,7 +39,7 @@ Some of these limitations have potential solutions, whereas others exist
 due to fundamental architectural differences generally related to the
 use of VMs.
 
-The [Kata Container runtime](https://github.com/kata-containers/runtime)
+The [Kata Container runtime](../src/runtime)
 launches each container within its own hardware isolated VM, and each VM has
 its own kernel. Due to this higher degree of isolation, certain container
 capabilities cannot be supported or are implicitly enabled through the VM.
@@ -78,7 +78,7 @@ The following link shows the latest list of limitations:
 If you would like to work on resolving a limitation, please refer to the
 [contributors guide](https://github.com/kata-containers/community/blob/master/CONTRIBUTING.md).
 If you wish to raise an issue for a new limitation, either
-[raise an issue directly on the runtime](https://github.com/kata-containers/runtime/issues/new)
+[raise an issue directly on the runtime](https://github.com/kata-containers/kata-containers/issues/new)
 or see the
 [project table of contents](https://github.com/kata-containers/kata-containers)
 for advice on which repository to raise the issue against.
@@ -270,11 +270,6 @@ The following examples outline some of the various areas constraints can be appl
 
     This can be achieved by specifying particular hypervisor configuration options.
 
-  - Constrain the [shim](https://github.com/kata-containers/shim) process.
-
-    This process represents the container workload running inside the VM.
-
-  - Constrain the [proxy](https://github.com/kata-containers/proxy) process.
 
 Note that in some circumstances it might be necessary to apply particular constraints
 to more than one of the previous areas to achieve the desired level of isolation and resource control.

docs/README.md

@@ -54,7 +54,7 @@ Documents that help to understand and contribute to Kata Containers.
 
 * [Developer Guide](Developer-Guide.md): Setup the Kata Containers developing environments
 * [How to contribute to Kata Containers](https://github.com/kata-containers/community/blob/master/CONTRIBUTING.md)
-* [Code of Conduct](CODE_OF_CONDUCT.md)
+* [Code of Conduct](../CODE_OF_CONDUCT.md)
 
 ### Code Licensing
 

docs/Release-Process.md

@@ -10,7 +10,6 @@
     - [Merge all bump version Pull requests](#merge-all-bump-version-pull-requests)
     - [Tag all Kata repositories](#tag-all-kata-repositories)
     - [Check Git-hub Actions](#check-git-hub-actions)
-    - [Create OBS Packages](#create-obs-packages)
     - [Create release notes](#create-release-notes)
     - [Announce the release](#announce-the-release)
 <!-- TOC END -->
@@ -42,7 +41,7 @@
 
   Alternatively, you can also bump the repositories using a script in the Kata packaging repo
   ```
-  $ cd ${GOPATH}/src/github.com/kata-containers/packaging/release
+  $ cd ${GOPATH}/src/github.com/kata-containers/kata-containers/tools/packaging/release
   $ export NEW_VERSION=<the-new-kata-version>
   $ export BRANCH=<the-branch-you-want-to-bump>
   $ ./update-repository-version.sh -p "$NEW_VERSION" "$BRANCH"
@@ -59,7 +58,7 @@
   Once all the pull requests to bump versions in all Kata repositories are merged,
   tag all the repositories as shown below.  
   ```
-  $ cd ${GOPATH}/src/github.com/kata-containers/packaging/release
+  $ cd ${GOPATH}/src/github.com/kata-containers/kata-containers/tools/packaging/release
   $ git checkout  <kata-branch-to-release>
   $ git pull
   $ ./tag_repos.sh -p -b "$BRANCH" tag
@@ -69,34 +68,7 @@
 
   We make use of [GitHub actions](https://github.com/features/actions) in this [file](https://github.com/kata-containers/kata-containers/blob/master/.github/workflows/main.yaml) in the `kata-containers/kata-containers` repository to build and upload release artifacts. This action is auto triggered with the above step when a new tag is pushed to the `kata-containers/kata-conatiners` repository.
 
-  Check the [actions status page](https://github.com/kata-containers/kata-containers/actions) to verify all steps in the actions workflow have completed successfully. On success, a static tarball containing Kata release artifacts will be uploaded to the [Release page](https://github.com/kata-containers/runtime/releases).
-
-### Create OBS Packages
-
-  - We have set up an [Azure Pipelines](https://azure.microsoft.com/en-us/services/devops/pipelines/) job
-  to trigger generation of Kata packages in [OBS](https://build.opensuse.org/).
-  Go to the [Azure Pipelines job that creates OBS packages](https://dev.azure.com/kata-containers/release-process/_release?_a=releases&view=mine&definitionId=1).
-  - Click on "Create release" (blue button, at top right corner).
-    It should prompt you for variables to be passed to the release job. They should look like:
-
-    ```
-    BRANCH="the-kata-branch-that-is-release"
-    BUILD_HEAD=false
-    OBS_BRANCH="the-kata-branch-that-is-release"
-    ```
-    Note: If the release is `Alpha` , `Beta` , or `RC` (that is part of a `master` release), please use `OBS_BRANCH=master`.
-
-    The above step shall create OBS packages for Kata for various distributions that Kata supports and test them as well.
-  - Verify that the packages have built successfully by checking the [Kata OBS  project page](https://build.opensuse.org/project/subprojects/home:katacontainers).
-  - Make sure packages work correctly. This can be done manually or via the [package testing pipeline](http://jenkins.katacontainers.io/job/package-release-testing).
-    You have to make sure the packages are already published by OBS before this step.
-    It should prompt you for variables to be passed to the pipeline:
-
-    ```
-    BRANCH="<kata-branch-to-release>"
-    NEW_VERSION=<the-version-you-expect-to-be-packaged|latest>
-    ```
-    Note: `latest` will verify that a package provides the latest Kata tag in that branch.
+  Check the [actions status page](https://github.com/kata-containers/kata-containers/actions) to verify all steps in the actions workflow have completed successfully. On success, a static tarball containing Kata release artifacts will be uploaded to the [Release page](https://github.com/kata-containers/kata-containers/releases).
 
 ### Create release notes
 
@@ -105,12 +77,12 @@
   Run the script as shown below:
 
   ```
-  $ cd ${GOPATH}/src/github.com/kata-containers/packaging/release
+  $ cd ${GOPATH}/src/github.com/kata-containers/kata-containers/tools/packaging/release
   # Note: OLD_VERSION is where the script should start to get changes.
   $ ./runtime-release-notes.sh ${OLD_VERSION} ${NEW_VERSION} > notes.md
   # Edit the `notes.md` file to review and make any changes to the release notes.
   # Add the release notes in GitHub runtime.
-  $ hub -C "${GOPATH}/src/github.com/kata-containers/runtime" release edit -F notes.md "${NEW_VERSION}"
+  $ hub release edit -F notes.md "${NEW_VERSION}"
   ```
 
 ### Announce the release

docs/Upgrading.md

@@ -68,7 +68,7 @@ $ for container in $(sudo docker ps -q); do sudo docker stop $container; done
 
 The automatic migration of
 [Clear Containers configuration](https://github.com/clearcontainers/runtime#configuration) to
-[Kata Containers configuration](https://github.com/kata-containers/runtime#configuration) is
+[Kata Containers configuration](../src/runtime/README.md#configuration) is
 not supported.
 
 If you have made changes to your Clear Containers configuration, you should
@@ -111,7 +111,7 @@ $ sudo rm /etc/systemd/system/docker.service.d/clear-containers.conf
 
 ## Install Kata Containers
 
-Follow one of the [installation guides](https://github.com/kata-containers/documentation/tree/master/install).
+Follow one of the [installation guides](install).
 
 ## Create a Kata Container
 
@@ -126,12 +126,12 @@ not configured to use the same container root storage. Currently, runV defaults
 defaults to `/var/run/kata-containers`.
 
 Now, to upgrade from runV you need to fresh install Kata Containers by following one of
-the [installation guides](https://github.com/kata-containers/documentation/tree/master/install).
+the [installation guides](install).
 
 # Upgrade Kata Containers
 
 As shown in the
-[installation instructions](https://github.com/kata-containers/documentation/blob/master/install),
+[installation instructions](install),
 Kata Containers provide binaries for popular distributions in their native
 packaging formats. This allows Kata Containers to be upgraded using the
 standard package management tools for your distribution.
@@ -150,7 +150,7 @@ Since the official assets are packaged, they are automatically upgraded when
 new package versions are published.
 
 > **Warning**: Note that if you use custom assets (by modifying the
-> [Kata Runtime configuration > file](https://github.com/kata-containers/runtime/#configuration)),
+> [Kata Runtime configuration > file](../src/runtime/README.md#configuration)),
 > it is your responsibility to ensure they are updated as necessary.
 
 ### Guest kernel
@@ -159,7 +159,7 @@ The `kata-linux-container` package contains a Linux\* kernel based on the
 latest vanilla version of the
 [long-term kernel](https://www.kernel.org/)
 plus a small number of
-[patches](https://github.com/kata-containers/packaging/tree/master/kernel).
+[patches](../tools/packaging/kernel).
 
 The `Longterm` branch is only updated with
 [important bug fixes](https://www.kernel.org/category/releases.html)
@@ -174,7 +174,7 @@ The `kata-containers-image` package is updated only when critical updates are
 available for the packages used to create it, such as:
 
 - systemd
-- [Kata Containers Agent](https://github.com/kata-containers/agent)
+- [Kata Containers Agent](../src/agent)
 
 ### Determining asset versions
 

docs/design/README.md

@@ -8,3 +8,4 @@ Kata Containers design documents:
 - [VSocks](VSocks.md)
 - [VCPU handling](vcpu-handling.md)
 - [Host cgroups](host-cgroups.md)
+- [Metrics(Kata 2.0)](kata-2-0-metrics.md)

docs/design/VSocks.md

@@ -1,7 +1,6 @@
 # Kata Containers and VSOCKs
 
 - [Introduction](#introduction)
-    - [proxy communication diagram](#proxy-communication-diagram)
     - [VSOCK communication diagram](#vsock-communication-diagram)
 - [System requirements](#system-requirements)
 - [Advantages of using VSOCKs](#advantages-of-using-vsocks)
@@ -16,46 +15,10 @@ processes in the virtual machine can read/write data from/to a serial port
 device and the processes in the host can read/write data from/to a Unix socket.
 Most GNU/Linux distributions have support for serial ports, making it the most
 portable solution. However, the serial link limits read/write access to one
-process at a time. To deal with this limitation the resources (serial port and
-Unix socket) must be multiplexed. In Kata Containers those resources are
-multiplexed by using [`kata-proxy`][2] and [Yamux][3], the following diagram shows
-how it's implemented.
-
-
-### proxy communication diagram
-
-```
-.----------------------.
-| .------------------. |
-| | .-----.  .-----. | |
-| | |cont1|  |cont2| | |
-| | `-----'  `-----' | |
-| |       \   /      | |
-| |    .---------.   | |
-| |    |  agent  |   | |
-| |    `---------'   | |
-| |         |        | |
-| |    .-----------. | |
-| |POD |serial port| | |
-| `----|-----------|-' |
-|      |  socket   |   |
-|      `-----------'   |
-|           |          |
-|       .-------.      |
-|       | proxy |      |
-|       `-------'      |
-|           |          |
-|  .------./ \.------. |
-|  | shim |   | shim | |
-|  `------'   `------' |
-| Host                 |
-`----------------------'
-```
-
-A newer, simpler method is [VSOCKs][4], which can accept connections from
-multiple clients and does not require multiplexers ([`kata-proxy`][2] and
-[Yamux][3]). The following diagram shows how it's implemented in Kata Containers.
+process at a time.
 
+A newer, simpler method is [VSOCKs][1], which can accept connections from
+multiple clients. The following diagram shows how it's implemented in Kata Containers.
 
 ### VSOCK communication diagram
 
@@ -95,6 +58,7 @@ The Kata Containers version must be greater than or equal to 1.2.0 and `use_vsoc
 must be set to `true` in the runtime [configuration file][1].
 
 ### With VMWare guest
+
 To use Kata Containers with VSOCKs in a VMWare guest environment, first stop the `vmware-tools` service and unload the VMWare Linux kernel module.
 ```
 sudo systemctl stop vmware-tools
@@ -107,28 +71,25 @@ sudo modprobe -i vhost_vsock
 ### High density
 
 Using a proxy for multiplexing the connections between the VM and the host uses
-4.5MB per [POD][5]. In a high density deployment this could add up to GBs of
+4.5MB per [POD][2]. In a high density deployment this could add up to GBs of
 memory that could have been used to host more PODs. When we talk about density
 each kilobyte matters and it might be the decisive factor between run another
 POD or not. For example if you have 500 PODs running in a server, the same
-amount of [`kata-proxy`][2] processes will be running and consuming for around
+amount of [`kata-proxy`][3] processes will be running and consuming for around
 2250MB of RAM. Before making the decision not to use VSOCKs, you should ask
 yourself, how many more containers can run with the memory RAM consumed by the
 Kata proxies?
 
 ### Reliability
 
-[`kata-proxy`][2] is in charge of multiplexing the connections between virtual
+[`kata-proxy`][3] is in charge of multiplexing the connections between virtual
 machine and host processes, if it dies all connections get broken. For example
-if you have a [POD][5] with 10 containers running, if `kata-proxy` dies it would
+if you have a [POD][2] with 10 containers running, if `kata-proxy` dies it would
 be impossible to contact your containers, though they would still be running.
 Since communication via VSOCKs is direct, the only way to lose communication
-with the containers is if the VM itself or the [shim][6] dies, if this happens
+with the containers is if the VM itself or the `containerd-shim-kata-v2` dies, if this happens
 the containers are removed automatically.
 
-[1]: https://github.com/kata-containers/runtime#configuration
-[2]: https://github.com/kata-containers/proxy
-[3]: https://github.com/hashicorp/yamux
-[4]: https://wiki.qemu.org/Features/VirtioVsock
-[5]: ./vcpu-handling.md#virtual-cpus-and-kubernetes-pods
-[6]: https://github.com/kata-containers/shim
+[1]: https://wiki.qemu.org/Features/VirtioVsock
+[2]: ./vcpu-handling.md#virtual-cpus-and-kubernetes-pods
+[3]: https://github.com/kata-containers/proxy

docs/design/arch-images/kata-2-metrics.drawio

@@ -0,0 +1 @@
+<mxfile host="Chrome" modified="2020-07-02T06:44:28.736Z" agent="5.0 (Macintosh; Intel Mac OS X 10_15_4) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/83.0.4103.116 Safari/537.36" etag="r7FpfnbGNK7jbg54Gu9x" version="13.3.5" type="device"><diagram id="XNV8G0dePIPkhS_Khqr4" name="Page-1">7VvZcuI4FP0aHqFky+sjkNCTqfR0qtLV6fTLlMDy0hiLscWWrx8Zy3iRQkhjQxbygnVlhK1zz9HRkg4cztZfYjT3vxIHhx0VOOsOvOqoKrRthX2kkU0WMTWYBbw4cLIQKAL3wRPOgkoeXQQOTngsC1FCQhrMq8EJiSI8oZUYimOyqt7mktCpBObIw0LgfoJCMfoQONTPoiqEdlHxFw48n/80hIA/+Qzld/NA4iOHrEoheN2Bw5gQml3N1kMcpr1X7ZjRM7W7J4txRA/5wrd/v5rDewTubvrjyZDYg1l/01W0rJklChf8lfnT0k3eBzFZRA5OW1E6cLDyA4rv52iS1q4Y6izm01nIq10SUQ4jwxAOvBg5AXvCIQlJvG0PmhgZOK1zgzAsxR2ELXfC4gmNyRSXaoyJhccuqxHfmXfDEscUr0sh3gdfMJlhGm/YLby2q+i6nn2J56QOzKy8KhDWctT8Eri7rEQ8q7xd60W/swve9a+BQW8PBlWTw+C6jm0YIgyu66oTKQyOMTZ0oykYNLsGQ/7OJRgYnUQYFENvCYYWUXgvZFDss5PBuHBBVc7OBQUKvY4dNjbyIompTzwSofC6iA4KXNKULu65JWTO0fiNKd1wONCCkipWeB3Qn6Xrx7Spns5LV2ve8raw4YUyy7R9iCRkEU/4u3i3328se/zw+97wp99Wf4fTh2I0pCj2MN3TOZwjaYfsBTjGIaLBsmomGodKhQJjKI3nEymAt2jMPFql01EYeBG7nrAOwyy/B2nqBswE9XnFLHCcDF+cBE9ovG0v7fo5CSK6fR190NGvDgJjb7YJpNlZO/6rFfMkJRPoKbahVUUtKx2MBm/8Ln27Ustqmonldrt2tQ3iugnLmzqeu4c8COK9mVkRRSNkXTpwgmUFZeO/RWopt0h0ky0UfXaDos3XWzzyenblpZ+sfykKIhw73cQPZt0poqi73DXPHnf7C9nNzY2Hmqi2yhgpWJWpLQDGdX/EW6jqM/trSoUts6bCUBwLFdPMk6Csw0YDg6EcePMV3H6D4szwiDc/y4XSt9Ji8bVtSSbq5nGibouivpdjL6o6TxjQA5ZuVjMGHKc0jQqJfKwQzdQHzpwj7YAkc+Sj17nswN7HLklGofHNCbglCrjhWKahyQQc9nUN5i20JeBMnMHLAi6bzLQm35r+megGjqKbeqhQw0OF+jR8U0W+3cVp2z5eJOmL45gl8ccmnm1XiGdIltQUS2uHePJx7py8K7j2WKbaC7wrqPaYt3eSYQ5KZr1yMTsb76QQi1Min9K5FPe3OelVnyHaq+e8oMfYVWXgkVPefIKrKL3aAlN71lRcxXgWz5PxWP2YRIYHEnmXXxBa1fSyj8uvRrMJ/XBvb7q/6A348c9DF/34nvjgzANA61lzgizJMX4jNguKer9dqpqRKKCkXX917pUpW1drS48yh6Xqp1yZ0kS9Tshk2hwOsl0xCwATynDo6wBooG0cLFibYFoiCjIQYGs2VxH6+43OL/9omNu32vLyqozxpuyqKurXe9uleZYyf+BYoa6rx3mInJWGUuFkVwOn86yKgOllW6Zp0TVr2zKaRHRPvS2jiWT+8IOfqcBeDQodqucd/8TdMeSl7/iRzaCm1bYpVREEWwZCW0dFLAGEnXilCtcsGJLTO7bpANOUEEbHliNdFbXUMczO+1SBGo0AGI2aAgqqdaC0XKTK0qWdkjB79oZYWJw0f1qsDMkgc1Kk8vN15fWwzRzHyyCRzHbZi9IqGNWOjEiEa73OQ4f7Shn61blE/aidT+LgKc2vsPPCssWrzizWBVB2ZlF2ZLE1qEQb6C1wkprUKY6j9Ez8u4CrmeEJVNGBsk1Y46TwiCdKP59L0HOhOpdLkJxkutgE2dCjw/PbBGW/p7v4hB1Y5tl9gmjpLj5B6hNka+Yn9QmqaOkuPmGHjtaeT2DF4h/tsrW/4v8V4fX/</diagram></mxfile>

docs/design/arch-images/kata-metrics-sequence-diagram.drawio

@@ -0,0 +1 @@
+<mxfile host="Chrome" modified="2020-07-02T06:45:31.744Z" agent="5.0 (Macintosh; Intel Mac OS X 10_15_4) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/83.0.4103.116 Safari/537.36" etag="f3JpMUEY9_WRpPV9i93y" version="13.3.5" type="device"><diagram id="XNV8G0dePIPkhS_Khqr4" name="Page-1">7VrbcqM4EP0aPzolIa6PTpzMPEyqspna2tl9SclGNmwwYoRw7P36lUDY3GzjBPA4NSlXBTWtC92nT7cEI3S32nxhOPIeqUuCkQbczQhNR5qGHAeKf1KyzSSWjjLBkvluJgJ7wXf/P5IJYS5NfJfESpaJOKUB96OycE7DkMx5SYYZo29ltQUN3JIgwktSE3yf46Au/ct3uZdJNYSc/Y2vxF96amqEgFr5CufaShB72KVvBRG6H6E7RinPrlabOxJI65UN83Dg7m5ljIS8TYd4pmuY/PvHjzj+Of06ftWeH/4Zm9koaxwk6omfGF0R7pEklmsmbE2YWj/f5lYRjxLJy2QVfPMXJPBD0bqNCPNFV6GPpoESP+1lt2+ez8n3CM9l1zeBFyHz+CoQLSguhQs5Fl3Yrh0EOIr9WTorEBJG5gmL/TV5JnGGFCmlCZcz3e0QIIULMZhCk7T07cIPgjsaUJY+ADJd2zJ1IY85o6+kcAdNDB1BNUJBvkj/hLxuduUJYShONgWRcsMXIg3KtkJF3R1rmm1lnVRYaKYCyVsRZErHK+DLzhWxAvZyN/re9eJCef8MJFg1JLxijscrGvqcfkIIwBmGRGuCAADm/eShZwhAxwKnIWCAISEAYQ0DNb8XvBVRP+TpGozbkTGtuJ8y7tElDXFQBEDVKXVTHgVna/tCrWxdqBs168IG4xqG0ZNx7QbbmgGXAIxwWDKy+TORSSE11jiD+UQoaFq0SU2W3xdXS14MG3cce/5qnAbueje+WG42Rab9O5S7DmXTaBHKdlMoG32FsmYMGcrQaB/K9rn2hcgoW7dlKOuwr1DWmmL5So2rV4xr1aHbaNzegOsc5EnXX+cclkQuFuwjyO5RzOrP4wLXFdRqXiGuqO5Vc2/4+720SGE48JehZD+yUDQ998Plt7Q1lXRDQnci9xiiOQvo/FWSGk1Cl7iKt4Sz2PaHGi5t/F1ntNS/ZONzqQhuHGSrdqp7A4Cj2tNNPqxsbAuNAnJS2XloiWnC5uSYTxRGOGZLcmzAPDCkmVtRJ7gRHK2XEDiGahBGAsxFeijRchPe1PBPMswKyK6ScpVrs8dWvfaoFR7F24Kait7D8+h64zwPh/Qt42P6VklfXGQrbu6NULkMqtiALhYx4aNq0O489f44RuDTciTSL8yRqF6n/5kS4nMScmGeCifOWM6HfjgPEleQmFzpVlh47cdybwdWeY/r50toa1fDl3mQnOTLjrkQaT1xYYWrUPVI6pS+foTbumImWGcmvCTqwX/vizrcF+32PMf2RaBpX9QfddYPPH+RtLSDZWsDO+Xg0bV2aalWBHRn3Kba/UqNC0HFuqYoxbXL5n29zlzZRuglJbCXq83i8AbosJzFQd7uPIvnLjuZxnOuOJ3GnYHSOCzzqa7vMdl1Jq/CXz+RydEH9c0hMr9Wix+P8yg9XX0QP4FmHHg0Fr6e2MABcpIDIVUIGMkpvug4UYEzo5zTVXOg7EIDnAqNgwn4JGzbojY/7e8bteNa+alXM9AB0Hbmd1TzOyM8YeFH3UojEsqDKRx7KftlhJe/xrYLDAdK7OYYnXo89+RJl5sX87htlgfJltqbx7W6x/usRKyuKuZd4ZvbDV34qEHTD5Qc0pYkWFzz2UG54gCwi4qj85DNkdx7zEK7mpJbsvS5pUUDyuHxYkH0qFY+5/dAAxQYWn13Kb0r8IBDd0Y3R6LlcgXFaQgag0GwAgwDVF7h9VwoaPWPYD5VoXA5T9qOM2gBgOqeTPfIN2LSx18uBDs8UR6qxBMVXtnDpj1sUY/qL+E/Vay2Pn0YLqiNSk61jGGren3Yqv6o/86p6qFVfkVsgwtX9fqhqv5lmZD4Cg8S31/c6IPV19WXXVZLwjy/vq7uvvOZ3ln7iub+K/VMff+xP7r/Hw==</diagram></mxfile>

docs/design/architecture.md

@@ -1,101 +1,61 @@
 # Kata Containers Architecture
 
 
-* [Overview](#overview)
-* [Virtualization](#virtualization)
-* [Guest assets](#guest-assets)
-    * [Guest kernel](#guest-kernel)
-    * [Guest Image](#guest-image)
-      * [Root filesystem image](#root-filesystem-image)
-      * [Initrd image](#initrd-image)
-* [Agent](#agent)
-* [Runtime](#runtime)
-    * [Configuration](#configuration)
-    * [Significant OCI commands](#significant-oci-commands)
-        * [create](#create)
-        * [start](#start)
-        * [exec](#exec)
-        * [kill](#kill)
-        * [delete](#delete)
-* [Proxy](#proxy)
-* [Shim](#shim)
-* [Networking](#networking)
-* [Storage](#storage)
-* [Kubernetes Support](#kubernetes-support)
-    * [Problem Statement](#problem-statement)
-    * [Containerd](#containerd)
-    * [CRI-O](#cri-o)
-        * [OCI Annotations](#oci-annotations)
-        * [Mixing VM based and namespace based runtimes](#mixing-vm-based-and-namespace-based-runtimes)
-* [Appendices](#appendices)
-    * [DAX](#dax)
+- [Kata Containers Architecture](#kata-containers-architecture)
+  - [Overview](#overview)
+  - [Virtualization](#virtualization)
+  - [Guest assets](#guest-assets)
+    - [Guest kernel](#guest-kernel)
+    - [Guest image](#guest-image)
+      - [Root filesystem image](#root-filesystem-image)
+      - [Initrd image](#initrd-image)
+  - [Agent](#agent)
+  - [Runtime](#runtime)
+    - [Configuration](#configuration)
+  - [Networking](#networking)
+    - [CNM](#cnm)
+    - [Network Hotplug](#network-hotplug)
+  - [Storage](#storage)
+  - [Kubernetes support](#kubernetes-support)
+      - [OCI annotations](#oci-annotations)
+      - [Mixing VM based and namespace based runtimes](#mixing-vm-based-and-namespace-based-runtimes)
+- [Appendices](#appendices)
+  - [DAX](#dax)
 
 ## Overview
 
-This is an architectural overview of Kata Containers, based on the 1.5.0 release.
+This is an architectural overview of Kata Containers, based on the 2.0 release.
 
-The two primary deliverables of the Kata Containers project are a container runtime
-and a CRI friendly shim. There is also a CRI friendly library API behind them.
+The primary deliverable of the Kata Containers project is a CRI friendly shim. There is also a CRI friendly library API behind them.
 
-The [Kata Containers runtime (`kata-runtime`)](https://github.com/kata-containers/runtime)
+The [Kata Containers runtime](../../src/runtime)
 is compatible with the [OCI](https://github.com/opencontainers) [runtime specification](https://github.com/opencontainers/runtime-spec)
-and therefore works seamlessly with the
-[Docker\* Engine](https://www.docker.com/products/docker-engine) pluggable runtime
-architecture. It also supports the [Kubernetes\* Container Runtime Interface (CRI)](https://github.com/kubernetes/community/blob/master/contributors/devel/sig-node/container-runtime-interface.md)
+and therefore works seamlessly with the [Kubernetes\* Container Runtime Interface (CRI)](https://github.com/kubernetes/community/blob/master/contributors/devel/sig-node/container-runtime-interface.md)
 through the [CRI-O\*](https://github.com/kubernetes-incubator/cri-o) and
-[Containerd CRI Plugin\*](https://github.com/containerd/cri) implementation. In other words, you can transparently
-select between the [default Docker and CRI shim runtime (runc)](https://github.com/opencontainers/runc)
-and `kata-runtime`.
+[Containerd\*](https://github.com/containerd/containerd) implementation.
 
-`kata-runtime` creates a QEMU\*/KVM virtual machine for each container or pod,
-the Docker engine or `kubelet` (Kubernetes) creates respectively.
+Kata Containers creates a QEMU\*/KVM virtual machine for pod that `kubelet` (Kubernetes) creates respectively.
 
-![Docker and Kata Containers](arch-images/docker-kata.png)
-
-The [`containerd-shim-kata-v2` (shown as `shimv2` from this point onwards)](https://github.com/kata-containers/runtime/tree/master/containerd-shim-v2) 
-is another Kata Containers entrypoint, which 
+The [`containerd-shim-kata-v2` (shown as `shimv2` from this point onwards)](../../src/runtime/containerd-shim-v2) 
+is the Kata Containers entrypoint, which
 implements the [Containerd Runtime V2 (Shim API)](https://github.com/containerd/containerd/tree/master/runtime/v2) for Kata.
-With `shimv2`, Kubernetes can launch Pod and OCI compatible containers with one shim (the `shimv2`) per Pod instead
-of `2N+1` shims (a `containerd-shim` and a `kata-shim` for each container and the Pod sandbox itself), and no standalone
-`kata-proxy` process even if no VSOCK is available.
+
+Before `shimv2` (as done in [Kata Containers 1.x releases](https://github.com/kata-containers/runtime/releases)), we need to create a `containerd-shim` and a [`kata-shim`](https://github.com/kata-containers/shim) for each container and the Pod sandbox itself, plus an optional [`kata-proxy`](https://github.com/kata-containers/proxy) when VSOCK is not available. With `shimv2`, Kubernetes can launch Pod and OCI compatible containers with one shim (the `shimv2`) per Pod instead of `2N+1` shims, and no standalone `kata-proxy` process even if no VSOCK is available.
 
 ![Kubernetes integration with shimv2](arch-images/shimv2.svg)
 
 The container process is then spawned by
-[agent](https://github.com/kata-containers/agent), an agent process running
-as a daemon inside the virtual machine. `kata-agent` runs a gRPC server in
+[`kata-agent`](../../src/agent), an agent process running
+as a daemon inside the virtual machine. `kata-agent` runs a [`ttRPC`](https://github.com/containerd/ttrpc-rust) server in
 the guest using a VIRTIO serial or VSOCK interface which QEMU exposes as a socket
-file on the host. `kata-runtime` uses a gRPC protocol to communicate with
+file on the host. `shimv2` uses a `ttRPC` protocol to communicate with
 the agent. This protocol allows the runtime to send container management
 commands to the agent. The protocol is also used to carry the I/O streams (stdout,
-stderr, stdin) between the containers and the manage engines (e.g. Docker Engine).
+stderr, stdin) between the containers and the manage engines (e.g. CRI-O or containerd).
 
 For any given container, both the init process and all potentially executed
 commands within that container, together with their related I/O streams, need
-to go through the VIRTIO serial or VSOCK interface exported by QEMU. 
-In the VIRTIO serial case, a [Kata Containers
-proxy (`kata-proxy`)](https://github.com/kata-containers/proxy) instance is
-launched for each virtual machine to handle multiplexing and demultiplexing
-those commands and streams.
-
-On the host, each container process's removal is handled by a reaper in the higher
-layers of the container stack. In the case of Docker or containerd it is handled by `containerd-shim`.
-In the case of CRI-O it is handled by `conmon`. For clarity, for the remainder
-of this document the term "container process reaper" will be used to refer to
-either reaper. As Kata Containers processes run inside their own  virtual machines,
-the container process reaper cannot monitor, control
-or reap them. `kata-runtime` fixes that issue by creating an [additional shim process
-(`kata-shim`)](https://github.com/kata-containers/shim) between the container process
-reaper and `kata-proxy`. A `kata-shim` instance will both forward signals and `stdin`
-streams to the container process on the guest and pass the container `stdout`
-and `stderr` streams back up the stack to the CRI shim or Docker via the container process
-reaper. `kata-runtime` creates a `kata-shim` daemon for each container and for each
-OCI command received to run within an already running container (example, `docker
-exec`).
-
-Since Kata Containers version 1.5, the new introduced `shimv2` has integrated the
-functionalities of the reaper, the `kata-runtime`, the `kata-shim`, and the `kata-proxy`.
-As a result, there will not be any of the additional processes previously listed.
+to go through the VSOCK interface exported by QEMU.
 
 The container workload, that is, the actual OCI bundle rootfs, is exported from the
 host to the virtual machine.  In the case where a block-based graph driver is
@@ -136,7 +96,7 @@ The only services running in the context of the mini O/S are the init daemon
 is created using libcontainer, creating a container in the same manner that is done
 by `runc`.
 
-For example, when `docker run -ti ubuntu date` is run:
+For example, when `ctr run -ti ubuntu date` is run:
 
 - The hypervisor will boot the mini-OS image using the guest kernel.
 - `systemd`, running inside the mini-OS context, will launch the `kata-agent` in
@@ -155,225 +115,36 @@ The only service running in the context of the initrd is the [Agent](#agent) as
 
 ## Agent
 
-[`kata-agent`](https://github.com/kata-containers/agent) is a process running in the
-guest as a supervisor for managing containers and processes running within
-those containers.
+[`kata-agent`](../../src/agent) is a process running in the guest as a supervisor for managing containers and processes running within those containers.
 
-The `kata-agent` execution unit is the sandbox. A `kata-agent` sandbox is a container sandbox defined by a set of namespaces (NS, UTS, IPC and PID). `kata-runtime` can
+For the 2.0 release, the `kata-agent` is rewritten in the [RUST programming language](https://www.rust-lang.org/) so that we can minimize its memory footprint while keeping the memory safety of the original GO version of [`kata-agent` used in Kata Container 1.x](https://github.com/kata-containers/agent). This memory footprint reduction is pretty impressive, from tens of megabytes down to less than 100 kilobytes, enabling Kata Containers in more use cases like functional computing and edge computing.
+
+The `kata-agent` execution unit is the sandbox. A `kata-agent` sandbox is a container sandbox defined by a set of namespaces (NS, UTS, IPC and PID). `shimv2` can
 run several containers per VM to support container engines that require multiple
-containers running inside a pod. In the case of docker, `kata-runtime` creates a
-single container per pod.
+containers running inside a pod.
 
-`kata-agent` communicates with the other Kata components over gRPC.
-It also runs a [`yamux`](https://github.com/hashicorp/yamux) server on the same gRPC URL.
-
-The `kata-agent` makes use of [`libcontainer`](https://github.com/opencontainers/runc/tree/master/libcontainer)
-to manage the lifecycle of the container. This way the `kata-agent` reuses most
-of the code used by [`runc`](https://github.com/opencontainers/runc).
-
-### Agent gRPC protocol
-
-placeholder
+`kata-agent` communicates with the other Kata components over `ttRPC`.
 
 ## Runtime
 
-`kata-runtime` is an OCI compatible container runtime and is responsible for handling
-all commands specified by
-[the OCI runtime specification](https://github.com/opencontainers/runtime-spec)
-and launching `kata-shim` instances.
+`containerd-shim-kata-v2` is a [containerd runtime shimv2](https://github.com/containerd/containerd/blob/v1.4.1/runtime/v2/README.md) implementation and is responsible for handling the `runtime v2 shim APIs`, which is similar to [the OCI runtime specification](https://github.com/opencontainers/runtime-spec) but simplifies the architecture by loading the runtime once and making RPC calls to handle the various container lifecycle commands. This refinement is an improvement on the OCI specification which requires the container manager call the runtime binary multiple times, at least once for each lifecycle command.
 
-`kata-runtime` heavily utilizes the
-[virtcontainers project](https://github.com/containers/virtcontainers), which
-provides a generic, runtime-specification agnostic, hardware-virtualized containers
-library.
+`containerd-shim-kata-v2` heavily utilizes the
+[virtcontainers package](../../src/runtime/virtcontainers/), which provides a generic, runtime-specification agnostic, hardware-virtualized containers library.
 
 ### Configuration
 
-The runtime uses a TOML format configuration file called `configuration.toml`. By
-default this file is installed in the `/usr/share/defaults/kata-containers`
-directory and contains various settings such as the paths to the hypervisor,
-the guest kernel and the mini-OS image.
+The runtime uses a TOML format configuration file called `configuration.toml`. By default this file is installed in the `/usr/share/defaults/kata-containers` directory and contains various settings such as the paths to the hypervisor, the guest kernel and the mini-OS image.
 
+The actual configuration file paths can be determined by running:
+```
+$ kata-runtime --kata-show-default-config-paths
+```
 Most users will not need to modify the configuration file.
 
-The file is well commented and provides a few "knobs" that can be used to modify
-the behavior of the runtime.
-
-The configuration file is also used to enable runtime [debug output](https://github.com/kata-containers/documentation/blob/master/Developer-Guide.md#enable-full-debug).
-
-### Significant OCI commands
-
-Here we describe how `kata-runtime` handles the most important OCI commands.
-
-#### `create`
-
-When handling the OCI
-[`create`](https://github.com/kata-containers/runtime/blob/master/cli/create.go)
-command, `kata-runtime` goes through the following steps:
-
-1. Create the network namespace where we will spawn VM and shims processes.
-2. Call into the pre-start hooks. One of them should be responsible for creating
-the `veth` network pair between the host network namespace and the network namespace
-freshly created.
-3. Scan the network from the new network namespace, and create a MACVTAP connection
- between the `veth` interface and a `tap` interface into the VM.
-4. Start the VM inside the network namespace by providing the `tap` interface
- previously created.
-5. Wait for the VM to be ready.
-6. Start `kata-proxy`, which will connect to the created VM. The `kata-proxy` process
-will take care of proxying all communications with the VM. Kata has a single proxy
-per VM.
-7. Communicate with `kata-agent` (through the proxy) to configure the sandbox
- inside the VM.
-8. Communicate with `kata-agent` to create the container, relying on the OCI
-configuration file `config.json` initially provided to `kata-runtime`. This
-spawns the container process inside the VM, leveraging the `libcontainer` package.
-9. Start `kata-shim`, which will connect to the gRPC server socket provided by the `kata-proxy`. `kata-shim`  will spawn a few Go routines to parallelize blocking calls `ReadStdout()` , `ReadStderr()` and `WaitProcess()`. Both `ReadStdout()` and `ReadStderr()` are run through infinite loops since `kata-shim` wants the output of those until the container process terminates. `WaitProcess()` is a unique call which returns the exit code of the container process when it terminates inside the VM. Note that `kata-shim` is started inside the network namespace, to allow upper layers to determine which network namespace has been created and by checking the `kata-shim` process. It also creates a new PID namespace by entering into it. This ensures that all `kata-shim` processes belonging to the same container will get killed when the `kata-shim` representing the container process terminates.
-
-At this point the container process is running inside of the VM, and it is represented
-on the host system by the `kata-shim` process.
-
-![`kata-oci-create`](arch-images/kata-oci-create.svg)
-
-#### `start`
-
-With traditional containers, [`start`](https://github.com/kata-containers/runtime/blob/master/cli/start.go) launches a container process in its own set of namespaces. With Kata Containers, the main task of `kata-runtime` is to ask [`kata-agent`](#agent) to start the container workload inside the virtual machine. `kata-runtime` will run through the following steps:
-
-1. Communicate with `kata-agent` (through the proxy) to start the container workload
- inside the VM. If, for example, the command to execute inside of the container is `top`,
- the `kata-shim`'s `ReadStdOut()` will start returning text output for top, and
-  `WaitProcess()` will continue to block as long as the `top` process runs.
-2. Call into the post-start hooks. Usually, this is a no-op since nothing is provided
-  (this needs clarification)
-
-![`kata-oci-start`](arch-images/kata-oci-start.svg)
-
-#### `exec`
-
-OCI [`exec`](https://github.com/kata-containers/runtime/blob/master/cli/exec.go) allows you to run an additional command within an already running
-container.  In Kata Containers, this is handled as follows:
-
-1. A request is sent to the `kata agent` (through the proxy) to start a new process
- inside an existing container running within the VM.
-2. A new `kata-shim` is created within the same network and PID namespaces as the
- original `kata-shim` representing the container process. This new `kata-shim` is
- used for the new exec process.
-
-Now the process started with `exec` is running within the VM, sharing `uts`, `pid`, `mnt` and `ipc` namespaces with the container process.
-
-![`kata-oci-exec`](arch-images/kata-oci-exec.svg)
-
-#### `kill`
-
-When sending the OCI [`kill`](https://github.com/kata-containers/runtime/blob/master/cli/kill.go) command, the container runtime should send a
-[UNIX signal](https://en.wikipedia.org/wiki/Unix_signal) to the container process.
-A `kill` sending a termination signal such as `SIGKILL` or `SIGTERM` is expected
-to terminate the container process.  In the context of a traditional container,
-this means stopping the container.  For `kata-runtime`, this translates to stopping
-the container and the VM associated with it.
-
-1. Send a request to kill the container process to the `kata-agent` (through the proxy).
-2. Wait for `kata-shim` process to exit.
-3. Force kill the container process if `kata-shim` process didn't return after a
- timeout. This is done by communicating with `kata-agent` (connecting the proxy),
- sending `SIGKILL` signal to the container process inside the VM.
-4. Wait for `kata-shim` process to exit, and return an error if we reach the
- timeout again.
-5. Communicate with `kata-agent` (through the proxy) to remove the container
- configuration from the VM.
-6. Communicate with `kata-agent` (through the proxy) to destroy the sandbox
- configuration from the VM.
-7. Stop the VM.
-8. Remove all network configurations inside the network namespace and delete the
- namespace.
-9. Execute post-stop hooks.
-
-If `kill` was invoked with a non-termination signal, this simply signals the container process. Otherwise, everything has been torn down, and the VM has been removed.
-
-#### `delete`
-
-[`delete`](https://github.com/kata-containers/runtime/blob/master/cli/delete.go) removes all internal resources related to a container. A running container
-cannot be deleted unless the OCI runtime is explicitly being asked to, by using
-`--force` flag.
-
-If the sandbox is not stopped, but the particular container process returned on
-its own already, the `kata-runtime` will first go through most of the steps a `kill`
-would go through for a termination signal. After this process, or if the `sandboxID` was already stopped to begin with, then `kata-runtime` will:
-
-1. Remove container resources. Every file kept under `/var/{lib,run}/virtcontainers/sandboxes/<sandboxID>/<containerID>`.
-2. Remove sandbox resources. Every file kept under `/var/{lib,run}/virtcontainers/sandboxes/<sandboxID>`.
-
-At this point, everything related to the container should have been removed from the host system, and no related process should be running.
-
-#### `state`
-
-[`state`](https://github.com/kata-containers/runtime/blob/master/cli/state.go)
-returns the status of the container. For `kata-runtime`, this means being
-able to detect if the container is still running by looking at the state of `kata-shim`
-process representing this container process.
-
-1. Ask the container status by checking information stored on disk. (clarification needed)
-2. Check `kata-shim` process representing the container.
-3. In case the container status on disk was supposed to be `ready` or `running`,
- and the `kata-shim` process no longer exists, this involves the detection of a
- stopped container. This means that before returning the container status,
- the container has to be properly stopped. Here are the steps involved in this detection:
-	1. Wait for `kata-shim` process to exit.
-	2. Force kill the container process if `kata-shim` process didn't return after a timeout. This is done by communicating with `kata-agent` (connecting the proxy), sending `SIGKILL` signal to the container process inside the VM.
-	3. Wait for `kata-shim` process to exit, and return an error if we reach the timeout again.
-	4. Communicate with `kata-agent` (connecting the proxy) to remove the container configuration from the VM.
-4. Return container status.
-
-## Proxy
-
-Communication with the VM can be achieved by either `virtio-serial` or, if the host
-kernel is newer than v4.8, a virtual socket, `vsock` can be used. The default is `virtio-serial`.
-
-The VM will likely be running multiple container processes.  In the event `virtio-serial`
-is used, the I/O streams associated with each process needs to be multiplexed and demultiplexed on the host. On systems with `vsock` support, this component becomes optional.
-
-`kata-proxy` is a process offering access to the VM [`kata-agent`](https://github.com/kata-containers/agent)
-to multiple `kata-shim` and `kata-runtime` clients associated with the VM. Its
-main role is to route the I/O streams and signals between each `kata-shim`
-instance and the `kata-agent`.
-`kata-proxy` connects to `kata-agent` on a Unix domain socket that `kata-runtime` provides
-while spawning `kata-proxy`.
-`kata-proxy` uses [`yamux`](https://github.com/hashicorp/yamux) to multiplex gRPC
-requests on its connection to the `kata-agent`.
-
-When proxy type is configured as `proxyBuiltIn`, we do not spawn a separate
-process to proxy gRPC connections. Instead a built-in Yamux gRPC dialer is used to connect
-directly to `kata-agent`. This is used by CRI container runtime server `frakti` which
-calls directly into `kata-runtime`.
-
-## Shim
-
-A container process reaper, such as Docker's `containerd-shim` or CRI-O's `conmon`,
-is designed around the assumption that it can monitor and reap the actual container
-process. As the container process reaper runs on the host, it cannot directly
-monitor a process running within a virtual machine. At most it can see the QEMU
-process, but that is not enough. With Kata Containers, `kata-shim` acts as the
-container process that the container process reaper can monitor. Therefore
-`kata-shim` needs to handle all container I/O streams (`stdout`, `stdin` and `stderr`)
-and forward all signals the container process reaper decides to send to the container
-process.
-
-`kata-shim` has an implicit knowledge about which VM agent will handle those streams
-and signals and thus acts as an encapsulation layer between the container process
-reaper and the `kata-agent`. `kata-shim`:
-
-- Connects to `kata-proxy` on a Unix domain socket. The socket URL is passed from
-  `kata-runtime` to `kata-shim` when the former spawns the latter along with a
-  `containerID` and `execID`. The `containerID` and `execID` are used to identify
-  the true container process that the shim process will be shadowing or representing.
-- Forwards the standard input stream from the container process reaper into
- `kata-proxy` using gRPC `WriteStdin` gRPC API.
-- Reads the standard output/error from the container process.
-- Forwards signals it receives from the container process reaper to `kata-proxy`
-  using `SignalProcessRequest` API.
-- Monitors terminal changes and forwards them to `kata-proxy` using gRPC `TtyWinResize`
-  API.
+The file is well commented and provides a few "knobs" that can be used to modify the behavior of the runtime and your chosen hypervisor.
 
+The configuration file is also used to enable runtime [debug output](../Developer-Guide.md#enable-full-debug).
 
 ## Networking
 
@@ -391,7 +162,7 @@ cannot handle `veth` interfaces. Typically, `TAP` interfaces are created for VM
 connectivity.
 
 To overcome incompatibility between typical container engines expectations
-and virtual machines, `kata-runtime` networking transparently connects `veth`
+and virtual machines, Kata Containers networking transparently connects `veth`
 interfaces with `TAP` ones using MACVTAP:
 
 ![Kata Containers networking](arch-images/network.png)
@@ -456,35 +227,14 @@ The following diagram illustrates the Kata Containers network hotplug workflow.
 ![Network Hotplug](arch-images/kata-containers-network-hotplug.png)
 
 ## Storage
-Container workloads are shared with the virtualized environment through [9pfs](https://www.kernel.org/doc/Documentation/filesystems/9p.txt).
-The devicemapper storage driver is a special case. The driver uses dedicated block
-devices rather than formatted filesystems, and operates at the block level rather
-than the file level. This knowledge is used to directly use the underlying block
-device instead of the overlay file system for the container root file system. The
-block device maps to the top read-write layer for the overlay. This approach gives
-much better I/O performance compared to using 9pfs to share the container file system.
+Container workloads are shared with the virtualized environment through [virtio-fs](https://virtio-fs.gitlab.io/).
 
-The approach above does introduce a limitation in terms of dynamic file copy
-in/out of the container using the `docker cp` operations. The copy operation from
-host to container accesses the mounted file system on the host-side. This is
-not expected to work and may lead to inconsistencies as the block device will
-be simultaneously written to from two different mounts. The copy operation from
-container to host will work, provided the user calls `sync(1)` from within the
-container prior to the copy to make sure any outstanding cached data is written
-to the block device.
+The [devicemapper `snapshotter`](https://github.com/containerd/containerd/tree/master/snapshots/devmapper) is a special case. The `snapshotter` uses dedicated block devices rather than formatted filesystems, and operates at the block level rather than the file level. This knowledge is used to directly use the underlying block device instead of the overlay file system for the container root file system. The block device maps to the top read-write layer for the overlay. This approach gives much better I/O performance compared to using `virtio-fs` to share the container file system.
 
-```
-docker cp [OPTIONS] CONTAINER:SRC_PATH HOST:DEST_PATH
-docker cp [OPTIONS] HOST:SRC_PATH CONTAINER:DEST_PATH
-```
+Kata Containers has the ability to hotplug and remove block devices, which makes it possible to use block devices for containers started after the VM has been launched.
 
-Kata Containers has the ability to hotplug and remove block devices, which makes it
-possible to use block devices for containers started after the VM has been launched.
-
-Users can check to see if the container uses the devicemapper block device as its
-rootfs by calling `mount(8)` within the container.  If the devicemapper block device
-is used, `/` will be mounted on `/dev/vda`.  Users can disable direct mounting
-of the underlying block device through the runtime configuration.
+Users can check to see if the container uses the devicemapper block device as its rootfs by calling `mount(8)` within the container.  If the devicemapper block device
+is used, `/` will be mounted on `/dev/vda`.  Users can disable direct mounting of the underlying block device through the runtime configuration.
 
 ## Kubernetes support
 
@@ -505,44 +255,13 @@ lifecycle management from container execution through the dedicated
 
 In other words, a Kubelet is a CRI client and expects a CRI implementation to
 handle the server side of the interface.
-[CRI-O\*](https://github.com/kubernetes-incubator/cri-o) and [Containerd CRI Plugin\*](https://github.com/containerd/cri) are CRI implementations that rely on [OCI](https://github.com/opencontainers/runtime-spec)
+[CRI-O\*](https://github.com/kubernetes-incubator/cri-o) and [Containerd\*](https://github.com/containerd/containerd/) are CRI implementations that rely on [OCI](https://github.com/opencontainers/runtime-spec)
 compatible runtimes for managing container instances.
 
-Kata Containers is an officially supported CRI-O and Containerd CRI Plugin runtime. It is OCI compatible and therefore aligns with project's architecture and requirements.
-However, due to the fact that Kubernetes execution units are sets of containers (also
-known as pods) rather than single containers, the Kata Containers runtime needs to
-get extra information to seamlessly integrate with Kubernetes.
+Kata Containers is an officially supported CRI-O and Containerd runtime. Refer to the following guides on how to set up Kata Containers with Kubernetes:
 
-### Problem statement
-
-The Kubernetes\* execution unit is a pod that has specifications detailing constraints
-such as namespaces, groups, hardware resources, security contents, *etc* shared by all
-the containers within that pod.
-By default the Kubelet will send a container creation request to its CRI runtime for
-each pod and container creation. Without additional metadata from the CRI runtime,
-the Kata Containers runtime will thus create one virtual machine for each pod and for
-each containers within a pod. However the task of providing the Kubernetes pod semantics
-when creating one virtual machine for each container within the same pod is complex given
-the resources of these virtual machines (such as networking or PID) need to be shared.
-
-The challenge with Kata Containers when working as a Kubernetes\* runtime is thus to know
-when to create a full virtual machine (for pods) and when to create a new container inside
-a previously created virtual machine. In both cases it will get called with very similar
-arguments, so it needs the help of the Kubernetes CRI runtime to be able to distinguish a
-pod creation request from a container one.
-
-### Containerd
-
-As of Kata Containers 1.5, using `shimv2` with containerd 1.2.0 or above is the preferred
-way to run Kata Containers with Kubernetes ([see the howto](https://github.com/kata-containers/documentation/blob/master/how-to/how-to-use-k8s-with-cri-containerd-and-kata.md#configure-containerd-to-use-kata-containers)).
-The CRI-O will catch up soon ([`kubernetes-sigs/cri-o#2024`](https://github.com/kubernetes-sigs/cri-o/issues/2024)).
-
-Refer to the following how-to guides:
-
-- [How to use Kata Containers and Containerd](/how-to/containerd-kata.md)
-- [How to use Kata Containers and CRI (containerd plugin) with Kubernetes](/how-to/how-to-use-k8s-with-cri-containerd-and-kata.md)
-
-### CRI-O
+- [How to use Kata Containers and Containerd](../how-to/containerd-kata.md)
+- [Run Kata Containers with Kubernetes](../how-to/run-kata-with-k8s.md)
 
 ####  OCI annotations
 
@@ -587,36 +306,10 @@ with a Kubernetes pod:
 
 #### Mixing VM based and namespace based runtimes
 
-> **Note:** Since Kubernetes 1.12, the [`Kubernetes RuntimeClass`](/how-to/containerd-kata.md#kubernetes-runtimeclass)
+> **Note:** Since Kubernetes 1.12, the [`Kubernetes RuntimeClass`](https://kubernetes.io/docs/concepts/containers/runtime-class/)
 > has been supported and the user can specify runtime without the non-standardized annotations.
 
-One interesting evolution of the CRI-O support for `kata-runtime` is the ability
-to run virtual machine based pods alongside namespace ones. With CRI-O and Kata
-Containers, one can introduce the concept of workload trust inside a Kubernetes
-cluster.
-
-A cluster operator can now tag (through Kubernetes annotations) container workloads
-as `trusted` or `untrusted`. The former labels known to be safe workloads while
-the latter describes potentially malicious or misbehaving workloads that need the
-highest degree of isolation. In a software development context, an example of a `trusted` workload would be a containerized continuous integration engine whereas all
-developers applications would be `untrusted` by default. Developers workloads can
-be buggy, unstable or even include malicious code and thus from a security perspective
-it makes sense to tag them as `untrusted`. A CRI-O and Kata Containers based
-Kubernetes cluster handles this use case transparently as long as the deployed
-containers are properly tagged. All `untrusted` containers will be handled by Kata Containers and thus run in a hardware virtualized secure sandbox while `runc`, for
-example, could  handle the `trusted` ones.
-
-CRI-O's default behavior is to trust all pods, except when they're annotated with
-`io.kubernetes.cri-o.TrustedSandbox` set to `false`. The default CRI-O trust level
-is set through its `configuration.toml` configuration file. Generally speaking,
-the CRI-O runtime selection between its trusted runtime (typically `runc`) and its untrusted one (`kata-runtime`) is a function of the pod `Privileged` setting, the `io.kubernetes.cri-o.TrustedSandbox` annotation value, and the default CRI-O trust
-level. When a pod is `Privileged`, the runtime will always be `runc`. However, when
-a pod is **not** `Privileged` the runtime selection is done as follows:
-
-|                                        | `io.kubernetes.cri-o.TrustedSandbox` not set   | `io.kubernetes.cri-o.TrustedSandbox` = `true` | `io.kubernetes.cri-o.TrustedSandbox` = `false` |
-| :---                                   |     :---:                                      |     :---:                                     |     :---:                                             |
-| Default CRI-O trust level: `trusted`   | runc                                           | runc                                          | Kata Containers |
-| Default CRI-O trust level: `untrusted` | Kata Containers                               | Kata Containers                              |  Kata Containers |
+With `RuntimeClass`, users can define Kata Containers as a `RuntimeClass` and then explicitly specify that a pod being created as a Kata Containers pod. For details, please refer to [How to use Kata Containers and Containerd](../../docs/how-to/containerd-kata.md).
 
 
 # Appendices

docs/design/host-cgroups.md

@@ -51,7 +51,7 @@ Kata Containers introduces a non-negligible overhead for running a sandbox (pod)
  2) Kata Containers do not fully constrain the VMM and associated processes, instead placing a subset of them outside of the pod-cgroup.
 
 Kata Containers provides two options for how cgroups are handled on the host. Selection of these options is done through
-the `SandboxCgroupOnly` flag within the Kata Containers [configuration](https://github.com/kata-containers/runtime#configuration)
+the `SandboxCgroupOnly` flag within the Kata Containers [configuration](../../src/runtime/README.md#configuration)
 file.
 
 ## `SandboxCgroupOnly` enabled

docs/design/kata-2-0-metrics.md

@@ -0,0 +1,352 @@
+# Kata 2.0 Metrics Design
+
+* [Limitations of Kata 1.x and the target of Kata 2.0](#limitations-of-kata-1x-and-the-target-of-kata-20)
+* [Metrics architecture](#metrics-architecture)
+  * [Kata monitor](#kata-monitor)
+  * [Kata runtime](#kata-runtime)
+  * [Kata agent](#kata-agent)
+  * [Performance and overhead](#performance-and-overhead)
+* [Metrics list](#metrics-list)
+  * [Metric types](#metric-types)
+  * [Kata agent metrics](#kata-agent-metrics)
+  * [Firecracker metrics](#firecracker-metrics)
+  * [Kata guest OS metrics](#kata-guest-os-metrics)
+  * [Hypervisor metrics](#hypervisor-metrics)
+  * [Kata monitor metrics](#kata-monitor-metrics)
+  * [Kata containerd shim v2 metrics](#kata-containerd-shim-v2-metrics)
+
+Kata implement CRI's API and support [`ContainerStats`](https://github.com/kubernetes/kubernetes/blob/release-1.18/staging/src/k8s.io/cri-api/pkg/apis/runtime/v1alpha2/api.proto#L101) and [`ListContainerStats`](https://github.com/kubernetes/kubernetes/blob/release-1.18/staging/src/k8s.io/cri-api/pkg/apis/runtime/v1alpha2/api.proto#L103) interfaces to expose containers metrics. User can use these interface to get basic metrics about container.
+
+But unlike `runc`, Kata is a VM-based runtime and has a different architecture.
+
+## Limitations of Kata 1.x and the target of Kata 2.0
+
+Kata 1.x has a number of limitations related to observability that may be obstacles to running Kata Containers at scale.
+
+In Kata 2.0, the following components will be able to provide more details about the system.
+
+- containerd shim v2 (effectively `kata-runtime`)
+- Hypervisor statistics
+- Agent process
+- Guest OS statistics
+
+> **Note**: In Kata 1.x, the main user-facing component was the runtime (`kata-runtime`). From 1.5, Kata then introduced the Kata containerd shim v2 (`containerd-shim-kata-v2`) which is essentially a modified runtime that is loaded by containerd to simplify and improve the way VM-based containers are created and managed.
+>
+> For Kata 2.0, the main component is the Kata containerd shim v2, although the deprecated `kata-runtime` binary will be maintained for a period of time.
+>
+> Any mention of the "Kata runtime" in this document should be taken to refer to the Kata containerd shim v2 unless explicitly noted otherwise (for example by referring to it explicitly as the `kata-runtime` binary).
+
+## Metrics architecture
+
+Kata 2.0 metrics strongly depend on [Prometheus](https://prometheus.io/), a graduated project from CNCF.
+
+Kata Containers 2.0 introduces a new Kata component called `kata-monitor` which is used to monitor the other Kata components on the host. It's the monitor interface with Kata runtime, and we can do something like these:
+
+- Get metrics
+- Get events
+
+In this document we will cover metrics only. And until now it only supports metrics function.
+
+This is the architecture overview metrics in Kata Containers 2.0.
+
+![Kata Containers 2.0 metrics](arch-images/kata-2-metrics.png)
+
+
+And the sequence diagram is shown below:
+
+![Kata Containers 2.0 metrics ](arch-images/kata-metrics-sequence-diagram.png)
+
+For a quick evaluation, you can check out [this how to](../how-to/how-to-set-prometheus-in-k8s.md).
+
+### Kata monitor
+
+`kata-monitor` is a management agent on one node, where many Kata containers are running. `kata-monitor`'s work include:
+
+> **Note**: node is a single host system or a node in K8s clusters.
+
+- Aggregate sandbox metrics running on this node, and add `sandbox_id` label
+- As a Prometheus target, all metrics from Kata shim on this node will be collected by Prometheus indirectly. This can easy the targets count in Prometheus, and also need not to expose shim's metrics by `ip:port`
+
+Only one `kata-monitor` process are running on one node.
+
+`kata-monitor` is using a different communication channel other than that `conatinerd` communicating with Kata shim, and Kata shim listen on a new socket address for communicating with `kata-monitor`.
+
+The way `kata-monitor` get shim's metrics socket file(`monitor_address`) like that `containerd` get shim address. The socket is an abstract socket and saved as file `abstract` with the same directory of `address` for `containerd`.
+
+> **Note**: If there is no Prometheus server is configured, i.e., there is no scrape operations, `kata-monitor` will do nothing initiative.
+
+### Kata runtime
+
+Runtime is responsible for:
+
+- Gather metrics about shim process
+- Gather metrics about hypervisor process
+- Gather metrics about running sandbox
+- Get metrics from Kata agent(through `ttrpc`)
+
+### Kata agent
+
+Agent is responsible for:
+
+- Gather agent process metrics
+- Gather guest OS metrics
+
+And in Kata 2.0, agent will add a new interface:
+
+```protobuf
+rpc GetMetrics(GetMetricsRequest) returns (Metrics);
+
+message GetMetricsRequest {}
+
+message Metrics {
+	string metrics = 1;
+}
+
+```
+
+The `metrics` field is Prometheus encoded content. This can avoid defining a fixed structure in protocol buffers.
+
+### Performance and overhead
+
+Metrics should not become the bottleneck of system, downgrade the performance, and run with minimal overhead.
+
+Requirements:
+
+* Metrics **MUST** be quick to collect
+* Metrics **MUST** be small.
+* Metrics **MUST** be generated only if there are subscribers to the Kata metrics service
+* Metrics **MUST** be stateless
+
+In Kata 2.0, metrics are collected mainly from `/proc` filesystem, and consumed by Prometheus, based on a pull mode, that is mean if there is no Prometheus collector is running, so there will be zero overhead if nobody cares the metrics.
+
+Metrics service also doesn't hold any metrics in memory.
+
+|\*|No Sandbox | 1 Sandbox | 2 Sandboxes |
+|---|---|---|---|
+|Metrics count| 39 | 106 | 173 |
+|Metrics size(bytes)| 9K | 144K | 283K |
+|Metrics size(`gzipped`, bytes)| 2K | 10K | 17K |
+
+*Metrics size*: Response size of one Prometheus scrape request.
+
+It's easy to estimated that if there are 10 sandboxes running in the host, the size of one metrics fetch request issued by Prometheus will be about to 9 + (144 - 9) * 10 = 1.35M (not `gzipped`) or 2 + (10 - 2) * 10 = 82K (`gzipped`). Of course Prometheus support `gzip` compression, that can reduce the response size of every request.
+
+And here is some test data:
+
+- End-to-end (from Prometheus server to `kata-monitor` and `kata-monitor` write response back): 20ms(avg)
+- Agent(RPC all from shim to agent): 3ms(avg)
+
+Test infrastructure:
+
+- OS: Ubuntu 20.04
+- Hardware: Intel(R) Core(TM) i5-8500 CPU @ 3.00GHz, 6 Cores, and 16GB memory.
+
+**Scrape interval**
+
+Prometheus default `scrape_interval` is 1 minute, and usually it is set to 15s. Small `scrape_interval` will cause more overhead, so user should set it on monitor demand.
+
+## Metrics list
+
+Here listed is all supported metrics by Kata 2.0. Some metrics is dependent on guest kernels in the VM, so there may be some different by your environment.
+
+Metrics is categorized by component where metrics are collected from and for.
+
+* [Metric types](#metric-types)
+* [Kata agent metrics](#kata-agent-metrics)
+* [Firecracker metrics](#firecracker-metrics)
+* [Kata guest OS metrics](#kata-guest-os-metrics)
+* [Hypervisor metrics](#hypervisor-metrics)
+* [Kata monitor metrics](#kata-monitor-metrics)
+* [Kata containerd shim v2 metrics](#kata-containerd-shim-v2-metrics)
+
+> **Note**:
+>  * Labels here are not include `instance` and `job` labels that added by Prometheus.
+>  * Notes about metrics unit
+>    * `Kibibytes`, abbreviated `KiB`. 1 `KiB` equals 1024 B.
+>    * For some metrics (like network devices statistics from file `/proc/net/dev`), unit is depend on label( for example `recv_bytes` and `recv_packets` are having different units).
+>    * Most of these metrics is collected from `/proc` filesystem, so the unit of metrics are keeping the same unit as `/proc`. See the `proc(5)` manual page for further details.
+
+### Metric types
+
+Prometheus offer four core metric types.
+
+- Counter: A counter is a cumulative metric that represents a single monotonically increasing counter whose value can only increase.
+
+- Gauge: A gauge metric represents a single numerical value that can go up and down, typically used for measured values like current memory usage.
+
+- Histogram: A histogram samples observations (usually things like request durations or response sizes) and counts them in configurable buckets.
+
+- Summary: A summary samples observations like histogram, it can calculate configurable quantiles over a sliding time window.
+
+See [Prometheus metric types](https://prometheus.io/docs/concepts/metric_types/) for detailed explanations about these metric types.
+
+### Kata agent metrics
+
+Agent's metrics contains metrics about agent process.
+
+| Metric name | Type | Units | Labels | Introduced in Kata version |
+|---|---|---|---|---|
+| `kata_agent_io_stat`: <br> Agent process IO stat. | `GAUGE` |  | <ul><li>`item` (see `/proc/<pid>/io`)<ul><li>`cancelled_write_byte`</li><li>`rchar`</li><li>`read_bytes`</li><li>`syscr`</li><li>`syscw`</li><li>`wchar`</li><li>`write_bytes`</li></ul></li><li>`sandbox_id`</li></ul> | 2.0.0 |
+| `kata_agent_proc_stat`: <br> Agent process stat. | `GAUGE` |  | <ul><li>`item` (see `/proc/<pid>/stat`)<ul><li>`cstime`</li><li>`cutime`</li><li>`stime`</li><li>`utime`</li></ul></li><li>`sandbox_id`</li></ul> | 2.0.0 |
+| `kata_agent_proc_status`: <br> Agent process status. | `GAUGE` |  | <ul><li>`item` (see `/proc/<pid>/status`)<ul><li>`hugetlbpages`</li><li>`nonvoluntary_ctxt_switches`</li><li>`rssanon`</li><li>`rssfile`</li><li>`rssshmem`</li><li>`vmdata`</li><li>`vmexe`</li><li>`vmhwm`</li><li>`vmlck`</li><li>`vmlib`</li><li>`vmpeak`</li><li>`vmpin`</li><li>`vmpte`</li><li>`vmrss`</li><li>`vmsize`</li><li>`vmstk`</li><li>`vmswap`</li><li>`voluntary_ctxt_switches`</li></ul></li><li>`sandbox_id`</li></ul> | 2.0.0 |
+| `kata_agent_process_cpu_seconds_total`: <br> Total user and system CPU time spent in seconds. | `COUNTER` | `seconds` | <ul><li>`sandbox_id`</li></ul> | 2.0.0 |
+| `kata_agent_process_max_fds`: <br> Maximum number of open file descriptors. | `GAUGE` |  | <ul><li>`sandbox_id`</li></ul> | 2.0.0 |
+| `kata_agent_process_open_fds`: <br> Number of open file descriptors. | `GAUGE` |  | <ul><li>`sandbox_id`</li></ul> | 2.0.0 |
+| `kata_agent_process_resident_memory_bytes`: <br> Resident memory size in bytes. | `GAUGE` | `bytes` | <ul><li>`sandbox_id`</li></ul> | 2.0.0 |
+| `kata_agent_process_start_time_seconds`: <br> Start time of the process since `unix` epoch in seconds. | `GAUGE` | `seconds` | <ul><li>`sandbox_id`</li></ul> | 2.0.0 |
+| `kata_agent_process_virtual_memory_bytes`: <br> Virtual memory size in bytes. | `GAUGE` | `bytes` | <ul><li>`sandbox_id`</li></ul> | 2.0.0 |
+| `kata_agent_scrape_count`: <br> Metrics scrape count | `COUNTER` |  | <ul><li>`sandbox_id`</li></ul> | 2.0.0 |
+| `kata_agent_total_rss`: <br> Agent process total `rss` size | `GAUGE` |  | <ul><li>`sandbox_id`</li></ul> | 2.0.0 |
+| `kata_agent_total_time`: <br> Agent process total time | `GAUGE` |  | <ul><li>`sandbox_id`</li></ul> | 2.0.0 |
+| `kata_agent_total_vm`: <br> Agent process total `vm` size | `GAUGE` |  | <ul><li>`sandbox_id`</li></ul> | 2.0.0 |
+
+### Firecracker metrics
+
+Metrics for Firecracker vmm.
+
+| Metric name | Type | Units | Labels | Introduced in Kata version |
+|---|---|---|---|---|
+| `kata_firecracker_api_server`: <br> Metrics related to the internal API server. | `GAUGE` |  | <ul><li>`item`<ul><li>`process_startup_time_cpu_us`</li><li>`process_startup_time_us`</li><li>`sync_response_fails`</li><li>`sync_vmm_send_timeout_count`</li></ul></li><li>`sandbox_id`</li></ul> | 2.0.0 |
+| `kata_firecracker_block`: <br> Block Device associated metrics. | `GAUGE` |  | <ul><li>`item`<ul><li>`activate_fails`</li><li>`cfg_fails`</li><li>`event_fails`</li><li>`execute_fails`</li><li>`flush_count`</li><li>`invalid_reqs_count`</li><li>`no_avail_buffer`</li><li>`queue_event_count`</li><li>`rate_limiter_event_count`</li><li>`rate_limiter_throttled_events`</li><li>`read_bytes`</li><li>`read_count`</li><li>`update_count`</li><li>`update_fails`</li><li>`write_bytes`</li><li>`write_count`</li></ul></li><li>`sandbox_id`</li></ul> | 2.0.0 |
+| `kata_firecracker_get_api_requests`: <br> Metrics specific to GET API Requests for counting user triggered actions and/or failures. | `GAUGE` |  | <ul><li>`item`<ul><li>`instance_info_count`</li><li>`instance_info_fails`</li><li>`machine_cfg_count`</li><li>`machine_cfg_fails`</li></ul></li><li>`sandbox_id`</li></ul> | 2.0.0 |
+| `kata_firecracker_i8042`: <br> Metrics specific to the i8042 device. | `GAUGE` |  | <ul><li>`item`<ul><li>`error_count`</li><li>`missed_read_count`</li><li>`missed_write_count`</li><li>`read_count`</li><li>`reset_count`</li><li>`write_count`</li></ul></li><li>`sandbox_id`</li></ul> | 2.0.0 |
+| `kata_firecracker_latencies_us`: <br> Performance metrics related for the moment only to snapshots. | `GAUGE` |  | <ul><li>`item`<ul><li>`diff_create_snapshot`</li><li>`full_create_snapshot`</li><li>`load_snapshot`</li><li>`pause_vm`</li><li>`resume_vm`</li><li>`vmm_diff_create_snapshot`</li><li>`vmm_full_create_snapshot`</li><li>`vmm_load_snapshot`</li><li>`vmm_pause_vm`</li><li>`vmm_resume_vm`</li></ul></li><li>`sandbox_id`</li></ul> | 2.0.0 |
+| `kata_firecracker_logger`: <br> Metrics for the logging subsystem. | `GAUGE` |  | <ul><li>`item`<ul><li>`log_fails`</li><li>`metrics_fails`</li><li>`missed_log_count`</li><li>`missed_metrics_count`</li></ul></li><li>`sandbox_id`</li></ul> | 2.0.0 |
+| `kata_firecracker_mmds`: <br> Metrics for the MMDS functionality. | `GAUGE` |  | <ul><li>`item`<ul><li>`connections_created`</li><li>`connections_destroyed`</li><li>`rx_accepted`</li><li>`rx_accepted_err`</li><li>`rx_accepted_unusual`</li><li>`rx_bad_eth`</li><li>`rx_count`</li><li>`tx_bytes`</li><li>`tx_count`</li><li>`tx_errors`</li><li>`tx_frames`</li></ul></li><li>`sandbox_id`</li></ul> | 2.0.0 |
+| `kata_firecracker_net`: <br> Network-related metrics. | `GAUGE` |  | <ul><li>`item`<ul><li>`activate_fails`</li><li>`cfg_fails`</li><li>`event_fails`</li><li>`mac_address_updates`</li><li>`no_rx_avail_buffer`</li><li>`no_tx_avail_buffer`</li><li>`rx_bytes_count`</li><li>`rx_count`</li><li>`rx_event_rate_limiter_count`</li><li>`rx_fails`</li><li>`rx_packets_count`</li><li>`rx_partial_writes`</li><li>`rx_queue_event_count`</li><li>`rx_rate_limiter_throttled`</li><li>`rx_tap_event_count`</li><li>`tap_read_fails`</li><li>`tap_write_fails`</li><li>`tx_bytes_count`</li><li>`tx_count`</li><li>`tx_fails`</li><li>`tx_malformed_frames`</li><li>`tx_packets_count`</li><li>`tx_partial_reads`</li><li>`tx_queue_event_count`</li><li>`tx_rate_limiter_event_count`</li><li>`tx_rate_limiter_throttled`</li><li>`tx_spoofed_mac_count`</li></ul></li><li>`sandbox_id`</li></ul> | 2.0.0 |
+| `kata_firecracker_patch_api_requests`: <br> Metrics specific to PATCH API Requests for counting user triggered actions and/or failures. | `GAUGE` |  | <ul><li>`item`<ul><li>`drive_count`</li><li>`drive_fails`</li><li>`machine_cfg_count`</li><li>`machine_cfg_fails`</li><li>`network_count`</li><li>`network_fails`</li></ul></li><li>`sandbox_id`</li></ul> | 2.0.0 |
+| `kata_firecracker_put_api_requests`: <br> Metrics specific to PUT API Requests for counting user triggered actions and/or failures. | `GAUGE` |  | <ul><li>`item`<ul><li>`actions_count`</li><li>`actions_fails`</li><li>`boot_source_count`</li><li>`boot_source_fails`</li><li>`drive_count`</li><li>`drive_fails`</li><li>`logger_count`</li><li>`logger_fails`</li><li>`machine_cfg_count`</li><li>`machine_cfg_fails`</li><li>`metrics_count`</li><li>`metrics_fails`</li><li>`network_count`</li><li>`network_fails`</li></ul></li><li>`sandbox_id`</li></ul> | 2.0.0 |
+| `kata_firecracker_rtc`: <br> Metrics specific to the RTC device. | `GAUGE` |  | <ul><li>`item`<ul><li>`error_count`</li><li>`missed_read_count`</li><li>`missed_write_count`</li></ul></li><li>`sandbox_id`</li></ul> | 2.0.0 |
+| `kata_firecracker_seccomp`: <br> Metrics for the seccomp filtering. | `GAUGE` |  | <ul><li>`item`<ul><li>`num_faults`</li></ul></li><li>`sandbox_id`</li></ul> | 2.0.0 |
+| `kata_firecracker_signals`: <br> Metrics related to signals. | `GAUGE` |  | <ul><li>`item`<ul><li>`sigbus`</li><li>`sigsegv`</li></ul></li><li>`sandbox_id`</li></ul> | 2.0.0 |
+| `kata_firecracker_uart`: <br> Metrics specific to the UART device. | `GAUGE` |  | <ul><li>`item`<ul><li>`error_count`</li><li>`flush_count`</li><li>`missed_read_count`</li><li>`missed_write_count`</li><li>`read_count`</li><li>`write_count`</li></ul></li><li>`sandbox_id`</li></ul> | 2.0.0 |
+| `kata_firecracker_vcpu`: <br> Metrics specific to VCPUs' mode of functioning. | `GAUGE` |  | <ul><li>`item`<ul><li>`exit_io_in`</li><li>`exit_io_out`</li><li>`exit_mmio_read`</li><li>`exit_mmio_write`</li><li>`failures`</li><li>`filter_cpuid`</li></ul></li><li>`sandbox_id`</li></ul> | 2.0.0 |
+| `kata_firecracker_vmm`: <br> Metrics specific to the machine manager as a whole. | `GAUGE` |  | <ul><li>`item`<ul><li>`device_events`</li><li>`panic_count`</li></ul></li><li>`sandbox_id`</li></ul> | 2.0.0 |
+| `kata_firecracker_vsock`: <br> Vsock-related metrics. | `GAUGE` |  | <ul><li>`item`<ul><li>`activate_fails`</li><li>`cfg_fails`</li><li>`conn_event_fails`</li><li>`conns_added`</li><li>`conns_killed`</li><li>`conns_removed`</li><li>`ev_queue_event_fails`</li><li>`killq_resync`</li><li>`muxer_event_fails`</li><li>`rx_bytes_count`</li><li>`rx_packets_count`</li><li>`rx_queue_event_count`</li><li>`rx_queue_event_fails`</li><li>`rx_read_fails`</li><li>`tx_bytes_count`</li><li>`tx_flush_fails`</li><li>`tx_packets_count`</li><li>`tx_queue_event_count`</li><li>`tx_queue_event_fails`</li><li>`tx_write_fails`</li></ul></li><li>`sandbox_id`</li></ul> | 2.0.0 |
+
+### Kata guest OS metrics
+
+Guest OS's metrics in hypervisor.
+
+| Metric name | Type | Units | Labels | Introduced in Kata version |
+|---|---|---|---|---|
+| `kata_guest_cpu_time`: <br> Guest CPU stat. | `GAUGE` |  | <ul><li>`cpu` (CPU no. and total for all CPUs)<ul><li>`0` (CPU 0)</li><li>`1` (CPU 1)</li><li>`total` (for all CPUs)</li></ul></li><li>`item` (Kernel/system statistics, from `/proc/stat`)<ul><li>`guest`</li><li>`guest_nice`</li><li>`idle`</li><li>`iowait`</li><li>`irq`</li><li>`nice`</li><li>`softirq`</li><li>`steal`</li><li>`system`</li><li>`user`</li></ul></li><li>`sandbox_id`</li></ul> | 2.0.0 |
+| `kata_guest_diskstat`: <br> Disks stat in system. | `GAUGE` |  | <ul><li>`disk` (disk name)</li><li>`item` (see `/proc/diskstats`)<ul><li>`discards`</li><li>`discards_merged`</li><li>`flushes`</li><li>`in_progress`</li><li>`merged`</li><li>`reads`</li><li>`sectors_discarded`</li><li>`sectors_read`</li><li>`sectors_written`</li><li>`time_discarding`</li><li>`time_flushing`</li><li>`time_in_progress`</li><li>`time_reading`</li><li>`time_writing`</li><li>`weighted_time_in_progress`</li><li>`writes`</li><li>`writes_merged`</li></ul></li><li>`sandbox_id`</li></ul> | 2.0.0 |
+| `kata_guest_load`: <br> Guest system load. | `GAUGE` |  | <ul><li>`item`<ul><li>`load1`</li><li>`load15`</li><li>`load5`</li></ul></li><li>`sandbox_id`</li></ul> | 2.0.0 |
+| `kata_guest_meminfo`: <br> Statistics about memory usage on the system. | `GAUGE` |  | <ul><li>`item` (see `/proc/meminfo`)<ul><li>`active`</li><li>`active_anon`</li><li>`active_file`</li><li>`anon_hugepages`</li><li>`anon_pages`</li><li>`bounce`</li><li>`buffers`</li><li>`cached`</li><li>`cma_free`</li><li>`cma_total`</li><li>`commit_limit`</li><li>`committed_as`</li><li>`direct_map_1G`</li><li>`direct_map_2M`</li><li>`direct_map_4M`</li><li>`direct_map_4k`</li><li>`dirty`</li><li>`hardware_corrupted`</li><li>`high_free`</li><li>`high_total`</li><li>`hugepages_free`</li><li>`hugepages_rsvd`</li><li>`hugepages_surp`</li><li>`hugepages_total`</li><li>`hugepagesize`</li><li>`hugetlb`</li><li>`inactive`</li><li>`inactive_anon`</li><li>`inactive_file`</li><li>`k_reclaimable`</li><li>`kernel_stack`</li><li>`low_free`</li><li>`low_total`</li><li>`mapped`</li><li>`mem_available`</li><li>`mem_free`</li><li>`mem_total`</li><li>`mlocked`</li><li>`mmap_copy`</li><li>`nfs_unstable`</li><li>`page_tables`</li><li>`per_cpu`</li><li>`quicklists`</li><li>`s_reclaimable`</li><li>`s_unreclaim`</li><li>`shmem`</li><li>`shmem_hugepages`</li><li>`shmem_pmd_mapped`</li><li>`slab`</li><li>`swap_cached`</li><li>`swap_free`</li><li>`swap_total`</li><li>`unevictable`</li><li>`vmalloc_chunk`</li><li>`vmalloc_total`</li><li>`vmalloc_used`</li><li>`writeback`</li><li>`writeback_tmp`</li></ul></li><li>`sandbox_id`</li></ul> | 2.0.0 |
+| `kata_guest_netdev_stat`: <br> Guest net devices stats. | `GAUGE` |  | <ul><li>`interface` (network device name)</li><li>`item` (see `/proc/net/dev`)<ul><li>`recv_bytes`</li><li>`recv_compressed`</li><li>`recv_drop`</li><li>`recv_errs`</li><li>`recv_fifo`</li><li>`recv_frame`</li><li>`recv_multicast`</li><li>`recv_packets`</li><li>`sent_bytes`</li><li>`sent_carrier`</li><li>`sent_colls`</li><li>`sent_compressed`</li><li>`sent_drop`</li><li>`sent_errs`</li><li>`sent_fifo`</li><li>`sent_packets`</li></ul></li><li>`sandbox_id`</li></ul> | 2.0.0 |
+| `kata_guest_tasks`: <br> Guest system load. | `GAUGE` |  | <ul><li>`item`<ul><li>`cur`</li><li>`max`</li></ul></li><li>`sandbox_id`</li></ul> | 2.0.0 |
+| `kata_guest_vm_stat`: <br> Guest virtual memory stat. | `GAUGE` |  | <ul><li>`item` (see `/proc/vmstat`)<ul><li>`allocstall_dma`</li><li>`allocstall_dma32`</li><li>`allocstall_movable`</li><li>`allocstall_normal`</li><li>`balloon_deflate`</li><li>`balloon_inflate`</li><li>`compact_daemon_free_scanned`</li><li>`compact_daemon_migrate_scanned`</li><li>`compact_daemon_wake`</li><li>`compact_fail`</li><li>`compact_free_scanned`</li><li>`compact_isolated`</li><li>`compact_migrate_scanned`</li><li>`compact_stall`</li><li>`compact_success`</li><li>`drop_pagecache`</li><li>`drop_slab`</li><li>`htlb_buddy_alloc_fail`</li><li>`htlb_buddy_alloc_success`</li><li>`kswapd_high_wmark_hit_quickly`</li><li>`kswapd_inodesteal`</li><li>`kswapd_low_wmark_hit_quickly`</li><li>`nr_active_anon`</li><li>`nr_active_file`</li><li>`nr_anon_pages`</li><li>`nr_anon_transparent_hugepages`</li><li>`nr_bounce`</li><li>`nr_dirtied`</li><li>`nr_dirty`</li><li>`nr_dirty_background_threshold`</li><li>`nr_dirty_threshold`</li><li>`nr_file_pages`</li><li>`nr_free_cma`</li><li>`nr_free_pages`</li><li>`nr_inactive_anon`</li><li>`nr_inactive_file`</li><li>`nr_isolated_anon`</li><li>`nr_isolated_file`</li><li>`nr_kernel_stack`</li><li>`nr_mapped`</li><li>`nr_mlock`</li><li>`nr_page_table_pages`</li><li>`nr_shmem`</li><li>`nr_shmem_hugepages`</li><li>`nr_shmem_pmdmapped`</li><li>`nr_slab_reclaimable`</li><li>`nr_slab_unreclaimable`</li><li>`nr_unevictable`</li><li>`nr_unstable`</li><li>`nr_vmscan_immediate_reclaim`</li><li>`nr_vmscan_write`</li><li>`nr_writeback`</li><li>`nr_writeback_temp`</li><li>`nr_written`</li><li>`nr_zone_active_anon`</li><li>`nr_zone_active_file`</li><li>`nr_zone_inactive_anon`</li><li>`nr_zone_inactive_file`</li><li>`nr_zone_unevictable`</li><li>`nr_zone_write_pending`</li><li>`oom_kill`</li><li>`pageoutrun`</li><li>`pgactivate`</li><li>`pgalloc_dma`</li><li>`pgalloc_dma32`</li><li>`pgalloc_movable`</li><li>`pgalloc_normal`</li><li>`pgdeactivate`</li><li>`pgfault`</li><li>`pgfree`</li><li>`pginodesteal`</li><li>`pglazyfree`</li><li>`pglazyfreed`</li><li>`pgmajfault`</li><li>`pgmigrate_fail`</li><li>`pgmigrate_success`</li><li>`pgpgin`</li><li>`pgpgout`</li><li>`pgrefill`</li><li>`pgrotated`</li><li>`pgscan_direct`</li><li>`pgscan_direct_throttle`</li><li>`pgscan_kswapd`</li><li>`pgskip_dma`</li><li>`pgskip_dma32`</li><li>`pgskip_movable`</li><li>`pgskip_normal`</li><li>`pgsteal_direct`</li><li>`pgsteal_kswapd`</li><li>`pswpin`</li><li>`pswpout`</li><li>`slabs_scanned`</li><li>`swap_ra`</li><li>`swap_ra_hit`</li><li>`unevictable_pgs_cleared`</li><li>`unevictable_pgs_culled`</li><li>`unevictable_pgs_mlocked`</li><li>`unevictable_pgs_munlocked`</li><li>`unevictable_pgs_rescued`</li><li>`unevictable_pgs_scanned`</li><li>`unevictable_pgs_stranded`</li><li>`workingset_activate`</li><li>`workingset_nodereclaim`</li><li>`workingset_refault`</li></ul></li><li>`sandbox_id`</li></ul> | 2.0.0 |
+
+### Hypervisor metrics
+
+Hypervisors metrics, collected mainly from `proc` filesystem of hypervisor process.
+
+| Metric name | Type | Units | Labels | Introduced in Kata version |
+|---|---|---|---|---|
+| `kata_hypervisor_fds`: <br> Open FDs for hypervisor. | `GAUGE` |  | <ul><li>`sandbox_id`</li></ul> | 2.0.0 |
+| `kata_hypervisor_io_stat`: <br> Process IO statistics. | `GAUGE` |  | <ul><li>`item` (see `/proc/<pid>/io`)<ul><li>`cancelledwritebytes`</li><li>`rchar`</li><li>`readbytes`</li><li>`syscr`</li><li>`syscw`</li><li>`wchar`</li><li>`writebytes`</li></ul></li><li>`sandbox_id`</li></ul> | 2.0.0 |
+| `kata_hypervisor_netdev`: <br> Net devices statistics. | `GAUGE` |  | <ul><li>`interface` (network device name)</li><li>`item` (see `/proc/net/dev`)<ul><li>`recv_bytes`</li><li>`recv_compressed`</li><li>`recv_drop`</li><li>`recv_errs`</li><li>`recv_fifo`</li><li>`recv_frame`</li><li>`recv_multicast`</li><li>`recv_packets`</li><li>`sent_bytes`</li><li>`sent_carrier`</li><li>`sent_colls`</li><li>`sent_compressed`</li><li>`sent_drop`</li><li>`sent_errs`</li><li>`sent_fifo`</li><li>`sent_packets`</li></ul></li><li>`sandbox_id`</li></ul> | 2.0.0 |
+| `kata_hypervisor_proc_stat`: <br> Hypervisor process statistics. | `GAUGE` |  | <ul><li>`item` (see `/proc/<pid>/stat`)<ul><li>`cstime`</li><li>`cutime`</li><li>`stime`</li><li>`utime`</li></ul></li><li>`sandbox_id`</li></ul> | 2.0.0 |
+| `kata_hypervisor_proc_status`: <br> Hypervisor process status. | `GAUGE` |  | <ul><li>`item` (see `/proc/<pid>/status`)<ul><li>`hugetlbpages`</li><li>`nonvoluntary_ctxt_switches`</li><li>`rssanon`</li><li>`rssfile`</li><li>`rssshmem`</li><li>`vmdata`</li><li>`vmexe`</li><li>`vmhwm`</li><li>`vmlck`</li><li>`vmlib`</li><li>`vmpeak`</li><li>`vmpin`</li><li>`vmpmd`</li><li>`vmpte`</li><li>`vmrss`</li><li>`vmsize`</li><li>`vmstk`</li><li>`vmswap`</li><li>`voluntary_ctxt_switches`</li></ul></li><li>`sandbox_id`</li></ul> | 2.0.0 |
+| `kata_hypervisor_threads`: <br> Hypervisor process threads. | `GAUGE` |  | <ul><li>`sandbox_id`</li></ul> | 2.0.0 |
+
+### Kata monitor metrics
+
+Metrics about monitor itself.
+
+| Metric name | Type | Units | Labels | Introduced in Kata version |
+|---|---|---|---|---|
+| `kata_monitor_go_gc_duration_seconds`: <br> A summary of the pause duration of garbage collection cycles. | `SUMMARY` | `seconds` |  | 2.0.0 |
+| `kata_monitor_go_goroutines`: <br> Number of goroutines that currently exist. | `GAUGE` |  |  | 2.0.0 |
+| `kata_monitor_go_info`: <br> Information about the Go environment. | `GAUGE` |  | <ul><li>`version` (golang version)<ul><li>`go1.13.9` (environment dependent variable)</li></ul></li></ul> | 2.0.0 |
+| `kata_monitor_go_memstats_alloc_bytes`: <br> Number of bytes allocated and still in use. | `GAUGE` | `bytes` |  | 2.0.0 |
+| `kata_monitor_go_memstats_alloc_bytes_total`: <br> Total number of bytes allocated, even if freed. | `COUNTER` | `bytes` |  | 2.0.0 |
+| `kata_monitor_go_memstats_buck_hash_sys_bytes`: <br> Number of bytes used by the profiling bucket hash table. | `GAUGE` | `bytes` |  | 2.0.0 |
+| `kata_monitor_go_memstats_frees_total`: <br> Total number of frees. | `COUNTER` |  |  | 2.0.0 |
+| `kata_monitor_go_memstats_gc_cpu_fraction`: <br> The fraction of this program's available CPU time used by the GC since the program started. | `GAUGE` |  |  | 2.0.0 |
+| `kata_monitor_go_memstats_gc_sys_bytes`: <br> Number of bytes used for garbage collection system metadata. | `GAUGE` | `bytes` |  | 2.0.0 |
+| `kata_monitor_go_memstats_heap_alloc_bytes`: <br> Number of heap bytes allocated and still in use. | `GAUGE` | `bytes` |  | 2.0.0 |
+| `kata_monitor_go_memstats_heap_idle_bytes`: <br> Number of heap bytes waiting to be used. | `GAUGE` | `bytes` |  | 2.0.0 |
+| `kata_monitor_go_memstats_heap_inuse_bytes`: <br> Number of heap bytes that are in use. | `GAUGE` | `bytes` |  | 2.0.0 |
+| `kata_monitor_go_memstats_heap_objects`: <br> Number of allocated objects. | `GAUGE` |  |  | 2.0.0 |
+| `kata_monitor_go_memstats_heap_released_bytes`: <br> Number of heap bytes released to OS. | `GAUGE` | `bytes` |  | 2.0.0 |
+| `kata_monitor_go_memstats_heap_sys_bytes`: <br> Number of heap bytes obtained from system. | `GAUGE` | `bytes` |  | 2.0.0 |
+| `kata_monitor_go_memstats_last_gc_time_seconds`: <br> Number of seconds since 1970 of last garbage collection. | `GAUGE` | `seconds` |  | 2.0.0 |
+| `kata_monitor_go_memstats_lookups_total`: <br> Total number of pointer lookups. | `COUNTER` |  |  | 2.0.0 |
+| `kata_monitor_go_memstats_mallocs_total`: <br> Total number of `mallocs`. | `COUNTER` |  |  | 2.0.0 |
+| `kata_monitor_go_memstats_mcache_inuse_bytes`: <br> Number of bytes in use by `mcache` structures. | `GAUGE` | `bytes` |  | 2.0.0 |
+| `kata_monitor_go_memstats_mcache_sys_bytes`: <br> Number of bytes used for `mcache` structures obtained from system. | `GAUGE` | `bytes` |  | 2.0.0 |
+| `kata_monitor_go_memstats_mspan_inuse_bytes`: <br> Number of bytes in use by `mspan` structures. | `GAUGE` | `bytes` |  | 2.0.0 |
+| `kata_monitor_go_memstats_mspan_sys_bytes`: <br> Number of bytes used for `mspan` structures obtained from system. | `GAUGE` | `bytes` |  | 2.0.0 |
+| `kata_monitor_go_memstats_next_gc_bytes`: <br> Number of heap bytes when next garbage collection will take place. | `GAUGE` | `bytes` |  | 2.0.0 |
+| `kata_monitor_go_memstats_other_sys_bytes`: <br> Number of bytes used for other system allocations. | `GAUGE` | `bytes` |  | 2.0.0 |
+| `kata_monitor_go_memstats_stack_inuse_bytes`: <br> Number of bytes in use by the stack allocator. | `GAUGE` | `bytes` |  | 2.0.0 |
+| `kata_monitor_go_memstats_stack_sys_bytes`: <br> Number of bytes obtained from system for stack allocator. | `GAUGE` | `bytes` |  | 2.0.0 |
+| `kata_monitor_go_memstats_sys_bytes`: <br> Number of bytes obtained from system. | `GAUGE` | `bytes` |  | 2.0.0 |
+| `kata_monitor_go_threads`: <br> Number of OS threads created. | `GAUGE` |  |  | 2.0.0 |
+| `kata_monitor_process_cpu_seconds_total`: <br> Total user and system CPU time spent in seconds. | `COUNTER` | `seconds` |  | 2.0.0 |
+| `kata_monitor_process_max_fds`: <br> Maximum number of open file descriptors. | `GAUGE` |  |  | 2.0.0 |
+| `kata_monitor_process_open_fds`: <br> Number of open file descriptors. | `GAUGE` |  |  | 2.0.0 |
+| `kata_monitor_process_resident_memory_bytes`: <br> Resident memory size in bytes. | `GAUGE` | `bytes` |  | 2.0.0 |
+| `kata_monitor_process_start_time_seconds`: <br> Start time of the process since `unix` epoch in seconds. | `GAUGE` | `seconds` |  | 2.0.0 |
+| `kata_monitor_process_virtual_memory_bytes`: <br> Virtual memory size in bytes. | `GAUGE` | `bytes` |  | 2.0.0 |
+| `kata_monitor_process_virtual_memory_max_bytes`: <br> Maximum amount of virtual memory available in bytes. | `GAUGE` | `bytes` |  | 2.0.0 |
+| `kata_monitor_running_shim_count`: <br> Running shim count(running sandboxes). | `GAUGE` |  |  | 2.0.0 |
+| `kata_monitor_scrape_count`: <br> Scape count. | `COUNTER` |  |  | 2.0.0 |
+| `kata_monitor_scrape_durations_histogram_milliseconds`: <br> Time used to scrape from shims | `HISTOGRAM` | `milliseconds` |  | 2.0.0 |
+| `kata_monitor_scrape_failed_count`: <br> Failed scape count. | `COUNTER` |  |  | 2.0.0 |
+
+### Kata containerd shim v2 metrics
+
+Metrics about Kata containerd shim v2 process.
+
+| Metric name | Type | Units | Labels | Introduced in Kata version |
+|---|---|---|---|---|
+| `kata_shim_agent_rpc_durations_histogram_milliseconds`: <br> RPC latency distributions. | `HISTOGRAM` | `milliseconds` | <ul><li>`action` (RPC actions of Kata agent)<ul><li>`grpc.CheckRequest`</li><li>`grpc.CloseStdinRequest`</li><li>`grpc.CopyFileRequest`</li><li>`grpc.CreateContainerRequest`</li><li>`grpc.CreateSandboxRequest`</li><li>`grpc.DestroySandboxRequest`</li><li>`grpc.ExecProcessRequest`</li><li>`grpc.GetMetricsRequest`</li><li>`grpc.GuestDetailsRequest`</li><li>`grpc.ListInterfacesRequest`</li><li>`grpc.ListProcessesRequest`</li><li>`grpc.ListRoutesRequest`</li><li>`grpc.MemHotplugByProbeRequest`</li><li>`grpc.OnlineCPUMemRequest`</li><li>`grpc.PauseContainerRequest`</li><li>`grpc.RemoveContainerRequest`</li><li>`grpc.ReseedRandomDevRequest`</li><li>`grpc.ResumeContainerRequest`</li><li>`grpc.SetGuestDateTimeRequest`</li><li>`grpc.SignalProcessRequest`</li><li>`grpc.StartContainerRequest`</li><li>`grpc.StartTracingRequest`</li><li>`grpc.StatsContainerRequest`</li><li>`grpc.StopTracingRequest`</li><li>`grpc.TtyWinResizeRequest`</li><li>`grpc.UpdateContainerRequest`</li><li>`grpc.UpdateInterfaceRequest`</li><li>`grpc.UpdateRoutesRequest`</li><li>`grpc.WaitProcessRequest`</li><li>`grpc.WriteStreamRequest`</li></ul></li><li>`sandbox_id`</li></ul> | 2.0.0 |
+| `kata_shim_fds`: <br> Kata containerd shim v2 open FDs. | `GAUGE` |  | <ul><li>`sandbox_id`</li></ul> | 2.0.0 |
+| `kata_shim_go_gc_duration_seconds`: <br> A summary of the pause duration of garbage collection cycles. | `SUMMARY` | `seconds` | <ul><li>`sandbox_id`</li></ul> | 2.0.0 |
+| `kata_shim_go_goroutines`: <br> Number of goroutines that currently exist. | `GAUGE` |  | <ul><li>`sandbox_id`</li></ul> | 2.0.0 |
+| `kata_shim_go_info`: <br> Information about the Go environment. | `GAUGE` |  | <ul><li>`sandbox_id`</li><li>`version` (golang version)<ul><li>`go1.13.9` (environment dependent variable)</li></ul></li></ul> | 2.0.0 |
+| `kata_shim_go_memstats_alloc_bytes`: <br> Number of bytes allocated and still in use. | `GAUGE` | `bytes` | <ul><li>`sandbox_id`</li></ul> | 2.0.0 |
+| `kata_shim_go_memstats_alloc_bytes_total`: <br> Total number of bytes allocated, even if freed. | `COUNTER` | `bytes` | <ul><li>`sandbox_id`</li></ul> | 2.0.0 |
+| `kata_shim_go_memstats_buck_hash_sys_bytes`: <br> Number of bytes used by the profiling bucket hash table. | `GAUGE` | `bytes` | <ul><li>`sandbox_id`</li></ul> | 2.0.0 |
+| `kata_shim_go_memstats_frees_total`: <br> Total number of frees. | `COUNTER` |  | <ul><li>`sandbox_id`</li></ul> | 2.0.0 |
+| `kata_shim_go_memstats_gc_cpu_fraction`: <br> The fraction of this program's available CPU time used by the GC since the program started. | `GAUGE` |  | <ul><li>`sandbox_id`</li></ul> | 2.0.0 |
+| `kata_shim_go_memstats_gc_sys_bytes`: <br> Number of bytes used for garbage collection system metadata. | `GAUGE` | `bytes` | <ul><li>`sandbox_id`</li></ul> | 2.0.0 |
+| `kata_shim_go_memstats_heap_alloc_bytes`: <br> Number of heap bytes allocated and still in use. | `GAUGE` | `bytes` | <ul><li>`sandbox_id`</li></ul> | 2.0.0 |
+| `kata_shim_go_memstats_heap_idle_bytes`: <br> Number of heap bytes waiting to be used. | `GAUGE` | `bytes` | <ul><li>`sandbox_id`</li></ul> | 2.0.0 |
+| `kata_shim_go_memstats_heap_inuse_bytes`: <br> Number of heap bytes that are in use. | `GAUGE` | `bytes` | <ul><li>`sandbox_id`</li></ul> | 2.0.0 |
+| `kata_shim_go_memstats_heap_objects`: <br> Number of allocated objects. | `GAUGE` |  | <ul><li>`sandbox_id`</li></ul> | 2.0.0 |
+| `kata_shim_go_memstats_heap_released_bytes`: <br> Number of heap bytes released to OS. | `GAUGE` | `bytes` | <ul><li>`sandbox_id`</li></ul> | 2.0.0 |
+| `kata_shim_go_memstats_heap_sys_bytes`: <br> Number of heap bytes obtained from system. | `GAUGE` | `bytes` | <ul><li>`sandbox_id`</li></ul> | 2.0.0 |
+| `kata_shim_go_memstats_last_gc_time_seconds`: <br> Number of seconds since 1970 of last garbage collection. | `GAUGE` | `seconds` | <ul><li>`sandbox_id`</li></ul> | 2.0.0 |
+| `kata_shim_go_memstats_lookups_total`: <br> Total number of pointer lookups. | `COUNTER` |  | <ul><li>`sandbox_id`</li></ul> | 2.0.0 |
+| `kata_shim_go_memstats_mallocs_total`: <br> Total number of `mallocs`. | `COUNTER` |  | <ul><li>`sandbox_id`</li></ul> | 2.0.0 |
+| `kata_shim_go_memstats_mcache_inuse_bytes`: <br> Number of bytes in use by `mcache` structures. | `GAUGE` | `bytes` | <ul><li>`sandbox_id`</li></ul> | 2.0.0 |
+| `kata_shim_go_memstats_mcache_sys_bytes`: <br> Number of bytes used for `mcache` structures obtained from system. | `GAUGE` | `bytes` | <ul><li>`sandbox_id`</li></ul> | 2.0.0 |
+| `kata_shim_go_memstats_mspan_inuse_bytes`: <br> Number of bytes in use by `mspan` structures. | `GAUGE` | `bytes` | <ul><li>`sandbox_id`</li></ul> | 2.0.0 |
+| `kata_shim_go_memstats_mspan_sys_bytes`: <br> Number of bytes used for `mspan` structures obtained from system. | `GAUGE` | `bytes` | <ul><li>`sandbox_id`</li></ul> | 2.0.0 |
+| `kata_shim_go_memstats_next_gc_bytes`: <br> Number of heap bytes when next garbage collection will take place. | `GAUGE` | `bytes` | <ul><li>`sandbox_id`</li></ul> | 2.0.0 |
+| `kata_shim_go_memstats_other_sys_bytes`: <br> Number of bytes used for other system allocations. | `GAUGE` | `bytes` | <ul><li>`sandbox_id`</li></ul> | 2.0.0 |
+| `kata_shim_go_memstats_stack_inuse_bytes`: <br> Number of bytes in use by the stack allocator. | `GAUGE` | `bytes` | <ul><li>`sandbox_id`</li></ul> | 2.0.0 |
+| `kata_shim_go_memstats_stack_sys_bytes`: <br> Number of bytes obtained from system for stack allocator. | `GAUGE` | `bytes` | <ul><li>`sandbox_id`</li></ul> | 2.0.0 |
+| `kata_shim_go_memstats_sys_bytes`: <br> Number of bytes obtained from system. | `GAUGE` | `bytes` | <ul><li>`sandbox_id`</li></ul> | 2.0.0 |
+| `kata_shim_go_threads`: <br> Number of OS threads created. | `GAUGE` |  | <ul><li>`sandbox_id`</li></ul> | 2.0.0 |
+| `kata_shim_io_stat`: <br> Kata containerd shim v2 process IO statistics. | `GAUGE` |  | <ul><li>`item` (see `/proc/<pid>/io`)<ul><li>`cancelledwritebytes`</li><li>`rchar`</li><li>`readbytes`</li><li>`syscr`</li><li>`syscw`</li><li>`wchar`</li><li>`writebytes`</li></ul></li><li>`sandbox_id`</li></ul> | 2.0.0 |
+| `kata_shim_netdev`: <br> Kata containerd shim v2 network devices statistics. | `GAUGE` |  | <ul><li>`interface` (network device name)</li><li>`item` (see `/proc/net/dev`)<ul><li>`recv_bytes`</li><li>`recv_compressed`</li><li>`recv_drop`</li><li>`recv_errs`</li><li>`recv_fifo`</li><li>`recv_frame`</li><li>`recv_multicast`</li><li>`recv_packets`</li><li>`sent_bytes`</li><li>`sent_carrier`</li><li>`sent_colls`</li><li>`sent_compressed`</li><li>`sent_drop`</li><li>`sent_errs`</li><li>`sent_fifo`</li><li>`sent_packets`</li></ul></li><li>`sandbox_id`</li></ul> | 2.0.0 |
+| `kata_shim_pod_overhead_cpu`: <br> Kata Pod overhead for CPU resources(percent). | `GAUGE` | percent | <ul><li>`sandbox_id`</li></ul> | 2.0.0 |
+| `kata_shim_pod_overhead_memory_in_bytes`: <br> Kata Pod overhead for memory resources(bytes). | `GAUGE` | `bytes` | <ul><li>`sandbox_id`</li></ul> | 2.0.0 |
+| `kata_shim_proc_stat`: <br> Kata containerd shim v2 process statistics. | `GAUGE` |  | <ul><li>`item` (see `/proc/<pid>/stat`)<ul><li>`cstime`</li><li>`cutime`</li><li>`stime`</li><li>`utime`</li></ul></li><li>`sandbox_id`</li></ul> | 2.0.0 |
+| `kata_shim_proc_status`: <br> Kata containerd shim v2 process status. | `GAUGE` |  | <ul><li>`item` (see `/proc/<pid>/status`)<ul><li>`hugetlbpages`</li><li>`nonvoluntary_ctxt_switches`</li><li>`rssanon`</li><li>`rssfile`</li><li>`rssshmem`</li><li>`vmdata`</li><li>`vmexe`</li><li>`vmhwm`</li><li>`vmlck`</li><li>`vmlib`</li><li>`vmpeak`</li><li>`vmpin`</li><li>`vmpmd`</li><li>`vmpte`</li><li>`vmrss`</li><li>`vmsize`</li><li>`vmstk`</li><li>`vmswap`</li><li>`voluntary_ctxt_switches`</li></ul></li><li>`sandbox_id`</li></ul> | 2.0.0 |
+| `kata_shim_process_cpu_seconds_total`: <br> Total user and system CPU time spent in seconds. | `COUNTER` | `seconds` | <ul><li>`sandbox_id`</li></ul> | 2.0.0 |
+| `kata_shim_process_max_fds`: <br> Maximum number of open file descriptors. | `GAUGE` |  | <ul><li>`sandbox_id`</li></ul> | 2.0.0 |
+| `kata_shim_process_open_fds`: <br> Number of open file descriptors. | `GAUGE` |  | <ul><li>`sandbox_id`</li></ul> | 2.0.0 |
+| `kata_shim_process_resident_memory_bytes`: <br> Resident memory size in bytes. | `GAUGE` | `bytes` | <ul><li>`sandbox_id`</li></ul> | 2.0.0 |
+| `kata_shim_process_start_time_seconds`: <br> Start time of the process since `unix` epoch in seconds. | `GAUGE` | `seconds` | <ul><li>`sandbox_id`</li></ul> | 2.0.0 |
+| `kata_shim_process_virtual_memory_bytes`: <br> Virtual memory size in bytes. | `GAUGE` | `bytes` | <ul><li>`sandbox_id`</li></ul> | 2.0.0 |
+| `kata_shim_process_virtual_memory_max_bytes`: <br> Maximum amount of virtual memory available in bytes. | `GAUGE` | `bytes` | <ul><li>`sandbox_id`</li></ul> | 2.0.0 |
+| `kata_shim_rpc_durations_histogram_milliseconds`: <br> RPC latency distributions. | `HISTOGRAM` | `milliseconds` | <ul><li>`action` (Kata shim v2 actions)<ul><li>`checkpoint`</li><li>`close_io`</li><li>`connect`</li><li>`create`</li><li>`delete`</li><li>`exec`</li><li>`kill`</li><li>`pause`</li><li>`pids`</li><li>`resize_pty`</li><li>`resume`</li><li>`shutdown`</li><li>`start`</li><li>`state`</li><li>`stats`</li><li>`update`</li><li>`wait`</li></ul></li><li>`sandbox_id`</li></ul> | 2.0.0 |
+| `kata_shim_threads`: <br> Kata containerd shim v2 process threads. | `GAUGE` |  | <ul><li>`sandbox_id`</li></ul> | 2.0.0 |
+
+

docs/design/kata-api-design.md

@@ -10,43 +10,56 @@ To fulfill the [Kata design requirements](kata-design-requirements.md), and base
 
 |Name|Description|
 |---|---|
-|`CreateSandbox(SandboxConfig)`| Create and start a sandbox, and return the sandbox structure.|
-|`FetchSandbox(ID)`| Connect to an existing sandbox and return the sandbox structure.|
-|`ListSandboxes()`| List all existing sandboxes with status. |
+|`CreateSandbox(SandboxConfig, Factory)`| Create a sandbox and its containers, base on `SandboxConfig` and `Factory`. Return the `Sandbox` structure, but do not start them.|
 
 ### Sandbox Operation API
 
 |Name|Description|
 |---|---|
-|`sandbox.Pause()`| Pause the sandbox.|
-|`sandbox.Resume()`| Resume the paused sandbox.|
-|`sandbox.Release()`| Release a sandbox data structure, close connections to the agent, and quit any goroutines associated with the sandbox. Mostly used for daemon restart.|
-|`sandbox.Delete()`| Destroy the sandbox and remove all persistent metadata.|
-|`sandbox.Status()`| Get the status of the sandbox and containers.|
+|`sandbox.Delete()`| Shut down the VM in which the sandbox, and destroy the sandbox and remove all persistent metadata.|
 |`sandbox.Monitor()`| Return a context handler for caller to monitor sandbox callbacks such as error termination.|
-|`sandbox.CreateContainer()`| Create new container in the sandbox.|
-|`sandbox.DeleteContainer()`| Delete a container from the sandbox.|
-|`sandbox.StartContainer()`| Start a container in the sandbox.|
-|`sandbox.StatusContainer()`| Get the status of a container in the sandbox.|
-|`sandbox.EnterContainer()`| Run a new process in a container.|
-|`sandbox.WaitProcess()`| Wait on a process to terminate.|
+|`sandbox.Release()`| Release a sandbox data structure, close connections to the agent, and quit any goroutines associated with the Sandbox. Mostly used for daemon restart.|
+|`sandbox.Start()`| Start a sandbox and the containers making the sandbox.|
+|`sandbox.Stats()`| Get the stats of a running sandbox, return a `SandboxStats` structure.|
+|`sandbox.Status()`| Get the status of the sandbox and containers, return a `SandboxStatus` structure.|
+|`sandbox.Stop(force)`| Stop a sandbox and Destroy the containers in the sandbox. When force is true, ignore guest related stop failures.|
+|`sandbox.CreateContainer(contConfig)`| Create new container in the sandbox with the `ContainerConfig` param. It will add new container config to `sandbox.config.Containers`.|
+|`sandbox.DeleteContainer(containerID)`| Delete a container from the sandbox by containerID, return a `Container` structure.|
+|`sandbox.EnterContainer(containerID, cmd)`| Run a new process in a container, executing customer's `types.Cmd` command.|
+|`sandbox.KillContainer(containerID, signal, all)`| Signal a container in the sandbox by the containerID.|
+|`sandbox.PauseContainer(containerID)`| Pause a running container in the sandbox by the containerID.|
+|`sandbox.ProcessListContainer(containerID, options)`| List every process running inside a specific container in the sandbox, return a `ProcessList` structure.|
+|`sandbox.ResumeContainer(containerID)`| Resume a paused container in the sandbox by the containerID.|
+|`sandbox.StartContainer(containerID)`| Start a container in the sandbox by the containerID.|
+|`sandbox.StatsContainer(containerID)`| Get the stats of a running container, return a `ContainerStats` structure.|
+|`sandbox.StatusContainer(containerID)`| Get the status of a container in the sandbox, return a `ContainerStatus` structure.|
+|`sandbox.StopContainer(containerID, force)`| Stop a container in the sandbox by the containerID.|
+|`sandbox.UpdateContainer(containerID, resources)`| Update a running container in the sandbox.|
+|`sandbox.WaitProcess(containerID, processID)`| Wait on a process to terminate.|
 ### Sandbox Hotplug API
 |Name|Description|
 |---|---|
-|`sandbox.AddDevice()`| Add new storage device to the sandbox.|
-|`sandbox.AddInterface()`| Add new NIC to the sandbox.|
-|`sandbox.RemoveInterface()`| Remove a NIC from the sandbox.|
-|`sandbox.ListInterfaces()`| List all NICs and their configurations in the sandbox.|
-|`sandbox.UpdateRoutes()`| Update the sandbox route table (e.g. for portmapping support).|
-|`sandbox.ListRoutes()`| List the sandbox route table.|
+|`sandbox.AddDevice(info)`| Add new storage device `DeviceInfo` to the sandbox, return a `Device` structure.|
+|`sandbox.AddInterface(inf)`| Add new NIC to the sandbox.|
+|`sandbox.RemoveInterface(inf)`| Remove a NIC from the sandbox.|
+|`sandbox.ListInterfaces()`| List all NICs and their configurations in the sandbox, return a `pbTypes.Interface` list.|
+|`sandbox.UpdateRoutes(routes)`| Update the sandbox route table (e.g. for portmapping support), return a `pbTypes.Route` list.|
+|`sandbox.ListRoutes()`| List the sandbox route table, return a `pbTypes.Route` list.|
 
 ### Sandbox Relay API
 |Name|Description|
 |---|---|
-|`sandbox.WinsizeProcess(containerID, processID, Height, Width)`|Relay TTY resize request to a process.|
+|`sandbox.WinsizeProcess(containerID, processID, Height, Width)`| Relay TTY resize request to a process.|
 |`sandbox.SignalProcess(containerID, processID, signalID, signalALL)`| Relay a signal to a process or all processes in a container.|
 |`sandbox.IOStream(containerID, processID)`| Relay a process stdio. Return stdin/stdout/stderr pipes to the process stdin/stdout/stderr streams.|
 
+### Sandbox Monitor API
+|Name|Description|
+|---|---|
+|`sandbox.GetOOMEvent()`| Monitor the OOM events that occur in the sandbox..|
+|`sandbox.UpdateRuntimeMetrics()`| Update the shim/hypervisor's metrics of the running sandbox.|
+|`sandbox.GetAgentMetrics()`| Get metrics of the agent and the guest in the running sandbox.|
+
 ## Plugin framework for external proprietary Kata runtime extensions
 ### Hypervisor plugin
 

docs/design/vcpu-handling.md

@@ -170,6 +170,6 @@ docker run --cpus 4 -ti debian bash -c "nproc; cat /sys/fs/cgroup/cpu,cpuacct/cp
 [2]: https://kubernetes.io/docs/tasks/configure-pod-container/assign-cpu-resource
 [3]: https://kubernetes.io/docs/concepts/workloads/pods/pod/
 [4]: https://docs.docker.com/engine/reference/commandline/update/
-[5]: https://github.com/kata-containers/agent
-[6]: https://github.com/kata-containers/runtime
-[7]: https://github.com/kata-containers/runtime#configuration
+[5]: ../../src/agent
+[6]: ../../src/runtime
+[7]: ../../src/runtime/README.md#configuration

docs/how-to/README.md

@@ -26,3 +26,4 @@
 - [How to load kernel modules in Kata Containers](how-to-load-kernel-modules-with-kata.md)
 - [How to use Kata Containers with `virtio-mem`](how-to-use-virtio-mem-with-kata.md)
 - [How to set sandbox Kata Containers configurations with pod annotations](how-to-set-sandbox-config-kata.md)
+- [How to monitor Kata Containers in K8s](how-to-set-prometheus-in-k8s.md)

docs/how-to/containerd-kata.md

@@ -57,7 +57,7 @@ use `RuntimeClass` instead of the deprecated annotations.
 
 ### Containerd Runtime V2 API: Shim V2 API
 
-The [`containerd-shim-kata-v2` (short as `shimv2` in this documentation)](https://github.com/kata-containers/runtime/tree/master/containerd-shim-v2) 
+The [`containerd-shim-kata-v2` (short as `shimv2` in this documentation)](../../src/runtime/containerd-shim-v2) 
 implements the [Containerd Runtime V2 (Shim API)](https://github.com/containerd/containerd/tree/master/runtime/v2) for Kata.
 With `shimv2`, Kubernetes can launch Pod and OCI-compatible containers with one shim per Pod. Prior to `shimv2`, `2N+1` 
 shims (i.e. a `containerd-shim` and a `kata-shim` for each container and the Pod sandbox itself) and no standalone `kata-proxy` 
@@ -72,7 +72,7 @@ is implemented in Kata Containers v1.5.0.
 
 ### Install Kata Containers
 
-Follow the instructions to [install Kata Containers](https://github.com/kata-containers/documentation/blob/master/install/README.md).
+Follow the instructions to [install Kata Containers](../install/README.md).
 
 ### Install containerd with CRI plugin
 
@@ -193,13 +193,16 @@ From Containerd v1.2.4 and Kata v1.6.0, there is a new runtime option supported,
 ```toml
       [plugins.cri.containerd.runtimes.kata]
          runtime_type = "io.containerd.kata.v2"
+	 privileged_without_host_devices = true
 	 [plugins.cri.containerd.runtimes.kata.options]
 	   ConfigPath = "/etc/kata-containers/config.toml"
 ```
 
+`privileged_without_host_devices` tells containerd that a privileged Kata container should not have direct access to all host devices. If unset, containerd will pass all host devices to Kata container, which may cause security issues.
+
 This `ConfigPath` option is optional. If you do not specify it, shimv2 first tries to get the configuration file from the environment variable `KATA_CONF_FILE`. If neither are set, shimv2 will use the default Kata configuration file paths (`/etc/kata-containers/configuration.toml` and `/usr/share/defaults/kata-containers/configuration.toml`).
 
-If you use Containerd older than v1.2.4 or a version of Kata older than v1.6.0  and also want to specify a configuration file, you can use the following workaround, since the shimv2 accepts an environment variable, `KATA_CONF_FILE` for the configuration file path. Then, you can create a 
+If you use Containerd older than v1.2.4 or a version of Kata older than v1.6.0  and also want to specify a configuration file, you can use the following workaround, since the shimv2 accepts an environment variable, `KATA_CONF_FILE` for the configuration file path. Then, you can create a
 shell script with the following:
 
 ```bash

docs/how-to/data/grafana.yml

@@ -0,0 +1,43 @@
+apiVersion: apps/v1
+kind: Deployment
+metadata:
+  name: grafana
+  namespace: prometheus
+  labels:
+    app: grafana
+spec:
+  replicas: 1
+  selector:
+    matchLabels:
+      app: grafana
+  template:
+    metadata:
+      labels:
+        app: grafana
+    spec:
+      containers:
+      - image: grafana/grafana:7.0.5
+        name: grafana
+        ports:
+        - containerPort: 3000
+          name: http
+
+---
+
+apiVersion: v1
+kind: Service
+metadata:
+  namespace: prometheus
+  name: grafana
+  labels:
+    app: grafana
+spec:
+  type: NodePort
+  selector:
+    app: grafana
+  ports:
+  - port: 3000
+    targetPort: 3000
+    name: http
+    nodePort: 30000
+    protocol: TCP

docs/how-to/data/kata-monitor-daemonset.yml

@@ -0,0 +1,55 @@
+apiVersion: v1
+kind: Namespace
+metadata:
+  name: kata-system
+
+---
+
+apiVersion: apps/v1
+kind: DaemonSet
+metadata:
+  labels:
+    app.kubernetes.io/name: kata-monitor
+  name: kata-monitor
+  namespace: kata-system
+spec:
+  selector:
+    matchLabels:
+      app.kubernetes.io/name: kata-monitor
+  template:
+    metadata:
+      labels:
+        app.kubernetes.io/name: kata-monitor
+      annotations:
+        prometheus.io/scrape: "true"
+    spec:
+      hostNetwork: true
+      containers:
+      - name: kata-monitor
+        image: docker.io/katadocker/kata-monitor:2.0.0
+        args: 
+          - -log-level=debug
+        ports:
+          - containerPort: 8090
+        resources:
+          limits:
+            cpu: 200m
+            memory: 300Mi
+          requests:
+            cpu: 200m
+            memory: 300Mi
+        volumeMounts:
+        - name: containerdtask
+          mountPath: /run/containerd/io.containerd.runtime.v2.task/
+          readOnly: true
+        - name: containerdsocket
+          mountPath: /run/containerd/containerd.sock
+          readOnly: true
+      terminationGracePeriodSeconds: 30
+      volumes:
+      - name: containerdtask
+        hostPath:
+          path: /run/containerd/io.containerd.runtime.v2.task/
+      - name: containerdsocket
+        hostPath:
+          path: /run/containerd/containerd.sock

docs/how-to/data/prometheus.yml

@@ -0,0 +1,132 @@
+apiVersion: v1
+kind: Namespace
+metadata:
+  name: prometheus
+
+---
+
+apiVersion: rbac.authorization.k8s.io/v1beta1
+kind: ClusterRole
+metadata:
+  name: prometheus
+rules:
+- apiGroups: [""]
+  resources:
+  - nodes
+  - nodes/proxy
+  - services
+  - endpoints
+  - pods
+  verbs: ["get", "list", "watch"]
+- apiGroups:
+  - extensions
+  resources:
+  - ingresses
+  verbs: ["get", "list", "watch"]
+- nonResourceURLs: ["/metrics"]
+  verbs: ["get"]
+
+---
+
+apiVersion: v1
+kind: ServiceAccount
+metadata:
+  name: prometheus
+  namespace: prometheus
+
+---
+
+apiVersion: rbac.authorization.k8s.io/v1beta1
+kind: ClusterRoleBinding
+metadata:
+  name: prometheus
+roleRef:
+  apiGroup: rbac.authorization.k8s.io
+  kind: ClusterRole
+  name: prometheus
+subjects:
+- kind: ServiceAccount
+  name: prometheus
+  namespace: prometheus
+
+---
+
+kind: Service
+apiVersion: v1
+metadata:
+  name: prometheus
+  namespace: prometheus
+  labels:
+    app: prometheus
+spec:
+  type: NodePort
+  selector:
+    app: prometheus
+  ports:
+  - port: 9090
+    targetPort: 9090
+    name: http
+    nodePort: 30909
+    protocol: TCP
+
+---
+
+kind: Deployment
+apiVersion: apps/v1
+metadata:
+  name: prometheus
+  namespace: prometheus
+spec:
+  replicas: 1
+  selector:
+    matchLabels: 
+      app: prometheus
+  template:
+    metadata:
+      labels:
+        app: prometheus
+    spec:
+      serviceAccountName: prometheus
+      containers:
+      - name: prometheus
+        image: prom/prometheus:v2.7.1
+        ports:
+        - containerPort: 9090
+        volumeMounts:
+        - name: prometheus-config-volume
+          mountPath: /etc/prometheus/prometheus.yml
+          subPath: prometheus.yml
+      volumes:
+      - name: prometheus-config-volume
+        configMap:
+          name: prometheus-config
+      restartPolicy: Always
+
+---
+
+kind: ConfigMap
+apiVersion: v1
+metadata:
+  name: prometheus-config
+  namespace: prometheus
+data:
+  prometheus.yml: |
+    # my global config
+    global:
+      scrape_interval:     15s # Set the scrape interval to every 15 seconds. Default is every 1 minute.
+      evaluation_interval: 15s # Evaluate rules every 15 seconds. The default is every 1 minute.
+
+    scrape_configs:
+    - job_name: 'kubernetes-pods'
+      kubernetes_sd_configs:
+      - role: pod
+        namespaces:
+          names:
+          - kata-system
+
+      relabel_configs:
+      # Example relabel to scrape only pods that have
+      # "prometheus.io/scrape: true" annotation.
+      - source_labels: [__meta_kubernetes_pod_annotation_prometheus_io_scrape]
+        action: keep
+        regex: true

docs/how-to/how-to-import-kata-logs-with-fluentd.md

@@ -33,7 +33,7 @@ also applies to the Kata `shimv2` runtime.  Differences pertaining to Kata `shim
 
 Kata generates logs. The logs can come from numerous parts of the Kata stack (the runtime, proxy, shim
 and even the agent). By default the logs
-[go to the system journal](https://github.com/kata-containers/runtime#logging),
+[go to the system journal](../../src/runtime/README.md#logging),
 but they can also be configured to be stored in files.
 
 The logs default format is in [`logfmt` structured logging](https://brandur.org/logfmt), but can be switched to
@@ -256,7 +256,7 @@ directly from Kata, that should make overall import and processing of the log en
 
 There are potentially two things we can do with Kata here:
 
-- Get Kata to [output its logs in `JSON` format](https://github.com/kata-containers/runtime#logging) rather
+- Get Kata to [output its logs in `JSON` format](../../src/runtime/README.md#logging) rather
   than `logfmt`.
 - Get Kata to log directly into a file, rather than via the system journal. This would allow us to not need
   to parse the systemd format files, and capture the Kata log lines directly. It would also avoid Fluentd

docs/how-to/how-to-load-kernel-modules-with-kata.md

@@ -101,6 +101,8 @@ spec:
     tty: true
 ```
 
-[1]: https://github.com/kata-containers/runtime
-[2]: https://github.com/kata-containers/agent
+> **Note**: To pass annotations to Kata containers, [cri must to be configurated correctly](how-to-set-sandbox-config-kata.md#cri-configuration)
+
+[1]: ../../src/runtime
+[2]: ../../src/agent
 [3]: https://kubernetes.io/docs/concepts/workloads/pods/pod/

docs/how-to/how-to-set-prometheus-in-k8s.md

@@ -0,0 +1,108 @@
+# How to monitor Kata Containers in Kubernetes clusters
+
+This document describes how to run `kata-monitor` in a Kubernetes cluster using Prometheus's service discovery to scrape metrics from `kata-agent`.
+
+- [Introduction](#introduction)
+- [Pre-requisites](#pre-requisites)
+- [Configure Prometheus](#configure-prometheus)
+- [Configure `kata-monitor`](#configure-kata-monitor)
+- [Setup Grafana](#setup-grafana)
+  * [Create `datasource`](#create-datasource)
+  * [Import dashboard](#import-dashboard)
+
+> **Warning**: This how-to is only for evaluation purpose, you **SHOULD NOT** running it in production using this configurations.
+
+## Introduction
+
+If you are running Kata containers in a Kubernetes cluster, the best way to run `kata-monitor` is using Kubernetes native `DaemonSet`, `kata-monitor` will run on desired Kubernetes nodes without other operations when new nodes joined the cluster.
+
+Prometheus also support a Kubernetes service discovery that can find scrape targets dynamically without explicitly setting `kata-monitor`'s metric endpoints.
+
+## Pre-requisites
+
+You must have a running Kubernetes cluster first. If not, [install a Kubernetes cluster](https://kubernetes.io/docs/setup/) first.
+
+Also you should ensure that `kubectl` working correctly.
+
+> **Note**: More information about Kubernetes integrations:
+>   - [Run Kata Containers with Kubernetes](run-kata-with-k8s.md)
+>   - [How to use Kata Containers and Containerd](containerd-kata.md)
+>   - [How to use Kata Containers and CRI (containerd plugin) with Kubernetes](how-to-use-k8s-with-cri-containerd-and-kata.md)
+
+## Configure Prometheus
+
+Start Prometheus by utilizing our sample manifest:
+
+```
+$ kubectl apply -f https://raw.githubusercontent.com/kata-containers/kata-containers/2.0-dev/docs/how-to/data/prometheus.yml
+```
+
+This will create a new namespace, `prometheus`, and create the following resources:
+
+* `ClusterRole`, `ServiceAccount`, `ClusterRoleBinding` to let Prometheus to access Kubernetes API server.
+* `ConfigMap` that contains minimum configurations to let Prometheus run Kubernetes service discovery.
+* `Deployment` that run Prometheus in `Pod`.
+* `Service` with `type` of `NodePort`(`30909` in this how to), that we can access Prometheus through `<hostIP>:30909`. In production environment, this `type` may be `LoadBalancer` or `Ingress` resource.
+
+After the Prometheus server is running, run `curl -s http://hostIP:NodePort:30909/metrics`, if Prometheus is working correctly, you will get response like these:
+
+```
+# HELP go_gc_duration_seconds A summary of the GC invocation durations.
+# TYPE go_gc_duration_seconds summary
+go_gc_duration_seconds{quantile="0"} 3.9403e-05
+go_gc_duration_seconds{quantile="0.25"} 0.000169907
+go_gc_duration_seconds{quantile="0.5"} 0.000207421
+go_gc_duration_seconds{quantile="0.75"} 0.000229911
+```
+
+## Configure `kata-monitor`
+
+`kata-monitor` can be started on the cluster as follows:
+
+```
+$ kubectl apply -f https://raw.githubusercontent.com/kata-containers/kata-containers/2.0-dev/docs/how-to/data/kata-monitor-daemonset.yml
+```
+
+This will create a new namespace `kata-system` and a `daemonset` in it.
+
+Once the `daemonset` is running, Prometheus should discover `kata-monitor` as a target. You can open `http://<hostIP>:30909/service-discovery` and find `kubernetes-pods` under the `Service Discovery` list
+
+
+## Setup Grafana
+
+Run this command to run Grafana in Kubernetes:
+
+```
+$ kubectl apply -f https://raw.githubusercontent.com/kata-containers/kata-containers/2.0-dev/docs/how-to/data/grafana.yml
+```
+
+This will create deployment and service for Grafana under namespace `prometheus`.
+
+After the Grafana deployment is ready, you can open `http://hostIP:NodePort:30000/` to access Grafana server. For Grafana 7.0.5, the default user/password is `admin/admin`. You can modify the default account and adjust other security settings by editing the [Grafana configuration](https://grafana.com/docs/grafana/latest/installation/configuration/#security).
+
+To use Grafana show data from Prometheus, you must create a Prometheus `datasource` and dashboard.
+
+### Create `datasource`
+
+Open `http://hostIP:NodePort:30000/datasources/new` in your browser, select Prometheus from time series databases list.
+
+Normally you only need to set `URL` to `http://hostIP:NodePort:30909` to let it work, and leave the name as `Prometheus` as default.
+
+### Import dashboard
+
+A [sample dashboard](data/dashboard.json) for Kata Containers metrics is provided which can be imported to Grafana for evaluation.
+
+You can import this dashboard using Grafana UI, or using `curl` command in console.
+
+
+```
+$ curl -XPOST -i localhost:3000/api/dashboards/import \
+    -u admin:admin \
+    -H "Content-Type: application/json" \
+	-d "{\"dashboard\":$(curl -sL https://raw.githubusercontent.com/kata-containers/kata-containers/2.0-dev/docs/how-to/data/dashboard.json )}"
+```
+
+## References
+
+- [Prometheus `kubernetes_sd_config`](https://prometheus.io/docs/prometheus/latest/configuration/configuration/#kubernetes_sd_config)
+

docs/how-to/how-to-set-sandbox-config-kata.md

@@ -79,13 +79,14 @@ There are several kinds of Kata configurations and they are listed below.
 # CRI Configuration
 
 In case of CRI-O, all annotations specified in the pod spec are passed down to Kata.
+
 For containerd, annotations specified in the pod spec are passed down to Kata
-starting with version `1.3.0`. Additionally, extra configuration is needed for containerd,
-by providing a `pod_annotations` field in the containerd config file.  The `pod_annotations`
-field is a list of annotations that can be passed down to Kata as OCI annotations. 
-It supports golang match patterns. Since annotations supported by Kata follow the pattern
-`io.katacontainers.*`, the following configuration would work for passing annotations to 
-Kata from containerd:
+starting with version `1.3.0` of containerd. Additionally, extra configuration is
+needed for containerd, by providing a `pod_annotations` field in the containerd config
+file.  The `pod_annotations` field is a list of annotations that can be passed down to
+Kata as OCI annotations. It supports golang match patterns. Since annotations supported
+by Kata follow the pattern `io.katacontainers.*`, the following configuration would work
+for passing annotations to Kata from containerd:
 
 ```
 $ cat /etc/containerd/config

docs/how-to/how-to-use-k8s-with-cri-containerd-and-kata.md

@@ -29,7 +29,7 @@ to launch Kata Containers. For the previous version of Kata Containers, the Pods
 
 > **Note:** For information about the supported versions of these components,
 > see the  Kata Containers
-> [`versions.yaml`](https://github.com/kata-containers/runtime/blob/master/versions.yaml)
+> [`versions.yaml`](../../versions.yaml)
 > file.
 
 ## Install and configure containerd
@@ -177,7 +177,7 @@ $ sudo -E kubectl taint nodes --all node-role.kubernetes.io/master-
 By default, all pods are created with the default runtime configured in CRI containerd plugin.
 
 If a pod has the `io.kubernetes.cri.untrusted-workload` annotation set to `"true"`, the CRI plugin runs the pod with the
-[Kata Containers runtime](https://github.com/kata-containers/runtime/blob/master/README.md).
+[Kata Containers runtime](../../src/runtime/README.md).
 
 - Create an untrusted pod configuration
 

docs/how-to/how-to-use-kata-containers-with-acrn.md

@@ -49,7 +49,7 @@ This document requires the presence of the ACRN hypervisor and Kata Containers o
   $ sudo sed -i "s/$kernel_img/bzImage/g" /mnt/loader/entries/$conf_file
   $ sync && sudo umount /mnt && sudo reboot
   ```
-- Kata Containers installation: Automated installation does not seem to be supported for Clear Linux, so please use [manual installation](https://github.com/kata-containers/documentation/blob/master/Developer-Guide.md) steps.
+- Kata Containers installation: Automated installation does not seem to be supported for Clear Linux, so please use [manual installation](../Developer-Guide.md) steps.
 
 > **Note:** Create rootfs image and not initrd image.
 
@@ -82,7 +82,7 @@ $ sudo systemctl daemon-reload
 $ sudo systemctl restart docker
 ```
 
-4. Configure [Docker](https://github.com/kata-containers/documentation/blob/master/Developer-Guide.md#update-the-docker-systemd-unit-file) to use `kata-runtime`.
+4. Configure [Docker](../Developer-Guide.md#update-the-docker-systemd-unit-file) to use `kata-runtime`.
 
 ## Configure Kata Containers with ACRN
 

docs/how-to/how-to-use-kata-containers-with-nemu.md

@@ -19,7 +19,7 @@ Kata Containers relies by default on the QEMU hypervisor in order to spawn the v
 This document describes how to run Kata Containers with NEMU, first by explaining how to download, build and install it. Then it walks through the steps needed to update your Kata Containers configuration in order to run with NEMU. 
 
 ## Pre-requisites
-This document requires Kata Containers to be [installed](https://github.com/kata-containers/documentation/blob/master/install/README.md) on your system.
+This document requires Kata Containers to be [installed](../install/README.md) on your system.
 
 Also, it's worth noting that NEMU only supports `x86_64` and `aarch64` architecture.
 

docs/how-to/how-to-use-virtio-fs-with-kata.md

@@ -13,7 +13,7 @@ As of the 1.7 release of Kata Containers, [9pfs](https://www.kernel.org/doc/Docu
 
 To help address these limitations, [virtio-fs](https://virtio-fs.gitlab.io/) has been developed. virtio-fs is a shared file system that lets virtual machines access a directory tree on the host. In Kata Containers, virtio-fs can be used to share container volumes, secrets, config-maps, configuration files (hostname, hosts, `resolv.conf`) and the container rootfs on the host with the guest.  virtio-fs provides significant performance and POSIX compliance improvements compared to 9pfs.
 
-Enabling of virtio-fs requires changes in the guest kernel as well as the VMM. For Kata Containers, experimental virtio-fs support is enabled through the [NEMU VMM](https://github.com/intel/nemu).
+Enabling of virtio-fs requires changes in the guest kernel as well as the VMM. For Kata Containers, experimental virtio-fs support is enabled through `qemu` and `cloud-hypervisor` VMMs.
 
 **Note: virtio-fs support is experimental in the 1.7 release of Kata Containers. Work is underway to improve stability, performance and upstream integration. This is available for early preview - use at your own risk**
 
@@ -21,31 +21,41 @@ This document describes how to get Kata Containers to work with virtio-fs.
 
 ## Pre-requisites
 
-* Before Kata 1.8 this feature required the host to have hugepages support enabled. Enable this with the `sysctl vm.nr_hugepages=1024` command on the host.
+Before Kata 1.8 this feature required the host to have hugepages support enabled. Enable this with the `sysctl vm.nr_hugepages=1024` command on the host.In later versions of Kata, virtio-fs leverages `/dev/shm` as the shared memory backend. The default size of `/dev/shm` on a system is typically half of the total system memory. This can pose a physical limit to the maximum number of pods that can be launched with virtio-fs. This can be overcome by increasing the size of `/dev/shm` as shown below:
 
+```bash
+$ mount -o remount,size=${desired_shm_size} /dev/shm
+```
+ 
 ## Install Kata Containers with virtio-fs support
 
-The Kata Containers NEMU configuration, the NEMU VMM and the `virtiofs` daemon are available in the [Kata Container release](https://github.com/kata-containers/runtime/releases) artifacts starting with the 1.7 release. While the feature is experimental, distribution packages are not supported, but installation is available through [`kata-deploy`](https://github.com/kata-containers/packaging/tree/master/kata-deploy).
+The Kata Containers `qemu` configuration with virtio-fs and the `virtiofs` daemon are available in the [Kata Container release](https://github.com/kata-containers/runtime/releases) artifacts starting with the 1.9 release. Installation is available through [distribution packages](https://github.com/kata-containers/documentation/blob/master/install/README.md#supported-distributions) as well through [`kata-deploy`](https://github.com/kata-containers/packaging/tree/master/kata-deploy).
 
-Install the latest release of Kata as follows:
+**Note: Support for virtio-fs was first introduced in `NEMU` hypervisor in Kata 1.8 release. This hypervisor has been deprecated.**
+
+Install the latest release of Kata with `kata-deploy` as follows:
 ```
 docker run --runtime=runc -v /opt/kata:/opt/kata -v /var/run/dbus:/var/run/dbus -v /run/systemd:/run/systemd -v /etc/docker:/etc/docker -it katadocker/kata-deploy kata-deploy-docker install
 ```
 
-This will place the Kata release artifacts in `/opt/kata`, and update Docker's configuration to include a runtime target, `kata-nemu`. Learn more about `kata-deploy` and how to use `kata-deploy` in Kubernetes [here](https://github.com/kata-containers/packaging/tree/master/kata-deploy#kubernetes-quick-start).
-
+This will place the Kata release artifacts in `/opt/kata`, and update Docker's configuration to include a runtime target, `kata-qemu-virtiofs`. Learn more about `kata-deploy` and how to use `kata-deploy` in Kubernetes [here](https://github.com/kata-containers/packaging/tree/master/kata-deploy#kubernetes-quick-start).
 
 ## Run a Kata Container utilizing virtio-fs
 
-Once installed, start a new container, utilizing NEMU + `virtiofs`:
+Once installed, start a new container, utilizing `qemu` + `virtiofs`:
 ```bash
-$ docker run --runtime=kata-nemu -it busybox
+$ docker run --runtime=kata-qemu-virtiofs -it busybox
 ```
 
-Verify the new container is running with the NEMU hypervisor as well as using `virtiofsd`. To do this look for the hypervisor path and the `virtiofs` daemon process on the host:
+Verify the new container is running with the `qemu` hypervisor as well as using `virtiofsd`. To do this look for the hypervisor path and the `virtiofs` daemon process on the host:
 ```bash
 $ ps -aux | grep virtiofs
 root ... /home/foo/build-x86_64_virt/x86_64_virt-softmmu/qemu-system-x86_64_virt
 ...  -machine virt,accel=kvm,kernel_irqchip,nvdimm ...
 root ... /home/foo/build-x86_64_virt/virtiofsd-x86_64 ...
 ```
+
+You can also try out virtio-fs using `cloud-hypervisor` VMM:
+```bash
+$ docker run --runtime=kata-clh -it busybox
+```

docs/how-to/privileged.md

@@ -75,5 +75,5 @@ See below example config:
   privileged_without_host_devices = true
 ```
 
- - [Kata Containers with CRI-O](https://github.com/kata-containers/documentation/blob/master/how-to/run-kata-with-k8s.md#cri-o)
+ - [Kata Containers with CRI-O](../how-to/run-kata-with-k8s.md#cri-o)
   

docs/how-to/run-kata-with-k8s.md

@@ -14,7 +14,7 @@
   * [Run a Kubernetes pod with Kata Containers](#run-a-kubernetes-pod-with-kata-containers)
 
 ## Prerequisites
-This guide requires Kata Containers available on your system, install-able by following [this guide](https://github.com/kata-containers/documentation/blob/master/install/README.md).
+This guide requires Kata Containers available on your system, install-able by following [this guide](../install/README.md).
 
 ## Install a CRI implementation
 
@@ -28,7 +28,7 @@ After choosing one CRI implementation, you must make the appropriate configurati
 to ensure it integrates with Kata Containers.
 
 Kata Containers 1.5 introduced the `shimv2` for containerd 1.2.0, reducing the components
-required to spawn pods and containers, and this is the preferred way to run Kata Containers with Kubernetes ([as documented here](https://github.com/kata-containers/documentation/blob/master/how-to/how-to-use-k8s-with-cri-containerd-and-kata.md#configure-containerd-to-use-kata-containers)).
+required to spawn pods and containers, and this is the preferred way to run Kata Containers with Kubernetes ([as documented here](../how-to/how-to-use-k8s-with-cri-containerd-and-kata.md#configure-containerd-to-use-kata-containers)).
 
 An equivalent shim implementation for CRI-O is planned.
 
@@ -78,7 +78,7 @@ a runtime to be used when the workload cannot be trusted and a higher level of s
 is required. An additional flag can be used to let CRI-O know if a workload
 should be considered _trusted_ or _untrusted_ by default.
 For further details, see the documentation
-[here](https://github.com/kata-containers/documentation/blob/master/design/architecture.md#mixing-vm-based-and-namespace-based-runtimes).
+[here](../design/architecture.md#mixing-vm-based-and-namespace-based-runtimes).
 
 ```toml
 # runtime is the OCI compatible runtime used for trusted container workloads.
@@ -132,7 +132,7 @@ to properly install it.
 
 To customize containerd to select Kata Containers runtime, follow our
 "Configure containerd to use Kata Containers" internal documentation
-[here](https://github.com/kata-containers/documentation/blob/master/how-to/how-to-use-k8s-with-cri-containerd-and-kata.md#configure-containerd-to-use-kata-containers).
+[here](../how-to/how-to-use-k8s-with-cri-containerd-and-kata.md#configure-containerd-to-use-kata-containers).
 
 ## Install Kubernetes
 
@@ -160,7 +160,7 @@ Environment="KUBELET_EXTRA_ARGS=--container-runtime=remote --runtime-request-tim
 Environment="KUBELET_EXTRA_ARGS=--container-runtime=remote --runtime-request-timeout=15m --container-runtime-endpoint=unix:///run/containerd/containerd.sock"
 ```
 For more information about containerd see the "Configure Kubelet to use containerd"
-documentation [here](https://github.com/kata-containers/documentation/blob/master/how-to/how-to-use-k8s-with-cri-containerd-and-kata.md#configure-kubelet-to-use-containerd).
+documentation [here](../how-to/how-to-use-k8s-with-cri-containerd-and-kata.md#configure-kubelet-to-use-containerd).
 
 ## Run a Kubernetes pod with Kata Containers
 

docs/how-to/service-mesh.md

@@ -48,7 +48,7 @@ as the proxy starts.
 
 ### Kata and Kubernetes
 
-Follow the [instructions](https://github.com/kata-containers/documentation/blob/master/install/README.md)
+Follow the [instructions](../install/README.md)
 to get Kata Containers properly installed and configured with Kubernetes.
 You can choose between CRI-O and CRI-containerd, both are supported
 through this document.

docs/how-to/what-is-vm-cache-and-how-do-I-use-it.md

@@ -10,7 +10,7 @@
 VMCache is a new function that creates VMs as caches before using it.
 It helps speed up new container creation.  
 The function consists of a server and some clients communicating
-through Unix socket.  The protocol is gRPC in [`protocols/cache/cache.proto`](https://github.com/kata-containers/runtime/blob/master/protocols/cache/cache.proto).  
+through Unix socket.  The protocol is gRPC in [`protocols/cache/cache.proto`](../../src/runtime/protocols/cache/cache.proto).  
 The VMCache server will create some VMs and cache them by factory cache.
 It will convert the VM to gRPC format and transport it when gets
 requested from clients.  
@@ -21,9 +21,9 @@ a new sandbox.
 
 ### How is this different to VM templating
 
-Both [VM templating](https://github.com/kata-containers/documentation/blob/master/how-to/what-is-vm-templating-and-how-do-I-use-it.md) and VMCache help speed up new container creation.  
+Both [VM templating](../how-to/what-is-vm-templating-and-how-do-I-use-it.md) and VMCache help speed up new container creation.  
 When VM templating enabled, new VMs are created by cloning from a pre-created template VM, and they will share the same initramfs, kernel and agent memory in readonly mode.  So it saves a lot of memory if there are many Kata Containers running on the same host.  
-VMCache is not vulnerable to [share memory CVE](https://github.com/kata-containers/documentation/blob/master/how-to/what-is-vm-templating-and-how-do-I-use-it.md#what-are-the-cons) because each VM doesn't share the memory.
+VMCache is not vulnerable to [share memory CVE](../how-to/what-is-vm-templating-and-how-do-I-use-it.md#what-are-the-cons) because each VM doesn't share the memory.
 
 ### How to enable VMCache
 

docs/how-to/what-is-vm-templating-and-how-do-I-use-it.md

@@ -8,7 +8,7 @@ same initramfs, kernel and agent memory in readonly mode. It is very
 much like a process fork done by the kernel but here we *fork* VMs.
 
 ### How is this different from VMCache
-Both [VMCache](https://github.com/kata-containers/documentation/blob/master/how-to/what-is-vm-cache-and-how-do-I-use-it.md) and VM templating help speed up new container creation.  
+Both [VMCache](../how-to/what-is-vm-cache-and-how-do-I-use-it.md) and VM templating help speed up new container creation.  
 When VMCache enabled, new VMs are created by the VMCache server.  So it is not vulnerable to share memory CVE because each VM doesn't share the memory.  
 VM templating saves a lot of memory if there are many Kata Containers running on the same host.
 

docs/install/README.md

@@ -1,16 +1,16 @@
 # Kata Containers installation user guides
 
-* [Prerequisites](#prerequisites)
-* [Packaged installation methods](#packaged-installation-methods)
-   * [Supported Distributions](#supported-distributions)
-      * [Official packages](#official-packages)
-   * [Automatic Installation](#automatic-installation)
-   * [Snap Installation](#snap-installation)
-   * [Scripted Installation](#scripted-installation)
-   * [Manual Installation](#manual-installation)
-* [Build from source installation](#build-from-source-installation)
-* [Installing on a Cloud Service Platform](#installing-on-a-cloud-service-platform)
-* [Further information](#further-information)
+- [Kata Containers installation user guides](#kata-containers-installation-user-guides)
+  - [Prerequisites](#prerequisites)
+  - [Packaged installation methods](#packaged-installation-methods)
+    - [Official packages](#official-packages)
+    - [Automatic Installation](#automatic-installation)
+    - [Snap Installation](#snap-installation)
+    - [Scripted Installation](#scripted-installation)
+    - [Manual Installation](#manual-installation)
+  - [Build from source installation](#build-from-source-installation)
+  - [Installing on a Cloud Service Platform](#installing-on-a-cloud-service-platform)
+  - [Further information](#further-information)
 
 The following is an overview of the different installation methods available. All of these methods equally result
 in a system configured to run Kata Containers.
@@ -18,7 +18,7 @@ in a system configured to run Kata Containers.
 ## Prerequisites
 Kata Containers requires nested virtualization or bare metal.
 See the
-[hardware requirements](https://github.com/kata-containers/runtime/blob/master/README.md#hardware-requirements)
+[hardware requirements](../../src/runtime/README.md#hardware-requirements)
 to see if your system is capable of running Kata Containers.
 
 ## Packaged installation methods
@@ -29,33 +29,22 @@ to see if your system is capable of running Kata Containers.
 
 | Installation method                                  | Description                                                                             | Distributions supported              |
 |------------------------------------------------------|-----------------------------------------------------------------------------------------|--------------------------------------|
-| [Automatic](#automatic-installation)                 |Run a single command to install a full system                                            |[see table](#supported-distributions) |
+| [Automatic](#automatic-installation)                 |Run a single command to install a full system                                            | |
 | [Using snap](#snap-installation)                     |Easy to install and automatic updates                                                    |any distro that supports snapd        |
-| [Using official distro packages](#official-packages) |Kata packages provided by Linux distributions official repositories                      |[see table](#supported-distributions) |
-| [Scripted](#scripted-installation)                   |Generates an installation script which will result in a working system when executed     |[see table](#supported-distributions) |
-| [Manual](#manual-installation)                       |Allows the user to read a brief document and execute the specified commands step-by-step |[see table](#supported-distributions) |
+| [Using official distro packages](#official-packages) |Kata packages provided by Linux distributions official repositories                      | |
+| [Scripted](#scripted-installation)                   |Generates an installation script which will result in a working system when executed     | |
+| [Manual](#manual-installation)                       |Allows the user to read a brief document and execute the specified commands step-by-step | |
 
-### Supported Distributions
-
-Kata is packaged by the Kata community for:
-
-|Distribution (link to installation guide)                        | Versions                                                                                                          |
-|-----------------------------------------------------------------|-------------------------------------------------------------------------------------------------------------------|
-|[CentOS](centos-installation-guide.md)                           | 7                                                                                                                 |
-|[Debian](debian-installation-guide.md)                           | 9, 10                                                                                                             |
-|[Fedora](fedora-installation-guide.md)                           | 28, 29, 30                                                                                                        |
-|[openSUSE](opensuse-installation-guide.md)                       | [Leap](opensuse-leap-installation-guide.md) (15, 15.1)<br>[Tumbleweed](opensuse-tumbleweed-installation-guide.md) |
-|[Red Hat Enterprise Linux (RHEL)](rhel-installation-guide.md)    | 7                                                                                                                 |
-|[SUSE Linux Enterprise Server (SLES)](sles-installation-guide.md)| SLES 12 SP3                                                                                                       |
-|[Ubuntu](ubuntu-installation-guide.md)                           | 16.04, 18.04                                                                                                      |
-
-#### Official packages
+### Official packages
 
 Kata packages are provided by official distribution repositories for:
 
-|Distribution (link to packages)                                  | Versions   |
-|-----------------------------------------------------------------|------------|
-|[openSUSE](https://software.opensuse.org/package/katacontainers) | Tumbleweed |
+| Distribution (link to packages)                          | Versions                                                                       | Contacts |
+| -------------------------------------------------------- | ------------------------------------------------------------------------------ | -------- |
+| [CentOS](centos-installation-guide.md)                   | 8                                                                              |          |
+| [Fedora](fedora-installation-guide.md)                   | 32, Rawhide                                                                    |          |
+| [SUSE Linux Enterprise (SLE)](sle-installation-guide.md) | SLE 15 SP1, 15 SP2                                                             |          |
+| [openSUSE](opensuse-installation-guide.md)               | [Leap 15.1](opensuse-leap-15.1-installation-guide.md)<br>Leap 15.2, Tumbleweed |          |
 
 
 ### Automatic Installation
@@ -72,13 +61,13 @@ Kata packages are provided by official distribution repositories for:
 [Use `kata-doc-to-script`](installing-with-kata-doc-to-script.md) to generate installation scripts that can be reviewed before they are executed.
 
 ### Manual Installation
-Manual installation instructions are available for [these distributions](#supported-distributions) and document how to:
+Manual installation instructions are available for [these distributions](#packaged-installation-methods) and document how to:
 1. Add the Kata Containers repository to your distro package manager, and import the packages signing key.
 2. Install the Kata Containers packages.
 3. Install a supported container manager.
-4. Configure the container manager to use `kata-runtime` as the default OCI runtime. Or, for Kata Containers 1.5.0 or above, configure the
+4. Configure the container manager to use Kata Containers as the default OCI runtime. Or, for Kata Containers 1.5.0 or above, configure the
    `io.containerd.kata.v2` to be the runtime shim (see [containerd runtime v2 (shim API)](https://github.com/containerd/containerd/tree/master/runtime/v2)
-   and [How to use Kata Containers and CRI (containerd plugin) with Kubernetes](https://github.com/kata-containers/documentation/blob/master/how-to/how-to-use-k8s-with-cri-containerd-and-kata.md)).
+   and [How to use Kata Containers and CRI (containerd plugin) with Kubernetes](../how-to/how-to-use-k8s-with-cri-containerd-and-kata.md)).
 
 > **Notes on upgrading**:
 > - If you are installing Kata Containers on a system that already has Clear Containers or `runv` installed,
@@ -87,7 +76,7 @@ Manual installation instructions are available for [these distributions](#suppor
 > **Notes on releases**:
 > - [This download server](http://download.opensuse.org/repositories/home:/katacontainers:/releases:/)
 > hosts the Kata Containers packages built by OBS for all the supported architectures.
-> Packages are available for the latest and stable releases (more info [here](https://github.com/kata-containers/documentation/blob/master/Stable-Branch-Strategy.md)).
+> Packages are available for the latest and stable releases (more info [here](../Stable-Branch-Strategy.md)).
 >
 > - The following guides apply to the latest Kata Containers release
 > (a.k.a. `master` release).
@@ -124,4 +113,4 @@ versions. This is not recommended for normal users.
 ## Further information
 * The [upgrading document](../Upgrading.md).
 * The [developer guide](../Developer-Guide.md).
-* The [runtime documentation](https://github.com/kata-containers/runtime/blob/master/README.md).
+* The [runtime documentation](../../src/runtime/README.md).

docs/install/aws-installation-guide.md

@@ -137,4 +137,4 @@ Go onto the next step.
 
 The process for installing Kata itself on bare metal is identical to that of a virtualization-enabled VM.
 
-For detailed information to install Kata on your distribution of choice, see the [Kata Containers installation user guides](https://github.com/kata-containers/documentation/blob/master/install/README.md).
+For detailed information to install Kata on your distribution of choice, see the [Kata Containers installation user guides](../install/README.md).

docs/install/azure-installation-guide.md

@@ -15,4 +15,4 @@ Create a new virtual machine with:
 
 ## Set up with distribution specific quick start
 
-Follow distribution specific [install guides](https://github.com/kata-containers/documentation/tree/master/install#supported-distributions).
+Follow distribution specific [install guides](../install/README.md#packaged-installation-methods).

docs/install/centos-installation-guide.md

@@ -4,14 +4,25 @@
 
    ```bash
    $ source /etc/os-release
-   $ sudo yum -y install yum-utils
-   $ ARCH=$(arch)
-   $ BRANCH="${BRANCH:-master}"
-   $ sudo -E yum-config-manager --add-repo "http://download.opensuse.org/repositories/home:/katacontainers:/releases:/${ARCH}:/${BRANCH}/CentOS_${VERSION_ID}/home:katacontainers:releases:${ARCH}:${BRANCH}.repo"
-   $ sudo -E yum -y install kata-runtime kata-proxy kata-shim
+   $ cat <<EOF | sudo -E tee /etc/yum.repos.d/advanced-virt.repo
+     [advanced-virt]
+     name=Advanced Virtualization
+     baseurl=http://mirror.centos.org/\$contentdir/\$releasever/virt/\$basearch/advanced-virtualization
+     enabled=1
+     gpgcheck=1
+     skip_if_unavailable=1
+     EOF
+   $ cat <<EOF | sudo -E tee /etc/yum.repos.d/kata-containers.repo
+     [kata-containers]
+     name=Kata Containers
+     baseurl=http://mirror.centos.org/\$contentdir/\$releasever/virt/\$basearch/kata-containers
+     enabled=1
+     gpgcheck=1
+     skip_if_unavailable=1
+     EOF
+   $ sudo -E dnf module disable -y virt:rhel
+   $ sudo -E dnf install -y kata-runtime
    ```
 
 2. Decide which container manager to use and select the corresponding link that follows:
-
-   - [Docker](docker/centos-docker-install.md)
-   - [Kubernetes](https://github.com/kata-containers/documentation/blob/master/Developer-Guide.md#run-kata-containers-with-kubernetes)
+   - [Kubernetes](../Developer-Guide.md#run-kata-containers-with-kubernetes)

docs/install/debian-installation-guide.md

@@ -1,22 +0,0 @@
-# Install Kata Containers on Debian
-
-1. Install the Kata Containers components with the following commands:
-
-   ```bash
-   $ export DEBIAN_FRONTEND=noninteractive
-   $ ARCH=$(arch)
-   $ BRANCH="${BRANCH:-master}"
-   $ source /etc/os-release
-   $ [ "$ID" = debian ] && [ -z "$VERSION_ID" ] && echo >&2 "ERROR: Debian unstable not supported.
-     You can try stable packages here:
-     http://download.opensuse.org/repositories/home:/katacontainers:/releases:/${ARCH}:/${BRANCH}" && exit 1
-   $ sudo sh -c "echo 'deb http://download.opensuse.org/repositories/home:/katacontainers:/releases:/${ARCH}:/${BRANCH}/Debian_${VERSION_ID}/ /' > /etc/apt/sources.list.d/kata-containers.list"
-   $ curl -sL  http://download.opensuse.org/repositories/home:/katacontainers:/releases:/${ARCH}:/${BRANCH}/Debian_${VERSION_ID}/Release.key | sudo apt-key add -
-   $ sudo -E apt-get update
-   $ sudo -E apt-get -y install kata-runtime kata-proxy kata-shim
-   ```
-
-2. Decide which container manager to use and select the corresponding link that follows:
-
-   - [Docker](docker/debian-docker-install.md)
-   - [Kubernetes](https://github.com/kata-containers/documentation/blob/master/Developer-Guide.md#run-kata-containers-with-kubernetes)

docs/install/docker/centos-docker-install.md

@@ -1,75 +0,0 @@
-# Install Docker for Kata Containers on CentOS
-
-> **Note:**
->
-> - This guide assumes you have
->   [already installed the Kata Containers packages](../centos-installation-guide.md).
-
-1. Install the latest version of Docker with the following commands:
-
-   > **Notes:**
-   >
-   > - This step is only required if Docker is not installed on the system.
-   > - Docker version 18.09 [removed devicemapper support](https://github.com/kata-containers/documentation/issues/373).
-   >   If you wish to use a block based backend, see the options listed on https://github.com/kata-containers/documentation/issues/407.
-
-   ```bash
-   $ sudo yum-config-manager --add-repo https://download.docker.com/linux/centos/docker-ce.repo
-   $ sudo yum -y install docker-ce
-   ```
-
-   For more information on installing Docker please refer to the
-   [Docker Guide](https://docs.docker.com/engine/installation/linux/centos).
-
-2. Configure Docker to use Kata Containers by default with **ONE** of the following methods:
-
-   1. systemd (this is the default and is applied automatically if you select the
-      [automatic installation](https://github.com/kata-containers/documentation/tree/master/install#automatic-installation) option)
-
-       ```bash
-       $ sudo mkdir -p /etc/systemd/system/docker.service.d/
-       $ cat <<EOF | sudo tee /etc/systemd/system/docker.service.d/kata-containers.conf
-       [Service]
-       ExecStart=
-       ExecStart=/usr/bin/dockerd -D --add-runtime kata-runtime=/usr/bin/kata-runtime --default-runtime=kata-runtime
-       EOF
-       ```
-
-   2. Docker `daemon.json`
-
-      Create docker configuration folder.
-
-      ```
-      $ sudo mkdir -p /etc/docker
-      ```
-
-      Add the following definitions to `/etc/docker/daemon.json`:
-
-      ```json
-      {
-        "default-runtime": "kata-runtime",
-        "runtimes": {
-          "kata-runtime": {
-            "path": "/usr/bin/kata-runtime"
-          }
-        }
-      }
-      ```
-
-3. Restart the Docker systemd service with the following commands:
-
-   ```bash
-   $ sudo systemctl daemon-reload
-   $ sudo systemctl restart docker
-   ```
-
-4. Run Kata Containers
-
-   You are now ready to run Kata Containers:
-
-   ```bash
-   $ sudo docker run busybox uname -a
-   ```
-
-   The previous command shows details of the kernel version running inside the
-   container, which is different to the host kernel version.

docs/install/docker/debian-docker-install.md

@@ -1,103 +0,0 @@
-# Install Docker for Kata Containers on Debian
-
-> **Note:**
->
-> - This guide assumes you have
->   [already installed the Kata Containers packages](../debian-installation-guide.md).
-> - This guide allows for installation with `systemd` or `sysVinit` init systems.
-
-1. Install the latest version of Docker with the following commands:
-
-   > **Notes:**
-   >
-   > - This step is only required if Docker is not installed on the system.
-   > - Docker version 18.09 [removed devicemapper support](https://github.com/kata-containers/documentation/issues/373).
-   >   If you wish to use a block based backend, see the options listed on https://github.com/kata-containers/documentation/issues/407.
-
-   ```bash
-   $ sudo apt-get -y install apt-transport-https ca-certificates curl gnupg2 software-properties-common
-   $ curl -fsSL https://download.docker.com/linux/$(. /etc/os-release; echo "$ID")/gpg | sudo apt-key add -
-   $ sudo add-apt-repository "deb https://download.docker.com/linux/$(. /etc/os-release; echo "$ID") $(lsb_release -cs) stable"
-   $ sudo apt-get update
-   $ sudo -E apt-get -y install docker-ce
-   ```
-
-   For more information on installing Docker please refer to the
-   [Docker Guide](https://docs.docker.com/engine/installation/linux/debian).
-
-2. Configure Docker to use Kata Containers by default with **ONE** of the following methods:
-
-a. `sysVinit`
-
-    - with `sysVinit`, docker config is stored in `/etc/default/docker`, edit the options similar to the following:
-
-    ```sh
-    $ sudo sh -c "echo '# specify docker runtime for kata-containers
-    DOCKER_OPTS=\"-D --add-runtime kata-runtime=/usr/bin/kata-runtime --default-runtime=kata-runtime\"' >> /etc/default/docker"
-    ```
-
-b. systemd (this is the default and is applied automatically if you select the
-      [automatic installation](https://github.com/kata-containers/documentation/tree/master/install#automatic-installation) option)
-
-    ```bash
-    $ sudo mkdir -p /etc/systemd/system/docker.service.d/
-    $ cat <<EOF | sudo tee /etc/systemd/system/docker.service.d/kata-containers.conf
-    [Service]
-    ExecStart=
-    ExecStart=/usr/bin/dockerd -D --add-runtime kata-runtime=/usr/bin/kata-runtime --default-runtime=kata-runtime
-    EOF
-    ```
-
-c. Docker `daemon.json`
-
-    Create docker configuration folder.
-
-    ```
-    $ sudo mkdir -p /etc/docker
-    ```
-
-    Add the following definitions to `/etc/docker/daemon.json`:
-
-    ```json
-    {
-      "default-runtime": "kata-runtime",
-      "runtimes": {
-        "kata-runtime": {
-          "path": "/usr/bin/kata-runtime"
-        }
-      }
-    }
-    ```
-
-3. Restart the Docker systemd service with one of the following (depending on init choice):
-
-    a. `sysVinit`
-
-    ```sh
-    $ sudo /etc/init.d/docker stop
-    $ sudo /etc/init.d/docker start
-    ```
-
-    To watch for errors:
-
-    ```sh
-    $ tail -f /var/log/docker.log
-    ```
-
-    b. systemd
-
-    ```bash
-    $ sudo systemctl daemon-reload
-    $ sudo systemctl restart docker
-    ```
-
-4. Run Kata Containers
-
-   You are now ready to run Kata Containers:
-
-   ```bash
-   $ sudo docker run busybox uname -a
-   ```
-
-   The previous command shows details of the kernel version running inside the
-   container, which is different to the host kernel version.

docs/install/docker/fedora-docker-install.md

@@ -1,77 +0,0 @@
-# Install Docker for Kata Containers on Fedora
-
-> **Note:**
->
-> - This guide assumes you have
->   [already installed the Kata Containers packages](../fedora-installation-guide.md).
-
-1. Install the latest version of Docker with the following commands:
-
-   > **Notes:**
-   >
-   > - This step is only required if Docker is not installed on the system.
-   > - Docker version 18.09 [removed devicemapper support](https://github.com/kata-containers/documentation/issues/373).
-   >   If you wish to use a block based backend, see the options listed on https://github.com/kata-containers/documentation/issues/407.
-
-   ```bash
-   $ source /etc/os-release
-   $ sudo dnf config-manager --add-repo https://download.docker.com/linux/fedora/docker-ce.repo
-   $ sudo dnf makecache
-   $ sudo dnf -y install docker-ce
-   ```
-
-   For more information on installing Docker please refer to the
-   [Docker Guide](https://docs.docker.com/engine/installation/linux/fedora).
-
-2. Configure Docker to use Kata Containers by default with **ONE** of the following methods:
-
-   1. systemd (this is the default and is applied automatically if you select the
-      [automatic installation](https://github.com/kata-containers/documentation/tree/master/install#automatic-installation) option)
-
-       ```bash
-       $ sudo mkdir -p /etc/systemd/system/docker.service.d/
-       $ cat <<EOF | sudo tee /etc/systemd/system/docker.service.d/kata-containers.conf
-       [Service]
-       ExecStart=
-       ExecStart=/usr/bin/dockerd -D --add-runtime kata-runtime=/usr/bin/kata-runtime --default-runtime=kata-runtime
-       EOF
-       ```
-
-   2. Docker `daemon.json`
-
-      Create docker configuration folder.
-
-      ```
-      $ sudo mkdir -p /etc/docker
-      ```
-
-      Add the following definitions to `/etc/docker/daemon.json`:
-
-      ```json
-      {
-        "default-runtime": "kata-runtime",
-        "runtimes": {
-          "kata-runtime": {
-            "path": "/usr/bin/kata-runtime"
-          }
-        }
-      }
-      ```
-
-3. Restart the Docker systemd service with the following commands:
-
-   ```bash
-   $ sudo systemctl daemon-reload
-   $ sudo systemctl restart docker
-   ```
-
-4. Run Kata Containers
-
-   You are now ready to run Kata Containers:
-
-   ```bash
-   $ sudo docker run busybox uname -a
-   ```
-
-   The previous command shows details of the kernel version running inside the
-   container, which is different to the host kernel version.

docs/install/docker/opensuse-docker-install.md

@@ -1,75 +0,0 @@
-# Install Docker for Kata Containers on openSUSE
-
-> **Note:**
->
-> - This guide assumes you have
->   [already installed the Kata Containers packages](../opensuse-installation-guide.md).
-
-1. Install the latest version of Docker with the following commands:
-
-   > **Notes:**
-   >
-   > - This step is only required if Docker is not installed on the system.
-   > - Docker version 18.09 [removed devicemapper support](https://github.com/kata-containers/documentation/issues/373).
-   >   If you wish to use a block based backend, see the options listed on https://github.com/kata-containers/documentation/issues/407.
-
-   ```bash
-   $ sudo zypper -n install docker
-   ```
-
-   For more information on installing Docker please refer to the
-   [Docker Guide](https://software.opensuse.org/package/docker).
-
-2. Configure Docker to use Kata Containers by default with **ONE** of the following methods:
-
-   1. Specify the runtime options in `/etc/sysconfig/docker` (this is the default and is applied automatically if you select the
-      [automatic installation](https://github.com/kata-containers/documentation/tree/master/install#automatic-installation) option)
-
-       ```bash
-       $ DOCKER_SYSCONFIG=/etc/sysconfig/docker
-       # Add kata-runtime to the list of available runtimes, if not already listed
-       $ grep -qE "^ *DOCKER_OPTS=.+--add-runtime[= ] *kata-runtime" $DOCKER_SYSCONFIG || sudo -E sed -i -E "s|^( *DOCKER_OPTS=.+)\" *$|\1 --add-runtime kata-runtime=/usr/bin/kata-runtime\"|g" $DOCKER_SYSCONFIG
-       # If a current default runtime is specified, overwrite it with kata-runtime
-       $ sudo -E sed -i -E "s|^( *DOCKER_OPTS=.+--default-runtime[= ] *)[^ \"]+(.*\"$)|\1kata-runtime\2|g" $DOCKER_SYSCONFIG
-       # Add kata-runtime as default runtime, if no default runtime is specified
-       $ grep -qE "^ *DOCKER_OPTS=.+--default-runtime" $DOCKER_SYSCONFIG || sudo -E sed -i -E "s|^( *DOCKER_OPTS=.+)(\"$)|\1 --default-runtime=kata-runtime\2|g" $DOCKER_SYSCONFIG
-       ```
-
-   2. Docker `daemon.json`
-
-      Create docker configuration folder.
-
-      ```
-      $ sudo mkdir -p /etc/docker
-      ```
-
-      Add the following definitions to `/etc/docker/daemon.json`:
-
-      ```json
-      {
-        "default-runtime": "kata-runtime",
-        "runtimes": {
-          "kata-runtime": {
-            "path": "/usr/bin/kata-runtime"
-          }
-        }
-      }
-      ```
-
-3. Restart the Docker systemd service with the following commands:
-
-   ```bash
-   $ sudo systemctl daemon-reload
-   $ sudo systemctl restart docker
-   ```
-
-4. Run Kata Containers
-
-   You are now ready to run Kata Containers:
-
-   ```bash
-   $ sudo docker run busybox uname -a
-   ```
-
-   The previous command shows details of the kernel version running inside the
-   container, which is different to the host kernel version.

docs/install/docker/opensuse-leap-docker-install.md

@@ -1,14 +0,0 @@
-# Install Docker for Kata Containers on openSUSE Leap
-
-Follow the instructions in the generic [openSUSE Docker install guide](opensuse-docker-install.md).
-<!--
-You can ignore the content of this comment.
-(test code run by test-install-docs.sh to validate code blocks this document)
-
-```bash
-$ echo "NOTE: this document is just a link to the generic openSUSE install guide located at:
-https://raw.githubusercontent.com/kata-containers/documentation/master/install/docker/opensuse-docker-install.md
-
-Please download this file and run kata-doc-to-script.sh again."
-```
--->

docs/install/docker/opensuse-tumbleweed-docker-install.md

@@ -1,14 +0,0 @@
-# Install Docker for Kata Containers on openSUSE Tumbleweed
-
-Follow the instructions in the generic [openSUSE Docker install guide](opensuse-docker-install.md).
-<!--
-You can ignore the content of this comment.
-(test code run by test-install-docs.sh to validate code blocks this document)
-
-```bash
-$ echo "NOTE: this document is just a link to the generic openSUSE install guide located at:
-https://raw.githubusercontent.com/kata-containers/documentation/master/install/docker/opensuse-docker-install.md
-
-Please download this file and run kata-doc-to-script.sh again."
-```
--->

docs/install/docker/rhel-docker-install.md

@@ -1,76 +0,0 @@
-# Install Docker for Kata Containers on RHEL
-
-> **Note:**
->
-> - This guide assumes you have
->   [already installed the Kata Containers packages](../rhel-installation-guide.md).
-
-1. Install the latest version of Docker with the following commands:
-
-   > **Notes:**
-   >
-   > - This step is only required if Docker is not installed on the system.
-   > - Docker version 18.09 [removed devicemapper support](https://github.com/kata-containers/documentation/issues/373).
-   >   If you wish to use a block based backend, see the options listed on https://github.com/kata-containers/documentation/issues/407.
-
-   ```bash
-   $ export rhel_devtoolset_version="7"
-   $ sudo subscription-manager repos --enable=rhel-${rhel_devtoolset_version}-server-extras-rpms
-   $ sudo yum -y install docker && systemctl enable --now docker
-   ```
-
-   For more information on installing Docker please refer to the
-   [Docker Guide](https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux_atomic_host/7/html-single/getting_started_with_containers/#getting_docker_in_rhel_7).
-
-2. Configure Docker to use Kata Containers by default with **ONE** of the following methods:
-
-   1. systemd (this is the default and is applied automatically if you select the
-      [automatic installation](https://github.com/kata-containers/documentation/tree/master/install#automatic-installation) option)
-
-       ```bash
-       $ sudo mkdir -p /etc/systemd/system/docker.service.d/
-       $ cat <<EOF | sudo tee /etc/systemd/system/docker.service.d/kata-containers.conf
-       [Service]
-       ExecStart=
-       ExecStart=/usr/bin/dockerd -D --add-runtime kata-runtime=/usr/bin/kata-runtime --default-runtime=kata-runtime
-       EOF
-       ```
-
-   2. Docker `daemon.json`
-
-      Create docker configuration folder.
-
-      ```
-      $ sudo mkdir -p /etc/docker
-      ```
-
-      Add the following definitions to `/etc/docker/daemon.json`:
-
-      ```json
-      {
-        "default-runtime": "kata-runtime",
-        "runtimes": {
-          "kata-runtime": {
-            "path": "/usr/bin/kata-runtime"
-          }
-        }
-      }
-      ```
-
-3. Restart the Docker systemd service with the following commands:
-
-   ```bash
-   $ sudo systemctl daemon-reload
-   $ sudo systemctl restart docker
-   ```
-
-4. Run Kata Containers
-
-   You are now ready to run Kata Containers:
-
-   ```bash
-   $ sudo docker run busybox uname -a
-   ```
-
-   The previous command shows details of the kernel version running inside the
-   container, which is different to the host kernel version.

docs/install/docker/sles-docker-install.md

@@ -1,74 +0,0 @@
-# Install Docker for Kata Containers on SLES
-
-> **Note:**
->
-> - This guide assumes you have
->   [already installed the Kata Containers packages](../sles-installation-guide.md).
-
-1. Install the latest version of Docker with the following commands:
-
-   > **Notes:**
-   >
-   > - This step is only required if Docker is not installed on the system.
-   > - Docker version 18.09 [removed devicemapper support](https://github.com/kata-containers/documentation/issues/373).
-   >   If you wish to use a block based backend, see the options listed on https://github.com/kata-containers/documentation/issues/407.
-
-   ```bash
-   $ sudo zypper -n install docker
-   ```
-
-   For more information on installing Docker please refer to the
-   [Docker Guide](https://www.suse.com/documentation/sles-12/singlehtml/book_sles_docker/book_sles_docker.html).
-
-2. Configure Docker to use Kata Containers by default with **ONE** of the following methods:
-
-   1. systemd (this is the default and is applied automatically if you select the
-      [automatic installation](https://github.com/kata-containers/documentation/tree/master/install#automatic-installation) option)
-
-       ```bash
-       $ sudo mkdir -p /etc/systemd/system/docker.service.d/
-       $ cat <<EOF | sudo tee /etc/systemd/system/docker.service.d/kata-containers.conf
-       [Service]
-       ExecStart=
-       ExecStart=/usr/bin/dockerd -D --add-runtime kata-runtime=/usr/bin/kata-runtime --default-runtime=kata-runtime
-       EOF
-       ```
-
-   2. Docker `daemon.json`
-
-      Create docker configuration folder.
-
-      ```
-      $ sudo mkdir -p /etc/docker
-      ```
-
-      Add the following definitions to `/etc/docker/daemon.json`:
-
-      ```json
-      {
-        "default-runtime": "kata-runtime",
-        "runtimes": {
-          "kata-runtime": {
-            "path": "/usr/bin/kata-runtime"
-          }
-        }
-      }
-      ```
-
-3. Restart the Docker systemd service with the following commands:
-
-   ```bash
-   $ sudo systemctl daemon-reload
-   $ sudo systemctl restart docker
-   ```
-
-4. Run Kata Containers
-
-   You are now ready to run Kata Containers:
-
-   ```bash
-   $ sudo docker run busybox uname -a
-   ```
-
-   The previous command shows details of the kernel version running inside the
-   container, which is different to the host kernel version.

docs/install/docker/ubuntu-docker-install.md

@@ -1,79 +0,0 @@
-# Install Docker for Kata Containers on Ubuntu
-
-> **Note:**
->
-> - This guide assumes you have
->   [already installed the Kata Containers packages](../ubuntu-installation-guide.md).
-
-1. Install the latest version of Docker with the following commands:
-
-   > **Notes:**
-   >
-   > - This step is only required if Docker is not installed on the system.
-   > - Docker version 18.09 [removed devicemapper support](https://github.com/kata-containers/documentation/issues/373).
-   >   If you wish to use a block based backend, see the options listed on https://github.com/kata-containers/documentation/issues/407.
-
-   ```bash
-   $ sudo -E apt-get -y install apt-transport-https ca-certificates software-properties-common
-   $ curl -sL https://download.docker.com/linux/ubuntu/gpg | sudo apt-key add -
-   $ arch=$(dpkg --print-architecture)
-   $ sudo -E add-apt-repository "deb [arch=${arch}] https://download.docker.com/linux/ubuntu $(lsb_release -cs) stable"
-   $ sudo -E apt-get update
-   $ sudo -E apt-get -y install docker-ce
-   ```
-
-   For more information on installing Docker please refer to the
-   [Docker Guide](https://docs.docker.com/engine/installation/linux/ubuntu).
-
-2. Configure Docker to use Kata Containers by default with **ONE** of the following methods:
-
-   1. systemd (this is the default and is applied automatically if you select the
-      [automatic installation](https://github.com/kata-containers/documentation/tree/master/install#automatic-installation) option)
-
-       ```bash
-       $ sudo mkdir -p /etc/systemd/system/docker.service.d/
-       $ cat <<EOF | sudo tee /etc/systemd/system/docker.service.d/kata-containers.conf
-       [Service]
-       ExecStart=
-       ExecStart=/usr/bin/dockerd -D --add-runtime kata-runtime=/usr/bin/kata-runtime --default-runtime=kata-runtime
-       EOF
-       ```
-
-   2. Docker `daemon.json`
-
-      Create docker configuration folder.
-
-      ```
-      $ sudo mkdir -p /etc/docker
-      ```
-
-      Add the following definitions to `/etc/docker/daemon.json`:
-
-      ```json
-      {
-        "default-runtime": "kata-runtime",
-        "runtimes": {
-          "kata-runtime": {
-            "path": "/usr/bin/kata-runtime"
-          }
-        }
-      }
-      ```
-
-3. Restart the Docker systemd service with the following commands:
-
-   ```bash
-   $ sudo systemctl daemon-reload
-   $ sudo systemctl restart docker
-   ```
-
-4. Run Kata Containers
-
-   You are now ready to run Kata Containers:
-
-   ```bash
-   $ sudo docker run busybox uname -a
-   ```
-
-   The previous command shows details of the kernel version running inside the
-   container, which is different to the host kernel version.

docs/install/fedora-installation-guide.md

@@ -3,15 +3,8 @@
 1. Install the Kata Containers components with the following commands:
 
    ```bash
-   $ source /etc/os-release
-   $ ARCH=$(arch)
-   $ BRANCH="${BRANCH:-master}"
-   $ sudo dnf -y install dnf-plugins-core
-   $ sudo -E dnf config-manager --add-repo "http://download.opensuse.org/repositories/home:/katacontainers:/releases:/${ARCH}:/${BRANCH}/Fedora_${VERSION_ID}/home:katacontainers:releases:${ARCH}:${BRANCH}.repo"
-   $ sudo -E dnf -y install kata-runtime kata-proxy kata-shim
+   $ sudo -E dnf -y install kata-runtime
    ```
 
 2. Decide which container manager to use and select the corresponding link that follows:
-
-   - [Docker](docker/fedora-docker-install.md)
-   - [Kubernetes](https://github.com/kata-containers/documentation/blob/master/Developer-Guide.md#run-kata-containers-with-kubernetes)
+   - [Kubernetes](../Developer-Guide.md#run-kata-containers-with-kubernetes)

docs/install/gce-installation-guide.md

@@ -101,7 +101,7 @@ If this fails, ensure you created your instance from the correct image and that
 
 The process for installing Kata itself on a virtualization-enabled VM is identical to that for bare metal.
 
-For detailed information to install Kata on your distribution of choice, see the [Kata Containers installation user guides](https://github.com/kata-containers/documentation/blob/master/install/README.md).
+For detailed information to install Kata on your distribution of choice, see the [Kata Containers installation user guides](../install/README.md).
 
 ## Create a Kata-enabled Image
 

docs/install/installing-with-kata-doc-to-script.md

@@ -5,7 +5,7 @@
 * [Docker Installation and Setup](#docker-installation-and-setup)
 
 ## Introduction
-Use [these installation instructions](README.md#supported-distributions) together with
+Use [these installation instructions](README.md#packaged-installation-methods) together with
 [`kata-doc-to-script`](https://github.com/kata-containers/tests/blob/master/.ci/kata-doc-to-script.sh)
 to generate installation bash scripts.
 

docs/install/installing-with-kata-manager.md

@@ -6,7 +6,7 @@
 * [Further Information](#further-information)
 
 ## Introduction
-`kata-manager` automates the Kata Containers installation procedure documented for [these Linux distributions](README.md#supported-distributions).
+`kata-manager` automates the Kata Containers installation procedure documented for [these Linux distributions](README.md#packaged-installation-methods).
 
 > **Note**:
 > - `kata-manager` requires `curl` and `sudo` installed on your system.

docs/install/minikube-installation-guide.md

@@ -54,7 +54,7 @@ to enable nested virtualization can be found on the
 [KVM Nested Guests page](https://www.linux-kvm.org/page/Nested_Guests)
 
 Alternatively, and for other architectures, the Kata Containers built in
-[`kata-check`](https://github.com/kata-containers/runtime#hardware-requirements)
+[`kata-check`](../../src/runtime/README.md#hardware-requirements)
 command can be used *inside Minikube* once Kata has been installed, to check for compatibility.
 
 ## Setting up Minikube
@@ -126,10 +126,10 @@ Containers components to help with this, and then use `kubectl` on the host (tha
 configured for you) to deploy them:
 
 ```sh
-$ git clone https://github.com/kata-containers/packaging.git
-$ cd packaging/kata-deploy
-$ kubectl apply -f kata-rbac.yaml
-$ kubectl apply -f kata-deploy.yaml
+$ git clone https://github.com/kata-containers/kata-containers.git
+$ cd kata-containers/tools/packaging/kata-deploy
+$ kubectl apply -f kata-rbac/base/kata-rbac.yaml
+$ kubectl apply -f kata-deploy/base/kata-deploy.yaml
 ```
 
 This installs the Kata Containers components into `/opt/kata` inside the Minikube node. It can take
@@ -166,7 +166,7 @@ $ kubectl apply -f https://raw.githubusercontent.com/kubernetes/node-api/master/
 Now register the `kata qemu` runtime with that class. This should result in no errors:
 
 ```sh
-$ cd packaging/kata-deploy/k8s-1.14
+$ cd kata-containers/tools/packaging/kata-deploy/k8s-1.14
 $ kubectl apply -f kata-qemu-runtimeClass.yaml
 ```
 
@@ -187,7 +187,7 @@ for more details.
 Perform the following action to launch a Kata Containers based Apache PHP pod:
 
 ```sh
-$ cd packaging/kata-deploy/examples
+$ cd kata-containers/tools/packaging/kata-deploy/examples
 $ kubectl apply -f test-deploy-kata-qemu.yaml
 ```
 

docs/install/opensuse-installation-guide.md

@@ -3,21 +3,8 @@
 1. Install the Kata Containers components with the following commands:
 
    ```bash
-   $ source /etc/os-release
-   $ DISTRO_REPO=$(sed "s/ /_/g" <<< "$NAME")
-   $ [ -n "$VERSION" ] && DISTRO_REPO+="_${VERSION}"
-   $ DISTRO_REPO=$(echo $DISTRO_REPO | tr -d ' ')
-   $ ARCH=$(arch)
-   $ BRANCH="${BRANCH:-master}"
-   $ REPO_ALIAS="kata-${BRANCH}"
-   $ PUBKEY="/tmp/rpm-signkey.pub"
-   $ curl -SsL -o "$PUBKEY" "https://raw.githubusercontent.com/kata-containers/tests/master/data/rpm-signkey.pub"
-   $ sudo -E rpm --import "$PUBKEY"
-   $ zypper lr "$REPO_ALIAS" && sudo -E zypper -n removerepo "$REPO_ALIAS"
-   $ sudo -E zypper addrepo --refresh "http://download.opensuse.org/repositories/home:/katacontainers:/releases:/${ARCH}:/${BRANCH}/${DISTRO_REPO}/" "$REPO_ALIAS"
-   $ sudo -E zypper -n install kata-runtime
+   $ sudo -E zypper -n install katacontainers
    ```
 
 2. Decide which container manager to use and select the corresponding link that follows:
-   - [Docker](docker/opensuse-docker-install.md)
    - [Kubernetes](../Developer-Guide.md#run-kata-containers-with-kubernetes)

docs/install/opensuse-leap-15.1-installation-guide.md

@@ -0,0 +1,11 @@
+# Install Kata Containers on openSUSE Leap 15.1
+
+1. Install the Kata Containers components with the following commands:
+
+   ```bash
+   $ sudo -E zypper addrepo --refresh "https://download.opensuse.org/repositories/devel:/kubic/openSUSE_Leap_15.1/devel:kubic.repo"
+   $ sudo -E zypper -n --gpg-auto-import-keys install katacontainers
+   ```
+
+2. Decide which container manager to use and select the corresponding link that follows:
+   - [Kubernetes](../Developer-Guide.md#run-kata-containers-with-kubernetes)

docs/install/opensuse-leap-installation-guide.md

@@ -1,19 +0,0 @@
-# Install Kata Containers on openSUSE Leap
-
-1. Install Kata Containers on openSUSE by following the instructions in the
-[openSUSE install guide](opensuse-installation-guide.md).
-<!--
-You can ignore the content of this comment.
-(test code run by test-install-docs.sh to validate code blocks this document)
-
-```bash
-$ echo "NOTE: this document is just a link to the generic openSUSE install guide located at:
-https://raw.githubusercontent.com/kata-containers/documentation/master/install/opensuse-installation-guide.md
-
-Please download this file and run kata-doc-to-script.sh again."
-```
--->
-
-2. Decide which container manager to use and select the corresponding link that follows:
-   - [Docker](docker/opensuse-leap-docker-install.md)
-   - [Kubernetes](../Developer-Guide.md#run-kata-containers-with-kubernetes)

docs/install/opensuse-tumbleweed-installation-guide.md

@@ -1,19 +0,0 @@
-# Install Kata Containers on openSUSE Tumbleweed
-
-1. Install Kata Containers on openSUSE by following the instructions in the
-[openSUSE install guide](opensuse-installation-guide.md).
-<!--
-You can ignore the content of this comment.
-(test code run by test-install-docs.sh to validate code blocks this document)
-
-```bash
-$ echo "NOTE: this document is just a link to the generic openSUSE install guide located at:
-https://raw.githubusercontent.com/kata-containers/documentation/master/install/opensuse-installation-guide.md
-
-Please download this file and run kata-doc-to-script.sh again."
-```
--->
-
-2. Decide which container manager to use and select the corresponding link that follows:
-   - [Docker](docker/opensuse-tumbleweed-docker-install.md)
-   - [Kubernetes](../Developer-Guide.md#run-kata-containers-with-kubernetes)

docs/install/rhel-installation-guide.md

@@ -1,16 +0,0 @@
-# Install Kata Containers on RHEL
-
-1. Install the Kata Containers components with the following commands:
-
-   ```bash
-   $ source /etc/os-release
-   $ ARCH=$(arch)
-   $ BRANCH="${BRANCH:-master}"
-   $ sudo -E yum-config-manager --add-repo "http://download.opensuse.org/repositories/home:/katacontainers:/releases:/${ARCH}:/${BRANCH}/RHEL_${VERSION_ID}/home:katacontainers:releases:${ARCH}:${BRANCH}.repo"
-   $ sudo -E yum -y install kata-runtime kata-proxy kata-shim
-   ```
-
-2. Decide which container manager to use and select the corresponding link that follows:
-
-   - [Docker](docker/rhel-docker-install.md)
-   - [Kubernetes](https://github.com/kata-containers/documentation/blob/master/Developer-Guide.md#run-kata-containers-with-kubernetes)

docs/install/sle-installation-guide.md

@@ -0,0 +1,13 @@
+# Install Kata Containers on SLE
+
+1. Install the Kata Containers components with the following commands:
+
+   ```bash
+   $ source /etc/os-release
+   $ DISTRO_VERSION=$(sed "s/-/_/g" <<< "$VERSION")
+   $ sudo -E zypper addrepo --refresh "https://download.opensuse.org/repositories/devel:/kubic/SLE_${DISTRO_VERSION}_Backports/devel:kubic.repo"
+   $ sudo -E zypper -n --gpg-auto-import-keys install katacontainers
+   ```
+
+2. Decide which container manager to use and select the corresponding link that follows:
+   - [Kubernetes](../Developer-Guide.md#run-kata-containers-with-kubernetes)

docs/install/sles-installation-guide.md

@@ -1,15 +0,0 @@
-# Install Kata Containers on SLES
-
-1. Install the Kata Containers components with the following commands:
-
-   ```bash
-   $ ARCH=$(arch)
-   $ BRANCH="${BRANCH:-master}"
-   $ sudo -E zypper addrepo "http://download.opensuse.org/repositories/home:/katacontainers:/releases:/${ARCH}:/${BRANCH}/SLE_15_SP1/home:katacontainers:releases:${ARCH}:${BRANCH}.repo"
-   $ sudo -E zypper -n --no-gpg-checks install kata-runtime kata-proxy kata-shim
-   ```
-
-2. Decide which container manager to use and select the corresponding link that follows:
-
-   - [Docker](docker/sles-docker-install.md)
-   - [Kubernetes](https://github.com/kata-containers/documentation/blob/master/Developer-Guide.md#run-kata-containers-with-kubernetes)

docs/install/ubuntu-installation-guide.md

@@ -12,6 +12,4 @@
    ```
 
 2. Decide which container manager to use and select the corresponding link that follows:
-
-   - [Docker](docker/ubuntu-docker-install.md)
-   - [Kubernetes](https://github.com/kata-containers/documentation/blob/master/Developer-Guide.md#run-kata-containers-with-kubernetes)
+   - [Kubernetes](../Developer-Guide.md#run-kata-containers-with-kubernetes)

docs/install/vexxhost-installation-guide.md

@@ -13,4 +13,4 @@ with v2).  The recommended machine type for container workloads is `v2-highcpu`
 
 ## Set up with distribution specific quick start
 
-Follow distribution specific [install guides](https://github.com/kata-containers/documentation/tree/master/install#supported-distributions).
+Follow distribution specific [install guides](../install/README.md#packaged-installation-methods).

docs/use-cases/Intel-GPU-passthrough-and-Kata.md

@@ -55,7 +55,7 @@ line.
 ## Install and configure Kata Containers
 
 To use this feature, you need Kata version 1.3.0 or above.
-Follow the [Kata Containers setup instructions](https://github.com/kata-containers/documentation/blob/master/install/README.md)
+Follow the [Kata Containers setup instructions](../install/README.md)
 to install the latest version of Kata.
 
 In order to pass a GPU to a Kata Container, you need to enable the `hotplug_vfio_on_root_bus`
@@ -82,12 +82,12 @@ CONFIG_DRM_I915_USERPTR=y
 ```
 
 Build the Kata Containers kernel with the previous config options, using the instructions
-described in [Building Kata Containers kernel](https://github.com/kata-containers/packaging/tree/master/kernel).
-For further details on building and installing guest kernels, see [the developer guide](https://github.com/kata-containers/documentation/blob/master/Developer-Guide.md#install-guest-kernel-images).
+described in [Building Kata Containers kernel](../../tools/packaging/kernel).
+For further details on building and installing guest kernels, see [the developer guide](../Developer-Guide.md#install-guest-kernel-images).
 
 There is an easy way to build a guest kernel that supports Intel GPU:
 ```
-## Build guest kernel with https://github.com/kata-containers/packaging/tree/master/kernel
+## Build guest kernel with ../../tools/packaging/kernel
 
 # Prepare (download guest kernel source, generate .config)
 $ ./build-kernel.sh -g intel -f setup

docs/use-cases/Nvidia-GPU-passthrough-and-Kata.md

@@ -72,7 +72,7 @@ Your host kernel needs to be booted with `intel_iommu=on` on the kernel command
 
 ## Install and configure Kata Containers
 To use non-large BARs devices (for example, Nvidia Tesla T4), you need Kata version 1.3.0 or above.
-Follow the [Kata Containers setup instructions](https://github.com/kata-containers/documentation/blob/master/install/README.md)
+Follow the [Kata Containers setup instructions](../install/README.md)
 to install the latest version of Kata.
 
 The following configuration in the Kata `configuration.toml` file as shown below can work:
@@ -131,13 +131,13 @@ It is worth checking that it is not enabled in your kernel configuration to prev
 
 
 Build the Kata Containers kernel with the previous config options,
-using the instructions described in [Building Kata Containers kernel](https://github.com/kata-containers/packaging/tree/master/kernel).
+using the instructions described in [Building Kata Containers kernel](../../tools/packaging/kernel).
 For further details on building and installing guest kernels,
-see [the developer guide](https://github.com/kata-containers/documentation/blob/master/Developer-Guide.md#install-guest-kernel-images).
+see [the developer guide](../Developer-Guide.md#install-guest-kernel-images).
 
 There is an easy way to build a guest kernel that supports Nvidia GPU:
 ```
-## Build guest kernel with https://github.com/kata-containers/packaging/tree/master/kernel
+## Build guest kernel with ../../tools/packaging/kernel
 
 # Prepare (download guest kernel source, generate .config)
 $ ./build-kernel.sh -v 4.19.86 -g nvidia -f setup

docs/use-cases/using-Intel-QAT-and-kata.md

@@ -194,9 +194,9 @@ $ ls -la /sys/bus/pci/drivers/vfio-pci
 
 This example automatically uses the latest Kata kernel supported by Kata. It
 follows the instructions from the
-[packaging kernel repository](https://github.com/kata-containers/packaging/tree/master/kernel)
+[packaging kernel repository](../../tools/packaging/kernel)
 and uses the latest Kata kernel
-[config](https://github.com/kata-containers/packaging/tree/master/kernel/configs).
+[config](../../tools/packaging/kernel/configs).
 There are some patches that must be installed as well, which the 
 `build-kernel.sh` script should automatically apply. If you are using a
 different kernel version, then you might need to manually apply them. Since

docs/use-cases/using-SPDK-vhostuser-and-kata.md

@@ -184,7 +184,7 @@ used for vhost-user devices.
 
 The base directory for vhost-user device is a configurable value,
 with the default being `/var/run/kata-containers/vhost-user`. It can be
-configured by parameter `vhost_user_store_path` in [Kata TOML configuration file](https://github.com/kata-containers/runtime/blob/master/README.md#configuration).
+configured by parameter `vhost_user_store_path` in [Kata TOML configuration file](../../src/runtime/README.md#configuration).
 
 Currently, the vhost-user storage device is not enabled by default, so
 the user should enable it explicitly inside the Kata TOML configuration

docs/use-cases/zun_kata.md

@@ -10,7 +10,7 @@ Currently, the instructions are based on the following links:
 
 - https://docs.openstack.org/zun/latest/admin/clear-containers.html
 
-- https://github.com/kata-containers/documentation/blob/master/install/ubuntu-installation-guide.md
+- ../install/ubuntu-installation-guide.md
 
 
 ## Install Git to use with DevStack
@@ -54,7 +54,7 @@ $ zun delete test
 
 ## Install Kata Containers
 
-Follow [these instructions](https://github.com/kata-containers/documentation/blob/master/install/ubuntu-installation-guide.md)
+Follow [these instructions](../install/ubuntu-installation-guide.md)
 to install the Kata Containers components.
 
 ## Update Docker with new Kata Containers runtime

pkg/logging/Cargo.toml

@@ -1,7 +1,7 @@
 [package]
 name = "logging"
 version = "0.1.0"
-authors = ["Tim Zhang <tim@hyper.sh>"]
+authors = ["The Kata Containers community <kata-dev@lists.katacontainers.io>"]
 edition = "2018"
 
 # See more keys and their definitions at https://doc.rust-lang.org/cargo/reference/manifest.html

snap/README.md

@@ -84,12 +84,12 @@ then a new configuration file can be [created](#configure-kata-containers)
 and [configured][7].
 
 [1]: https://docs.snapcraft.io/snaps/intro
-[2]: https://github.com/kata-containers/documentation/blob/master/design/architecture.md#root-filesystem-image
+[2]: ../docs/design/architecture.md#root-filesystem-image
 [3]: https://docs.snapcraft.io/reference/confinement#classic
 [4]: https://github.com/kata-containers/runtime#configuration
 [5]: https://docs.docker.com/engine/reference/commandline/dockerd
-[6]: https://github.com/kata-containers/documentation/blob/master/install/docker/ubuntu-docker-install.md
-[7]: https://github.com/kata-containers/documentation/blob/master/Developer-Guide.md#configure-to-use-initrd-or-rootfs-image
+[6]: ../docs/install/docker/ubuntu-docker-install.md
+[7]: ../docs/Developer-Guide.md#configure-to-use-initrd-or-rootfs-image
 [8]: https://snapcraft.io/kata-containers
-[9]: https://github.com/kata-containers/documentation/blob/master/Developer-Guide.md#run-kata-containers-with-docker
-[10]: https://github.com/kata-containers/documentation/blob/master/Developer-Guide.md#run-kata-containers-with-kubernetes
+[9]: ../docs/Developer-Guide.md#run-kata-containers-with-docker
+[10]: ../docs/Developer-Guide.md#run-kata-containers-with-kubernetes

snap/snapcraft.yaml

@@ -0,0 +1,323 @@
+name: kata-containers
+summary: Build lightweight VMs that seamlessly plug into the containers ecosystem
+description: |
+  Kata Containers is an open source project and community working to build a
+  standard implementation of lightweight Virtual Machines (VMs) that feel and
+  perform like containers, but provide the workload isolation and security
+  advantages of VMs
+confinement: classic
+adopt-info: metadata
+base: core20
+
+parts:
+  metadata:
+    plugin: nil
+    prime:
+      - -*
+    build-packages:
+      - git
+      - git-extras
+    override-pull: |
+      version="9999"
+      kata_url="https://github.com/kata-containers/kata-containers"
+
+      image_info="${SNAPCRAFT_IMAGE_INFO:-}"
+      snap_env="$(echo "${image_info}" | egrep -o "build_url.*" | egrep -o "snap.*build" | cut -d/ -f2)"
+
+      case "${snap_env}" in
+        stable)
+          # Get the latest stable version
+          version=$(git ls-remote --tags ${kata_url}  | egrep -o "refs.*" | egrep -v "\-alpha|\-rc|{}" | egrep -o "[[:digit:]]+\.[[:digit:]]+\.[[:digit:]]+" | sort -V -r | head -1)
+          git checkout ${version}
+        ;;
+
+        *-dev)
+          version="${snap_env}"
+        ;;
+      esac
+
+      snapcraftctl set-grade "stable"
+      snapcraftctl set-version "${version}"
+
+      # setup GOPATH - this repo dir should be there
+      export GOPATH=${SNAPCRAFT_STAGE}/gopath
+      kata_dir=${GOPATH}/src/github.com/${SNAPCRAFT_PROJECT_NAME}/${SNAPCRAFT_PROJECT_NAME}
+      mkdir -p $(dirname ${kata_dir})
+      ln -sf $(realpath "${SNAPCRAFT_STAGE}/..") ${kata_dir}
+
+  godeps:
+    after: [metadata]
+    plugin: nil
+    prime:
+      - -*
+    build-packages:
+      - curl
+    override-build: |
+      # put everything in stage
+      cd ${SNAPCRAFT_STAGE}
+
+      yq_path="./yq"
+      yq_pkg="github.com/mikefarah/yq"
+      goos="linux"
+      case "$(uname -m)" in
+        aarch64) goarch="arm64";;
+        ppc64le) goarch="ppc64le";;
+        x86_64) goarch="amd64";;
+        s390x) goarch="s390x";;
+        *) echo "unsupported architecture: $(uname -m)"; exit 1;;
+      esac
+
+      # Workaround to get latest release from github (to not use github token).
+      # Get the redirection to latest release on github.
+      yq_latest_url=$(curl -Ls -o /dev/null -w %{url_effective} "https://${yq_pkg}/releases/latest")
+      # The redirected url should include the latest release version
+      # https://github.com/mikefarah/yq/releases/tag/<VERSION-HERE>
+      yq_version=$(basename "${yq_latest_url}")
+      yq_url="https://${yq_pkg}/releases/download/${yq_version}/yq_${goos}_${goarch}"
+      curl -o "${yq_path}" -LSsf ${yq_url}
+      chmod +x ${yq_path}
+
+      kata_dir=gopath/src/github.com/${SNAPCRAFT_PROJECT_NAME}/${SNAPCRAFT_PROJECT_NAME}
+      version="$(${yq_path} r ${kata_dir}/versions.yaml languages.golang.meta.newest-version)"
+      tarfile="go${version}.${goos}-${goarch}.tar.gz"
+      curl -LO https://golang.org/dl/${tarfile}
+      tar -xf ${tarfile} --strip-components=1
+
+  image:
+    after: [godeps]
+    plugin: nil
+    build-packages:
+      - docker.io
+      - cpio
+      - git
+      - iptables
+      - software-properties-common
+      - uidmap
+      - gnupg2
+    override-build: |
+      yq=${SNAPCRAFT_STAGE}/yq
+
+      # set GOPATH
+      export GOPATH=${SNAPCRAFT_STAGE}/gopath
+      kata_dir=${GOPATH}/src/github.com/${SNAPCRAFT_PROJECT_NAME}/${SNAPCRAFT_PROJECT_NAME}
+
+      export GOROOT=${SNAPCRAFT_STAGE}
+      export PATH="${GOROOT}/bin:${PATH}"
+
+      if [ -n "$http_proxy" ]; then
+        echo "Setting proxy $http_proxy"
+        sudo -E systemctl set-environment http_proxy=$http_proxy || true
+        sudo -E systemctl set-environment https_proxy=$https_proxy || true
+      fi
+
+      # Copy yq binary. It's used in the container
+      mkdir -p "${GOPATH}/bin/"
+      cp -a "${yq}" "${GOPATH}/bin/"
+
+      echo "Unmasking docker service"
+      sudo -E systemctl unmask docker.service || true
+      sudo -E systemctl unmask docker.socket || true
+      echo "Adding $USER into docker group"
+      sudo -E gpasswd -a $USER docker
+      echo "Starting docker"
+      sudo -E systemctl start docker || true
+
+      cd ${kata_dir}/tools/osbuilder
+
+      # build image
+      export AGENT_VERSION=$(cat ${kata_dir}/VERSION)
+      export AGENT_INIT=yes
+      export USE_DOCKER=1
+      export DEBUG=1
+      case "$(uname -m)" in
+        aarch64|ppc64le|s390x)
+          sudo -E PATH=$PATH make initrd DISTRO=alpine
+        ;;
+        x86_64)
+          # In some build systems it's impossible to build a rootfs image, try with the initrd image
+          sudo -E PATH=$PATH make image DISTRO=clearlinux || sudo -E PATH=$PATH make initrd DISTRO=alpine
+        ;;
+        *) echo "unsupported architecture: $(uname -m)"; exit 1;;
+      esac
+
+      # Install image
+      kata_image_dir=${SNAPCRAFT_PART_INSTALL}/usr/share/kata-containers
+      mkdir -p ${kata_image_dir}
+      cp kata-containers*.img ${kata_image_dir}
+
+  runtime:
+    after: [godeps, image]
+    plugin: nil
+    build-attributes: [no-patchelf]
+    override-build: |
+      # set GOPATH
+      export GOPATH=${SNAPCRAFT_STAGE}/gopath
+      export GOROOT=${SNAPCRAFT_STAGE}
+      export PATH="${GOROOT}/bin:${PATH}"
+      kata_dir=${GOPATH}/src/github.com/${SNAPCRAFT_PROJECT_NAME}/${SNAPCRAFT_PROJECT_NAME}
+
+      cd ${kata_dir}/src/runtime
+
+      # setup arch
+      arch=$(uname -m)
+      if [ ${arch} = "ppc64le" ]; then
+        arch="ppc64"
+      fi
+
+      # build and install runtime
+      make \
+        PREFIX=/snap/${SNAPCRAFT_PROJECT_NAME}/current/usr \
+        SKIP_GO_VERSION_CHECK=1 \
+        QEMUCMD=qemu-system-$arch
+      make install \
+        PREFIX=/usr \
+        DESTDIR=${SNAPCRAFT_PART_INSTALL} \
+        SKIP_GO_VERSION_CHECK=1 \
+        QEMUCMD=qemu-system-$arch
+
+      if [ -e ${SNAPCRAFT_PART_INSTALL}/../../image/install/usr/share/kata-containers/kata-containers.img ]; then
+        # Use rootfs image by default
+        sed -i -e '/^initrd =/d' ${SNAPCRAFT_PART_INSTALL}/usr/share/defaults/${SNAPCRAFT_PROJECT_NAME}/configuration.toml
+      else
+        # Use initrd by default
+        sed -i -e '/^image =/d' ${SNAPCRAFT_PART_INSTALL}/usr/share/defaults/${SNAPCRAFT_PROJECT_NAME}/configuration.toml
+      fi
+
+  kernel:
+    after: [godeps, image]
+    plugin: nil
+    build-packages:
+      - libelf-dev
+      - curl
+      - build-essential
+      - bison
+      - flex
+    override-build: |
+      export GOPATH=${SNAPCRAFT_STAGE}/gopath
+      kata_dir=${GOPATH}/src/github.com/${SNAPCRAFT_PROJECT_NAME}/${SNAPCRAFT_PROJECT_NAME}
+
+      cd ${kata_dir}/tools/packaging/kernel
+
+      # Say 'no' to everithing, fix issues with incomplete .config files
+      yes "n" | ./build-kernel.sh setup
+      kernel_dir_prefix="kata-linux-"
+      cd ${kernel_dir_prefix}*
+      version=$(basename ${PWD} | sed 's|'"${kernel_dir_prefix}"'||' | cut -d- -f1)
+      make -j $(($(nproc)-1)) EXTRAVERSION=".container"
+
+      kernel_suffix=${version}.container
+      kata_kernel_dir=${SNAPCRAFT_PART_INSTALL}/usr/share/kata-containers
+      mkdir -p ${kata_kernel_dir}
+
+      # Install bz kernel
+      make install INSTALL_PATH=${kata_kernel_dir} EXTRAVERSION=".container" || true
+      vmlinuz_name=vmlinuz-${kernel_suffix}
+      ln -sf ${vmlinuz_name} ${kata_kernel_dir}/vmlinuz.container
+
+      # Install raw kernel
+      vmlinux_name=vmlinux-${kernel_suffix}
+      cp vmlinux ${kata_kernel_dir}/${vmlinux_name}
+      ln -sf ${vmlinux_name} ${kata_kernel_dir}/vmlinux.container
+
+  qemu:
+    plugin: make
+    after: [godeps, runtime]
+    build-packages:
+      - gcc
+      - python
+      - zlib1g-dev
+      - libcap-ng-dev
+      - libglib2.0-dev
+      - libpixman-1-dev
+      - libnuma-dev
+      - libltdl-dev
+      - libcap-dev
+      - libattr1-dev
+      - libfdt-dev
+      - curl
+      - libcapstone-dev
+      - bc
+      - libblkid-dev
+      - libffi-dev
+      - libmount-dev
+      - libselinux1-dev
+    override-build: |
+      yq=${SNAPCRAFT_STAGE}/yq
+      export GOPATH=${SNAPCRAFT_STAGE}/gopath
+      kata_dir=${GOPATH}/src/github.com/${SNAPCRAFT_PROJECT_NAME}/${SNAPCRAFT_PROJECT_NAME}
+
+      versions_file="${kata_dir}/versions.yaml"
+      # arch-specific definition
+      case "$(uname -m)" in
+        "aarch64")
+          branch="$(${yq} r ${versions_file} assets.hypervisor.qemu.architecture.aarch64.branch)"
+          url="$(${yq} r ${versions_file} assets.hypervisor.qemu.url)"
+          commit="$(${yq} r ${versions_file} assets.hypervisor.qemu.architecture.aarch64.commit)"
+          patches_dir="${kata_dir}/tools/packaging/obs-packaging/qemu-aarch64/patches/"
+        ;;
+
+        *)
+          branch="$(${yq} r ${versions_file} assets.hypervisor.qemu.tag)"
+          url="$(${yq} r ${versions_file} assets.hypervisor.qemu.url)"
+          commit=""
+          patches_dir="${kata_dir}/tools/packaging/qemu/patches/$(echo ${branch} | sed -e 's/.[[:digit:]]*$//' -e 's/^v//').x"
+        ;;
+      esac
+
+      # download source
+      qemu_dir=${SNAPCRAFT_STAGE}/qemu
+      git clone --branch ${branch} --single-branch ${url} "${qemu_dir}"
+      cd ${qemu_dir}
+      [ -z "${commit}" ] || git checkout ${commit}
+
+      [ -n "$(ls -A ui/keycodemapdb)" ] || git clone https://github.com/qemu/keycodemapdb ui/keycodemapdb/
+      [ -n "$(ls -A capstone)" ] || git clone https://github.com/qemu/capstone capstone
+
+      # Apply patches
+      for patch in ${patches_dir}/*.patch; do
+        echo "Applying $(basename "$patch") ..."
+        patch \
+          --batch \
+          --forward \
+          --strip 1 \
+          --input "$patch"
+      done
+
+      # Only x86_64 supports libpmem
+      [ "$(uname -m)" = "x86_64" ] && sudo apt-get --no-install-recommends install -y apt-utils ca-certificates libpmem-dev
+
+      configure_hypervisor=${kata_dir}/tools/packaging/scripts/configure-hypervisor.sh
+      chmod +x ${configure_hypervisor}
+      # static build
+      echo "$(${configure_hypervisor} -s qemu) \
+        --disable-rbd
+        --prefix=/snap/${SNAPCRAFT_PROJECT_NAME}/current/usr \
+        --datadir=/snap/${SNAPCRAFT_PROJECT_NAME}/current/usr/share \
+        --libexecdir=/snap/${SNAPCRAFT_PROJECT_NAME}/current/usr/libexec/qemu" \
+        | xargs ./configure
+
+      # Copy QEMU configurations (Kconfigs)
+      cp -a ${kata_dir}/tools/packaging/qemu/default-configs/* default-configs/
+
+      # build and install
+      make -j $(($(nproc)-1))
+      make install DESTDIR=${SNAPCRAFT_PART_INSTALL}
+    prime:
+      - -snap/
+      - -usr/bin/qemu-ga
+      - -usr/bin/qemu-pr-helper
+      - -usr/bin/virtfs-proxy-helper
+      - -usr/include/
+      - -usr/libexec/
+      - -usr/share/applications/
+      - -usr/share/icons/
+      - -usr/var/
+      - usr/*
+      - lib/*
+    organize:
+      # Hack: move qemu to /
+      "snap/kata-containers/current/": "./"
+
+apps:
+  runtime:
+    command: usr/bin/containerd-shim-kata-v2

src/agent/Cargo.lock

@@ -1,13 +1,10 @@
 # This file is automatically @generated by Cargo.
 # It is not intended for manual editing.
 [[package]]
-name = "addr2line"
-version = "0.12.1"
+name = "adler32"
+version = "1.0.4"
 source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "a49806b9dadc843c61e7c97e72490ad7f7220ae249012fbda9ad0609457c0543"
-dependencies = [
- "gimli",
-]
+checksum = "5d2e7343e7fc9de883d1b0341e0b13970f764c14101234857d2ddafa1cb1cac2"
 
 [[package]]
 name = "aho-corasick"
@@ -18,12 +15,30 @@ dependencies = [
  "memchr",
 ]
 
+[[package]]
+name = "anyhow"
+version = "1.0.32"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "6b602bfe940d21c130f3895acd65221e8a61270debe89d628b9cb4e3ccb8569b"
+
 [[package]]
 name = "arc-swap"
 version = "0.4.6"
 source = "registry+https://github.com/rust-lang/crates.io-index"
 checksum = "b585a98a234c46fc563103e9278c9391fde1f4e6850334da895d27edb9580f62"
 
+[[package]]
+name = "arrayref"
+version = "0.3.6"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "a4c527152e37cf757a3f78aae5a06fbeefdb07ccc535c980a3208ee3060dd544"
+
+[[package]]
+name = "arrayvec"
+version = "0.5.1"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "cff77d8686867eceff3105329d4698d96c2391c176d5d03adc90c7389162b5b8"
+
 [[package]]
 name = "autocfg"
 version = "1.0.0"
@@ -31,17 +46,10 @@ source = "registry+https://github.com/rust-lang/crates.io-index"
 checksum = "f8aac770f1885fd7e387acedd76065302551364496e46b3dd00860b2f8359b9d"
 
 [[package]]
-name = "backtrace"
-version = "0.3.48"
+name = "base64"
+version = "0.11.0"
 source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "0df2f85c8a2abbe3b7d7e748052fdd9b76a0458fdeb16ad4223f5eca78c7c130"
-dependencies = [
- "addr2line",
- "cfg-if",
- "libc",
- "object",
- "rustc-demangle",
-]
+checksum = "b41b7ea54a0c9d92199de89e20e58d49f02f8e699814ef3fdf266f6f748d15c7"
 
 [[package]]
 name = "bitflags"
@@ -49,6 +57,17 @@ version = "1.2.1"
 source = "registry+https://github.com/rust-lang/crates.io-index"
 checksum = "cf1de2fe8c75bc145a2f577add951f8134889b4795d47466a54a5c846d691693"
 
+[[package]]
+name = "blake2b_simd"
+version = "0.5.10"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "d8fb2d74254a3a0b5cac33ac9f8ed0e44aa50378d9dbb2e5d83bd21ed1dc2c8a"
+dependencies = [
+ "arrayref",
+ "arrayvec",
+ "constant_time_eq",
+]
+
 [[package]]
 name = "byteorder"
 version = "1.3.4"
@@ -78,6 +97,17 @@ version = "0.1.10"
 source = "registry+https://github.com/rust-lang/crates.io-index"
 checksum = "4785bdd1c96b2a846b2bd7cc02e86b6b3dbf14e7e53446c4f54c92a361040822"
 
+[[package]]
+name = "cgroups"
+version = "0.1.1-alpha.0"
+source = "git+https://github.com/kata-containers/cgroups-rs?branch=stable-0.1.1#8717524f2c95aacd30768b6f0f7d7f2fddef5cac"
+dependencies = [
+ "libc",
+ "log",
+ "nix 0.18.0",
+ "regex",
+]
+
 [[package]]
 name = "chrono"
 version = "0.4.11"
@@ -89,6 +119,30 @@ dependencies = [
  "time",
 ]
 
+[[package]]
+name = "cloudabi"
+version = "0.0.3"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "ddfc5b9aa5d4507acaf872de71051dfd0e309860e88966e1051e462a077aac4f"
+dependencies = [
+ "bitflags",
+]
+
+[[package]]
+name = "constant_time_eq"
+version = "0.1.5"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "245097e9a4535ee1e3e3931fcfcd55a796a44c643e8596ff6566d68f09b87bbc"
+
+[[package]]
+name = "crc32fast"
+version = "1.2.0"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "ba125de2af0df55319f41944744ad91c71113bf74a4646efff39afe1f6842db1"
+dependencies = [
+ "cfg-if",
+]
+
 [[package]]
 name = "crossbeam-channel"
 version = "0.4.2"
@@ -110,6 +164,26 @@ dependencies = [
  "lazy_static",
 ]
 
+[[package]]
+name = "dirs"
+version = "3.0.1"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "142995ed02755914747cc6ca76fc7e4583cd18578746716d0508ea6ed558b9ff"
+dependencies = [
+ "dirs-sys",
+]
+
+[[package]]
+name = "dirs-sys"
+version = "0.3.5"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "8e93d7f5705de3e49895a2b5e0b8855a1c27f080192ae9c32a6432d50741a57a"
+dependencies = [
+ "libc",
+ "redox_users",
+ "winapi",
+]
+
 [[package]]
 name = "errno"
 version = "0.2.5"
@@ -137,10 +211,15 @@ version = "0.12.2"
 source = "registry+https://github.com/rust-lang/crates.io-index"
 checksum = "d371106cc88ffdfb1eabd7111e432da544f16f3e2d7bf1dfe8bf575f1df045cd"
 dependencies = [
- "backtrace",
  "version_check",
 ]
 
+[[package]]
+name = "fnv"
+version = "1.0.7"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "3f9eec918d3f24069decb9af1554cad7c880e2da24a9afd88aca000531ab82c1"
+
 [[package]]
 name = "futures"
 version = "0.1.29"
@@ -165,10 +244,10 @@ dependencies = [
 ]
 
 [[package]]
-name = "gimli"
-version = "0.21.0"
+name = "hex"
+version = "0.4.2"
 source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "bcc8e0c9bce37868955864dbecd2b1ab2bdf967e6f28066d65aaac620444b65c"
+checksum = "644f9158b2f133fd50f5fb3242878846d9eb792e445c893805ff0e3824006e35"
 
 [[package]]
 name = "itoa"
@@ -180,7 +259,8 @@ checksum = "b8b7a7c0c47db5545ed3fef7468ee7bb5b74691498139e4b3f6a20685dc6dd8e"
 name = "kata-agent"
 version = "0.1.0"
 dependencies = [
- "error-chain",
+ "anyhow",
+ "cgroups",
  "lazy_static",
  "libc",
  "logging",
@@ -188,6 +268,8 @@ dependencies = [
  "nix 0.17.0",
  "oci",
  "prctl",
+ "procfs",
+ "prometheus",
  "protobuf",
  "protocols",
  "regex",
@@ -210,9 +292,36 @@ checksum = "e2abad23fbc42b3700f2f279844dc832adb2b2eb069b2df918f455c4e18cc646"
 
 [[package]]
 name = "libc"
-version = "0.2.70"
+version = "0.2.77"
 source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "3baa92041a6fec78c687fa0cc2b3fae8884f743d672cf551bed1d6dac6988d0f"
+checksum = "f2f96b10ec2560088a8e76961b00d47107b3a625fecb76dedb29ee7ccbf98235"
+
+[[package]]
+name = "libflate"
+version = "1.0.0"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "a1fbe6b967a94346446d37ace319ae85be7eca261bb8149325811ac435d35d64"
+dependencies = [
+ "adler32",
+ "crc32fast",
+ "libflate_lz77",
+ "rle-decode-fast",
+]
+
+[[package]]
+name = "libflate_lz77"
+version = "1.0.0"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "3286f09f7d4926fc486334f28d8d2e6ebe4f7f9994494b6dab27ddfad2c9b11b"
+
+[[package]]
+name = "lock_api"
+version = "0.3.4"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "c4da24a77a3d8a6d4862d95f72e6fdb9c09a643ecdb402d754004a557f2bec75"
+dependencies = [
+ "scopeguard",
+]
 
 [[package]]
 name = "log"
@@ -284,6 +393,18 @@ dependencies = [
  "void",
 ]
 
+[[package]]
+name = "nix"
+version = "0.18.0"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "83450fe6a6142ddd95fb064b746083fc4ef1705fe81f64a64e1d4b39f54a1055"
+dependencies = [
+ "bitflags",
+ "cc",
+ "cfg-if",
+ "libc",
+]
+
 [[package]]
 name = "num-integer"
 version = "0.1.42"
@@ -303,12 +424,6 @@ dependencies = [
  "autocfg",
 ]
 
-[[package]]
-name = "object"
-version = "0.19.0"
-source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "9cbca9424c482ee628fa549d9c812e2cd22f1180b9222c9200fdfa6eb31aecb2"
-
 [[package]]
 name = "oci"
 version = "0.1.0"
@@ -319,6 +434,30 @@ dependencies = [
  "serde_json",
 ]
 
+[[package]]
+name = "parking_lot"
+version = "0.10.2"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "d3a704eb390aafdc107b0e392f56a82b668e3a71366993b5340f5833fd62505e"
+dependencies = [
+ "lock_api",
+ "parking_lot_core",
+]
+
+[[package]]
+name = "parking_lot_core"
+version = "0.7.2"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "d58c7c768d4ba344e3e8d72518ac13e259d7c7ade24167003b8488e10b6740a3"
+dependencies = [
+ "cfg-if",
+ "cloudabi",
+ "libc",
+ "redox_syscall",
+ "smallvec",
+ "winapi",
+]
+
 [[package]]
 name = "path-absolutize"
 version = "1.2.1"
@@ -351,7 +490,7 @@ source = "registry+https://github.com/rust-lang/crates.io-index"
 checksum = "059a34f111a9dee2ce1ac2826a68b24601c4298cfeb1a587c3cb493d5ab46f52"
 dependencies = [
  "libc",
- "nix 0.17.0",
+ "nix 0.18.0",
 ]
 
 [[package]]
@@ -363,6 +502,37 @@ dependencies = [
  "unicode-xid",
 ]
 
+[[package]]
+name = "procfs"
+version = "0.7.9"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "c434e93ef69c216e68e4f417c927b4f31502c3560b72cfdb6827e2321c5c6b3e"
+dependencies = [
+ "bitflags",
+ "byteorder",
+ "chrono",
+ "hex",
+ "lazy_static",
+ "libc",
+ "libflate",
+]
+
+[[package]]
+name = "prometheus"
+version = "0.9.0"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "dd0ced56dee39a6e960c15c74dc48849d614586db2eaada6497477af7c7811cd"
+dependencies = [
+ "cfg-if",
+ "fnv",
+ "lazy_static",
+ "libc",
+ "procfs",
+ "protobuf",
+ "spin",
+ "thiserror",
+]
+
 [[package]]
 name = "protobuf"
 version = "2.14.0"
@@ -453,6 +623,17 @@ version = "0.1.56"
 source = "registry+https://github.com/rust-lang/crates.io-index"
 checksum = "2439c63f3f6139d1b57529d16bc3b8bb855230c8efcc5d3a896c8bea7c3b1e84"
 
+[[package]]
+name = "redox_users"
+version = "0.3.4"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "09b23093265f8d200fa7b4c2c76297f47e681c655f6f1285a8780d6a022f7431"
+dependencies = [
+ "getrandom",
+ "redox_syscall",
+ "rust-argon2",
+]
+
 [[package]]
 name = "regex"
 version = "1.3.7"
@@ -481,17 +662,31 @@ dependencies = [
 ]
 
 [[package]]
-name = "rustc-demangle"
-version = "0.1.16"
+name = "rle-decode-fast"
+version = "1.0.1"
 source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "4c691c0e608126e00913e33f0ccf3727d5fc84573623b8d65b2df340b5201783"
+checksum = "cabe4fa914dec5870285fa7f71f602645da47c486e68486d2b4ceb4a343e90ac"
+
+[[package]]
+name = "rust-argon2"
+version = "0.7.0"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "2bc8af4bda8e1ff4932523b94d3dd20ee30a87232323eda55903ffd71d2fb017"
+dependencies = [
+ "base64",
+ "blake2b_simd",
+ "constant_time_eq",
+ "crossbeam-utils",
+]
 
 [[package]]
 name = "rustjail"
 version = "0.1.0"
 dependencies = [
+ "anyhow",
  "caps",
- "error-chain",
+ "cgroups",
+ "dirs",
  "lazy_static",
  "libc",
  "nix 0.17.0",
@@ -506,8 +701,10 @@ dependencies = [
  "serde",
  "serde_derive",
  "serde_json",
+ "serial_test",
  "slog",
  "slog-scope",
+ "tempfile",
 ]
 
 [[package]]
@@ -559,6 +756,28 @@ dependencies = [
  "serde",
 ]
 
+[[package]]
+name = "serial_test"
+version = "0.5.0"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "1b15f74add9a9d4a3eb2bf739c9a427d266d3895b53d992c3a7c234fec2ff1f1"
+dependencies = [
+ "lazy_static",
+ "parking_lot",
+ "serial_test_derive",
+]
+
+[[package]]
+name = "serial_test_derive"
+version = "0.5.0"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "65f59259be9fc1bf677d06cc1456e97756004a1a5a577480f71430bd7c17ba33"
+dependencies = [
+ "proc-macro2",
+ "quote",
+ "syn",
+]
+
 [[package]]
 name = "signal-hook"
 version = "0.1.15"
@@ -626,6 +845,18 @@ dependencies = [
  "slog",
 ]
 
+[[package]]
+name = "smallvec"
+version = "1.4.2"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "fbee7696b84bbf3d89a1c2eccff0850e3047ed46bfcd2e92c29a2d074d57e252"
+
+[[package]]
+name = "spin"
+version = "0.5.2"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "6e63cff320ae2c57904679ba7cb63280a3dc4613885beafb148ee7bf9aa9042d"
+
 [[package]]
 name = "syn"
 version = "1.0.25"
@@ -657,6 +888,26 @@ dependencies = [
  "winapi",
 ]
 
+[[package]]
+name = "thiserror"
+version = "1.0.19"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "b13f926965ad00595dd129fa12823b04bbf866e9085ab0a5f2b05b850fbfc344"
+dependencies = [
+ "thiserror-impl",
+]
+
+[[package]]
+name = "thiserror-impl"
+version = "1.0.19"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "893582086c2f98cde18f906265a65b5030a074b1046c674ae898be6519a7f479"
+dependencies = [
+ "proc-macro2",
+ "quote",
+ "syn",
+]
+
 [[package]]
 name = "thread_local"
 version = "1.0.1"

src/agent/Cargo.toml

@@ -1,7 +1,7 @@
 [package]
 name = "kata-agent"
 version = "0.1.0"
-authors = ["Yang Bo <bo@hyper.sh>"]
+authors = ["The Kata Containers community <kata-dev@lists.katacontainers.io>"]
 edition = "2018"
 
 [dependencies]
@@ -11,7 +11,6 @@ rustjail = { path = "rustjail" }
 protocols = { path = "protocols" }
 netlink = { path = "netlink", features = ["with-log", "with-agent-handler"] }
 lazy_static = "1.3.0"
-error-chain = "0.12.1"
 ttrpc = { git = "https://github.com/containerd/ttrpc-rust.git", branch="0.3.0" }
 protobuf = "=2.14.0"
 libc = "0.2.58"
@@ -30,6 +29,10 @@ slog = { version = "2.5.2", features = ["dynamic-keys", "max_level_trace", "rele
 slog-scope = "4.1.2"
 # for testing
 tempfile = "3.1.0"
+prometheus = { version = "0.9.0", features = ["process"] }
+procfs = "0.7.9"
+anyhow = "1.0.32"
+cgroups = { git = "https://github.com/kata-containers/cgroups-rs", branch = "stable-0.1.1"}
 
 [workspace]
 members = [

src/agent/Makefile

@@ -31,7 +31,21 @@ ifdef proto
 endif
 
 ARCH = $(shell uname -m)
-LIBC = musl
+LIBC ?= musl
+ifneq ($(LIBC),musl)
+    ifeq ($(LIBC),gnu)
+        override LIBC = gnu
+    else
+        $(error "ERROR: A non supported LIBC value was passed. Supported values are musl and gnu")
+    endif
+endif
+
+ifeq ($(ARCH), ppc64le)
+    override ARCH = powerpc64le
+    override LIBC = gnu
+    $(warning "WARNING: powerpc64le-unknown-linux-musl target is unavailable")
+endif
+
 TRIPLE = $(ARCH)-unknown-linux-$(LIBC)
 
 TARGET_PATH = target/$(TRIPLE)/$(BUILD_TYPE)/$(TARGET)
@@ -45,12 +59,27 @@ INIT := no
 # Path to systemd unit directory if installed as not init.
 UNIT_DIR := /usr/lib/systemd/system
 
+GENERATED_CODE = src/version.rs
+
+AGENT_NAME=$(TARGET)
+API_VERSION=0.0.1
+AGENT_VERSION=$(VERSION)
+
+GENERATED_REPLACEMENTS= \
+    AGENT_NAME \
+    AGENT_VERSION \
+    API_VERSION \
+    BINDIR \
+    COMMIT \
+    VERSION_COMMIT
 GENERATED_FILES :=
 
+GENERATED_FILES += $(GENERATED_CODE)
+
 ifeq ($(INIT),no)
     # Unit file to start kata agent in systemd systems
     UNIT_FILES = kata-agent.service
-    GENERATED_FILES := $(UNIT_FILES)
+    GENERATED_FILES += $(UNIT_FILES)
     # Target to be reached in systemd services
     UNIT_FILES += kata-containers.target
 endif
@@ -74,29 +103,32 @@ endef
 
 default: $(TARGET) show-header
 
-$(TARGET): $(TARGET_PATH)
+$(TARGET): $(GENERATED_CODE) $(TARGET_PATH)
 
 $(TARGET_PATH): $(SOURCES) | show-summary
 	@cargo build --target $(TRIPLE) --$(BUILD_TYPE)
 
+optimize: $(SOURCES) | show-summary show-header
+	@RUSTFLAGS='-C link-arg=-s' cargo build --target $(TRIPLE) --$(BUILD_TYPE)
+
 show-header:
 	@printf "%s - version %s (commit %s)\n\n" "$(TARGET)" "$(VERSION)" "$(COMMIT_MSG)"
 
 $(GENERATED_FILES): %: %.in
-	@sed \
-		-e 's|[@]bindir[@]|$(BINDIR)|g' \
-		-e 's|[@]kata-agent[@]|$(TARGET)|g' \
-		"$<" > "$@"
+	@sed $(foreach r,$(GENERATED_REPLACEMENTS),-e 's|@$r@|$($r)|g') "$<" > "$@"
 
 install: build-service
 	@install -D $(TARGET_PATH) $(DESTDIR)/$(BINDIR)/$(TARGET)
 
 clean:
 	@cargo clean
+	@rm -f $(GENERATED_FILES)
 
-check:
+test:
 	@cargo test --all --target $(TRIPLE)
 
+check: test
+
 run:
 	@cargo run --target $(TRIPLE)
 
@@ -126,7 +158,8 @@ help: show-summary
 .PHONY: \
 	help \
 	show-header \
-	show-summary
+	show-summary \
+	optimize
 
 generate-protocols:
 	protocols/hack/update-generated-proto.sh "${PROTO_FILE}"

src/agent/README.md

@@ -1,10 +1,10 @@
 # Kata Agent in Rust
 
-This is a rust version of the [`kata-agent`](https://github.com/kata-containers/kata-agent).
+This is a rust version of the [`kata-agent`](https://github.com/kata-containers/agent).
 
 In Denver PTG, [we discussed about re-writing agent in rust](https://etherpad.openstack.org/p/katacontainers-2019-ptg-denver-agenda):
 
-> In general, we all think about re-write agent in rust to reduce the footprint of agent. Moreover, Eric mentioned the possibility to stop using gRPC, which may have some impact on footprint. We may begin to do some PoC to show how much we could save by re-writing agent in rust.
+> In general, we all think about re-write agent in rust to reduce the footprint of agent. Moreover, Eric mentioned the possibility to stop using gRPC, which may have some impact on footprint. We may begin to do some POC to show how much we could save by re-writing agent in rust.
 
 After that, we drafted the initial code here, and any contributions are welcome.
 
@@ -18,7 +18,7 @@ After that, we drafted the initial code here, and any contributions are welcome.
 | exec/list process       | :white_check_mark: |
 | I/O stream              | :white_check_mark: |
 | Cgroups                 | :white_check_mark: |
-| Capabilities, rlimit, readonly path, masked path, users | :white_check_mark: |
+| Capabilities, `rlimit`, readonly path, masked path, users | :white_check_mark: |
 | container stats (`stats_container`)                     | :white_check_mark: |
 | Hooks                   | :white_check_mark: |
 | **Agent Features & APIs** |
@@ -28,7 +28,7 @@ After that, we drafted the initial code here, and any contributions are welcome.
 | network, interface/routes (`update_container`)   | :white_check_mark: |
 | File transfer API (`copy_file`)                  | :white_check_mark: |
 | Device APIs (`reseed_random_device`, , `online_cpu_memory`, `mem_hotplug_probe`, `set_guet_data_time`) | :white_check_mark: |
-| vsock support                                    | :white_check_mark: |
+| VSOCK support                                    | :white_check_mark: |
 | virtio-serial support                            | :heavy_multiplication_x: |
 | OCI Spec validator                               | :white_check_mark: |
 | **Infrastructures**|
@@ -38,30 +38,25 @@ After that, we drafted the initial code here, and any contributions are welcome.
 
 ## Getting Started
 
-### Dependencies
-The `rust-agent` depends on [`grpc-rs`](https://github.com/pingcap/grpc-rs) by PingCAP. However, the upstream `grpc-rs` and [gRPC](https://github.com/grpc/grpc) need some changes to be used here, which may take some time to be landed. Therefore, we created a temporary fork or `grpc-rs` here:
-- https://github.com/alipay/grpc-rs/tree/rust_agent
-
 ### Build from Source
-The rust-agent need to be built with rust nightly, and static linked with musl.
+The rust-agent need to be built with rust newer than 1.37, and static linked with `musl`.
 ```bash
 rustup target add x86_64-unknown-linux-musl
-git submodule update --init --recursive  
 sudo ln -s /usr/bin/g++ /bin/musl-g++  
 cargo build --target x86_64-unknown-linux-musl --release
 ```
 
 ## Run Kata CI with rust-agent
-   * Firstly, install kata as noted by ["how to install Kata"](https://github.com/kata-containers/documentation/blob/master/install/README.md)
-   * Secondly, build your own kata initrd/image following the steps in ["how to build your own initrd/image"](https://github.com/kata-containers/documentation/blob/master/Developer-Guide.md#create-and-install-rootfs-and-initrd-image).
+   * Firstly, install Kata as noted by ["how to install Kata"](../../docs/install/README.md)
+   * Secondly, build your own Kata initrd/image following the steps in ["how to build your own initrd/image"](../../docs/Developer-Guide.md#create-and-install-rootfs-and-initrd-image).
 notes: Please use your rust agent instead of the go agent when building your initrd/image.
-   * Clone the kata ci test cases from: https://github.com/kata-containers/tests.git, and then run the cri test with: 
+   * Clone the Kata CI test cases from: https://github.com/kata-containers/tests.git, and then run the CRI test with: 
 
 ```bash
 $sudo -E PATH=$PATH -E GOPATH=$GOPATH integration/containerd/shimv2/shimv2-tests.sh
 ```
 
 ## Mini Benchmark
-The memory of 'RssAnon' consumed by the go-agent and rust-agent as below:
+The memory of `RssAnon` consumed by the go-agent and rust-agent as below:
 go-agent: about 11M
 rust-agent: about 1.1M