docs: fix up developer guide for 2.0

1. Until we restore docker/moby support, we should use crictl as developer example. 2. Most of the hyperlinks should point to kata-containers repository. 3. There is no more standalone mode. Fixes: #767 Signed-off-by: Peng Tao <bergwolf@hyper.sh>
2025-06-27 15:57:09 +00:00 · 2020-09-21 13:57:04 +08:00 · 2020-09-21 13:57:04 +08:00 · f6877fa4de
commit f6877fa4de
parent 6d326f217c
1 changed files with 39 additions and 124 deletions
--- a/docs/Developer-Guide.md
+++ b/docs/Developer-Guide.md
@ -13,7 +13,6 @@
        * [journald rate limiting](#journald-rate-limiting)
            * [`systemd-journald` suppressing messages](#systemd-journald-suppressing-messages)
            * [Disabling `systemd-journald` rate limiting](#disabling-systemd-journald-rate-limiting)
-* [Build and install Kata shim](#build-and-install-kata-shim)
 * [Create and install rootfs and initrd image](#create-and-install-rootfs-and-initrd-image)
    * [Build a custom Kata agent - OPTIONAL](#build-a-custom-kata-agent---optional)
    * [Get the osbuilder](#get-the-osbuilder)
@ -30,9 +29,7 @@
 * [Install a hypervisor](#install-a-hypervisor)
    * [Build a custom QEMU](#build-a-custom-qemu)
        * [Build a custom QEMU for aarch64/arm64 - REQUIRED](#build-a-custom-qemu-for-aarch64arm64---required)
-* [Run Kata Containers with Docker](#run-kata-containers-with-docker)
-    * [Update the Docker systemd unit file](#update-the-docker-systemd-unit-file)
-    * [Create a container using Kata](#create-a-container-using-kata)
+* [Run Kata Containers with Containerd](#run-kata-containers-with-containerd)
 * [Run Kata Containers with Kubernetes](#run-kata-containers-with-kubernetes)
 * [Troubleshoot Kata Containers](#troubleshoot-kata-containers)
 * [Appendices](#appendices)
@ -47,9 +44,6 @@
        * [Connect to the virtual machine using the debug console](#connect-to-the-virtual-machine-using-the-debug-console)
        * [Obtain details of the image](#obtain-details-of-the-image)
    * [Capturing kernel boot logs](#capturing-kernel-boot-logs)
-    * [Running standalone](#running-standalone)
-        * [Create an OCI bundle](#create-an-oci-bundle)
-        * [Launch the runtime to create a container](#launch-the-runtime-to-create-a-container)

 # Warning

@ -76,7 +70,7 @@ You need to install the following to build Kata Containers components:
 - [golang](https://golang.org/dl)

  To view the versions of go known to work, see the `golang` entry in the
-  [versions database](https://github.com/kata-containers/runtime/blob/master/versions.yaml).
+  [versions database](../versions.yaml).

 - `make`.
 - `gcc` (required for building the shim and runtime).
@ -84,14 +78,14 @@ You need to install the following to build Kata Containers components:
 # Build and install the Kata Containers runtime

 ```
-$ go get -d -u github.com/kata-containers/runtime
-$ cd $GOPATH/src/github.com/kata-containers/runtime
+$ go get -d -u github.com/kata-containers/kata-containers
+$ cd $GOPATH/src/github.com/kata-containers/kata-containers/src/runtime
 $ make && sudo -E PATH=$PATH make install
 ```

 The build will create the following:

- runtime binary: `/usr/local/bin/kata-runtime`
+- runtime binary: `/usr/local/bin/kata-runtime` and `/usr/local/bin/containerd-shim-kata-v2`
 - configuration file: `/usr/share/defaults/kata-containers/configuration.toml`

 # Check hardware requirements
@ -242,13 +236,6 @@ Restart `systemd-journald` for the changes to take effect:
 $ sudo systemctl restart systemd-journald
 ```

-# Build and install Kata shim
-
-```
-$ go get -d -u github.com/kata-containers/shim
-$ cd $GOPATH/src/github.com/kata-containers/shim && make && sudo make install
-```
-
 # Create and install rootfs and initrd image

 ## Build a custom Kata agent - OPTIONAL
@ -258,14 +245,15 @@ $ cd $GOPATH/src/github.com/kata-containers/shim && make && sudo make install
 > - You should only do this step if you are testing with the latest version of the agent.

 ```
-$ go get -d -u github.com/kata-containers/agent
-$ cd $GOPATH/src/github.com/kata-containers/agent && make
+$ go get -d -u github.com/kata-containers/kata-containers
+$ cd $GOPATH/src/github.com/kata-containers/kata-containers/src/agent && make
 ```

 ## Get the osbuilder

 ```
-$ go get -d -u github.com/kata-containers/osbuilder
+$ go get -d -u github.com/kata-containers/kata-containers
+$ cd $GOPATH/src/github.com/kata-containers/kata-containers/tools/osbuilder
 ```

 ## Create a rootfs image
@ -276,16 +264,16 @@ able to run the `rootfs.sh` script with `USE_DOCKER=true` as expected in
 the following example.

 ```
-$ export ROOTFS_DIR=${GOPATH}/src/github.com/kata-containers/osbuilder/rootfs-builder/rootfs
+$ export ROOTFS_DIR=${GOPATH}/src/github.com/kata-containers/kata-containers/tools/osbuilder/rootfs-builder/rootfs
 $ sudo rm -rf ${ROOTFS_DIR}
-$ cd $GOPATH/src/github.com/kata-containers/osbuilder/rootfs-builder
+$ cd $GOPATH/src/github.com/kata-containers/kata-containers/tools/osbuilder/rootfs-builder
 $ script -fec 'sudo -E GOPATH=$GOPATH USE_DOCKER=true SECCOMP=no ./rootfs.sh ${distro}'
 ```
 You MUST choose one of `alpine`, `centos`, `clearlinux`, `debian`, `euleros`, `fedora`, `suse`, and `ubuntu` for `${distro}`. By default `seccomp` packages are not included in the rootfs image. Set `SECCOMP` to `yes` to include them.

 > **Note:**
 >
-> - Check the [compatibility matrix](https://github.com/kata-containers/osbuilder#platform-distro-compatibility-matrix) before creating rootfs.
+> - Check the [compatibility matrix](../tools/osbuilder/README.md#platform-distro-compatibility-matrix) before creating rootfs.
 > - You must ensure that the *default Docker runtime* is `runc` to make use of
 >   the `USE_DOCKER` variable. If that is not the case, remove the variable
 >   from the previous command. See [Checking Docker default runtime](#checking-docker-default-runtime).
@ -305,7 +293,7 @@ $ sudo install -o root -g root -m 0440 ../../agent/kata-containers.target ${ROOT
 ### Build a rootfs image

 ```
-$ cd $GOPATH/src/github.com/kata-containers/osbuilder/image-builder
+$ cd $GOPATH/src/github.com/kata-containers/kata-containers/tools/osbuilder/image-builder
 $ script -fec 'sudo -E USE_DOCKER=true ./image_builder.sh ${ROOTFS_DIR}'
 ```

@ -332,9 +320,9 @@ $ (cd /usr/share/kata-containers && sudo ln -sf "$image" kata-containers.img)
 ## Create an initrd image - OPTIONAL
 ### Create a local rootfs for initrd image
 ```
-$ export ROOTFS_DIR="${GOPATH}/src/github.com/kata-containers/osbuilder/rootfs-builder/rootfs"
+$ export ROOTFS_DIR="${GOPATH}/src/github.com/kata-containers/kata-containers/tools/osbuilder/rootfs-builder/rootfs"
 $ sudo rm -rf ${ROOTFS_DIR}
-$ cd $GOPATH/src/github.com/kata-containers/osbuilder/rootfs-builder
+$ cd $GOPATH/src/github.com/kata-containers/kata-containers/tools/osbuilder/rootfs-builder
 $ script -fec 'sudo -E GOPATH=$GOPATH AGENT_INIT=yes USE_DOCKER=true SECCOMP=no ./rootfs.sh ${distro}'
 ```
 `AGENT_INIT` controls if the guest image uses the Kata agent as the guest `init` process. When you create an initrd image,
@ -344,7 +332,7 @@ You MUST choose one of `alpine`, `centos`, `clearlinux`, `euleros`, and `fedora`

 > **Note:**
 >
-> - Check the [compatibility matrix](https://github.com/kata-containers/osbuilder#platform-distro-compatibility-matrix) before creating rootfs.
+> - Check the [compatibility matrix](../tools/osbuilder/README.md#platform-distro-compatibility-matrix) before creating rootfs.

 Optionally, add your custom agent binary to the rootfs with the following:
 ```
@ -354,7 +342,7 @@ $ sudo install -o root -g root -m 0550 -T ../../agent/kata-agent ${ROOTFS_DIR}/s
 ### Build an initrd image

 ```
-$ cd $GOPATH/src/github.com/kata-containers/osbuilder/initrd-builder
+$ cd $GOPATH/src/github.com/kata-containers/kata-containers/tools/osbuilder/initrd-builder
 $ script -fec 'sudo -E AGENT_INIT=yes USE_DOCKER=true ./initrd_builder.sh ${ROOTFS_DIR}'
 ```

@ -382,7 +370,7 @@ Your QEMU directory need to be prepared with source code. Alternatively, you can

 ```
 $ go get -d github.com/kata-containers/qemu
-$ qemu_branch=$(grep qemu-lite- ${GOPATH}/src/github.com/kata-containers/runtime/versions.yaml | cut -d '"' -f2)
+$ qemu_branch=$(grep qemu-lite- ${GOPATH}/src/github.com/kata-containers/kata-containers/versions.yaml | cut -d '"' -f2)
 $ cd ${GOPATH}/src/github.com/kata-containers/qemu
 $ git checkout -b $qemu_branch remotes/origin/$qemu_branch
 $ your_qemu_directory=${GOPATH}/src/github.com/kata-containers/qemu
@ -391,9 +379,9 @@ $ your_qemu_directory=${GOPATH}/src/github.com/kata-containers/qemu
 To build a version of QEMU using the same options as the default `qemu-lite` version , you could use the `configure-hypervisor.sh` script:

 ```
-$ go get -d github.com/kata-containers/packaging
+$ go get -d github.com/kata-containers/kata-containers/tools/packaging
 $ cd $your_qemu_directory
-$ ${GOPATH}/src/github.com/kata-containers/packaging/scripts/configure-hypervisor.sh qemu > kata.cfg
+$ ${GOPATH}/src/github.com/kata-containers/kata-containers/tools/packaging/scripts/configure-hypervisor.sh qemu > kata.cfg
 $ eval ./configure "$(cat kata.cfg)"
 $ make -j $(nproc)
 $ sudo -E make install
@ -412,27 +400,11 @@ $ go get -d github.com/kata-containers/tests
 $ script -fec 'sudo -E ${GOPATH}/src/github.com/kata-containers/tests/.ci/install_qemu.sh'
 ```

-# Run Kata Containers with Docker
-
-## Update the Docker systemd unit file
-
-```
-$ dockerUnit=$(systemctl show -p FragmentPath docker.service | cut -d "=" -f 2)
-$ unitFile=${dockerUnit:-/etc/systemd/system/docker.service.d/kata-containers.conf}
-$ test -e "$unitFile" || { sudo mkdir -p "$(dirname $unitFile)"; echo -e "[Service]\nType=simple\nExecStart=\nExecStart=/usr/bin/dockerd -D --default-runtime runc" | sudo tee "$unitFile"; }
-$ grep -q "kata-runtime=" $unitFile || sudo sed -i 's!^\(ExecStart=[^$].*$\)!\1 --add-runtime kata-runtime=/usr/local/bin/kata-runtime!g' "$unitFile"
-$ sudo systemctl daemon-reload
-$ sudo systemctl restart docker
-```
-
-## Create a container using Kata
-
-```
-$ sudo docker run -ti --runtime kata-runtime busybox sh
-```
+# Run Kata Containers with Containerd
+Refer to the [How to use Kata Containers and Containerd](how-to/containerd-kata.md) how-to guide.

 # Run Kata Containers with Kubernetes
-Refer to to the [Run Kata Containers with Kubernetes](how-to/run-kata-with-k8s.md) how-to guide.
+Refer to the [Run Kata Containers with Kubernetes](how-to/run-kata-with-k8s.md) how-to guide.

 # Troubleshoot Kata Containers

@ -452,18 +424,6 @@ To perform analysis on Kata logs, use the
 [`kata-log-parser`](https://github.com/kata-containers/tests/tree/master/cmd/log-parser)
 tool, which can convert the logs into formats (e.g. JSON, TOML, XML, and YAML).

-To obtain a full backtrace for the agent, proxy, runtime, or shim send the
-`SIGUSR1` signal to the process ID of the component. The component will send a
-backtrace to the system log on the host system and continue to run without
-interruption.
-
-For example, to obtain a backtrace for `kata-proxy`:
-
-```
-$ sudo kill -USR1 $kata_proxy_pid
-$ sudo journalctl -t kata-proxy
-```
-
 See [Set up a debug console](#set-up-a-debug-console).

 # Appendices
@ -486,12 +446,12 @@ the following steps (using rootfs or initrd image).

 > **Note:** The following debug console instructions assume a systemd-based guest
 > O/S image. This means you must create a rootfs for a distro that supports systemd.
-> Currently, all distros supported by [osbuilder](https://github.com/kata-containers/osbuilder) support systemd
+> Currently, all distros supported by [osbuilder](../tools/osbuilder) support systemd
 > except for Alpine Linux.
 >
 > Look for `INIT_PROCESS=systemd` in the `config.sh` osbuilder rootfs config file
 > to verify an osbuilder distro supports systemd for the distro you want to build rootfs for.
-> For an example, see the [Clear Linux config.sh file](https://github.com/kata-containers/osbuilder/blob/master/rootfs-builder/clearlinux/config.sh).
+> For an example, see the [Clear Linux config.sh file](../tools/osbuilder/blob/master/rootfs-builder/clearlinux/config.sh).
 >
 > For a non-systemd-based distro, create an equivalent system
 > service using that distro’s init system syntax. Alternatively, you can build a distro
@ -511,8 +471,8 @@ an additional `coreutils` package.
 For example using CentOS:

 ```
-$ cd $GOPATH/src/github.com/kata-containers/osbuilder/rootfs-builder
-$ export ROOTFS_DIR=${GOPATH}/src/github.com/kata-containers/osbuilder/rootfs-builder/rootfs
+$ cd $GOPATH/src/github.com/kata-containers/kata-containers/tools/osbuilder/rootfs-builder
+$ export ROOTFS_DIR=${GOPATH}/src/github.com/kata-containers/kata-containers/tools/osbuilder/rootfs-builder/rootfs
 $ script -fec 'sudo -E GOPATH=$GOPATH USE_DOCKER=true EXTRA_PKGS="bash coreutils" ./rootfs.sh centos'
 ```

@ -575,31 +535,18 @@ $ (cd /usr/share/kata-containers && sudo ln -sf "$name" kata-containers.img)
 **Note**: You should take care to undo this change after you finish debugging
 to avoid all subsequently created containers from using the debug image.

-### Ensure debug options are valid
-
-For the debug console to work, you **must** ensure that proxy debug is
-**disabled** in the configuration file. If proxy debug is enabled, you will
-not see any output when you connect to the virtual machine:
-
-```
-$ sudo mkdir -p /etc/kata-containers/
-$ sudo install -o root -g root -m 0640 /usr/share/defaults/kata-containers/configuration.toml /etc/kata-containers
-$ sudo awk '{if (/^\[proxy\.kata\]/) {got=1}; if (got == 1 && /^.*enable_debug/) {print "#enable_debug = true"; got=0; next; } else {print}}' /etc/kata-containers/configuration.toml > /tmp/configuration.toml
-$ sudo install -o root -g root -m 0640 /tmp/configuration.toml /etc/kata-containers/
-```
-
 ### Create a container

-Create a container as normal. For example using Docker:
+Create a container as normal. For example using crictl:

 ```
-$ sudo docker run -ti busybox sh
+$ sudo crictl run -r kata container.yaml pod.yaml
 ```

 ### Connect to the virtual machine using the debug console

 ```
-$ id=$(sudo docker ps -q --no-trunc)
+$ id=$(sudo crictl pods --no-trunc -q)
 $ console="/var/run/vc/vm/${id}/console.sock"
 $ sudo socat "stdin,raw,echo=0,escape=0x11" "unix-connect:${console}"
 ```
@ -612,7 +559,7 @@ To disconnect from the virtual machine, type `CONTROL+q` (hold down the
 ### Obtain details of the image

 If the image is created using
-[osbuilder](https://github.com/kata-containers/osbuilder), the following YAML
+[osbuilder](../tools/osbuilder), the following YAML
 file exists and contains details of the image and how it was created:

 ```
@ -629,54 +576,22 @@ command inside the container to view the kernel boot logs.
 If however you are unable to `exec` into the container, you can enable some debug
 options to have the kernel boot messages logged into the system journal.

-Which debug options you enable depends on if you are using the hypervisor `vsock` mode
-or not, as defined by the `use_vsock` setting in the `[hypervisor.qemu]` section of
-the configuration file. The following details the settings:
-
- For `use_vsock = false`:
-    - Set `enable_debug = true` in both the `[hypervisor.qemu]` and `[proxy.kata]` sections
- For `use_vsock = true`:
-    - Set `enable_debug = true` in both the `[hypervisor.qemu]` and `[shim.kata]` sections
+- Set `enable_debug = true` in the `[hypervisor.qemu]` and `[runtime]` sections

 For generic information on enabling debug in the configuration file, see the
 [Enable full debug](#enable-full-debug) section.

-The kernel boot messages will appear in the `kata-proxy` or `kata-shim` log appropriately,
+The kernel boot messages will appear in the `containerd` or `CRI-O` log appropriately,
 such as:

 ```bash
-$ sudo journalctl -t kata-proxy
+$ sudo journalctl -t containerd
 -- Logs begin at Thu 2020-02-13 16:20:40 UTC, end at Thu 2020-02-13 16:30:23 UTC. --
 ...
-Feb 13 16:20:56 minikube kata-proxy[17371]: time="2020-02-13T16:20:56.608714324Z" level=info msg="[    1.418768] brd: module loaded\n" name=kata-proxy pid=17371 sandbox=a13ffb2b9b5a66f7787bdae9a427fa954a4d21ec4031d0179eee2573986a8a6e source=agent
-Feb 13 16:20:56 minikube kata-proxy[17371]: time="2020-02-13T16:20:56.628493231Z" level=info msg="[    1.438612] loop: module loaded\n" name=kata-proxy pid=17371 sandbox=a13ffb2b9b5a66f7787bdae9a427fa954a4d21ec4031d0179eee2573986a8a6e source=agent
-Feb 13 16:20:56 minikube kata-proxy[17371]: time="2020-02-13T16:20:56.67707956Z" level=info msg="[    1.487165]  pmem0: p1\n" name=kata-proxy pid=17371 sandbox=a13ffb2b9b5a66f7787bdae9a427fa954a4d21ec4031d0179eee2573986a8a6e source=agent
+time="2020-09-15T14:56:23.095113803+08:00" level=debug msg="reading guest console" console-protocol=unix console-url=/run/vc/vm/ab9f633385d4987828d342e47554fc6442445b32039023eeddaa971c1bb56791/console.sock pid=107642 sandbox=ab9f633385d4987828d342e47554fc6442445b32039023eeddaa971c1bb56791 source=virtcontainers subsystem=sandbox vmconsole="[    0.395399] brd: module loaded"
+time="2020-09-15T14:56:23.102633107+08:00" level=debug msg="reading guest console" console-protocol=unix console-url=/run/vc/vm/ab9f633385d4987828d342e47554fc6442445b32039023eeddaa971c1bb56791/console.sock pid=107642 sandbox=ab9f633385d4987828d342e47554fc6442445b32039023eeddaa971c1bb56791 source=virtcontainers subsystem=sandbox vmconsole="[    0.402845] random: fast init done"
+time="2020-09-15T14:56:23.103125469+08:00" level=debug msg="reading guest console" console-protocol=unix console-url=/run/vc/vm/ab9f633385d4987828d342e47554fc6442445b32039023eeddaa971c1bb56791/console.sock pid=107642 sandbox=ab9f633385d4987828d342e47554fc6442445b32039023eeddaa971c1bb56791 source=virtcontainers subsystem=sandbox vmconsole="[    0.403544] random: crng init done"
+time="2020-09-15T14:56:23.105268162+08:00" level=debug msg="reading guest console" console-protocol=unix console-url=/run/vc/vm/ab9f633385d4987828d342e47554fc6442445b32039023eeddaa971c1bb56791/console.sock pid=107642 sandbox=ab9f633385d4987828d342e47554fc6442445b32039023eeddaa971c1bb56791 source=virtcontainers subsystem=sandbox vmconsole="[    0.405599] loop: module loaded"
+time="2020-09-15T14:56:23.121121598+08:00" level=debug msg="reading guest console" console-protocol=unix console-url=/run/vc/vm/ab9f633385d4987828d342e47554fc6442445b32039023eeddaa971c1bb56791/console.sock pid=107642 sandbox=ab9f633385d4987828d342e47554fc6442445b32039023eeddaa971c1bb56791 source=virtcontainers subsystem=sandbox vmconsole="[    0.421324] memmap_init_zone_device initialised 32768 pages in 12ms"
 ...
 ```
-
-## Running standalone
-
-It is possible to start the runtime without a container manager. This is
-mostly useful for testing and debugging purposes.
-
-### Create an OCI bundle
-
-To build an
-[OCI bundle](https://github.com/opencontainers/runtime-spec/blob/master/bundle.md),
-required by the runtime:
-
-```
-$ bundle="/tmp/bundle"
-$ rootfs="$bundle/rootfs"
-$ mkdir -p "$rootfs" && (cd "$bundle" && kata-runtime spec)
-$ sudo docker export $(sudo docker create busybox) | tar -C "$rootfs" -xvf -
-```
-
-### Launch the runtime to create a container
-
-Run the runtime standalone by providing it with the path to the
-previously-created [OCI bundle](#create-an-oci-bundle):
-
-```
-$ sudo kata-runtime --log=/dev/stdout run --bundle "$bundle" foo
-```