diff --git a/docs/.nav.yml b/docs/.nav.yml index 15dab995cc..7dc1b12238 100644 --- a/docs/.nav.yml +++ b/docs/.nav.yml @@ -15,7 +15,7 @@ nav: - Use Cases: - NVIDIA GPU Passthrough: use-cases/NVIDIA-GPU-passthrough-and-Kata-QEMU.md - NVIDIA vGPU: use-cases/NVIDIA-GPU-passthrough-and-Kata.md - - Intel Discrete GPU: use-cases/Intel-Discrete-GPU-passthrough-and-Kata.md + - Intel QAT: use-cases/using-Intel-QAT-and-kata.md - Contributing: - Documentation: doc-contributing.md - Misc: diff --git a/docs/use-cases/GPU-passthrough-and-Kata.md b/docs/use-cases/GPU-passthrough-and-Kata.md index 40b5297eea..8fdbfad6f4 100644 --- a/docs/use-cases/GPU-passthrough-and-Kata.md +++ b/docs/use-cases/GPU-passthrough-and-Kata.md @@ -2,5 +2,5 @@ Kata Containers supports passing certain GPUs from the host into the container. Select the GPU vendor for detailed information: -- [Intel Discrete GPUs](Intel-Discrete-GPU-passthrough-and-Kata.md)/[Intel Integrated GPUs](Intel-GPU-passthrough-and-Kata.md) - [NVIDIA GPUs](NVIDIA-GPU-passthrough-and-Kata.md) and [Enabling NVIDIA GPU workloads using GPU passthrough with Kata Containers](NVIDIA-GPU-passthrough-and-Kata-QEMU.md) +- PLACE HOLDER: for other GPU vendors (e.g., AMD, Intel) diff --git a/docs/use-cases/Intel-Discrete-GPU-passthrough-and-Kata.md b/docs/use-cases/Intel-Discrete-GPU-passthrough-and-Kata.md deleted file mode 100644 index 73ccf613c5..0000000000 --- a/docs/use-cases/Intel-Discrete-GPU-passthrough-and-Kata.md +++ /dev/null @@ -1,274 +0,0 @@ -# Using Intel Discrete GPU device with Kata Containers - -This guide covers the use case for passing Intel Discrete GPUs to Kata. -These include the Intel® Data Center GPU Max Series and Intel® Data Center GPU Flex Series. -For integrated GPUs please refer to [Integrate-Intel-GPUs-with-Kata](Intel-GPU-passthrough-and-Kata.md) - -> **Note:** These instructions are for a system that has an x86_64 CPU. - -An Intel Discrete GPU can be passed to a Kata Container using GPU passthrough, -or SR-IOV passthrough. - -In Intel GPU pass-through mode, an entire physical GPU is directly assigned to one VM. -In this mode of operation, the GPU is accessed exclusively by the Intel driver running in -the VM to which it is assigned. The GPU is not shared among VMs. - -With SR-IOV mode, it is possible to pass a Virtual GPU instance to a virtual machine. -With this, multiple Virtual GPU instances can be carved out of a single physical GPU -and be passed to different VMs, allowing the GPU to be shared. - -| Technology | Description | -|-|-| -| GPU passthrough | Physical GPU assigned to a single VM | -| SR-IOV passthrough | Physical GPU shared by multiple VMs | - -## Hardware Requirements - -Intel GPUs Recommended for Virtualization: - -- Intel® Data Center GPU Max Series (`Ponte Vecchio`) -- Intel® Data Center GPU Flex Series (`Arctic Sound-M`) -- Intel® Data Center GPU Arc Series - -The following steps outline the workflow for using an Intel Graphics device with Kata Containers. - -## Host BIOS requirements - -Hardware such as Intel Max and Flex series require larger PCI BARs. - -For large BAR devices, MMIO mapping above the 4GB address space should be enabled in the PCI configuration of the BIOS. - -Some hardware vendors use a different name in the BIOS, such as: - -- Above 4GB Decoding -- Memory Hole for PCI MMIO -- Memory Mapped I/O above 4GB - -## Host Kernel Requirements - -For device passthrough to work with the Max and Flex Series, an out of tree kernel driver is required. - -For Ubuntu 22.04 server, follow these instructions to install the out of tree GPU driver: -```bash -$ sudo apt update -$ sudo apt install -y gpg-agent wget -$ wget -qO - https://repositories.intel.com/gpu/intel-graphics.key | \ - sudo gpg --dearmor --output /usr/share/keyrings/intel-graphics.gpg -$ source /etc/os-release -$ echo "deb [arch=amd64 signed-by=/usr/share/keyrings/intel-graphics.gpg] https://repositories.intel.com/gpu/ubuntu ${VERSION_CODENAME}/lts/2350 unified" | \ - sudo tee /etc/apt/sources.list.d/intel-gpu-${VERSION_CODENAME}.list -$ sudo apt update -$ sudo apt install -y linux-headers-"$(uname -r)" flex bison intel-fw-gpu intel-i915-dkms xpu-smi -$ sudo reboot -``` -For support on other distributions, please refer to [DGPU-docs](https://dgpu-docs.intel.com/driver/installation.html) - -You can also install the driver from source which is maintained at [intel-gpu-i915-backports](https://github.com/intel-gpu/intel-gpu-i915-backports) -Detailed instructions for reference can be found at: https://github.com/intel-gpu/intel-gpu-i915-backports/blob/backport/main/docs/README_ubuntu.md. - -Below are the steps for installing the driver from source on an Ubuntu 22.04 LTS system: -```bash -$ export I915_BRANCH="backport/main" -$ git clone -b ${I915_BRANCH} --depth 1 https://github.com/intel-gpu/intel-gpu-i915-backports.git -$ cd intel-gpu-i915-backports/ -$ sudo apt install -y dkms make debhelper devscripts build-essential flex bison mawk -$ sudo apt install -y linux-headers-"$(uname -r)" linux-image-unsigned-"$(uname -r)" -$ make i915dkmsdeb-pkg -``` -The above make command will create Debian package in parent folder: `intel-i915-dkms_..deb` -Install the package as: -```bash -$ sudo dpkg -i intel-i915-dkms_..deb -$ sudo reboot -``` - -Additionally, verify that the following kernel configs are enabled for your host kernel: -``` -CONFIG_VFIO -CONFIG_VFIO_IOMMU_TYPE1 -CONFIG_VFIO_PCI -``` - -## Host kernel command line - -Your host kernel needs to be booted with `intel_iommu=on` and `i915.enable_iaf=0` on the kernel command -line. - -1. Run the following to change the kernel command line using grub: -```bash -$ sudo vim /etc/default/grub -``` - -2. At the end of the GRUB_CMDLINE_LINUX_DEFAULT append the below line: - -`intel_iommu=on iommu=pt i915.max_vfs=63 i915.enable_iaf=0` - -3. Update grub as per OS distribution: - -For Ubuntu: -```bash -$ sudo update-grub -``` - -For CentOS/RHEL: -```bash -$ sudo grub2-mkconfig -o /boot/grub2/grub.cfg -``` - -4. Reboot the system -```bash -$ sudo reboot -``` - -## Install and configure Kata Containers - -To use this feature, you need Kata version 1.3.0 or above. -Follow the [Kata Containers setup instructions](../install/README.md) -to install the latest version of Kata. - -To use large BARs devices (for example, NVIDIA Tesla P100), you need Kata version 1.11.0 or above. - -In order to pass a GPU to a Kata Container, you need to enable the `hotplug_vfio_on_root_bus` -configuration in the Kata `configuration.toml` file as shown below. - -```bash -$ sudo sed -i -e 's/^# *\(hotplug_vfio_on_root_bus\).*=.*$/\1 = true/g' /usr/share/defaults/kata-containers/configuration.toml -``` - -Make sure you are using the `q35` machine type by verifying `machine_type = "q35"` is -set in the `configuration.toml`. Make sure `pcie_root_port` is set to a positive value. - -After making the above changes, configuration in the `configuration.toml` should look like this: -``` -machine_type = "q35" - -hotplug_vfio_on_root_bus = true -pcie_root_port = 1 -``` - -## GPU passthrough with Kata Containers - -Use the following steps to pass an Intel discrete GPU with Kata: - -1. Find the Bus-Device-Function (BDF) for GPU device: - - ``` - $ sudo lspci -nn -D | grep Display - ``` - - Run the previous command to determine the BDF for the GPU device on host.
- From the previous output, PCI address `0000:29:00.0` is assigned to the hardware GPU device.
- We choose this BDF to use it later to unbind the GPU device from the host for the purpose of demonstration.
- -2. Find the IOMMU group for the GPU device: - - ```bash - $ BDF="0000:29:00.0" - $ readlink -e /sys/bus/pci/devices/$BDF/iommu_group - /sys/kernel/iommu_groups/27 - ``` - - The previous output shows that the GPU belongs to IOMMU group 27. - -3. Bind the GPU to the `vfio-pci` device driver: - - ```bash - $ BDF="0000:29:00.0" - $ DEV="/sys/bus/pci/devices/$BDF" - $ echo "vfio-pci" | sudo tee "$DEV"/driver_override - $ echo $BDF | sudo tee "$DEV"/driver/unbind - $ echo "$BDF" | sudo tee "/sys/bus/pci/drivers_probe" - ``` - - After you run the previous commands, the GPU is bound to `vfio-pci` driver.
- A new directory with the IOMMU group number is created under `/dev/vfio`: - - ```bash - $ ls -l /dev/vfio - total 0 - crw------- 1 root root 241, 0 May 18 15:38 27 - crw-rw-rw- 1 root root 10, 196 May 18 15:37 vfio - ``` - - Later, to return the device to the standard driver, we simply clear the - `driver_override` and re-probe the device, ex: - - ```bash - $ echo | sudo tee "$DEV/preferred_driver" - $ echo $BDF | sudo tee $DEV/driver/unbind - $ echo $BDF | sudo tee /sys/bus/pci/drivers_probe - ``` - -5. Start a Kata container with GPU device: - - ```bash - $ sudo ctr --debug run --runtime "io.containerd.kata.v2" --device "/dev/vfio/27" --rm -t "docker.io/library/archlinux:latest" arch uname -r - - ``` - - Run `lspci` within the container to verify the GPU device is seen in the list of - the PCI devices. Note the vendor-device id of the GPU ("8086:0bd5") in the `lspci` output. - -## SR-IOV mode for Intel Discrete GPUs - -Use the following steps to pass an Intel Graphics device in SR-IOV mode to a Kata Container: - -1. Find the BDF for GPU device: - - ```sh - $ sudo lspci -nn -D | grep Display - 0000:29:00.0 Display controller [0380]: Intel Corporation Ponte Vecchio 1T [8086:0bd5] (rev 2f) - 0000:3a:00.0 Display controller [0380]: Intel Corporation Ponte Vecchio 1T [8086:0bd5] (rev 2f) - 0000:9a:00.0 Display controller [0380]: Intel Corporation Ponte Vecchio 1T [8086:0bd5] (rev 2f) - 0000:ca:00.0 Display controller [0380]: Intel Corporation Ponte Vecchio 1T [8086:0bd5] (rev 2f) - ``` - - Run the previous command to find out the BDF for the GPU device on host. - We choose the GPU with PCI address "0000:3a:00.0" to assign a GPU SR-IOV interface. - -2. Carve out SR-IOV slice for the GPU: - - List our total possible SR-IOV virtual interfaces for the GPU: - - ```bash - $ BDF="0000:3a:00.0" - $ cat "/sys/bus/pci/devices/$BDF/sriov_totalvfs" - 63 - ``` - - Create SR-IOV interfaces for the GPU: - ```sh - $ echo 4 | sudo tee /sys/bus/pci/devices/$BDF/sriov_numvfs - 4 - $ sudo lspci | grep Display - 29:00.0 Display controller: Intel Corporation Ponte Vecchio 1T (rev 2f) - 3a:00.0 Display controller: Intel Corporation Ponte Vecchio 1T (rev 2f) - 3a:00.1 Display controller: Intel Corporation Ponte Vecchio 1T (rev 2f) - 3a:00.2 Display controller: Intel Corporation Ponte Vecchio 1T (rev 2f) - 3a:00.3 Display controller: Intel Corporation Ponte Vecchio 1T (rev 2f) - 3a:00.4 Display controller: Intel Corporation Ponte Vecchio 1T (rev 2f) - 9a:00.0 Display controller: Intel Corporation Ponte Vecchio 1T (rev 2f) - ca:00.0 Display controller: Intel Corporation Ponte Vecchio 1T (rev 2f) - ``` - The above output shows the SR-IOV interfaces created for the GPU. - -3. Find the IOMMU group for the GPU SR-IOV interface(VGPU): - - ```bash - $ BDF="0000:3a:00:1" - $ readlink -e "/sys/bus/pci/devices/$BDF/iommu_group" - /sys/kernel/iommu_groups/437 - $ ls -l /dev/vfio - total 0 - crw------- 1 root root 241, 0 May 18 11:30 437 - crw-rw-rw- 1 root root 10, 196 May 18 11:29 vfio - ``` - - Now you can use the device node `/dev/vfio/437` in docker command line to pass - the VGPU to a Kata Container. - -4. Start a Kata Containers container with GPU device enabled: - - ```bash - $ sudo ctr --debug run --runtime "io.containerd.kata.v2" --device /dev/vfio/437 --rm -t "docker.io/library/archlinux:latest" arch uname -r - ``` diff --git a/docs/use-cases/Intel-GPU-passthrough-and-Kata.md b/docs/use-cases/Intel-GPU-passthrough-and-Kata.md deleted file mode 100644 index ce18ead804..0000000000 --- a/docs/use-cases/Intel-GPU-passthrough-and-Kata.md +++ /dev/null @@ -1,287 +0,0 @@ -# Using Intel GPU device with Kata Containers - -An Intel Graphics device can be passed to a Kata Containers container using GPU -passthrough (Intel GVT-d) as well as GPU mediated passthrough (Intel GVT-g). - -Intel GVT-d (one VM to one physical GPU) also named as Intel-Graphics-Device -passthrough feature is one flavor of graphics virtualization approach. -This flavor allows direct assignment of an entire GPU to a single user, -passing the native driver capabilities through the hypervisor without any limitations. - -Intel GVT-g (multiple VMs to one physical GPU) is a full GPU virtualization solution -with mediated pass-through.
-A virtual GPU instance is maintained for each VM, with part of performance critical -resources, directly assigned. The ability to run a native graphics driver inside a -VM without hypervisor intervention in performance critical paths, achieves a good -balance among performance, feature, and sharing capability. - -| Technology | Description | Behaviour | Detail | -|-|-|-|-| -| Intel GVT-d | GPU passthrough | Physical GPU assigned to a single VM | Direct GPU assignment to VM without limitation | -| Intel GVT-g | GPU sharing | Physical GPU shared by multiple VMs | Mediated passthrough | - -## Hardware Requirements - - - For client platforms, 5th generation Intel® Core Processor Graphics or higher are required. - - For server platforms, E3_v4 or higher Xeon Processor Graphics are required. - -The following steps outline the workflow for using an Intel Graphics device with Kata. - -## Host Kernel Requirements - -The following configurations need to be enabled on your host kernel: - -``` -CONFIG_VFIO_IOMMU_TYPE1=m -CONFIG_VFIO=m -CONFIG_VFIO_PCI=m -CONFIG_VFIO_MDEV=m -CONFIG_VFIO_MDEV_DEVICE=m -CONFIG_DRM_I915_GVT=m -CONFIG_DRM_I915_GVT_KVMGT=m -``` - -Your host kernel needs to be booted with `intel_iommu=on` on the kernel command -line. - -## Install and configure Kata Containers - -To use this feature, you need Kata version 1.3.0 or above. -Follow the [Kata Containers setup instructions](../install/README.md) -to install the latest version of Kata. - -In order to pass a GPU to a Kata Container, you need to enable the `hotplug_vfio_on_root_bus` -configuration in the Kata `configuration.toml` file as shown below. - -``` -$ sudo sed -i -e 's/^# *\(hotplug_vfio_on_root_bus\).*=.*$/\1 = true/g' /usr/share/defaults/kata-containers/configuration.toml -``` - -Make sure you are using the `q35` machine type by verifying `machine_type = "q35"` is -set in the `configuration.toml`. Make sure `pcie_root_port` is set to a positive value. - -## Build Kata Containers kernel with GPU support - -The default guest kernel installed with Kata Containers does not provide GPU support. -To use an Intel GPU with Kata Containers, you need to build a kernel with the necessary -GPU support. - -The following i915 kernel config options need to be enabled: -``` -CONFIG_DRM=y -CONFIG_DRM_I915=y -CONFIG_DRM_I915_USERPTR=y -``` - -Build the Kata Containers kernel with the previous config options, using the instructions -described in [Building Kata Containers kernel](../../tools/packaging/kernel). -For further details on building and installing guest kernels, see [the developer guide](../Developer-Guide.md#install-guest-kernel-images). - -There is an easy way to build a guest kernel that supports Intel GPU: -``` -## Build guest kernel with ../../tools/packaging/kernel - -# Prepare (download guest kernel source, generate .config) -$ ./build-kernel.sh -g intel -f setup - -# Build guest kernel -$ ./build-kernel.sh -g intel build - -# Install guest kernel -$ sudo -E ./build-kernel.sh -g intel install -/usr/share/kata-containers/vmlinux-intel-gpu.container -> vmlinux-5.4.15-70-intel-gpu -/usr/share/kata-containers/vmlinuz-intel-gpu.container -> vmlinuz-5.4.15-70-intel-gpu -``` - -Before using the new guest kernel, please update the `kernel` parameters in `configuration.toml`. -``` -kernel = "/usr/share/kata-containers/vmlinuz-intel-gpu.container" -``` - -## GVT-d with Kata Containers - -Use the following steps to pass an Intel Graphics device in GVT-d mode with Kata: - -1. Find the Bus-Device-Function (BDF) for GPU device: - - ``` - $ sudo lspci -nn -D | grep Graphics - 0000:00:02.0 VGA compatible controller [0300]: Intel Corporation Broadwell-U Integrated Graphics [8086:1616] (rev 09) - ``` - - Run the previous command to determine the BDF for the GPU device on host.
- From the previous output, PCI address `0000:00:02.0` is assigned to the hardware GPU device.
- This BDF is used later to unbind the GPU device from the host.
- "8086 1616" is the device ID of the hardware GPU device. It is used later to - rebind the GPU device to `vfio-pci` driver. - -2. Find the IOMMU group for the GPU device: - - ``` - $ BDF="0000:00:02.0" - $ readlink -e /sys/bus/pci/devices/$BDF/iommu_group - /sys/kernel/iommu_groups/1 - ``` - - The previous output shows that the GPU belongs to IOMMU group 1. - -3. Unbind the GPU: - - ``` - $ echo $BDF | sudo tee /sys/bus/pci/devices/$BDF/driver/unbind - ``` - -4. Bind the GPU to the `vfio-pci` device driver: - - ``` - $ sudo modprobe vfio-pci - $ echo 8086 1616 | sudo tee /sys/bus/pci/drivers/vfio-pci/new_id - $ echo $BDF | sudo tee --append /sys/bus/pci/drivers/vfio-pci/bind - ``` - - After you run the previous commands, the GPU is bound to `vfio-pci` driver.
- A new directory with the IOMMU group number is created under `/dev/vfio`: - - ``` - $ ls -l /dev/vfio - total 0 - crw------- 1 root root 241, 0 May 18 15:38 1 - crw-rw-rw- 1 root root 10, 196 May 18 15:37 vfio - ``` - -5. Start a Kata container with GPU device: - - ``` - $ sudo docker run -it --runtime=kata-runtime --rm --device /dev/vfio/1 -v /dev:/dev debian /bin/bash - ``` - - Run `lspci` within the container to verify the GPU device is seen in the list of - the PCI devices. Note the vendor-device id of the GPU ("8086:1616") in the `lspci` output. - - ``` - $ lspci -nn -D - 0000:00:00.0 Class [0600]: Device [8086:1237] (rev 02) - 0000:00:01.0 Class [0601]: Device [8086:7000] - 0000:00:01.1 Class [0101]: Device [8086:7010] - 0000:00:01.3 Class [0680]: Device [8086:7113] (rev 03) - 0000:00:02.0 Class [0604]: Device [1b36:0001] - 0000:00:03.0 Class [0780]: Device [1af4:1003] - 0000:00:04.0 Class [0100]: Device [1af4:1004] - 0000:00:05.0 Class [0002]: Device [1af4:1009] - 0000:00:06.0 Class [0200]: Device [1af4:1000] - 0000:00:0f.0 Class [0300]: Device [8086:1616] (rev 09) - ``` - - Additionally, you can access the device node for the graphics device: - - ``` - $ ls /dev/dri - card0 renderD128 - ``` - -## GVT-g with Kata Containers - -For GVT-g, you append `i915.enable_gvt=1` in addition to `intel_iommu=on` -on your host kernel command line and then reboot your host. - -Use the following steps to pass an Intel Graphics device in GVT-g mode to a Kata Container: - -1. Find the BDF for GPU device: - - ``` - $ sudo lspci -nn -D | grep Graphics - 0000:00:02.0 VGA compatible controller [0300]: Intel Corporation Broadwell-U Integrated Graphics [8086:1616] (rev 09) - ``` - - Run the previous command to find out the BDF for the GPU device on host. - The previous output shows PCI address "0000:00:02.0" is assigned to the GPU device. - -2. Choose the MDEV (Mediated Device) type for VGPU (Virtual GPU): - - For background on `mdev` types, please follow this [kernel documentation](https://github.com/torvalds/linux/blob/master/Documentation/driver-api/vfio-mediated-device.rst). - - * List out the `mdev` types for the VGPU: - - ``` - $ BDF="0000:00:02.0" - - $ ls /sys/devices/pci0000:00/$BDF/mdev_supported_types - i915-GVTg_V4_1 i915-GVTg_V4_2 i915-GVTg_V4_4 i915-GVTg_V4_8 - ``` - - * Inspect the `mdev` types and choose one that fits your requirement: - - ``` - $ cd /sys/devices/pci0000:00/0000:00:02.0/mdev_supported_types/i915-GVTg_V4_8 && ls - available_instances create description device_api devices - - $ cat description - low_gm_size: 64MB - high_gm_size: 384MB - fence: 4 - resolution: 1024x768 - weight: 2 - - $ cat available_instances - 7 - ``` - - The output of file `description` represents the GPU resources that are - assigned to the VGPU with specified MDEV type.The output of file `available_instances` - represents the remaining amount of VGPUs you can create with specified MDEV type. - -3. Create a VGPU: - - * Generate a UUID: - - ``` - $ gpu_uuid=$(uuid) - ``` - - * Write the UUID to the `create` file under the chosen `mdev` type: - - ``` - $ echo $(gpu_uuid) | sudo tee /sys/devices/pci0000:00/0000:00:02.0/mdev_supported_types/i915-GVTg_V4_8/create - ``` - -4. Find the IOMMU group for the VGPU: - - ``` - $ ls -la /sys/devices/pci0000:00/0000:00:02.0/mdev_supported_types/i915-GVTg_V4_8/devices/${gpu_uuid}/iommu_group - lrwxrwxrwx 1 root root 0 May 18 14:35 devices/bbc4aafe-5807-11e8-a43e-03533cceae7d/iommu_group -> ../../../../kernel/iommu_groups/0 - - $ ls -l /dev/vfio - total 0 - crw------- 1 root root 241, 0 May 18 11:30 0 - crw-rw-rw- 1 root root 10, 196 May 18 11:29 vfio - ``` - - The IOMMU group "0" is created from the previous output.
- Now you can use the device node `/dev/vfio/0` in docker command line to pass - the VGPU to a Kata Container. - -5. Start Kata container with GPU device enabled: - - ``` - $ sudo docker run -it --runtime=kata-runtime --rm --device /dev/vfio/0 -v /dev:/dev debian /bin/bash - $ lspci -nn -D - 0000:00:00.0 Class [0600]: Device [8086:1237] (rev 02) - 0000:00:01.0 Class [0601]: Device [8086:7000] - 0000:00:01.1 Class [0101]: Device [8086:7010] - 0000:00:01.3 Class [0680]: Device [8086:7113] (rev 03) - 0000:00:02.0 Class [0604]: Device [1b36:0001] - 0000:00:03.0 Class [0780]: Device [1af4:1003] - 0000:00:04.0 Class [0100]: Device [1af4:1004] - 0000:00:05.0 Class [0002]: Device [1af4:1009] - 0000:00:06.0 Class [0200]: Device [1af4:1000] - 0000:00:0f.0 Class [0300]: Device [8086:1616] (rev 09) - ``` - - BDF "0000:00:0f.0" is assigned to the VGPU device. - - Additionally, you can access the device node for the graphics device: - - ``` - $ ls /dev/dri - card0 renderD128 - ```