github/kata-containers
1
0
Fork 2
mirror of https://github.com/kata-containers/kata-containers.git synced 2025-09-04 10:24:37 +00:00
Code Issues Projects Releases Wiki Activity

use-cases: Reorganize GPU related documents

Browse Source 
- Index the use-case of Intel and Nvidia GPUs
- Fix link to vfio-mediated-device in Intel GPU use-cases

Fixes #616

Signed-off-by: Jimmy Xu <junming.xjm@antfin.com>
This commit is contained in:
Jimmy Xu
2020-03-18 02:23:08 +08:00
parent d0768d5c3c
commit 16197d6855
3 changed files with 311 additions and 308 deletions
Download Patch File Download Diff File Expand all files Collapse all files

276
use-cases/GPU-passthrough-and-Kata.md
View File

@@ -1,274 +1,6 @@
# Using Intel GPU device with Kata Containers # Using GPUs with Kata Containers
- [Using Intel GPU device with Kata Containers](#using-intel-gpu-device-with-kata-containers) Kata Containers supports passing certain GPUs from the host into the container. Select the GPU vendor for detailed information:
- [Hardware Requirements](#hardware-requirements)
- [Host Kernel Requirements](#host-kernel-requirements)
- [Install and configure Kata Containers](#install-and-configure-kata-containers)
- [Build Kata Containers kernel with GPU support](#build-kata-containers-kernel-with-gpu-support)
- [GVT-d with Kata Containers](#gvt-d-with-kata-containers)
- [GVT-g with Kata Containers](#gvt-g-with-kata-containers)
An Intel Graphics device can be passed to a Kata Containers container using GPU - [Intel](Intel-GPU-passthrough-and-Kata.md)
passthrough (Intel GVT-d) as well as GPU mediated passthrough (Intel GVT-g). - [Nvidia](Nvidia-GPU-passthrough-and-Kata.md)
Intel GVT-d (one VM to one physical GPU) also named as Intel-Graphics-Device
passthrough feature is one flavor of graphics virtualization approach.
This flavor allows direct assignment of an entire GPU to a single user,
passing the native driver capabilities through the hypervisor without any limitations.
Intel GVT-g (multiple VMs to one physical GPU) is a full GPU virtualization solution
with mediated pass-through.<br/>
A virtual GPU instance is maintained for each VM, with part of performance critical
resources, directly assigned. The ability to run a native graphics driver inside a
VM without hypervisor intervention in performance critical paths, achieves a good
balance among performance, feature, and sharing capability.
| Technology | Description | Behaviour | Detail |
|-|-|-|-|
| Intel GVT-d | GPU passthrough | Physical GPU assigned to a single VM | Direct GPU assignment to VM without limitation |
| Intel GVT-g | GPU sharing | Physical GPU shared by multiple VMs | Mediated passthrough |
## Hardware Requirements
- For client platforms, 5th generation Intel® Core Processor Graphics or higher are required.
- For server platforms, E3_v4 or higher Xeon Processor Graphics are required.
The following steps outline the workflow for using an Intel Graphics device with Kata.
## Host Kernel Requirements
The following configurations need to be enabled on your host kernel:
```
CONFIG_VFIO_IOMMU_TYPE1=m
CONFIG_VFIO=m
CONFIG_VFIO_PCI=m
CONFIG_VFIO_MDEV=m
CONFIG_VFIO_MDEV_DEVICE=m
CONFIG_DRM_I915_GVT=m
CONFIG_DRM_I915_GVT_KVMGT=m
```
Your host kernel needs to be booted with `intel_iommu=on` on the kernel command
line.
## Install and configure Kata Containers
To use this feature, you need Kata version 1.3.0 or above.
Follow the [Kata Containers setup instructions](https://github.com/kata-containers/documentation/blob/master/install/README.md)
to install the latest version of Kata.
In order to pass a GPU to a Kata Container, you need to enable the `hotplug_vfio_on_root_bus`
configuration in the Kata `configuration.toml` file as shown below.
```
$ sudo sed -i -e 's/^# *\(hotplug_vfio_on_root_bus\).*=.*$/\1 = true/g' /usr/share/defaults/kata-containers/configuration.toml
```
Make sure you are using the `pc` machine type by verifying `machine_type = "pc"` is
set in the `configuration.toml`.
## Build Kata Containers kernel with GPU support
The default guest kernel installed with Kata Containers does not provide GPU support.
To use an Intel GPU with Kata Containers, you need to build a kernel with the necessary
GPU support.
The following i915 kernel config options need to be enabled:
```
CONFIG_DRM=y
CONFIG_DRM_I915=y
CONFIG_DRM_I915_USERPTR=y
```
Build the Kata Containers kernel with the previous config options, using the instructions
described in [Building Kata Containers kernel](https://github.com/kata-containers/packaging/tree/master/kernel).
For further details on building and installing guest kernels, see [the developer guide](https://github.com/kata-containers/documentation/blob/master/Developer-Guide.md#install-guest-kernel-images).
## GVT-d with Kata Containers
Use the following steps to pass an Intel Graphics device in GVT-d mode with Kata:
1. Find the Bus-Device-Function (BDF) for GPU device:
```
$ sudo lspci -nn -D | grep Graphics
0000:00:02.0 VGA compatible controller [0300]: Intel Corporation Broadwell-U Integrated Graphics [8086:1616] (rev 09)
```
Run the previous command to determine the BDF for the GPU device on host.<br/>
From the previous output, PCI address `0000:00:02.0` is assigned to the hardware GPU device.<br/>
This BDF is used later to unbind the GPU device from the host.<br/>
"8086 1616" is the device ID of the hardware GPU device. It is used later to
rebind the GPU device to `vfio-pci` driver.
2. Find the IOMMU group for the GPU device:
```
$ BDF="0000:00:02.0"
$ readlink -e /sys/bus/pci/devices/$BDF/iommu_group
/sys/kernel/iommu_groups/1
```
The previous output shows that the GPU belongs to IOMMU group 1.
3. Unbind the GPU:
```
$ echo $BDF | sudo tee /sys/bus/pci/devices/$BDF/driver/unbind
```
4. Bind the GPU to the `vfio-pci` device driver:
```
$ sudo modprobe vfio-pci
$ echo 8086 1616 | sudo tee /sys/bus/pci/drivers/vfio-pci/new_id
$ echo $BDF | sudo tee --append /sys/bus/pci/drivers/vfio-pci/bind
```
After you run the previous commands, the GPU is bound to `vfio-pci` driver.<br/>
A new directory with the IOMMU group number is created under `/dev/vfio`:
```
$ ls -l /dev/vfio
total 0
crw------- 1 root root 241, 0 May 18 15:38 1
crw-rw-rw- 1 root root 10, 196 May 18 15:37 vfio
```
5. Start a Kata container with GPU device:
```
$ sudo docker run -it --runtime=kata-runtime --rm --device /dev/vfio/1 -v /dev:/dev debian /bin/bash
```
Run `lspci` within the container to verify the GPU device is seen in the list of
the PCI devices. Note the vendor-device id of the GPU ("8086:1616") in the `lspci` output.
```
$ lspci -nn -D
0000:00:00.0 Class [0600]: Device [8086:1237] (rev 02)
0000:00:01.0 Class [0601]: Device [8086:7000]
0000:00:01.1 Class [0101]: Device [8086:7010]
0000:00:01.3 Class [0680]: Device [8086:7113] (rev 03)
0000:00:02.0 Class [0604]: Device [1b36:0001]
0000:00:03.0 Class [0780]: Device [1af4:1003]
0000:00:04.0 Class [0100]: Device [1af4:1004]
0000:00:05.0 Class [0002]: Device [1af4:1009]
0000:00:06.0 Class [0200]: Device [1af4:1000]
0000:00:0f.0 Class [0300]: Device [8086:1616] (rev 09)
```
Additionally, you can access the device node for the graphics device:
```
$ ls /dev/dri
card0 renderD128
```
## GVT-g with Kata Containers
For GVT-g, you append `i915.enable_gvt=1` in addition to `intel_iommu=on`
on your host kernel command line and then reboot your host.
Use the following steps to pass an Intel Graphics device in GVT-g mode to a Kata Container:
1. Find the BDF for GPU device:
```
$ sudo lspci -nn -D | grep Graphics
0000:00:02.0 VGA compatible controller [0300]: Intel Corporation Broadwell-U Integrated Graphics [8086:1616] (rev 09)
```
Run the previous command to find out the BDF for the GPU device on host.
The previous output shows PCI address "0000:00:02.0" is assigned to the GPU device.
2. Choose the MDEV (Mediated Device) type for VGPU (Virtual GPU):
For background on `mdev` types, please follow this [kernel documentation](https://github.com/torvalds/linux/blob/master/Documentation/vfio-mediated-device.txt).
* List out the `mdev` types for the VGPU:
```
$ BDF="0000:00:02.0"
$ ls /sys/devices/pci0000:00/$BDF/mdev_supported_types
i915-GVTg_V4_1 i915-GVTg_V4_2 i915-GVTg_V4_4 i915-GVTg_V4_8
```
* Inspect the `mdev` types and choose one that fits your requirement:
```
$ cd /sys/devices/pci0000:00/0000:00:02.0/mdev_supported_types/i915-GVTg_V4_8 && ls
available_instances create description device_api devices
$ cat description
low_gm_size: 64MB
high_gm_size: 384MB
fence: 4
resolution: 1024x768
weight: 2
$ cat available_instances
7
```
The output of file `description` represents the GPU resources that are
assigned to the VGPU with specified MDEV type.The output of file `available_instances`
represents the remaining amount of VGPUs you can create with specified MDEV type.
3. Create a VGPU:
* Generate a UUID:
```
$ gpu_uuid=$(uuid)
```
* Write the UUID to the `create` file under the chosen `mdev` type:
```
$ echo $(gpu_uuid) | sudo tee /sys/devices/pci0000:00/0000:00:02.0/mdev_supported_types/i915-GVTg_V4_8/create
```
4. Find the IOMMU group for the VGPU:
```
$ ls -la /sys/devices/pci0000:00/0000:00:02.0/mdev_supported_types/i915-GVTg_V4_8/devices/${gpu_uuid}/iommu_group
lrwxrwxrwx 1 root root 0 May 18 14:35 devices/bbc4aafe-5807-11e8-a43e-03533cceae7d/iommu_group -> ../../../../kernel/iommu_groups/0
$ ls -l /dev/vfio
total 0
crw------- 1 root root 241, 0 May 18 11:30 0
crw-rw-rw- 1 root root 10, 196 May 18 11:29 vfio
```
The IOMMU group "0" is created from the previous output.<br/>
Now you can use the device node `/dev/vfio/0` in docker command line to pass
the VGPU to a Kata Container.
5. Start Kata container with GPU device enabled:
```
$ sudo docker run -it --runtime=kata-runtime --rm --device /dev/vfio/0 -v /dev:/dev debian /bin/bash
$ lspci -nn -D
0000:00:00.0 Class [0600]: Device [8086:1237] (rev 02)
0000:00:01.0 Class [0601]: Device [8086:7000]
0000:00:01.1 Class [0101]: Device [8086:7010]
0000:00:01.3 Class [0680]: Device [8086:7113] (rev 03)
0000:00:02.0 Class [0604]: Device [1b36:0001]
0000:00:03.0 Class [0780]: Device [1af4:1003]
0000:00:04.0 Class [0100]: Device [1af4:1004]
0000:00:05.0 Class [0002]: Device [1af4:1009]
0000:00:06.0 Class [0200]: Device [1af4:1000]
0000:00:0f.0 Class [0300]: Device [8086:1616] (rev 09)
```
BDF "0000:00:0f.0" is assigned to the VGPU device.
Additionally, you can access the device node for the graphics device:
```
$ ls /dev/dri
card0 renderD128
```

274
use-cases/Intel-GPU-passthrough-and-Kata.md Normal file
View File

@@ -0,0 +1,274 @@
# Using Intel GPU device with Kata Containers
- [Using Intel GPU device with Kata Containers](#using-intel-gpu-device-with-kata-containers)
- [Hardware Requirements](#hardware-requirements)
- [Host Kernel Requirements](#host-kernel-requirements)
- [Install and configure Kata Containers](#install-and-configure-kata-containers)
- [Build Kata Containers kernel with GPU support](#build-kata-containers-kernel-with-gpu-support)
- [GVT-d with Kata Containers](#gvt-d-with-kata-containers)
- [GVT-g with Kata Containers](#gvt-g-with-kata-containers)
An Intel Graphics device can be passed to a Kata Containers container using GPU
passthrough (Intel GVT-d) as well as GPU mediated passthrough (Intel GVT-g).
Intel GVT-d (one VM to one physical GPU) also named as Intel-Graphics-Device
passthrough feature is one flavor of graphics virtualization approach.
This flavor allows direct assignment of an entire GPU to a single user,
passing the native driver capabilities through the hypervisor without any limitations.
Intel GVT-g (multiple VMs to one physical GPU) is a full GPU virtualization solution
with mediated pass-through.<br/>
A virtual GPU instance is maintained for each VM, with part of performance critical
resources, directly assigned. The ability to run a native graphics driver inside a
VM without hypervisor intervention in performance critical paths, achieves a good
balance among performance, feature, and sharing capability.
| Technology | Description | Behaviour | Detail |
|-|-|-|-|
| Intel GVT-d | GPU passthrough | Physical GPU assigned to a single VM | Direct GPU assignment to VM without limitation |
| Intel GVT-g | GPU sharing | Physical GPU shared by multiple VMs | Mediated passthrough |
## Hardware Requirements
- For client platforms, 5th generation Intel® Core Processor Graphics or higher are required.
- For server platforms, E3_v4 or higher Xeon Processor Graphics are required.
The following steps outline the workflow for using an Intel Graphics device with Kata.
## Host Kernel Requirements
The following configurations need to be enabled on your host kernel:
```
CONFIG_VFIO_IOMMU_TYPE1=m
CONFIG_VFIO=m
CONFIG_VFIO_PCI=m
CONFIG_VFIO_MDEV=m
CONFIG_VFIO_MDEV_DEVICE=m
CONFIG_DRM_I915_GVT=m
CONFIG_DRM_I915_GVT_KVMGT=m
```
Your host kernel needs to be booted with `intel_iommu=on` on the kernel command
line.
## Install and configure Kata Containers
To use this feature, you need Kata version 1.3.0 or above.
Follow the [Kata Containers setup instructions](https://github.com/kata-containers/documentation/blob/master/install/README.md)
to install the latest version of Kata.
In order to pass a GPU to a Kata Container, you need to enable the `hotplug_vfio_on_root_bus`
configuration in the Kata `configuration.toml` file as shown below.
```
$ sudo sed -i -e 's/^# *\(hotplug_vfio_on_root_bus\).*=.*$/\1 = true/g' /usr/share/defaults/kata-containers/configuration.toml
```
Make sure you are using the `pc` machine type by verifying `machine_type = "pc"` is
set in the `configuration.toml`.
## Build Kata Containers kernel with GPU support
The default guest kernel installed with Kata Containers does not provide GPU support.
To use an Intel GPU with Kata Containers, you need to build a kernel with the necessary
GPU support.
The following i915 kernel config options need to be enabled:
```
CONFIG_DRM=y
CONFIG_DRM_I915=y
CONFIG_DRM_I915_USERPTR=y
```
Build the Kata Containers kernel with the previous config options, using the instructions
described in [Building Kata Containers kernel](https://github.com/kata-containers/packaging/tree/master/kernel).
For further details on building and installing guest kernels, see [the developer guide](https://github.com/kata-containers/documentation/blob/master/Developer-Guide.md#install-guest-kernel-images).
## GVT-d with Kata Containers
Use the following steps to pass an Intel Graphics device in GVT-d mode with Kata:
1. Find the Bus-Device-Function (BDF) for GPU device:
```
$ sudo lspci -nn -D | grep Graphics
0000:00:02.0 VGA compatible controller [0300]: Intel Corporation Broadwell-U Integrated Graphics [8086:1616] (rev 09)
```
Run the previous command to determine the BDF for the GPU device on host.<br/>
From the previous output, PCI address `0000:00:02.0` is assigned to the hardware GPU device.<br/>
This BDF is used later to unbind the GPU device from the host.<br/>
"8086 1616" is the device ID of the hardware GPU device. It is used later to
rebind the GPU device to `vfio-pci` driver.
2. Find the IOMMU group for the GPU device:
```
$ BDF="0000:00:02.0"
$ readlink -e /sys/bus/pci/devices/$BDF/iommu_group
/sys/kernel/iommu_groups/1
```
The previous output shows that the GPU belongs to IOMMU group 1.
3. Unbind the GPU:
```
$ echo $BDF | sudo tee /sys/bus/pci/devices/$BDF/driver/unbind
```
4. Bind the GPU to the `vfio-pci` device driver:
```
$ sudo modprobe vfio-pci
$ echo 8086 1616 | sudo tee /sys/bus/pci/drivers/vfio-pci/new_id
$ echo $BDF | sudo tee --append /sys/bus/pci/drivers/vfio-pci/bind
```
After you run the previous commands, the GPU is bound to `vfio-pci` driver.<br/>
A new directory with the IOMMU group number is created under `/dev/vfio`:
```
$ ls -l /dev/vfio
total 0
crw------- 1 root root 241, 0 May 18 15:38 1
crw-rw-rw- 1 root root 10, 196 May 18 15:37 vfio
```
5. Start a Kata container with GPU device:
```
$ sudo docker run -it --runtime=kata-runtime --rm --device /dev/vfio/1 -v /dev:/dev debian /bin/bash
```
Run `lspci` within the container to verify the GPU device is seen in the list of
the PCI devices. Note the vendor-device id of the GPU ("8086:1616") in the `lspci` output.
```
$ lspci -nn -D
0000:00:00.0 Class [0600]: Device [8086:1237] (rev 02)
0000:00:01.0 Class [0601]: Device [8086:7000]
0000:00:01.1 Class [0101]: Device [8086:7010]
0000:00:01.3 Class [0680]: Device [8086:7113] (rev 03)
0000:00:02.0 Class [0604]: Device [1b36:0001]
0000:00:03.0 Class [0780]: Device [1af4:1003]
0000:00:04.0 Class [0100]: Device [1af4:1004]
0000:00:05.0 Class [0002]: Device [1af4:1009]
0000:00:06.0 Class [0200]: Device [1af4:1000]
0000:00:0f.0 Class [0300]: Device [8086:1616] (rev 09)
```
Additionally, you can access the device node for the graphics device:
```
$ ls /dev/dri
card0 renderD128
```
## GVT-g with Kata Containers
For GVT-g, you append `i915.enable_gvt=1` in addition to `intel_iommu=on`
on your host kernel command line and then reboot your host.
Use the following steps to pass an Intel Graphics device in GVT-g mode to a Kata Container:
1. Find the BDF for GPU device:
```
$ sudo lspci -nn -D | grep Graphics
0000:00:02.0 VGA compatible controller [0300]: Intel Corporation Broadwell-U Integrated Graphics [8086:1616] (rev 09)
```
Run the previous command to find out the BDF for the GPU device on host.
The previous output shows PCI address "0000:00:02.0" is assigned to the GPU device.
2. Choose the MDEV (Mediated Device) type for VGPU (Virtual GPU):
For background on `mdev` types, please follow this [kernel documentation](https://github.com/torvalds/linux/blob/master/Documentation/driver-api/vfio-mediated-device.rst).
* List out the `mdev` types for the VGPU:
```
$ BDF="0000:00:02.0"
$ ls /sys/devices/pci0000:00/$BDF/mdev_supported_types
i915-GVTg_V4_1 i915-GVTg_V4_2 i915-GVTg_V4_4 i915-GVTg_V4_8
```
* Inspect the `mdev` types and choose one that fits your requirement:
```
$ cd /sys/devices/pci0000:00/0000:00:02.0/mdev_supported_types/i915-GVTg_V4_8 && ls
available_instances create description device_api devices
$ cat description
low_gm_size: 64MB
high_gm_size: 384MB
fence: 4
resolution: 1024x768
weight: 2
$ cat available_instances
7
```
The output of file `description` represents the GPU resources that are
assigned to the VGPU with specified MDEV type.The output of file `available_instances`
represents the remaining amount of VGPUs you can create with specified MDEV type.
3. Create a VGPU:
* Generate a UUID:
```
$ gpu_uuid=$(uuid)
```
* Write the UUID to the `create` file under the chosen `mdev` type:
```
$ echo $(gpu_uuid) | sudo tee /sys/devices/pci0000:00/0000:00:02.0/mdev_supported_types/i915-GVTg_V4_8/create
```
4. Find the IOMMU group for the VGPU:
```
$ ls -la /sys/devices/pci0000:00/0000:00:02.0/mdev_supported_types/i915-GVTg_V4_8/devices/${gpu_uuid}/iommu_group
lrwxrwxrwx 1 root root 0 May 18 14:35 devices/bbc4aafe-5807-11e8-a43e-03533cceae7d/iommu_group -> ../../../../kernel/iommu_groups/0
$ ls -l /dev/vfio
total 0
crw------- 1 root root 241, 0 May 18 11:30 0
crw-rw-rw- 1 root root 10, 196 May 18 11:29 vfio
```
The IOMMU group "0" is created from the previous output.<br/>
Now you can use the device node `/dev/vfio/0` in docker command line to pass
the VGPU to a Kata Container.
5. Start Kata container with GPU device enabled:
```
$ sudo docker run -it --runtime=kata-runtime --rm --device /dev/vfio/0 -v /dev:/dev debian /bin/bash
$ lspci -nn -D
0000:00:00.0 Class [0600]: Device [8086:1237] (rev 02)
0000:00:01.0 Class [0601]: Device [8086:7000]
0000:00:01.1 Class [0101]: Device [8086:7010]
0000:00:01.3 Class [0680]: Device [8086:7113] (rev 03)
0000:00:02.0 Class [0604]: Device [1b36:0001]
0000:00:03.0 Class [0780]: Device [1af4:1003]
0000:00:04.0 Class [0100]: Device [1af4:1004]
0000:00:05.0 Class [0002]: Device [1af4:1009]
0000:00:06.0 Class [0200]: Device [1af4:1000]
0000:00:0f.0 Class [0300]: Device [8086:1616] (rev 09)
```
BDF "0000:00:0f.0" is assigned to the VGPU device.
Additionally, you can access the device node for the graphics device:
```
$ ls /dev/dri
card0 renderD128
```

69
use-cases/Nvidia-GPU-passthrough-and-Kata.md
View File

@@ -12,20 +12,20 @@
- [References](#references) - [References](#references)
A Nvidia GPU device can be passed to a Kata Containers container using GPU passthrough An Nvidia GPU device can be passed to a Kata Containers container using GPU passthrough
(Nvidia GPU pass-through mode) as well as GPU mediated passthrough (Nvidia vGPU mode).  (Nvidia GPU pass-through mode) as well as GPU mediated passthrough (Nvidia vGPU mode). 
Nvidia GPU pass-through mode, an entire physical GPU is directly assigned to one VM, Nvidia GPU pass-through mode, an entire physical GPU is directly assigned to one VM,
bypassing the Nvidia Virtual GPU Manager. In this mode of operation, the GPU is accessed bypassing the Nvidia Virtual GPU Manager. In this mode of operation, the GPU is accessed
exclusively by the Nvidia driver running in the VM to which it is assigned. exclusively by the Nvidia driver running in the VM to which it is assigned.
The GPU is not shared among VMs. The GPU is not shared among VMs.
Nvidia Virtual GPU (vGPU) enables multiple virtual machines (VMs) to have simultaneous, Nvidia Virtual GPU (vGPU) enables multiple virtual machines (VMs) to have simultaneous,
direct access to a single physical GPU, using the same Nvidia graphics drivers that are direct access to a single physical GPU, using the same Nvidia graphics drivers that are
deployed on non-virtualized operating systems. By doing this, Nvidia vGPU provides VMs deployed on non-virtualized operating systems. By doing this, Nvidia vGPU provides VMs
with unparalleled graphics performance, compute performance, and application compatibility, with unparalleled graphics performance, compute performance, and application compatibility,
together with the cost-effectiveness and scalability brought about by sharing a GPU together with the cost-effectiveness and scalability brought about by sharing a GPU
among multiple workloads. among multiple workloads.
| Technology | Description | Behaviour | Detail | | Technology | Description | Behaviour | Detail |
| --- | --- | --- | --- | | --- | --- | --- | --- |
@@ -35,7 +35,7 @@ Nvidia Virtual GPU (vGPU) enables multiple virtual machines (VMs) to have simult
## Hardware Requirements ## Hardware Requirements
Nvidia GPUs Recommended for Virtualization: Nvidia GPUs Recommended for Virtualization:
- Nvidia Tesla (T4, M10, P6, V100...) - Nvidia Tesla (T4, M10, P6, V100 or newer)
- Nvidia Quadro RTX 6000/8000 - Nvidia Quadro RTX 6000/8000
## Host BIOS Requirements ## Host BIOS Requirements
@@ -49,7 +49,7 @@ $ lspci -s 04:00.0 -vv | grep Region
``` ```
For large BARs devices, MMIO mapping above 4G address space should be `enabled` For large BARs devices, MMIO mapping above 4G address space should be `enabled`
in the PCI configuration of the BIOS. in the PCI configuration of the BIOS.
Some hardware vendors use different name in BIOS, such as: Some hardware vendors use different name in BIOS, such as:
@@ -57,14 +57,13 @@ Some hardware vendors use different name in BIOS, such as:
- Memory Hole for PCI MMIO - Memory Hole for PCI MMIO
- Memory Mapped I/O above 4GB - Memory Mapped I/O above 4GB
The following steps outline the workflow for using an Nvidia GPU with Kata. The following steps outline the workflow for using an Nvidia GPU with Kata.
## Host Kernel Requirements ## Host Kernel Requirements
The following configurations need to be enabled on your host kernel: The following configurations need to be enabled on your host kernel:
- `CONFIG_VFIO` - `CONFIG_VFIO`
- `CONFIG_VFIO_IOMMU_TYPE` - `CONFIG_VFIO_IOMMU_TYPE1`
- `CONFIG_VFIO_MDEV` - `CONFIG_VFIO_MDEV`
- `CONFIG_VFIO_MDEV_DEVICE` - `CONFIG_VFIO_MDEV_DEVICE`
- `CONFIG_VFIO_PCI` - `CONFIG_VFIO_PCI`
@@ -73,8 +72,8 @@ Your host kernel needs to be booted with `intel_iommu=on` on the kernel command
## Install and configure Kata Containers ## Install and configure Kata Containers
To use non-large BARs devices (for example, Nvidia Tesla T4), you need Kata version 1.3.0 or above. To use non-large BARs devices (for example, Nvidia Tesla T4), you need Kata version 1.3.0 or above.
Follow the [Kata Containers setup instructions](https://github.com/kata-containers/documentation/blob/master/install/README.md) Follow the [Kata Containers setup instructions](https://github.com/kata-containers/documentation/blob/master/install/README.md)
to install the latest version of Kata. to install the latest version of Kata.
The following configuration in the Kata `configuration.toml` file as shown below can work: The following configuration in the Kata `configuration.toml` file as shown below can work:
``` ```
@@ -84,7 +83,6 @@ hotplug_vfio_on_root_bus = true
``` ```
To use large BARs devices (for example, Nvidia Tesla P100), you need Kata version 1.11.0 or above. To use large BARs devices (for example, Nvidia Tesla P100), you need Kata version 1.11.0 or above.
([related PR](https://github.com/kata-containers/runtime/pull/2461))
The following configuration in the Kata `configuration.toml` file as shown below can work: The following configuration in the Kata `configuration.toml` file as shown below can work:
@@ -96,7 +94,6 @@ hotplug_vfio_on_root_bus = false
``` ```
Hotplug for PCIe devices by `pciehp` (Linux's PCIe Hotplug driver): Hotplug for PCIe devices by `pciehp` (Linux's PCIe Hotplug driver):
[related PR](https://github.com/kata-containers/runtime/pull/2410)
``` ```
machine_type = "q35" machine_type = "q35"
@@ -106,8 +103,8 @@ pcie_root_port = 1
## Build Kata Containers kernel with GPU support ## Build Kata Containers kernel with GPU support
The default guest kernel installed with Kata Containers does not provide GPU support. The default guest kernel installed with Kata Containers does not provide GPU support.
To use an Nvidia GPU with Kata Containers, you need to build a kernel with the To use an Nvidia GPU with Kata Containers, you need to build a kernel with the
necessary GPU support. necessary GPU support.
The following kernel config options need to be enabled: The following kernel config options need to be enabled:
``` ```
@@ -130,15 +127,15 @@ The following kernel config options need to be disabled:
CONFIG_DRM_NOUVEAU=n CONFIG_DRM_NOUVEAU=n
``` ```
> **Note**: `CONFIG_DRM_NOUVEAU` is normally disabled by default. > **Note**: `CONFIG_DRM_NOUVEAU` is normally disabled by default.
It is worth checking that it is not enabled in your kernel configuration to prevent any conflicts. It is worth checking that it is not enabled in your kernel configuration to prevent any conflicts.
Build the Kata Containers kernel with the previous config options, Build the Kata Containers kernel with the previous config options,
using the instructions described in [Building Kata Containers kernel](https://github.com/kata-containers/packaging/tree/master/kernel). using the instructions described in [Building Kata Containers kernel](https://github.com/kata-containers/packaging/tree/master/kernel).
For further details on building and installing guest kernels, For further details on building and installing guest kernels,
see [the developer guide](https://github.com/kata-containers/documentation/blob/master/Developer-Guide.md#install-guest-kernel-images). see [the developer guide](https://github.com/kata-containers/documentation/blob/master/Developer-Guide.md#install-guest-kernel-images).
There is an easy way to build a guest kernel that supports Nvidia GPU ([related PR](https://github.com/kata-containers/packaging/pull/938)): There is an easy way to build a guest kernel that supports Nvidia GPU:
``` ```
## Build guest kernel with http://github.com/kata-containers/packaging ## Build guest kernel with http://github.com/kata-containers/packaging
@@ -202,7 +199,7 @@ Use the following steps to pass an Nvidia GPU device in pass-through mode with K
4. Start a Kata container with GPU device: 4. Start a Kata container with GPU device:
``` ```
sudo docker run -it --runtime=kata-runtime --rm --device /dev/vfio/45 centos /bin/bash $ sudo docker run -it --runtime=kata-runtime --rm --device /dev/vfio/45 centos /bin/bash
``` ```
5. Run `lspci` within the container to verify the GPU device is seen in the list 5. Run `lspci` within the container to verify the GPU device is seen in the list
@@ -225,29 +222,29 @@ Use the following steps to pass an Nvidia GPU device in pass-through mode with K
## Nvidia vGPU mode with Kata Containers ## Nvidia vGPU mode with Kata Containers
Nvidia vGPU is a licensed product on all supported GPU boards. A software license Nvidia vGPU is a licensed product on all supported GPU boards. A software license
is required to enable all vGPU features within the guest VM. is required to enable all vGPU features within the guest VM.
> **Note**: There is no suitable test environment, so it is not written here. > **Note**: There is no suitable test environment, so it is not written here.
## Install Nvidia Driver in Kata Containers ## Install Nvidia Driver in Kata Containers
Download the official Nvidia driver from Download the official Nvidia driver from
[https://www.nvidia.com/Download/index.aspx](https://www.nvidia.com/Download/index.aspx), [https://www.nvidia.com/Download/index.aspx](https://www.nvidia.com/Download/index.aspx),
for example `NVIDIA-Linux-x86_64-418.87.01.run`. for example `NVIDIA-Linux-x86_64-418.87.01.run`.
Install the `kernel-devel`(generated in the previous steps) for guest kernel: Install the `kernel-devel`(generated in the previous steps) for guest kernel:
``` ```
$ rpm -ivh kernel-devel-4.19.86_gpu-1.x86_64.rpm $ sudo rpm -ivh kernel-devel-4.19.86_gpu-1.x86_64.rpm
``` ```
Here is an example to extract, compile and install Nvidia driver: Here is an example to extract, compile and install Nvidia driver:
``` ```
## Extract ## Extract
$ ./NVIDIA-Linux-x86_64-418.87.01.run -x $ sh ./NVIDIA-Linux-x86_64-418.87.01.run -x
## Compile and install (It will take some time) ## Compile and install (It will take some time)
$ cd NVIDIA-Linux-x86_64-418.87.01NVIDIA-Linux-x86_64-418.87.01 $ cd NVIDIA-Linux-x86_64-418.87.01
$ ./nvidia-installer -a -q --ui=none \ $ sudo ./nvidia-installer -a -q --ui=none \
--no-cc-version-check \ --no-cc-version-check \
--no-opengl-files --no-install-libglvnd \ --no-opengl-files --no-install-libglvnd \
--kernel-source-path=/usr/src/kernels/`uname -r` --kernel-source-path=/usr/src/kernels/`uname -r`
@@ -255,7 +252,7 @@ $ ./nvidia-installer -a -q --ui=none \
Or just run one command line: Or just run one command line:
``` ```
$ ./NVIDIA-Linux-x86_64-418.87.01.run -a -q --ui=none \ $ sudo sh ./NVIDIA-Linux-x86_64-418.87.01.run -a -q --ui=none \
--no-cc-version-check \ --no-cc-version-check \
--no-opengl-files --no-install-libglvnd \ --no-opengl-files --no-install-libglvnd \
--kernel-source-path=/usr/src/kernels/`uname -r` --kernel-source-path=/usr/src/kernels/`uname -r`
@@ -269,10 +266,10 @@ $ tail -f /var/log/nvidia-installer.log
Load Nvidia driver module manually Load Nvidia driver module manually
``` ```
# Optional（generate modules.dep and map files for Nvidia driver） # Optional（generate modules.dep and map files for Nvidia driver）
$ depmod $ sudo depmod
# Load module # Load module
$ modprobe nvidia-drm $ sudo modprobe nvidia-drm
# Check module # Check module
$ lsmod | grep nvidia $ lsmod | grep nvidia