mirror of
https://github.com/kata-containers/kata-containers.git
synced 2025-07-12 06:38:11 +00:00
use-cases: Add documentation for using Intel GPU with Kata
This document decsribes how an Intel GPU can be used with Kata Containers in GVT-g and GVT-d mode. An example of an actual workload will be added in the future. Fixes #260 Signed-off-by: Archana Shinde <archana.m.shinde@intel.com>
This commit is contained in:
Archana Shinde
parent
61c1c44127
commit
15d38d390f
274
use-cases/GPU-passthrough-and-Kata.md
Normal file
274
use-cases/GPU-passthrough-and-Kata.md
Normal file
@ -0,0 +1,274 @@
|
|||||||
|
# Using Intel GPU device with Kata Containers
|
||||||
|
|
||||||
|
- [Using Intel GPU device with Kata Containers](#using-intel-gpu-device-with-kata-containers)
|
||||||
|
- [Hardware Requirements](#hardware-requirements)
|
||||||
|
- [Host Kernel Requirements](#host-kernel-requirements)
|
||||||
|
- [Install and configure Kata Containers](#install-and-configure-kata-containers)
|
||||||
|
- [Build Kata Containers kernel with GPU support](#build-kata-containers-kernel-with-gpu-support)
|
||||||
|
- [GVT-d with Kata Containers](#gvt-d-with-kata-containers)
|
||||||
|
- [GVT-g with Kata Containers](#gvt-g-with-kata-containers)
|
||||||
|
|
||||||
|
An Intel Graphics device can be passed to a Kata Containers container using GPU
|
||||||
|
passthrough (Intel GVT-d) as well as GPU mediated passthrough (Intel GVT-g).
|
||||||
|
|
||||||
|
Intel GVT-d (one VM to one physical GPU) also named as Intel-Graphics-Device
|
||||||
|
passthrough feature is one flavor of graphics virtualization approach.
|
||||||
|
This flavor allows direct assignment of an entire GPU to a single user,
|
||||||
|
passing the native driver capabilities through the hypervisor without any limitations.
|
||||||
|
|
||||||
|
Intel GVT-g (multiple VMs to one physical GPU) is a full GPU virtualization solution
|
||||||
|
with mediated pass-through.<br/>
|
||||||
|
A virtual GPU instance is maintained for each VM, with part of performance critical
|
||||||
|
resources, directly assigned. The ability to run a native graphics driver inside a
|
||||||
|
VM without hypervisor intervention in performance critical paths, achieves a good
|
||||||
|
balance among performance, feature, and sharing capability.
|
||||||
|
|
||||||
|
| Technology | Description | Behaviour | Detail |
|
||||||
|
|-|-|-|-|
|
||||||
|
| Intel GVT-d | GPU passthrough | Physical GPU assigned to a single VM | Direct GPU assignment to VM without limitation |
|
||||||
|
| Intel GVT-g | GPU sharing | Physical GPU shared by multiple VMs | Mediated passthrough |
|
||||||
|
|
||||||
|
## Hardware Requirements
|
||||||
|
|
||||||
|
- For client platforms, 5th generation Intel® Core Processor Graphics or higher are required.
|
||||||
|
- For server platforms, E3_v4 or higher Xeon Processor Graphics are required.
|
||||||
|
|
||||||
|
The following steps outline the workflow for using an Intel Graphics device with Kata.
|
||||||
|
|
||||||
|
## Host Kernel Requirements
|
||||||
|
|
||||||
|
The following configurations need to be enabled on your host kernel:
|
||||||
|
|
||||||
|
```
|
||||||
|
CONFIG_VFIO_IOMMU_TYPE1=m
|
||||||
|
CONFIG_VFIO=m
|
||||||
|
CONFIG_VFIO_PCI=m
|
||||||
|
CONFIG_VFIO_MDEV=m
|
||||||
|
CONFIG_VFIO_MDEV_DEVICE=m
|
||||||
|
CONFIG_DRM_I915_GVT=m
|
||||||
|
CONFIG_DRM_I915_GVT_KVMGT=m
|
||||||
|
```
|
||||||
|
|
||||||
|
Your host kernel needs to be booted with `intel_iommu=on` on the kernel command
|
||||||
|
line.
|
||||||
|
|
||||||
|
## Install and configure Kata Containers
|
||||||
|
|
||||||
|
To use this feature, you need Kata version 1.3.0 or above.
|
||||||
|
Follow the [Kata Containers' setup instructions](https://github.com/kata-containers/documentation/blob/master/install/README.md)
|
||||||
|
to install the latest version of Kata.
|
||||||
|
|
||||||
|
In order to pass a GPU to a Kata Container, you need to enable the `hotplug_vfio_on_root_bus`
|
||||||
|
configuration in the Kata `configuration.toml` file as shown below.
|
||||||
|
|
||||||
|
```
|
||||||
|
$ sudo sed -i -e 's/^# *\(hotplug_vfio_on_root_bus\).*=.*$/\1 = true/g' /usr/share/defaults/kata-containers/configuration.toml
|
||||||
|
```
|
||||||
|
|
||||||
|
Make sure you are using the `pc` machine type by verifying `machine_type = "pc"` is
|
||||||
|
set in the configuration.toml.
|
||||||
|
|
||||||
|
## Build Kata Containers kernel with GPU support
|
||||||
|
|
||||||
|
The default guest kernel installed with Kata Containers does not provide GPU support.
|
||||||
|
To use an Intel GPU with Kata Containers, you need to build a kernel with the necessary
|
||||||
|
GPU support.
|
||||||
|
|
||||||
|
The following i915 kernel config options need to be enabled:
|
||||||
|
```
|
||||||
|
CONFIG_DRM=y
|
||||||
|
CONFIG_DRM_I915=y
|
||||||
|
CONFIG_DRM_I915_USERPTR=y
|
||||||
|
```
|
||||||
|
|
||||||
|
Build the Kata Containers kernel with the previous config options, using the instructions
|
||||||
|
described in [Building Kata Containers kernel](https://github.com/kata-containers/packaging/tree/master/kernel).
|
||||||
|
For further details on building and installing guest kernels, see [the developer guide](https://github.com/kata-containers/documentation/blob/master/Developer-Guide.md#install-guest-kernel-images).
|
||||||
|
|
||||||
|
## GVT-d with Kata Containers
|
||||||
|
|
||||||
|
Use the following steps to pass an Intel Graphics device in GVT-d mode with Kata:
|
||||||
|
|
||||||
|
1. Find the Bus-Device-Function (BDF) for GPU device:
|
||||||
|
|
||||||
|
```
|
||||||
|
$ sudo lspci -nn -D | grep Graphics
|
||||||
|
0000:00:02.0 VGA compatible controller [0300]: Intel Corporation Broadwell-U Integrated Graphics [8086:1616] (rev 09)
|
||||||
|
```
|
||||||
|
|
||||||
|
Run the previous command to determine the BDF for the GPU device on host.<br/>
|
||||||
|
From the previous output, PCI addres "0000:00:02.0" is assigned to the hardware GPU device.<br/>
|
||||||
|
This BDF is used later to unbind the GPU device from the host.<br/>
|
||||||
|
"8086 1616" is the device ID of the hardware GPU device. It is used later to
|
||||||
|
rebind the GPU device to `vfio-pci` driver.
|
||||||
|
|
||||||
|
2. Find the IOMMU group for the GPU device:
|
||||||
|
|
||||||
|
```
|
||||||
|
$ BDF="0000:00:02.0"
|
||||||
|
$ readlink -e /sys/bus/pci/devices/$BDF/iommu_group
|
||||||
|
/sys/kernel/iommu_groups/1
|
||||||
|
```
|
||||||
|
|
||||||
|
The previous output shows that the GPU belongs to IOMMU group 1.
|
||||||
|
|
||||||
|
3. Unbind the GPU:
|
||||||
|
|
||||||
|
```
|
||||||
|
$ echo $BDF | sudo tee /sys/bus/pci/devices/$BDF/driver/unbind
|
||||||
|
```
|
||||||
|
|
||||||
|
4. Bind the GPU to the `vfio-pci` device driver:
|
||||||
|
|
||||||
|
```
|
||||||
|
$ sudo modprobe vfio-pci
|
||||||
|
$ echo 8086 1616 | sudo tee /sys/bus/pci/drivers/vfio-pci/new_id
|
||||||
|
$ echo $BDF | sudo tee --append /sys/bus/pci/drivers/vfio-pci/bind
|
||||||
|
```
|
||||||
|
|
||||||
|
After you run the previous commands, the GPU is bound to `vfio-pci` driver.<br/>
|
||||||
|
A new directory with the IOMMU group number is created under `/dev/vfio`:
|
||||||
|
|
||||||
|
```
|
||||||
|
$ ls -l /dev/vfio
|
||||||
|
total 0
|
||||||
|
crw------- 1 root root 241, 0 May 18 15:38 1
|
||||||
|
crw-rw-rw- 1 root root 10, 196 May 18 15:37 vfio
|
||||||
|
```
|
||||||
|
|
||||||
|
5. Start a Kata container with GPU device:
|
||||||
|
|
||||||
|
```
|
||||||
|
$ sudo docker run -it --runtime=kata-runtime --rm --device /dev/vfio/1 -v /dev:/dev debian /bin/bash
|
||||||
|
```
|
||||||
|
|
||||||
|
Run `lspci` within the container to verify the GPU device is seen in the list of
|
||||||
|
the PCI devices. Note the vendor-device id of the GPU ("8086:1616") in the `lspci` output.
|
||||||
|
|
||||||
|
```
|
||||||
|
$ lspci -nn -D
|
||||||
|
0000:00:00.0 Class [0600]: Device [8086:1237] (rev 02)
|
||||||
|
0000:00:01.0 Class [0601]: Device [8086:7000]
|
||||||
|
0000:00:01.1 Class [0101]: Device [8086:7010]
|
||||||
|
0000:00:01.3 Class [0680]: Device [8086:7113] (rev 03)
|
||||||
|
0000:00:02.0 Class [0604]: Device [1b36:0001]
|
||||||
|
0000:00:03.0 Class [0780]: Device [1af4:1003]
|
||||||
|
0000:00:04.0 Class [0100]: Device [1af4:1004]
|
||||||
|
0000:00:05.0 Class [0002]: Device [1af4:1009]
|
||||||
|
0000:00:06.0 Class [0200]: Device [1af4:1000]
|
||||||
|
0000:00:0f.0 Class [0300]: Device [8086:1616] (rev 09)
|
||||||
|
```
|
||||||
|
|
||||||
|
Additionally, you can access the device node for the graphics device:
|
||||||
|
|
||||||
|
```
|
||||||
|
$ ls /dev/dri
|
||||||
|
card0 renderD128
|
||||||
|
```
|
||||||
|
|
||||||
|
## GVT-g with Kata Containers
|
||||||
|
|
||||||
|
For GVT-g, you append `i915.enable_gvt=1` in addition to `intel_iommu=on`
|
||||||
|
on your host kernel command line and then reboot your host.
|
||||||
|
|
||||||
|
Use the following steps to pass an Intel Graphics device in GVT-g mode to a Kata Container:
|
||||||
|
|
||||||
|
1. Find the BDF for GPU device:
|
||||||
|
|
||||||
|
```
|
||||||
|
$ sudo lspci -nn -D | grep Graphics
|
||||||
|
0000:00:02.0 VGA compatible controller [0300]: Intel Corporation Broadwell-U Integrated Graphics [8086:1616] (rev 09)
|
||||||
|
```
|
||||||
|
|
||||||
|
Run the previous command to find out the BDF for the GPU device on host.
|
||||||
|
The previous output shows PCI address "0000:00:02.0" is assigned to the GPU device.
|
||||||
|
|
||||||
|
2. Choose the MDEV (Mediated Device) type for VGPU (Virtual GPU):
|
||||||
|
|
||||||
|
For background on `mdev` types, please follow this [kernel documentation](https://github.com/torvalds/linux/blob/master/Documentation/vfio-mediated-device.txt).
|
||||||
|
|
||||||
|
* List out the `mdev` types for the VGPU:
|
||||||
|
|
||||||
|
```
|
||||||
|
$ BDF="0000:00:02.0"
|
||||||
|
|
||||||
|
$ ls /sys/devices/pci0000:00/$BDF/mdev_supported_types
|
||||||
|
i915-GVTg_V4_1 i915-GVTg_V4_2 i915-GVTg_V4_4 i915-GVTg_V4_8
|
||||||
|
```
|
||||||
|
|
||||||
|
* Inspect the `mdev` types and choose one that fits your requirement:
|
||||||
|
|
||||||
|
```
|
||||||
|
$ cd /sys/devices/pci0000:00/0000:00:02.0/mdev_supported_types/i915-GVTg_V4_8 && ls
|
||||||
|
available_instances create description device_api devices
|
||||||
|
|
||||||
|
$ cat description
|
||||||
|
low_gm_size: 64MB
|
||||||
|
high_gm_size: 384MB
|
||||||
|
fence: 4
|
||||||
|
resolution: 1024x768
|
||||||
|
weight: 2
|
||||||
|
|
||||||
|
$ cat available_instances
|
||||||
|
7
|
||||||
|
```
|
||||||
|
|
||||||
|
The output of file `description` represents the GPU resources that are
|
||||||
|
assigned to the VGPU with specified MDEV type.The output of file `available_instances`
|
||||||
|
represents the remaining amount of VGPUs you can create with specified MDEV type.
|
||||||
|
|
||||||
|
3. Create a VGPU:
|
||||||
|
|
||||||
|
* Generate a uuid:
|
||||||
|
|
||||||
|
```
|
||||||
|
$ gpu_uuid=$(uuid)
|
||||||
|
```
|
||||||
|
|
||||||
|
* Write the uuid to the `create` file under the chosen mdev type:
|
||||||
|
|
||||||
|
```
|
||||||
|
$ echo $(gpu_uuid) | sudo tee /sys/devices/pci0000:00/0000:00:02.0/mdev_supported_types/i915-GVTg_V4_8/create
|
||||||
|
```
|
||||||
|
|
||||||
|
4. Find the IOMMU group for the VGPU:
|
||||||
|
|
||||||
|
```
|
||||||
|
$ ls -la /sys/devices/pci0000:00/0000:00:02.0/mdev_supported_types/i915-GVTg_V4_8/devices/${gpu_uuid}/iommu_group
|
||||||
|
lrwxrwxrwx 1 root root 0 May 18 14:35 devices/bbc4aafe-5807-11e8-a43e-03533cceae7d/iommu_group -> ../../../../kernel/iommu_groups/0
|
||||||
|
|
||||||
|
$ ls -l /dev/vfio
|
||||||
|
total 0
|
||||||
|
crw------- 1 root root 241, 0 May 18 11:30 0
|
||||||
|
crw-rw-rw- 1 root root 10, 196 May 18 11:29 vfio
|
||||||
|
```
|
||||||
|
|
||||||
|
The IOMMU group "0" is created from the previous output.<br/>
|
||||||
|
Now you can use the device node `/dev/vfio/0` in docker command line to pass
|
||||||
|
the VGPU to a Kata Container.
|
||||||
|
|
||||||
|
5. Start Kata container with GPU device enabled:
|
||||||
|
|
||||||
|
```
|
||||||
|
$ sudo docker run -it --runtime=kata-runtime --rm --device /dev/vfio/0 -v /dev:/dev debian /bin/bash
|
||||||
|
$ lspci -nn -D
|
||||||
|
0000:00:00.0 Class [0600]: Device [8086:1237] (rev 02)
|
||||||
|
0000:00:01.0 Class [0601]: Device [8086:7000]
|
||||||
|
0000:00:01.1 Class [0101]: Device [8086:7010]
|
||||||
|
0000:00:01.3 Class [0680]: Device [8086:7113] (rev 03)
|
||||||
|
0000:00:02.0 Class [0604]: Device [1b36:0001]
|
||||||
|
0000:00:03.0 Class [0780]: Device [1af4:1003]
|
||||||
|
0000:00:04.0 Class [0100]: Device [1af4:1004]
|
||||||
|
0000:00:05.0 Class [0002]: Device [1af4:1009]
|
||||||
|
0000:00:06.0 Class [0200]: Device [1af4:1000]
|
||||||
|
0000:00:0f.0 Class [0300]: Device [8086:1616] (rev 09)
|
||||||
|
```
|
||||||
|
|
||||||
|
BDF "0000:00:0f.0" is assigned to the VGPU device.
|
||||||
|
|
||||||
|
Additionally, you can access the device node for the graphics device:
|
||||||
|
|
||||||
|
```
|
||||||
|
$ ls /dev/dri
|
||||||
|
card0 renderD128
|
||||||
|
```
|
||||||
Loading…
Reference in New Issue
Block a user