Since the network device hotplug is an asynchronous operation,
it's possible that the hotplug operation had returned, but
the network device hasn't ready in guest, thus it's better to
retry on this operation to wait until the device ready in guest.
Signed-off-by: Fupan Li <fupan.lfp@antgroup.com>
Add comments to make the "EnableIOThreads" flag as a switch
for virtio-blk(based on IndepIOThreads) driver.
Signed-off-by: zhangchen.kidd <zhangchen.kidd@jd.com>
Make hotplug virtio-blk device attach to Independent IOThread 0 as default
when enabled the EnableIOThreads and IndepIOThreads.
Signed-off-by: zhangchen.kidd <zhangchen.kidd@jd.com>
Qemu already support the device_add with iothread args.
Make KATA have ability to hotplug PCI device with IOThreads.
Currently, just support QEMU as the hypervisor, not sure it
works for stratovirt.
Signed-off-by: zhangchen.kidd <zhangchen.kidd@jd.com>
Make the original virtio-scsi iothread and the new independent
iothread to a dedicated method for handing the related logics.
Signed-off-by: zhangchen.kidd <zhangchen.kidd@jd.com>
Introduce independent IOThread framework for Kata container.
What is the indep_iothreads:
This new feature introduce a way to pre-alloc IOThreads
for QEMU hypervisor (maybe other hypervisor can support too).
Independent IOThreads enables IO to be processed in a separate thread.
To generally improve the performance of each module, avoid them
running in the QEMU main loop.
Why need indep_iothreads:
In Kata container implementation, many devices based on hotplug
mechanism. The real workload container may not sync the same
lifecycle with the VM. It may require to hotplug/unplug new disks
or other devices without destroying the VM. So we can keep the
IOThread with the VM as a IOThread pool(some devices need multi iothreads
for performance like virtio-blk vq-mapping), the hotplug devices
can attach/detach with the IOThread according to business needs.
At the same time, QEMU also support the "x-blockdev-set-iothread"
to change iothreads(but it need stop VM for data secure).
Current QEMU have many devices support iothread, virtio-blk,
virtio-scsi, virtio-balloon, monitor, colo-compare...etc...
How it works:
Add new item in hypervisor struct named "indep_iothreads" in toml.
The default value is 0, it reused the original "enable_iothreads" as
the switch. If the "indep_iothreads" != 0 and "enable_iothreads" = true
it will add qmp object -iothread indepIOThreadsPrefix_No when VM startup.
The first user is the virtio-blk, it will attach the indep_iothread_0
as default when enable iothread for virtio-blk.
Thanks
Chen
Signed-off-by: zhangchen.kidd <zhangchen.kidd@jd.com>
It was just missed when adding those configurations.
Signed-off-by: Manuel Huber <manuelh@nvidia.com>
Signed-off-by: Fabiano Fidêncio <ffidencio@nvidia.com>
While the primary goal of this change is to detect regressions to the
NVIDIA SNP GPU scenario, various improvements to reflect a more
realistic CC setting are planned in subsequent changes, such as:
* moving away from the overlayfs snapshotter
* disabling filesystem sharing
* applying a pod security policy
* activating the GPUs only after attestation
* using a refined approach for GPU cold-plugging without requiring
annotations
* revisiting pod timeout and overhead parameters (the podOverhead value
was increased due to CUDA vectorAdd requiring about 6Gi of
podOverhead, as well as the inference and embedqa requiring at least
12Gi, respectively, 14Gi of podOverhead to run without invoking the
host's oom-killer. We will revisit this aspect after addressing
points 1. and 2.)
Signed-off-by: Manuel Huber <manuelh@nvidia.com>
Signed-off-by: Fabiano Fidêncio <ffidencio@nvidia.com>
Pod annotations from the outer runtime are being used for cold-plugging
CDI devices. We need to ensure that these annotations don't leak into
the inner runtime for which specific container (sibling) annotations
are being created. Without this change, the inner runtime receives both
annotations, leading to failing CDI injection as an outer runtime
annotation observed in the guest translates to an unresolvable CDI
device, for example, cdi.k8s.io/gpu: "nvidia.com/pgpu=0".
Signed-off-by: Manuel Huber <manuelh@nvidia.com>
In order to fix:
```
=== Running govulncheck on containerd-shim-kata-v2 ===
Vulnerabilities found in containerd-shim-kata-v2:
=== Symbol Results ===
Vulnerability #1: GO-2025-4015
Excessive CPU consumption in Reader.ReadResponse in net/textproto
More info: https://pkg.go.dev/vuln/GO-2025-4015
Standard library
Found in: net/textproto@go1.24.6
Fixed in: net/textproto@go1.24.8
Vulnerable symbols found:
#1: textproto.Reader.ReadResponse
Vulnerability #2: GO-2025-4014
Unbounded allocation when parsing GNU sparse map in archive/tar
More info: https://pkg.go.dev/vuln/GO-2025-4014
Standard library
Found in: archive/tar@go1.24.6
Fixed in: archive/tar@go1.24.8
Vulnerable symbols found:
#1: tar.Reader.Next
Vulnerability #3: GO-2025-4013
Panic when validating certificates with DSA public keys in crypto/x509
More info: https://pkg.go.dev/vuln/GO-2025-4013
Standard library
Found in: crypto/x509@go1.24.6
Fixed in: crypto/x509@go1.24.8
Vulnerable symbols found:
#1: x509.Certificate.Verify
#2: x509.Certificate.Verify
Vulnerability #4: GO-2025-4012
Lack of limit when parsing cookies can cause memory exhaustion in net/http
More info: https://pkg.go.dev/vuln/GO-2025-4012
Standard library
Found in: net/http@go1.24.6
Fixed in: net/http@go1.24.8
Vulnerable symbols found:
#1: http.Client.Do
#2: http.Client.Get
#3: http.Client.Head
#4: http.Client.Post
#5: http.Client.PostForm
Use '-show traces' to see the other 9 found symbols
Vulnerability #5: GO-2025-4011
Parsing DER payload can cause memory exhaustion in encoding/asn1
More info: https://pkg.go.dev/vuln/GO-2025-4011
Standard library
Found in: encoding/asn1@go1.24.6
Fixed in: encoding/asn1@go1.24.8
Vulnerable symbols found:
#1: asn1.Unmarshal
#2: asn1.UnmarshalWithParams
Vulnerability #6: GO-2025-4010
Insufficient validation of bracketed IPv6 hostnames in net/url
More info: https://pkg.go.dev/vuln/GO-2025-4010
Standard library
Found in: net/url@go1.24.6
Fixed in: net/url@go1.24.8
Vulnerable symbols found:
#1: url.JoinPath
#2: url.Parse
#3: url.ParseRequestURI
#4: url.URL.Parse
#5: url.URL.UnmarshalBinary
Vulnerability #7: GO-2025-4009
Quadratic complexity when parsing some invalid inputs in encoding/pem
More info: https://pkg.go.dev/vuln/GO-2025-4009
Standard library
Found in: encoding/pem@go1.24.6
Fixed in: encoding/pem@go1.24.8
Vulnerable symbols found:
#1: pem.Decode
Vulnerability #8: GO-2025-4008
ALPN negotiation error contains attacker controlled information in
crypto/tls
More info: https://pkg.go.dev/vuln/GO-2025-4008
Standard library
Found in: crypto/tls@go1.24.6
Fixed in: crypto/tls@go1.24.8
Vulnerable symbols found:
#1: tls.Conn.Handshake
#2: tls.Conn.HandshakeContext
#3: tls.Conn.Read
#4: tls.Conn.Write
#5: tls.Dial
Use '-show traces' to see the other 4 found symbols
Vulnerability #9: GO-2025-4007
Quadratic complexity when checking name constraints in crypto/x509
More info: https://pkg.go.dev/vuln/GO-2025-4007
Standard library
Found in: crypto/x509@go1.24.6
Fixed in: crypto/x509@go1.24.9
Vulnerable symbols found:
#1: x509.CertPool.AppendCertsFromPEM
#2: x509.Certificate.CheckCRLSignature
#3: x509.Certificate.CheckSignature
#4: x509.Certificate.CheckSignatureFrom
#5: x509.Certificate.CreateCRL
Use '-show traces' to see the other 27 found symbols
Vulnerability #10: GO-2025-4006
Excessive CPU consumption in ParseAddress in net/mail
More info: https://pkg.go.dev/vuln/GO-2025-4006
Standard library
Found in: net/mail@go1.24.6
Fixed in: net/mail@go1.24.8
Vulnerable symbols found:
#1: mail.AddressParser.Parse
#2: mail.AddressParser.ParseList
#3: mail.Header.AddressList
#4: mail.ParseAddress
#5: mail.ParseAddressList
```
Signed-off-by: Fabiano Fidêncio <ffidencio@nvidia.com>
We have here either /dev/vfio/<num> or /dev/vfio/devices/vfio<num>,
for IOMMUFD format /dev/vfio/devices/vfio<num>, strip "vfio" prefix
/dev/vfio/123 - basename "123" - vfioNum = "123" - cdi.k8s.io/vfio123
/dev/vfio/devices/vfio123 - basename "vfio123" - strip - vfioNum = "123" - cdi.k8s.io/vfio123
Signed-off-by: Zvonko Kaiser <zkaiser@nvidia.com>
Add TDX QGS quote-generation-socket TDX QEMU object params for
attestation to work in NVGPU+TDX environment.
Signed-off-by: Mikko Ylinen <mikko.ylinen@intel.com>
This change crystallizes and simplifies the current handling of /dev
hostPath mounts with virtually no functional change.
Before this change:
- If a mount DESTINATION is in /dev and it is a non-regular file on the HOST,
the shim passes the OCI bind mount as is to the guest (e.g.
/dev/kmsg:/dev/kmsg). The container rightfully sees the GUEST device.
- If the mount DESTINATION does not exist on the host, the shim relies on
k8s/containerd to automatically create a directory (ie. non-regular file) on
the HOST. The shim then also passes the OCI bind mount as is to the guest. The
container rightfully sees the GUEST device.
- For other /dev mounts, the shim passes the device major/minor to the guest
over virtio-fs. The container rightfully sees the GUEST device.
After this change:
- If a mount SOURCE is in /dev and it is a non-regular file on the HOST,
the shim passes the OCI bind mount as is to the guest. The container
rightfully sees the GUEST device.
- The shim does not anymore rely on k8s/containerd to create missing mount
directories. Instead it explicitely handles missing mount SOURCES, and
treats them like the previous bullet point.
- The shim no longer uses virtio-fs to pass /dev device major/minor to the
guest, instead it passes the OCI bind mount as is.
Signed-off-by: Aurélien Bombo <abombo@microsoft.com>
85f3391bc added the support for TDX QGS port=0 but missed
defaultQgsPort in the default config. defaultQgsPort overrides
user provided tdx_quote_generation_service_socket_port=0.
After this change, defaultQgsPort is not needed anymore since
there's no default: any positive integer is OK and negative or
unset value becomes a parse error.
QEMUTDXQUOTEGENERATIONSERVICESOCKETPORT in the Makefile is used
to provide a sane default when tdx_quote_generation_service_socket_port
gets set in the configuration.
Signed-off-by: Mikko Ylinen <mikko.ylinen@intel.com>
This fixes that error everywhere by adding a `name:` field to all jobs that
were missing it. We keep the same name as the job ID to ensure no
disturbance to the required job names.
Signed-off-by: Aurélien Bombo <abombo@microsoft.com>
On s390x, QEMU fails if maxmem is set to 0:
```
invalid value of maxmem: maximum memory size (0x0) must be at least the initial memory size
```
This commit sets maxmem to the initial memory size for s390x when hotplug is disabled,
resolving the error while still ensuring that memory hotplug remains off.
Signed-off-by: Hyounggyu Choi <Hyounggyu.Choi@ibm.com>
The setting '-m xM,slots=y,maxmem=zM' where maxmem is from
the host's memory capacity is failing with confidential VMs
on hosts having 1T+ of RAM.
slots/maxmem are necessary for setups where the container
memory is hotplugged to the VM during container creation based
on createContainer info.
This is not the case with CoCo since StaticResourceManagement
is enabled and memory hotplug flows have not been checked.
To avoid unexpeted errors with maxmem, disable slots/maxmem
in case ConfidentialGuest is requested.
Signed-off-by: Mikko Ylinen <mikko.ylinen@intel.com>
This commit introduces changes to parse the PciDeviceInfo received
in response payload when adding a network device to the VM with cloud
hypervisor. When hotplugging a network device for a given endpoint,
it rightly sets the PciPath of the plugged-in device in the endpoint.
In calls like virtcontainers/sandbox.go:AddInterface, the later call
to agent sends the pci info for uevents (instead of empty value) to
rightly update the interfaces instead of failing with `Link not found`
Signed-off-by: Sumedh Alok Sharma <sumsharma@microsoft.com>
Delete annotation from OCI spec and sandbox config. This is done after the optional initdata annotation value has been read.
Signed-off-by: Saul Paredes <saulparedes@microsoft.com>
Bump cri-o to 1.34.0 to try and remediate security advisories
CVE-2025-0750 and CVE-2025-4437.
Note: Running
```
go get github.com/cri-o/cri-o@v1.34.0
```
seems to bump a lot of other go modules, hence the size of the
vendor diff
Signed-off-by: stevenhorsman <steven@uk.ibm.com>
In google.golang.org/grpc v1.72.0, `DialContext`, is deprecated, so
switch to use `NewClient` instead.
`grpc.WithBlock()` is deprecated and not recommend, so remove this
Signed-off-by: stevenhorsman <steven@uk.ibm.com>
In `github.com/prometheus/common v0.62.0` expfmt.FmtText
is deprecated, so replace with `expfmt.NewFormat(expfmt.TypeTextPlain)`.
Signed-off-by: stevenhorsman <steven@uk.ibm.com>
In `github.com/prometheus/common v0.62.0` expfmt.FmtText
is deprecated, so replace with `expfmt.NewFormat(expfmt.TypeTextPlain)`.
Signed-off-by: stevenhorsman <steven@uk.ibm.com>
Since the cloud hypervisor's resize vCPU is an asynchronous operation,
it's possible that the previous resize operation hasn't completed when
the request is sent, causing the current call to return an error.
Therefore, several retries can be performed to avoid this error.
Signed-off-by: Fupan Li <fupan.lfp@antgroup.com>
```
Experimental fw_cfg Device Support
This feature enables passing configuration data and files, such as VM
boot configurations (kernel, kernel cmdline, e820 memory map, and ACPI
tables), from the host to the guest. (#7117)
Experimental ivshmem Device Support
Support for inter-VM shared memory has been added. For more information,
please refer to the ivshmem documentation. (#6703)
Firmware Boot Support on riscv64
In addition to direct kernel boot, firmware boot support has been added
on riscv64 hosts. (#7249)
Increased vCPU Limit on x86_64/kvm
The maximum number of supported vCPUs on x86_64 hosts using KVM has been
raised from 254 to 8192. (#7299)
Improved Block Performance with Small Block Sizes
Performance for virtio-blk with small block sizes (16KB and below)
is enhanced via submitting async IO requests in batches. (#7146)
Faster VM Pause Operation
The VM pause operation now is significantly faster particularly for VMs
with a large number of vCPUs. (#7290)
Updated Documentation on Windows Guest Support
Our Windows documentation now includes instructions to run Windows 11
guests, in addition to Windows Server guests. (#7218)
Policy on AI Generated Code
We will decline any contributions known to contain contents generated or
derived from using Large Language Models (LLMs). Details can be found
in our contributing documentation. (#7162)
Removed SGX Support
The SGX support has been removed, as announced in the deprecation notice two
release cycles ago. (#7093)
Notable Bug Fixes
Seccomp filter fixes with glibc v2.42 (#7327)
Various fixes related to (#7331, #7334, #7335)
```
From https://github.com/cloud-hypervisor/cloud-hypervisor/releases/tag/v48.0
Signed-off-by: Fabiano Fidêncio <fabiano@fidencio.org>
Use @DEFENABLEANNOTATIONS_COCO@ in configuration-qemu-snp.toml,
for consistency with the tdx and coco-dev configuration files.
k8s-initdata.bats was failing during CI on SNP without this change,
because the cc_init_data annotation was disabled.
Signed-off-by: Dan Mihai <dmihai@microsoft.com>
ef642fe890 added a special case to avoid
moving cgroups that are on the "default" slice in case of deletion.
However, this special check should be done in the Parent() method
instead, which ensures that the default resource controller ID is
returned, instead of ".".
Fixes: #11599
Signed-off-by: Fabiano Fidêncio <fidencio@northflank.com>
Similar to what we've done for Cloud Hypervisor in the commit
9f76467cb7, we're backporting a runtime-rs
feature that would be benificial to have as part of the go runtime.
This allows users to use virito-balloon for the hypervisor to reclaim
memory freed by the guest.
Signed-off-by: Fabiano Fidêncio <fidencio@northflank.com>
As part of the go 1.24.6 bump there are errors about the incorrect
use of a errorf, so switch to the non-formatting version, or add
the format string as appropriate
Signed-off-by: stevenhorsman <steven@uk.ibm.com>
golang 1.25 has been released, so 1.23 is EoL,
so we should update to ensure we don't end up with security issues
Signed-off-by: stevenhorsman <steven@uk.ibm.com>
In #11693 the cc_init_data annotation was changes to be hypervisor
scoped, so each hypervisor needs to explicitly allow it in order to
use it now, so add this to both the go and rust runtime's remote
configurations
Signed-off-by: stevenhorsman <steven@uk.ibm.com>
