mirror of
https://github.com/k8snetworkplumbingwg/multus-cni.git
synced 2026-07-16 17:17:15 +00:00
Merge pull request #1492 from rollandf/dra-resourcename
feat: DRA resource.k8s.io/v1 integration via draclient
This commit is contained in:
Sebastian Scheinkman
GitHub
@@ -70,7 +70,18 @@ rules:
|
||||
- pods/status
|
||||
verbs:
|
||||
- get
|
||||
- list
|
||||
- update
|
||||
- watch
|
||||
- apiGroups:
|
||||
- "resource.k8s.io"
|
||||
resources:
|
||||
- resourceclaims
|
||||
- resourceclaims/status
|
||||
- resourceslices
|
||||
verbs:
|
||||
- get
|
||||
- list
|
||||
- apiGroups:
|
||||
- ""
|
||||
- events.k8s.io
|
||||
|
||||
@@ -73,6 +73,15 @@ rules:
|
||||
- list
|
||||
- update
|
||||
- watch
|
||||
- apiGroups:
|
||||
- "resource.k8s.io"
|
||||
resources:
|
||||
- resourceclaims
|
||||
- resourceclaims/status
|
||||
- resourceslices
|
||||
verbs:
|
||||
- get
|
||||
- list
|
||||
- apiGroups:
|
||||
- ""
|
||||
- events.k8s.io
|
||||
|
||||
@@ -70,7 +70,18 @@ rules:
|
||||
- pods/status
|
||||
verbs:
|
||||
- get
|
||||
- list
|
||||
- update
|
||||
- watch
|
||||
- apiGroups:
|
||||
- "resource.k8s.io"
|
||||
resources:
|
||||
- resourceclaims
|
||||
- resourceclaims/status
|
||||
- resourceslices
|
||||
verbs:
|
||||
- get
|
||||
- list
|
||||
- apiGroups:
|
||||
- ""
|
||||
- events.k8s.io
|
||||
|
||||
@@ -645,112 +645,144 @@ If you wish to have auto configuration use the `readinessindicatorfile` in the c
|
||||
|
||||
### Run pod with network annotation and Dynamic Resource Allocation driver
|
||||
|
||||
> :warning: Dynamic Resource Allocation (DRA) is [currently an alpha](https://kubernetes.io/docs/concepts/scheduling-eviction/dynamic-resource-allocation/),
|
||||
> and is subject to change. Please consider this functionality as a preview. The architecture and usage of DRA in
|
||||
> Multus CNI may change in the future as this technology matures.
|
||||
>
|
||||
> The current DRA integration is based on the DRA API for Kubernetes 1.26 to 1.30. With Kubernetes 1.31, the DRA API
|
||||
> will change and multus doesn't integrate with the new API yet.
|
||||
|
||||
Dynamic Resource Allocation is alternative mechanism to device plugin which allows to requests pod and container
|
||||
resources.
|
||||
Dynamic Resource Allocation is an alternative mechanism to device plugin which allows pods to request pod and container
|
||||
resources dynamically.
|
||||
|
||||
The following sections describe how to use DRA with multus and NVIDIA DRA driver. Other DRA networking driver vendors
|
||||
should follow similar concepts to make use of multus DRA support.
|
||||
The following sections describe how to use DRA with Multus. DRA networking driver vendors should follow similar
|
||||
concepts to make use of Multus DRA support.
|
||||
|
||||
#### Prerequisite
|
||||
|
||||
1. Kubernetes 1.27
|
||||
2. Container Runtime with CDI support enabled
|
||||
3. Kubernetes runtime-config=resource.k8s.io/v1alpha2
|
||||
4. Kubernetes feature-gates=DynamicResourceAllocation=True,KubeletPodResourcesDynamicResources=true
|
||||
1. Kubernetes 1.34+
|
||||
|
||||
#### Install DRA driver
|
||||
|
||||
The current example uses NVIDIA DRA driver for networking. This DRA driver is not publicly available. An alternative to
|
||||
this DRA driver is available at [dra-example-driver](https://github.com/kubernetes-sigs/dra-example-driver).
|
||||
You need to install a DRA driver that provides network devices. For example, you can use the [SR-IOV DRA](https://github.com/k8snetworkplumbingwg/dra-driver-sriov) driver or
|
||||
other DRA networking drivers. Refer to your DRA driver documentation for installation instructions.
|
||||
|
||||
#### Create dynamic resource class with NVIDIA network DRA driver
|
||||
The DRA driver MUST expose the following attributes on each allocated **network** device in `ResourceSlice`:
|
||||
- `k8s.cni.cncf.io/deviceID`: device ID that Multus passes to the CNI plugin.
|
||||
- `k8s.cni.cncf.io/resourceName`: **must exactly match** the value you put on the NetworkAttachmentDefinition
|
||||
`k8s.v1.cni.cncf.io/resourceName` annotation (same style as classic extended resources, e.g. `intel.com/sriov_vf`).
|
||||
|
||||
The `ResourceClass` defines the resource pool of `sf-pool-1`.
|
||||
Devices that lack `k8s.cni.cncf.io/deviceID` or `k8s.cni.cncf.io/resourceName` are **silently skipped** in the
|
||||
normal claims path — this is intentional so that mixed pods with both network and non-network DRA claims (e.g. a
|
||||
GPU claim alongside an SR-IOV claim) work correctly. The existing kubelet / device-plugin resource-map entries for
|
||||
such pods are preserved unchanged.
|
||||
|
||||
```
|
||||
# Execute following command at Kubernetes master
|
||||
cat <<EOF | kubectl create -f -
|
||||
apiVersion: resource.k8s.io/v1alpha2
|
||||
kind: ResourceClass
|
||||
metadata:
|
||||
name: sf-pool-1
|
||||
driverName: net.resource.nvidia.com
|
||||
EOF
|
||||
```
|
||||
For pods that use the **extended resource** feature gate, Multus uses `pod.status.extendedResourceClaimStatus`
|
||||
request mappings: the NAD `resourceName` matches `requestMappings[].resourceName`. In this path the device
|
||||
attribute `k8s.cni.cncf.io/resourceName` **must** be present and **equal** `requestMappings[].resourceName`
|
||||
exactly — a missing or mismatched value is treated as an error, because the extended-resource mapping explicitly
|
||||
names the expected device. Device lookup uses the same `ResourceClaim` / `ResourceSlice` flow as the normal path.
|
||||
|
||||
#### Create network attachment definition with resource name
|
||||
|
||||
The `k8s.v1.cni.cncf.io/resourceName` should match the `ResourceClass` name defined in the section above.
|
||||
In this example it is `sf-pool-1`. Multus query the K8s PodResource API to fetch the `resourceClass` name and also
|
||||
query the NetworkAttachmentDefinition `k8s.v1.cni.cncf.io/resourceName`. If both has the same name multus send the
|
||||
CDI device name in the DeviceID argument.
|
||||
The `k8s.v1.cni.cncf.io/resourceName` annotation must be the **same string** as the `k8s.cni.cncf.io/resourceName`
|
||||
device attribute published by your DRA driver for that allocation. Multiple secondary networks use distinct values
|
||||
on each device (or the same value when multiple device IDs should be consumed from one NAD, matching the device-plugin
|
||||
model).
|
||||
|
||||
##### NetworkAttachmentDefinition for ovn-kubernetes example:
|
||||
Multus queries the ResourceClaim and ResourceSlices APIs. When the NAD annotation equals the device’s
|
||||
`k8s.cni.cncf.io/resourceName`, Multus passes the corresponding `k8s.cni.cncf.io/deviceID` to the CNI plugin.
|
||||
|
||||
Following command creates NetworkAttachmentDefinition. CNI config is in `config:` field.
|
||||
##### NetworkAttachmentDefinition for SR-IOV example:
|
||||
|
||||
Following command creates a NetworkAttachmentDefinition for SR-IOV. The `resourceName` annotation must match what the
|
||||
DRA driver sets on the allocated device (here `intel.com/sriov_vf`):
|
||||
|
||||
```
|
||||
# Execute following command at Kubernetes master
|
||||
cat <<EOF | kubectl create -f -
|
||||
apiVersion: "k8s.cni.cncf.io/v1"
|
||||
apiVersion: k8s.cni.cncf.io/v1
|
||||
kind: NetworkAttachmentDefinition
|
||||
metadata:
|
||||
name: default
|
||||
name: sriov-net
|
||||
namespace: default
|
||||
annotations:
|
||||
k8s.v1.cni.cncf.io/resourceName: sf-pool-1
|
||||
k8s.v1.cni.cncf.io/resourceName: intel.com/sriov_vf
|
||||
spec:
|
||||
config: '{
|
||||
"cniVersion": "0.4.0",
|
||||
"dns": {},
|
||||
"ipam": {},
|
||||
"logFile": "/var/log/ovn-kubernetes/ovn-k8s-cni-overlay.log",
|
||||
"logLevel": "4",
|
||||
"logfile-maxage": 5,
|
||||
"logfile-maxbackups": 5,
|
||||
"logfile-maxsize": 100,
|
||||
"name": "ovn-kubernetes",
|
||||
"type": "ovn-k8s-cni-overlay"
|
||||
}'
|
||||
config: |-
|
||||
{
|
||||
"cniVersion": "1.0.0",
|
||||
"name": "sriov-net",
|
||||
"type": "sriov",
|
||||
"vlan": 0,
|
||||
"spoofchk": "on",
|
||||
"trust": "on",
|
||||
"vlanQoS": 0,
|
||||
"logLevel": "info",
|
||||
"ipam": {
|
||||
"type": "host-local",
|
||||
"ranges": [
|
||||
[
|
||||
{
|
||||
"subnet": "10.0.2.0/24"
|
||||
}
|
||||
]
|
||||
]
|
||||
}
|
||||
}
|
||||
EOF
|
||||
```
|
||||
|
||||
#### Create DRA Resource Claim
|
||||
#### Create Device Class
|
||||
|
||||
Following command creates `ResourceClaim` `sf` which request resource from `ResourceClass` `sf-pool-1`.
|
||||
Following command creates a `DeviceClass` for the `ResourceClaimTemplate` to request devices from.
|
||||
|
||||
```
|
||||
# Execute following command at Kubernetes master
|
||||
cat <<EOF | kubectl create -f -
|
||||
apiVersion: resource.k8s.io/v1alpha2
|
||||
kind: ResourceClaim
|
||||
apiVersion: resource.k8s.io/v1
|
||||
kind: DeviceClass
|
||||
metadata:
|
||||
name: sriovnetwork.k8snetworkplumbingwg.io
|
||||
spec:
|
||||
selectors:
|
||||
- cel:
|
||||
expression: device.driver == 'sriovnetwork.k8snetworkplumbingwg.io'
|
||||
EOF
|
||||
```
|
||||
|
||||
#### Create DRA Resource Claim Template
|
||||
|
||||
Following command creates a `ResourceClaimTemplate` that requests a VF device from the SR-IOV device class.
|
||||
The device request is named `vf`; the DRA driver should publish `k8s.cni.cncf.io/resourceName` on the device
|
||||
so it matches your NAD (e.g. `intel.com/sriov_vf`).
|
||||
|
||||
```
|
||||
# Execute following command at Kubernetes master
|
||||
cat <<EOF | kubectl create -f -
|
||||
apiVersion: resource.k8s.io/v1
|
||||
kind: ResourceClaimTemplate
|
||||
metadata:
|
||||
namespace: default
|
||||
name: sf
|
||||
name: sriov-template
|
||||
spec:
|
||||
spec:
|
||||
resourceClassName: sf-pool-1
|
||||
devices:
|
||||
requests:
|
||||
- name: vf
|
||||
deviceClassName: sriovnetwork.k8snetworkplumbingwg.io
|
||||
EOF
|
||||
```
|
||||
|
||||
#### Launch pod with DRA Resource Claim
|
||||
|
||||
Following command Launch a Pod with primiry network `default` and `ResourceClaim` `sf`.
|
||||
Following command launches a Pod with the secondary network `sriov-net` and a DRA resource claim named `sriov`.
|
||||
The NAD `resourceName` must match the driver’s `k8s.cni.cncf.io/resourceName` on the allocated device.
|
||||
|
||||
```
|
||||
# Execute following command at Kubernetes master
|
||||
cat <<EOF | kubectl create -f -
|
||||
apiVersion: v1
|
||||
kind: Pod
|
||||
metadata:
|
||||
namespace: default
|
||||
name: test-sf-claim
|
||||
name: sriov-pod
|
||||
annotations:
|
||||
v1.multus-cni.io/default-network: default
|
||||
k8s.v1.cni.cncf.io/networks: sriov-net
|
||||
spec:
|
||||
restartPolicy: Always
|
||||
containers:
|
||||
@@ -759,9 +791,16 @@ spec:
|
||||
command: ["/bin/sh", "-ec", "while :; do echo '.'; sleep 5 ; done"]
|
||||
resources:
|
||||
claims:
|
||||
- name: resource
|
||||
- name: sriov
|
||||
resourceClaims:
|
||||
- name: resource
|
||||
source:
|
||||
resourceClaimName: sf
|
||||
- name: sriov
|
||||
resourceClaimTemplateName: sriov-template
|
||||
EOF
|
||||
```
|
||||
|
||||
In this example:
|
||||
- The pod has a resourceClaim named `sriov` that uses the `sriov-template`
|
||||
- The ResourceClaimTemplate has a device request named `vf`
|
||||
- The DRA driver must set `k8s.cni.cncf.io/resourceName: intel.com/sriov_vf` (and `deviceID`) on the allocated device
|
||||
- The NetworkAttachmentDefinition uses `resourceName: intel.com/sriov_vf` to match that device attribute
|
||||
- Multus will match these and provide the allocated deviceID to the SR-IOV CNI plugin
|
||||
|
||||
363
pkg/draclient/draclient.go
Normal file
363
pkg/draclient/draclient.go
Normal file
@@ -0,0 +1,363 @@
|
||||
// Copyright (c) 2026 Multus Authors
|
||||
//
|
||||
// Licensed under the Apache License, Version 2.0 (the "License");
|
||||
// you may not use this file except in compliance with the License.
|
||||
// You may obtain a copy of the License at
|
||||
//
|
||||
// http://www.apache.org/licenses/LICENSE-2.0
|
||||
//
|
||||
// Unless required by applicable law or agreed to in writing, software
|
||||
// distributed under the License is distributed on an "AS IS" BASIS,
|
||||
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
// See the License for the specific language governing permissions and
|
||||
// limitations under the License.
|
||||
|
||||
package draclient
|
||||
|
||||
import (
|
||||
"context"
|
||||
"errors"
|
||||
"fmt"
|
||||
"time"
|
||||
|
||||
v1 "k8s.io/api/core/v1"
|
||||
resourcev1api "k8s.io/api/resource/v1"
|
||||
metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
|
||||
resourcev1 "k8s.io/client-go/kubernetes/typed/resource/v1"
|
||||
|
||||
"gopkg.in/k8snetworkplumbingwg/multus-cni.v4/pkg/logging"
|
||||
"gopkg.in/k8snetworkplumbingwg/multus-cni.v4/pkg/types"
|
||||
)
|
||||
|
||||
const (
|
||||
multusDeviceIDAttr = "k8s.cni.cncf.io/deviceID"
|
||||
multusResourceNameAttr = "k8s.cni.cncf.io/resourceName"
|
||||
)
|
||||
|
||||
// errDeviceNotInAnySlice is returned when allocation names a device that does not appear
|
||||
// in any ResourceSlice for that driver/pool (wrapped in getDeviceInfo). Callers may skip
|
||||
// individual results so multi-device claims (e.g. SR-IOV + GPU) still succeed for CNI.
|
||||
var errDeviceNotInAnySlice = errors.New("device not in any matching resource slice")
|
||||
|
||||
// namespacedClaimCacheKey avoids cache collisions: ResourceClaim is namespaced.
|
||||
func namespacedClaimCacheKey(namespace, claimName string) string {
|
||||
return namespace + "/" + claimName
|
||||
}
|
||||
|
||||
type deviceInfo struct {
|
||||
DeviceID string
|
||||
ResourceName string
|
||||
}
|
||||
|
||||
// deviceInfoCacheKey uniquely identifies a device within a driver/pool combination.
|
||||
type deviceInfoCacheKey struct {
|
||||
driverPool string // "driver/pool"
|
||||
deviceName string
|
||||
}
|
||||
|
||||
type ClientInterface interface {
|
||||
GetPodResourceMap(ctx context.Context, pod *v1.Pod, resourceMap map[string]*types.ResourceInfo) error
|
||||
}
|
||||
|
||||
type draClient struct {
|
||||
client resourcev1.ResourceV1Interface
|
||||
// deviceInfoCache stores lightweight device attributes extracted from ResourceSlices.
|
||||
// Keys are (driver/pool, deviceName); only the two attributes Multus reads are kept,
|
||||
// so full ResourceSlice objects (~400KB each) are GC'd immediately after listing.
|
||||
deviceInfoCache map[deviceInfoCacheKey]*deviceInfo
|
||||
// populatedDrivers tracks which "nodeName/driverName" combinations have already been
|
||||
// fetched from the API, preventing redundant List calls within a client's lifetime.
|
||||
populatedDrivers map[string]bool
|
||||
// Keys are namespace/claimName (ResourceClaim is namespaced).
|
||||
resourceClaimCache map[string]*resourcev1api.ResourceClaim
|
||||
}
|
||||
|
||||
// getOrFetchResourceClaim returns the ResourceClaim for the given namespace/name,
|
||||
// fetching it from the API server on the first call and caching it for subsequent calls.
|
||||
func (d *draClient) getOrFetchResourceClaim(ctx context.Context, namespace, claimName string) (*resourcev1api.ResourceClaim, error) {
|
||||
cacheKey := namespacedClaimCacheKey(namespace, claimName)
|
||||
if rc, ok := d.resourceClaimCache[cacheKey]; ok {
|
||||
logging.Debugf("getOrFetchResourceClaim: using cached ResourceClaim %s", cacheKey)
|
||||
return rc, nil
|
||||
}
|
||||
logging.Debugf("getOrFetchResourceClaim: ResourceClaim %s not in cache, fetching from API", cacheKey)
|
||||
rc, err := d.client.ResourceClaims(namespace).Get(ctx, claimName, metav1.GetOptions{})
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
d.resourceClaimCache[cacheKey] = rc
|
||||
logging.Debugf("getOrFetchResourceClaim: cached ResourceClaim %s", cacheKey)
|
||||
return rc, nil
|
||||
}
|
||||
|
||||
func NewClient(client resourcev1.ResourceV1Interface) ClientInterface {
|
||||
logging.Debugf("NewClient: creating new DRA client")
|
||||
return &draClient{
|
||||
client: client,
|
||||
deviceInfoCache: make(map[deviceInfoCacheKey]*deviceInfo),
|
||||
populatedDrivers: make(map[string]bool),
|
||||
resourceClaimCache: make(map[string]*resourcev1api.ResourceClaim),
|
||||
}
|
||||
}
|
||||
|
||||
// GetPodResourceMap populates resourceMap with device IDs for all DRA-allocated devices
|
||||
// that Multus needs to configure networking for the given pod.
|
||||
//
|
||||
// It walks pod.Status.ResourceClaimStatuses and, for each claim, fetches the
|
||||
// ResourceClaim (cached after the first fetch) to discover which devices were allocated.
|
||||
// For each allocated device it calls getDeviceInfo, which lazily lists ResourceSlices
|
||||
// for that driver scoped to pod.Spec.NodeName via server-side field selectors
|
||||
// (spec.nodeName + spec.driver) and extracts only the two attributes Multus reads
|
||||
// (k8s.cni.cncf.io/deviceID and k8s.cni.cncf.io/resourceName). Full ResourceSlice
|
||||
// objects are discarded immediately after extraction to keep memory usage minimal.
|
||||
//
|
||||
// Devices that lack k8s.cni.cncf.io/deviceID (e.g. GPU allocations not intended for
|
||||
// CNI) are silently skipped, allowing mixed SR-IOV + GPU claims to succeed. Devices
|
||||
// missing k8s.cni.cncf.io/resourceName are also skipped since there is no NAD to map
|
||||
// them to. If pod.Status.ExtendedResourceClaimStatus is set, it is processed with the
|
||||
// same logic but strict validation: device resourceName must match the extended
|
||||
// resource mapping exactly or an error is returned.
|
||||
//
|
||||
// ctx is respected for cancellation; a 20-second safety timeout is applied on top.
|
||||
func (d *draClient) GetPodResourceMap(ctx context.Context, pod *v1.Pod, resourceMap map[string]*types.ResourceInfo) error {
|
||||
logging.Verbosef("GetPodResourceMap: processing DRA resources for pod %s/%s", pod.Namespace, pod.Name)
|
||||
|
||||
ctx, cancel := context.WithTimeout(ctx, 20*time.Second)
|
||||
defer cancel()
|
||||
|
||||
nodeName := pod.Spec.NodeName
|
||||
|
||||
for _, claimResource := range pod.Status.ResourceClaimStatuses {
|
||||
if claimResource.ResourceClaimName == nil {
|
||||
logging.Errorf("GetPodResourceMap: resource claim status has nil ResourceClaimName")
|
||||
continue
|
||||
}
|
||||
claimName := *claimResource.ResourceClaimName
|
||||
// claimResource.Name is the pod-local reference name (pod.spec.resourceClaims[].name);
|
||||
// claimName is the actual ResourceClaim object name in the API — the two differ when
|
||||
// the scheduler generates a per-pod ResourceClaim from a ResourceClaimTemplate.
|
||||
logging.Debugf("GetPodResourceMap: processing ResourceClaim %q (pod-local ref: %q)", claimName, claimResource.Name)
|
||||
|
||||
resourceClaim, err := d.getOrFetchResourceClaim(ctx, pod.Namespace, claimName)
|
||||
if err != nil {
|
||||
logging.Errorf("GetPodResourceMap: failed to get resource claim %s: %v", claimName, err)
|
||||
return err
|
||||
}
|
||||
|
||||
if resourceClaim.Status.Allocation == nil || resourceClaim.Status.Allocation.Devices.Results == nil {
|
||||
logging.Errorf("GetPodResourceMap: claim %s has no device allocation", claimName)
|
||||
return fmt.Errorf("claim %s has no device allocation", claimName)
|
||||
}
|
||||
|
||||
results := resourceClaim.Status.Allocation.Devices.Results
|
||||
resolvedCount := 0
|
||||
for _, result := range results {
|
||||
logging.Debugf("GetPodResourceMap: processing device allocation - driver: %s, pool: %s, device: %s, request: %s",
|
||||
result.Driver, result.Pool, result.Device, result.Request)
|
||||
|
||||
info, err := d.getDeviceInfo(ctx, nodeName, result)
|
||||
if err != nil {
|
||||
if errors.Is(err, errDeviceNotInAnySlice) {
|
||||
logging.Warningf(
|
||||
"GetPodResourceMap: skipping allocation result for claim %s (driver=%s pool=%s device=%s): %v",
|
||||
claimName, result.Driver, result.Pool, result.Device, err)
|
||||
continue
|
||||
}
|
||||
logging.Errorf("GetPodResourceMap: failed to get device info for claim %s: %v", claimName, err)
|
||||
return err
|
||||
}
|
||||
|
||||
if info.ResourceName == "" {
|
||||
logging.Warningf(
|
||||
"GetPodResourceMap: skipping allocation result for claim %s (driver=%s pool=%s device=%s): no %q (only devices published for CNI are mapped)",
|
||||
claimName, result.Driver, result.Pool, result.Device, multusResourceNameAttr)
|
||||
continue
|
||||
}
|
||||
|
||||
resourceMapKey := info.ResourceName
|
||||
if rInfo, ok := resourceMap[resourceMapKey]; ok {
|
||||
rInfo.DeviceIDs = append(rInfo.DeviceIDs, info.DeviceID)
|
||||
logging.Debugf("GetPodResourceMap: appended device ID %s to existing resource map entry %s", info.DeviceID, resourceMapKey)
|
||||
} else {
|
||||
resourceMap[resourceMapKey] = &types.ResourceInfo{DeviceIDs: []string{info.DeviceID}}
|
||||
logging.Debugf("GetPodResourceMap: created new resource map entry %s with device ID %s", resourceMapKey, info.DeviceID)
|
||||
}
|
||||
resolvedCount++
|
||||
}
|
||||
if resolvedCount == 0 && len(results) > 0 {
|
||||
logging.Warningf(
|
||||
"GetPodResourceMap: claim %s had no allocation results mapped for Multus (skipping this claim; existing kubelet/device-plugin map entries are kept). "+
|
||||
"Fix DRA ResourceSlices or Multus attributes if this claim should contribute to CNI.",
|
||||
claimName)
|
||||
continue
|
||||
}
|
||||
logging.Debugf("GetPodResourceMap: successfully processed resource claim %s", claimName)
|
||||
}
|
||||
|
||||
if pod.Status.ExtendedResourceClaimStatus != nil {
|
||||
if err := d.processExtendedResourceClaimStatus(ctx, nodeName, pod, resourceMap); err != nil {
|
||||
return err
|
||||
}
|
||||
}
|
||||
|
||||
types.SortDeviceIDs(resourceMap)
|
||||
logging.Verbosef("GetPodResourceMap: successfully processed all DRA resources for pod %s/%s, total resources: %d",
|
||||
pod.Namespace, pod.Name, len(resourceMap))
|
||||
return nil
|
||||
}
|
||||
|
||||
// processExtendedResourceClaimStatus fills the resource map for pods that use
|
||||
// the extended resource feature gate (pod.Status.ExtendedResourceClaimStatus).
|
||||
// Keys come from requestMappings[].resourceName (same as NAD annotation).
|
||||
func (d *draClient) processExtendedResourceClaimStatus(ctx context.Context, nodeName string, pod *v1.Pod, resourceMap map[string]*types.ResourceInfo) error {
|
||||
extStatus := pod.Status.ExtendedResourceClaimStatus
|
||||
claimName := extStatus.ResourceClaimName
|
||||
logging.Debugf("GetPodResourceMap: processing extended resource claim: %s/%s", pod.Namespace, claimName)
|
||||
|
||||
resourceClaim, err := d.getOrFetchResourceClaim(ctx, pod.Namespace, claimName)
|
||||
if err != nil {
|
||||
logging.Errorf("GetPodResourceMap: failed to get extended resource claim %s/%s: %v", pod.Namespace, claimName, err)
|
||||
return err
|
||||
}
|
||||
|
||||
if resourceClaim.Status.Allocation == nil || resourceClaim.Status.Allocation.Devices.Results == nil {
|
||||
logging.Errorf("GetPodResourceMap: claim %s has no device allocation", claimName)
|
||||
return fmt.Errorf("claim %s has no device allocation", claimName)
|
||||
}
|
||||
|
||||
resultsByRequest := make(map[string][]resourcev1api.DeviceRequestAllocationResult)
|
||||
for _, result := range resourceClaim.Status.Allocation.Devices.Results {
|
||||
resultsByRequest[result.Request] = append(resultsByRequest[result.Request], result)
|
||||
}
|
||||
|
||||
for _, mapping := range extStatus.RequestMappings {
|
||||
results, ok := resultsByRequest[mapping.RequestName]
|
||||
if !ok || len(results) == 0 {
|
||||
logging.Errorf("GetPodResourceMap: extended resource request %s not found in claim %s", mapping.RequestName, claimName)
|
||||
return fmt.Errorf("request %s not found in claim %s", mapping.RequestName, claimName)
|
||||
}
|
||||
|
||||
resourceMapKey := mapping.ResourceName
|
||||
for _, result := range results {
|
||||
info, err := d.getDeviceInfo(ctx, nodeName, result)
|
||||
if err != nil {
|
||||
logging.Errorf("GetPodResourceMap: failed to get device info for extended resource claim %s request %s: %v", claimName, mapping.RequestName, err)
|
||||
return err
|
||||
}
|
||||
|
||||
if info.ResourceName == "" {
|
||||
resErr := fmt.Errorf("device %s missing required attribute %s (must match NAD k8s.v1.cni.cncf.io/resourceName and extended mapping %q)",
|
||||
result.Device, multusResourceNameAttr, resourceMapKey)
|
||||
logging.Errorf("GetPodResourceMap: %v", resErr)
|
||||
return resErr
|
||||
}
|
||||
if info.ResourceName != mapping.ResourceName {
|
||||
resErr := fmt.Errorf("device %s: %s is %q but extended resource mapping for request %q is %q",
|
||||
result.Device, multusResourceNameAttr, info.ResourceName, mapping.RequestName, mapping.ResourceName)
|
||||
logging.Errorf("GetPodResourceMap: %v", resErr)
|
||||
return resErr
|
||||
}
|
||||
|
||||
if rInfo, ok := resourceMap[resourceMapKey]; ok {
|
||||
rInfo.DeviceIDs = append(rInfo.DeviceIDs, info.DeviceID)
|
||||
logging.Debugf("GetPodResourceMap: appended device ID %s to extended resource map entry %s", info.DeviceID, resourceMapKey)
|
||||
} else {
|
||||
resourceMap[resourceMapKey] = &types.ResourceInfo{DeviceIDs: []string{info.DeviceID}}
|
||||
logging.Debugf("GetPodResourceMap: created new extended resource map entry %s with device ID %s", resourceMapKey, info.DeviceID)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
logging.Debugf("GetPodResourceMap: successfully processed extended resource claim %s", claimName)
|
||||
return nil
|
||||
}
|
||||
|
||||
// ensureDriverCachePopulated lists ResourceSlices for the given node and driver (using server-side
|
||||
// field selectors) and extracts only the two attributes Multus needs into deviceInfoCache.
|
||||
// Full ResourceSlice objects are discarded after extraction, keeping memory usage minimal.
|
||||
// Subsequent calls for the same node/driver combination are no-ops.
|
||||
func (d *draClient) ensureDriverCachePopulated(ctx context.Context, nodeName, driverName string) error {
|
||||
populatedKey := nodeName + "/" + driverName
|
||||
if d.populatedDrivers[populatedKey] {
|
||||
return nil
|
||||
}
|
||||
|
||||
// driverName is always non-empty (Driver is +required in DeviceRequestAllocationResult).
|
||||
// Always filter by driver to avoid loading unrelated slices into the cache.
|
||||
// Also filter by node when available to further scope the list.
|
||||
listOptions := metav1.ListOptions{
|
||||
FieldSelector: fmt.Sprintf("%s=%s", resourcev1api.ResourceSliceSelectorDriver, driverName),
|
||||
}
|
||||
if nodeName != "" {
|
||||
listOptions.FieldSelector = fmt.Sprintf("%s=%s,%s=%s",
|
||||
resourcev1api.ResourceSliceSelectorNodeName, nodeName,
|
||||
resourcev1api.ResourceSliceSelectorDriver, driverName)
|
||||
}
|
||||
|
||||
logging.Debugf("ensureDriverCachePopulated: listing ResourceSlices (fieldSelector=%q)", listOptions.FieldSelector)
|
||||
slices, err := d.client.ResourceSlices().List(ctx, listOptions)
|
||||
if err != nil {
|
||||
logging.Errorf("ensureDriverCachePopulated: failed to list resource slices: %v", err)
|
||||
return err
|
||||
}
|
||||
logging.Debugf("ensureDriverCachePopulated: listed %d ResourceSlice(s) for node=%q driver=%q", len(slices.Items), nodeName, driverName)
|
||||
|
||||
for i := range slices.Items {
|
||||
slice := &slices.Items[i]
|
||||
driverPool := fmt.Sprintf("%s/%s", slice.Spec.Driver, slice.Spec.Pool.Name)
|
||||
for _, device := range slice.Spec.Devices {
|
||||
key := deviceInfoCacheKey{driverPool: driverPool, deviceName: device.Name}
|
||||
info := &deviceInfo{}
|
||||
if attr, ok := device.Attributes[multusDeviceIDAttr]; ok && attr.StringValue != nil {
|
||||
info.DeviceID = *attr.StringValue
|
||||
}
|
||||
if attr, ok := device.Attributes[multusResourceNameAttr]; ok && attr.StringValue != nil {
|
||||
info.ResourceName = *attr.StringValue
|
||||
}
|
||||
if info.DeviceID != "" {
|
||||
d.deviceInfoCache[key] = info
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
d.populatedDrivers[populatedKey] = true
|
||||
return nil
|
||||
}
|
||||
|
||||
// getDeviceInfo returns the cached deviceInfo for the device named in result.
|
||||
// On the first call for a given driver it triggers ensureDriverCachePopulated,
|
||||
// which lists ResourceSlices scoped to nodeName and result.Driver via server-side
|
||||
// field selectors and extracts only the two attributes Multus needs. Subsequent
|
||||
// calls for the same driver are O(1) map lookups with no API traffic.
|
||||
// Returns errDeviceNotInAnySlice (wrapped) if the device is absent from the cache
|
||||
// or lacks a deviceID attribute; callers in the normal claims path treat this as
|
||||
// a skip, while the extended-resource path treats it as a hard error.
|
||||
func (d *draClient) getDeviceInfo(ctx context.Context, nodeName string, result resourcev1api.DeviceRequestAllocationResult) (*deviceInfo, error) {
|
||||
driverPool := fmt.Sprintf("%s/%s", result.Driver, result.Pool)
|
||||
logging.Debugf("getDeviceInfo: looking up device for driver/pool: %s, device: %s", driverPool, result.Device)
|
||||
|
||||
if err := d.ensureDriverCachePopulated(ctx, nodeName, result.Driver); err != nil {
|
||||
return nil, err
|
||||
}
|
||||
|
||||
key := deviceInfoCacheKey{driverPool: driverPool, deviceName: result.Device}
|
||||
info, ok := d.deviceInfoCache[key]
|
||||
if !ok {
|
||||
notFoundErr := fmt.Errorf("%w: device %s not found for claim resource %s/%s in any matching resource slice",
|
||||
errDeviceNotInAnySlice, result.Device, result.Driver, result.Pool)
|
||||
logging.Errorf("getDeviceInfo: %v", notFoundErr)
|
||||
return nil, notFoundErr
|
||||
}
|
||||
|
||||
if info.DeviceID == "" {
|
||||
logging.Warningf(
|
||||
"getDeviceInfo: device %q (driver %q, pool %q) has no %q in ResourceSlice; skipping allocation result",
|
||||
result.Device, result.Driver, result.Pool, multusDeviceIDAttr)
|
||||
return nil, fmt.Errorf("%w: device %q present in slice but missing %q",
|
||||
errDeviceNotInAnySlice, result.Device, multusDeviceIDAttr)
|
||||
}
|
||||
|
||||
logging.Verbosef("getDeviceInfo: successfully retrieved info for device %s (driver/pool: %s): deviceID=%s, resourceName=%s",
|
||||
result.Device, driverPool, info.DeviceID, info.ResourceName)
|
||||
return info, nil
|
||||
}
|
||||
29
pkg/draclient/draclient_suite_test.go
Normal file
29
pkg/draclient/draclient_suite_test.go
Normal file
@@ -0,0 +1,29 @@
|
||||
// Copyright (c) 2025 Multus Authors
|
||||
//
|
||||
// Licensed under the Apache License, Version 2.0 (the "License");
|
||||
// you may not use this file except in compliance with the License.
|
||||
// You may obtain a copy of the License at
|
||||
//
|
||||
// http://www.apache.org/licenses/LICENSE-2.0
|
||||
//
|
||||
// Unless required by applicable law or agreed to in writing, software
|
||||
// distributed under the License is distributed on an "AS IS" BASIS,
|
||||
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
// See the License for the specific language governing permissions and
|
||||
// limitations under the License.
|
||||
|
||||
package draclient
|
||||
|
||||
// disable dot-imports only for testing
|
||||
//revive:disable:dot-imports
|
||||
import (
|
||||
. "github.com/onsi/ginkgo/v2"
|
||||
. "github.com/onsi/gomega"
|
||||
|
||||
"testing"
|
||||
)
|
||||
|
||||
func TestDRAClient(t *testing.T) {
|
||||
RegisterFailHandler(Fail)
|
||||
RunSpecs(t, "draclient")
|
||||
}
|
||||
1979
pkg/draclient/draclient_test.go
Normal file
1979
pkg/draclient/draclient_test.go
Normal file
File diff suppressed because it is too large
Load Diff
@@ -42,6 +42,7 @@ import (
|
||||
netclient "github.com/k8snetworkplumbingwg/network-attachment-definition-client/pkg/client/clientset/versioned"
|
||||
netlister "github.com/k8snetworkplumbingwg/network-attachment-definition-client/pkg/client/listers/k8s.cni.cncf.io/v1"
|
||||
netutils "github.com/k8snetworkplumbingwg/network-attachment-definition-client/pkg/utils"
|
||||
"gopkg.in/k8snetworkplumbingwg/multus-cni.v4/pkg/draclient"
|
||||
"gopkg.in/k8snetworkplumbingwg/multus-cni.v4/pkg/kubeletclient"
|
||||
"gopkg.in/k8snetworkplumbingwg/multus-cni.v4/pkg/logging"
|
||||
"gopkg.in/k8snetworkplumbingwg/multus-cni.v4/pkg/types"
|
||||
@@ -53,6 +54,10 @@ const (
|
||||
networkAttachmentAnnot = "k8s.v1.cni.cncf.io/networks"
|
||||
)
|
||||
|
||||
// getResourceClientFunc returns kubelet / device-plugin resource info for a pod.
|
||||
// It defaults to kubeletclient.GetResourceClient; unit tests replace it to avoid a real kubelet checkpoint/socket.
|
||||
var getResourceClientFunc = kubeletclient.GetResourceClient
|
||||
|
||||
// NoK8sNetworkError indicates error, no network in kubernetes
|
||||
type NoK8sNetworkError struct {
|
||||
message string
|
||||
@@ -309,7 +314,7 @@ func getKubernetesDelegate(client *ClientInfo, net *types.NetworkSelectionElemen
|
||||
logging.Debugf("getKubernetesDelegate: found resourceName annotation : %s", resourceName)
|
||||
|
||||
if resourceMap == nil {
|
||||
ck, err := kubeletclient.GetResourceClient("")
|
||||
ck, err := getResourceClientFunc("")
|
||||
if err != nil {
|
||||
return nil, resourceMap, logging.Errorf("getKubernetesDelegate: failed to get a ResourceClient instance: %v", err)
|
||||
}
|
||||
@@ -317,6 +322,13 @@ func getKubernetesDelegate(client *ClientInfo, net *types.NetworkSelectionElemen
|
||||
if err != nil {
|
||||
return nil, resourceMap, logging.Errorf("getKubernetesDelegate: failed to get resourceMap from ResourceClient: %v", err)
|
||||
}
|
||||
|
||||
dc := draclient.NewClient(client.Client.ResourceV1())
|
||||
err = dc.GetPodResourceMap(context.TODO(), pod, resourceMap)
|
||||
if err != nil {
|
||||
return nil, resourceMap, logging.Errorf("getKubernetesDelegate: failed to get resourceMap from DRA client: %v", err)
|
||||
}
|
||||
|
||||
logging.Debugf("getKubernetesDelegate: resourceMap instance: %+v", resourceMap)
|
||||
}
|
||||
|
||||
|
||||
@@ -18,6 +18,7 @@ package k8sclient
|
||||
// disable dot-imports only for testing
|
||||
//revive:disable:dot-imports
|
||||
import (
|
||||
"context"
|
||||
"fmt"
|
||||
"os"
|
||||
"path/filepath"
|
||||
@@ -33,6 +34,9 @@ import (
|
||||
netfake "github.com/k8snetworkplumbingwg/network-attachment-definition-client/pkg/client/clientset/versioned/fake"
|
||||
netutils "github.com/k8snetworkplumbingwg/network-attachment-definition-client/pkg/utils"
|
||||
|
||||
v1 "k8s.io/api/core/v1"
|
||||
resourcev1api "k8s.io/api/resource/v1"
|
||||
metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
|
||||
"k8s.io/client-go/kubernetes/fake"
|
||||
|
||||
. "github.com/onsi/ginkgo/v2"
|
||||
@@ -52,6 +56,28 @@ func NewFakeClientInfo() *ClientInfo {
|
||||
}
|
||||
}
|
||||
|
||||
// fakeEmptyResourceClient implements types.ResourceClient with no device allocations (stubs kubelet in tests).
|
||||
type fakeEmptyResourceClient struct{}
|
||||
|
||||
func (*fakeEmptyResourceClient) GetPodResourceMap(*v1.Pod) (map[string]*types.ResourceInfo, error) {
|
||||
return make(map[string]*types.ResourceInfo), nil
|
||||
}
|
||||
|
||||
// fakeResourceClient implements types.ResourceClient with a pre-seeded device-plugin resource map,
|
||||
// simulating what the kubelet returns for device-plugin (non-DRA) allocations.
|
||||
type fakeResourceClient struct {
|
||||
resourceMap map[string]*types.ResourceInfo
|
||||
}
|
||||
|
||||
func (f *fakeResourceClient) GetPodResourceMap(*v1.Pod) (map[string]*types.ResourceInfo, error) {
|
||||
result := make(map[string]*types.ResourceInfo)
|
||||
for k, v := range f.resourceMap {
|
||||
cp := &types.ResourceInfo{DeviceIDs: append([]string(nil), v.DeviceIDs...)}
|
||||
result[k] = cp
|
||||
}
|
||||
return result, nil
|
||||
}
|
||||
|
||||
var _ = Describe("k8sclient operations", func() {
|
||||
var tmpDir string
|
||||
var err error
|
||||
@@ -1553,4 +1579,697 @@ users:
|
||||
Expect(err).NotTo(HaveOccurred())
|
||||
})
|
||||
})
|
||||
|
||||
Describe("DRA (Dynamic Resource Allocation) integration", func() {
|
||||
var tmpDir string
|
||||
var err error
|
||||
|
||||
BeforeEach(func() {
|
||||
tmpDir, err = os.MkdirTemp("", "multus_dra_test")
|
||||
Expect(err).NotTo(HaveOccurred())
|
||||
})
|
||||
|
||||
AfterEach(func() {
|
||||
err := os.RemoveAll(tmpDir)
|
||||
Expect(err).NotTo(HaveOccurred())
|
||||
})
|
||||
|
||||
Context("when pod has a DRA claim and the NAD carries a resourceName annotation", func() {
|
||||
// Override getResourceClientFunc so the kubelet stub returns an empty map,
|
||||
// letting us isolate the DRA branch in getKubernetesDelegate.
|
||||
var origGetResourceClient func(string) (types.ResourceClient, error)
|
||||
BeforeEach(func() {
|
||||
origGetResourceClient = getResourceClientFunc
|
||||
getResourceClientFunc = func(string) (types.ResourceClient, error) {
|
||||
return &fakeEmptyResourceClient{}, nil
|
||||
}
|
||||
})
|
||||
AfterEach(func() { getResourceClientFunc = origGetResourceClient })
|
||||
|
||||
It("should populate resourceMap from the DRA ResourceSlice when resourceMap is nil", func() {
|
||||
const fakePodName = "dra-test-pod"
|
||||
const fakeNamespace = "default"
|
||||
const sriovResName = "sriovnetwork.k8snetworkplumbingwg.io/sriov_vf"
|
||||
|
||||
netAttachDef := `{"name":"sriov-net","type":"sriov","cniVersion":"0.3.1"}`
|
||||
claimName := "sriov-claim"
|
||||
claimNamePtr := claimName
|
||||
driverName := "sriovnetwork.k8snetworkplumbingwg.io"
|
||||
poolName := "sriov-pool"
|
||||
deviceName := "vf-1"
|
||||
deviceID := "pci:0000:00:01.0"
|
||||
|
||||
fakePod := testutils.NewFakePod(fakePodName, "sriov-net", "")
|
||||
fakePod.Namespace = fakeNamespace
|
||||
fakePod.Status.ResourceClaimStatuses = []v1.PodResourceClaimStatus{
|
||||
{Name: claimName, ResourceClaimName: &claimNamePtr},
|
||||
}
|
||||
|
||||
clientInfo := NewFakeClientInfo()
|
||||
_, err = clientInfo.AddPod(fakePod)
|
||||
Expect(err).NotTo(HaveOccurred())
|
||||
|
||||
// NAD carries the resourceName annotation — this triggers the DRA lookup path.
|
||||
nad := testutils.NewFakeNetAttachDef(fakeNamespace, "sriov-net", netAttachDef)
|
||||
nad.Annotations = map[string]string{resourceNameAnnot: sriovResName}
|
||||
_, err = clientInfo.AddNetAttachDef(nad)
|
||||
Expect(err).NotTo(HaveOccurred())
|
||||
|
||||
deviceIDVal := deviceID
|
||||
resNameVal := sriovResName
|
||||
resourceSlice := &resourcev1api.ResourceSlice{
|
||||
ObjectMeta: metav1.ObjectMeta{Name: "sriov-slice"},
|
||||
Spec: resourcev1api.ResourceSliceSpec{
|
||||
Driver: driverName,
|
||||
Pool: resourcev1api.ResourcePool{Name: poolName, ResourceSliceCount: 1},
|
||||
Devices: []resourcev1api.Device{
|
||||
{Name: deviceName, Attributes: map[resourcev1api.QualifiedName]resourcev1api.DeviceAttribute{
|
||||
"k8s.cni.cncf.io/deviceID": {StringValue: &deviceIDVal},
|
||||
"k8s.cni.cncf.io/resourceName": {StringValue: &resNameVal},
|
||||
}},
|
||||
},
|
||||
},
|
||||
}
|
||||
resourceClaim := &resourcev1api.ResourceClaim{
|
||||
ObjectMeta: metav1.ObjectMeta{Name: claimName, Namespace: fakeNamespace},
|
||||
Status: resourcev1api.ResourceClaimStatus{Allocation: &resourcev1api.AllocationResult{
|
||||
Devices: resourcev1api.DeviceAllocationResult{Results: []resourcev1api.DeviceRequestAllocationResult{
|
||||
{Request: "sriov", Driver: driverName, Pool: poolName, Device: deviceName},
|
||||
}},
|
||||
}},
|
||||
}
|
||||
|
||||
_, err = clientInfo.Client.ResourceV1().ResourceClaims(fakeNamespace).Create(context.TODO(), resourceClaim, metav1.CreateOptions{})
|
||||
Expect(err).NotTo(HaveOccurred())
|
||||
_, err = clientInfo.Client.ResourceV1().ResourceSlices().Create(context.TODO(), resourceSlice, metav1.CreateOptions{})
|
||||
Expect(err).NotTo(HaveOccurred())
|
||||
|
||||
net := &types.NetworkSelectionElement{Name: "sriov-net", Namespace: fakeNamespace}
|
||||
// Pass nil so getKubernetesDelegate enters the DRA lookup branch.
|
||||
delegate, resourceMap, err := getKubernetesDelegate(clientInfo, net, tmpDir, fakePod, nil)
|
||||
Expect(err).NotTo(HaveOccurred())
|
||||
Expect(delegate).NotTo(BeNil())
|
||||
|
||||
// The DRA device ID must be in the resource map under the NAD resourceName key.
|
||||
Expect(resourceMap).To(HaveKey(sriovResName))
|
||||
Expect(resourceMap[sriovResName].DeviceIDs).To(ContainElement(deviceID))
|
||||
})
|
||||
})
|
||||
|
||||
Context("when GetNetworkDelegates is called with DRA-enabled pod", func() {
|
||||
It("should successfully retrieve delegates with DRA resources", func() {
|
||||
const fakePodName string = "dra-network-pod"
|
||||
const fakeNamespace string = "default"
|
||||
|
||||
// Create network attachment definition
|
||||
netAttachDef := `{
|
||||
"name": "dra-network",
|
||||
"type": "bridge",
|
||||
"cniVersion": "0.3.1"
|
||||
}`
|
||||
|
||||
// Create fake pod with DRA resource claims
|
||||
claimName := "network-claim"
|
||||
claimNamePtr := &claimName
|
||||
fakePod := testutils.NewFakePod(fakePodName, "dra-network", "")
|
||||
fakePod.Namespace = fakeNamespace
|
||||
fakePod.Status.ResourceClaimStatuses = []v1.PodResourceClaimStatus{
|
||||
{
|
||||
ResourceClaimName: claimNamePtr,
|
||||
},
|
||||
}
|
||||
|
||||
// Setup client
|
||||
clientInfo := NewFakeClientInfo()
|
||||
_, err = clientInfo.AddPod(fakePod)
|
||||
Expect(err).NotTo(HaveOccurred())
|
||||
|
||||
// Create network attachment definition with resource name annotation
|
||||
nad := testutils.NewFakeNetAttachDef(fakeNamespace, "dra-network", netAttachDef)
|
||||
nad.Annotations = map[string]string{
|
||||
resourceNameAnnot: "dra-network-claim/gpu",
|
||||
}
|
||||
_, err = clientInfo.AddNetAttachDef(nad)
|
||||
Expect(err).NotTo(HaveOccurred())
|
||||
|
||||
// Create DRA resources
|
||||
deviceName := "nic-1"
|
||||
driverName := "network.example.com"
|
||||
poolName := "network-pool"
|
||||
requestName := "gpu"
|
||||
deviceID := "pci:0000:00:02.0"
|
||||
// NAD k8s.v1.cni.cncf.io/resourceName must match device attribute k8s.cni.cncf.io/resourceName
|
||||
nadResourceKey := "dra-network-claim/gpu"
|
||||
|
||||
deviceIDValue := deviceID
|
||||
nadResourceKeyVal := nadResourceKey
|
||||
resourceSlice := &resourcev1api.ResourceSlice{
|
||||
ObjectMeta: metav1.ObjectMeta{
|
||||
Name: "network-resource-slice",
|
||||
},
|
||||
Spec: resourcev1api.ResourceSliceSpec{
|
||||
Driver: driverName,
|
||||
Pool: resourcev1api.ResourcePool{
|
||||
Name: poolName,
|
||||
ResourceSliceCount: 1,
|
||||
},
|
||||
Devices: []resourcev1api.Device{
|
||||
{
|
||||
Name: deviceName,
|
||||
Attributes: map[resourcev1api.QualifiedName]resourcev1api.DeviceAttribute{
|
||||
"k8s.cni.cncf.io/deviceID": {
|
||||
StringValue: &deviceIDValue,
|
||||
},
|
||||
"k8s.cni.cncf.io/resourceName": {
|
||||
StringValue: &nadResourceKeyVal,
|
||||
},
|
||||
},
|
||||
},
|
||||
},
|
||||
},
|
||||
}
|
||||
|
||||
resourceClaim := &resourcev1api.ResourceClaim{
|
||||
ObjectMeta: metav1.ObjectMeta{
|
||||
Name: claimName,
|
||||
Namespace: fakeNamespace,
|
||||
},
|
||||
Status: resourcev1api.ResourceClaimStatus{
|
||||
Allocation: &resourcev1api.AllocationResult{
|
||||
Devices: resourcev1api.DeviceAllocationResult{
|
||||
Results: []resourcev1api.DeviceRequestAllocationResult{
|
||||
{
|
||||
Request: requestName,
|
||||
Driver: driverName,
|
||||
Pool: poolName,
|
||||
Device: deviceName,
|
||||
},
|
||||
},
|
||||
},
|
||||
},
|
||||
},
|
||||
}
|
||||
|
||||
// Add DRA resources to fake client
|
||||
_, err = clientInfo.Client.ResourceV1().ResourceClaims(fakeNamespace).Create(context.TODO(), resourceClaim, metav1.CreateOptions{})
|
||||
Expect(err).NotTo(HaveOccurred())
|
||||
_, err = clientInfo.Client.ResourceV1().ResourceSlices().Create(context.TODO(), resourceSlice, metav1.CreateOptions{})
|
||||
Expect(err).NotTo(HaveOccurred())
|
||||
|
||||
// Get pod network
|
||||
networks, err := GetPodNetwork(fakePod)
|
||||
Expect(err).NotTo(HaveOccurred())
|
||||
Expect(networks).NotTo(BeEmpty())
|
||||
|
||||
// Create NetConf
|
||||
conf := &types.NetConf{
|
||||
ConfDir: tmpDir,
|
||||
}
|
||||
|
||||
// Get network delegates
|
||||
resourceMap := make(map[string]*types.ResourceInfo)
|
||||
delegates, err := GetNetworkDelegates(clientInfo, fakePod, networks, conf, resourceMap)
|
||||
Expect(err).NotTo(HaveOccurred())
|
||||
Expect(delegates).NotTo(BeEmpty())
|
||||
Expect(delegates[0]).NotTo(BeNil())
|
||||
})
|
||||
})
|
||||
|
||||
Context("when DRA resources are used in TryLoadPodDelegates", func() {
|
||||
It("should successfully load delegates with DRA resources", func() {
|
||||
const fakePodName string = "dra-delegate-pod"
|
||||
const fakeNamespace string = "default"
|
||||
|
||||
// Create network attachment definition
|
||||
netAttachDef := `{
|
||||
"name": "dra-delegate-network",
|
||||
"type": "macvlan",
|
||||
"cniVersion": "0.3.1"
|
||||
}`
|
||||
|
||||
// Create fake pod with DRA resource claims
|
||||
claimName := "delegate-claim"
|
||||
claimNamePtr := &claimName
|
||||
fakePod := testutils.NewFakePod(fakePodName, "dra-delegate-network", "")
|
||||
fakePod.Namespace = fakeNamespace
|
||||
fakePod.Status.ResourceClaimStatuses = []v1.PodResourceClaimStatus{
|
||||
{
|
||||
ResourceClaimName: claimNamePtr,
|
||||
},
|
||||
}
|
||||
|
||||
// Setup client
|
||||
clientInfo := NewFakeClientInfo()
|
||||
_, err = clientInfo.AddPod(fakePod)
|
||||
Expect(err).NotTo(HaveOccurred())
|
||||
|
||||
// Create network attachment definition
|
||||
nad := testutils.NewFakeNetAttachDef(fakeNamespace, "dra-delegate-network", netAttachDef)
|
||||
_, err = clientInfo.AddNetAttachDef(nad)
|
||||
Expect(err).NotTo(HaveOccurred())
|
||||
|
||||
// Create DRA resources
|
||||
deviceName := "macvlan-device"
|
||||
driverName := "macvlan.example.com"
|
||||
poolName := "macvlan-pool"
|
||||
requestName := "nic"
|
||||
deviceID := "pci:0000:00:03.0"
|
||||
|
||||
deviceIDValue := deviceID
|
||||
resourceSlice := &resourcev1api.ResourceSlice{
|
||||
ObjectMeta: metav1.ObjectMeta{
|
||||
Name: "macvlan-resource-slice",
|
||||
},
|
||||
Spec: resourcev1api.ResourceSliceSpec{
|
||||
Driver: driverName,
|
||||
Pool: resourcev1api.ResourcePool{
|
||||
Name: poolName,
|
||||
ResourceSliceCount: 1,
|
||||
},
|
||||
Devices: []resourcev1api.Device{
|
||||
{
|
||||
Name: deviceName,
|
||||
Attributes: map[resourcev1api.QualifiedName]resourcev1api.DeviceAttribute{
|
||||
"k8s.cni.cncf.io/deviceID": {
|
||||
StringValue: &deviceIDValue,
|
||||
},
|
||||
},
|
||||
},
|
||||
},
|
||||
},
|
||||
}
|
||||
|
||||
resourceClaim := &resourcev1api.ResourceClaim{
|
||||
ObjectMeta: metav1.ObjectMeta{
|
||||
Name: claimName,
|
||||
Namespace: fakeNamespace,
|
||||
},
|
||||
Status: resourcev1api.ResourceClaimStatus{
|
||||
Allocation: &resourcev1api.AllocationResult{
|
||||
Devices: resourcev1api.DeviceAllocationResult{
|
||||
Results: []resourcev1api.DeviceRequestAllocationResult{
|
||||
{
|
||||
Request: requestName,
|
||||
Driver: driverName,
|
||||
Pool: poolName,
|
||||
Device: deviceName,
|
||||
},
|
||||
},
|
||||
},
|
||||
},
|
||||
},
|
||||
}
|
||||
|
||||
// Add DRA resources to fake client
|
||||
_, err = clientInfo.Client.ResourceV1().ResourceClaims(fakeNamespace).Create(context.TODO(), resourceClaim, metav1.CreateOptions{})
|
||||
Expect(err).NotTo(HaveOccurred())
|
||||
_, err = clientInfo.Client.ResourceV1().ResourceSlices().Create(context.TODO(), resourceSlice, metav1.CreateOptions{})
|
||||
Expect(err).NotTo(HaveOccurred())
|
||||
|
||||
// Create NetConf with a default delegate
|
||||
confStr := `{
|
||||
"name": "test-network",
|
||||
"type": "multus",
|
||||
"delegates": [{
|
||||
"name": "default-network",
|
||||
"type": "bridge"
|
||||
}],
|
||||
"confDir": "` + tmpDir + `"
|
||||
}`
|
||||
|
||||
conf, err := types.LoadNetConf([]byte(confStr))
|
||||
Expect(err).NotTo(HaveOccurred())
|
||||
|
||||
// Try loading pod delegates
|
||||
resourceMap := make(map[string]*types.ResourceInfo)
|
||||
count, updatedClient, err := TryLoadPodDelegates(fakePod, conf, clientInfo, resourceMap)
|
||||
Expect(err).NotTo(HaveOccurred())
|
||||
Expect(updatedClient).NotTo(BeNil())
|
||||
Expect(count).To(BeNumerically(">", 0))
|
||||
})
|
||||
})
|
||||
|
||||
Context("when pod has no DRA resources", func() {
|
||||
It("should handle pods without DRA resources gracefully", func() {
|
||||
const fakePodName string = "no-dra-pod"
|
||||
const fakeNamespace string = "default"
|
||||
|
||||
// Create network attachment definition
|
||||
netAttachDef := `{
|
||||
"name": "simple-network",
|
||||
"type": "bridge",
|
||||
"cniVersion": "0.3.1"
|
||||
}`
|
||||
|
||||
// Create fake pod WITHOUT DRA resource claims
|
||||
fakePod := testutils.NewFakePod(fakePodName, "simple-network", "")
|
||||
fakePod.Namespace = fakeNamespace
|
||||
fakePod.Status.ResourceClaimStatuses = []v1.PodResourceClaimStatus{} // Empty
|
||||
|
||||
// Setup client
|
||||
clientInfo := NewFakeClientInfo()
|
||||
_, err = clientInfo.AddPod(fakePod)
|
||||
Expect(err).NotTo(HaveOccurred())
|
||||
|
||||
// Create network attachment definition
|
||||
nad := testutils.NewFakeNetAttachDef(fakeNamespace, "simple-network", netAttachDef)
|
||||
_, err = clientInfo.AddNetAttachDef(nad)
|
||||
Expect(err).NotTo(HaveOccurred())
|
||||
|
||||
// Get the network selection element
|
||||
net := &types.NetworkSelectionElement{
|
||||
Name: "simple-network",
|
||||
Namespace: fakeNamespace,
|
||||
}
|
||||
|
||||
// Call getKubernetesDelegate - should work without DRA
|
||||
delegate, resourceMap, err := getKubernetesDelegate(clientInfo, net, tmpDir, fakePod, nil)
|
||||
Expect(err).NotTo(HaveOccurred())
|
||||
Expect(delegate).NotTo(BeNil())
|
||||
// ResourceMap should be empty (no DRA resources)
|
||||
Expect(len(resourceMap)).To(Equal(0))
|
||||
})
|
||||
})
|
||||
|
||||
Context("when DRA client fails to get resources", func() {
|
||||
var origGetResourceClient func(string) (types.ResourceClient, error)
|
||||
|
||||
BeforeEach(func() {
|
||||
origGetResourceClient = getResourceClientFunc
|
||||
// Stub kubelet so we get past device-plugin checkpoint/socket and exercise draclient.
|
||||
getResourceClientFunc = func(string) (types.ResourceClient, error) {
|
||||
return &fakeEmptyResourceClient{}, nil
|
||||
}
|
||||
})
|
||||
|
||||
AfterEach(func() {
|
||||
getResourceClientFunc = origGetResourceClient
|
||||
})
|
||||
|
||||
It("should return an error from getKubernetesDelegate", func() {
|
||||
const fakePodName string = "dra-fail-pod"
|
||||
const fakeNamespace string = "default"
|
||||
const resourceName = "example.com/gpu"
|
||||
|
||||
// Create network attachment definition
|
||||
netAttachDef := `{
|
||||
"name": "fail-network",
|
||||
"type": "bridge",
|
||||
"cniVersion": "0.3.1"
|
||||
}`
|
||||
|
||||
// Create fake pod with DRA resource claims that don't exist in the API
|
||||
claimName := "non-existent-claim"
|
||||
claimNamePtr := &claimName
|
||||
fakePod := testutils.NewFakePod(fakePodName, "fail-network", "")
|
||||
fakePod.Namespace = fakeNamespace
|
||||
fakePod.Status.ResourceClaimStatuses = []v1.PodResourceClaimStatus{
|
||||
{
|
||||
Name: claimName,
|
||||
ResourceClaimName: claimNamePtr,
|
||||
},
|
||||
}
|
||||
|
||||
// Setup client
|
||||
clientInfo := NewFakeClientInfo()
|
||||
_, err = clientInfo.AddPod(fakePod)
|
||||
Expect(err).NotTo(HaveOccurred())
|
||||
|
||||
// NAD must have resourceName annotation so getKubernetesDelegate runs the DRA client path
|
||||
nad := testutils.NewFakeNetAttachDef(fakeNamespace, "fail-network", netAttachDef)
|
||||
nad.Annotations = map[string]string{
|
||||
resourceNameAnnot: resourceName,
|
||||
}
|
||||
_, err = clientInfo.AddNetAttachDef(nad)
|
||||
Expect(err).NotTo(HaveOccurred())
|
||||
|
||||
// Do NOT create the ResourceClaim — draclient.GetPodResourceMap fails on API Get.
|
||||
|
||||
net := &types.NetworkSelectionElement{
|
||||
Name: "fail-network",
|
||||
Namespace: fakeNamespace,
|
||||
}
|
||||
|
||||
_, _, err = getKubernetesDelegate(clientInfo, net, tmpDir, fakePod, nil)
|
||||
Expect(err).To(HaveOccurred())
|
||||
Expect(err.Error()).To(ContainSubstring("failed to get resourceMap from DRA client"))
|
||||
Expect(err.Error()).To(ContainSubstring("not found"))
|
||||
})
|
||||
})
|
||||
|
||||
Context("when using getNetDelegate with DRA resources", func() {
|
||||
It("should retrieve delegates with DRA resources populated", func() {
|
||||
const fakePodName string = "netdelegate-dra-pod"
|
||||
const fakeNamespace string = "default"
|
||||
|
||||
// Create network attachment definition
|
||||
netAttachDef := `{
|
||||
"name": "netdelegate-network",
|
||||
"type": "ipvlan",
|
||||
"cniVersion": "0.3.1"
|
||||
}`
|
||||
|
||||
// Create fake pod with DRA resource claims
|
||||
claimName := "netdelegate-claim"
|
||||
claimNamePtr := &claimName
|
||||
fakePod := testutils.NewFakePod(fakePodName, "", "")
|
||||
fakePod.Namespace = fakeNamespace
|
||||
fakePod.Status.ResourceClaimStatuses = []v1.PodResourceClaimStatus{
|
||||
{
|
||||
ResourceClaimName: claimNamePtr,
|
||||
},
|
||||
}
|
||||
|
||||
// Setup client
|
||||
clientInfo := NewFakeClientInfo()
|
||||
_, err = clientInfo.AddPod(fakePod)
|
||||
Expect(err).NotTo(HaveOccurred())
|
||||
|
||||
// Create network attachment definition
|
||||
nad := testutils.NewFakeNetAttachDef(fakeNamespace, "netdelegate-network", netAttachDef)
|
||||
_, err = clientInfo.AddNetAttachDef(nad)
|
||||
Expect(err).NotTo(HaveOccurred())
|
||||
|
||||
// Create DRA resources
|
||||
deviceName := "ipvlan-device"
|
||||
driverName := "ipvlan.example.com"
|
||||
poolName := "ipvlan-pool"
|
||||
requestName := "vnic"
|
||||
deviceID := "pci:0000:00:04.0"
|
||||
|
||||
deviceIDValue := deviceID
|
||||
resourceSlice := &resourcev1api.ResourceSlice{
|
||||
ObjectMeta: metav1.ObjectMeta{
|
||||
Name: "ipvlan-resource-slice",
|
||||
},
|
||||
Spec: resourcev1api.ResourceSliceSpec{
|
||||
Driver: driverName,
|
||||
Pool: resourcev1api.ResourcePool{
|
||||
Name: poolName,
|
||||
ResourceSliceCount: 1,
|
||||
},
|
||||
Devices: []resourcev1api.Device{
|
||||
{
|
||||
Name: deviceName,
|
||||
Attributes: map[resourcev1api.QualifiedName]resourcev1api.DeviceAttribute{
|
||||
"k8s.cni.cncf.io/deviceID": {
|
||||
StringValue: &deviceIDValue,
|
||||
},
|
||||
},
|
||||
},
|
||||
},
|
||||
},
|
||||
}
|
||||
|
||||
resourceClaim := &resourcev1api.ResourceClaim{
|
||||
ObjectMeta: metav1.ObjectMeta{
|
||||
Name: claimName,
|
||||
Namespace: fakeNamespace,
|
||||
},
|
||||
Status: resourcev1api.ResourceClaimStatus{
|
||||
Allocation: &resourcev1api.AllocationResult{
|
||||
Devices: resourcev1api.DeviceAllocationResult{
|
||||
Results: []resourcev1api.DeviceRequestAllocationResult{
|
||||
{
|
||||
Request: requestName,
|
||||
Driver: driverName,
|
||||
Pool: poolName,
|
||||
Device: deviceName,
|
||||
},
|
||||
},
|
||||
},
|
||||
},
|
||||
},
|
||||
}
|
||||
|
||||
// Add DRA resources to fake client
|
||||
_, err = clientInfo.Client.ResourceV1().ResourceClaims(fakeNamespace).Create(context.TODO(), resourceClaim, metav1.CreateOptions{})
|
||||
Expect(err).NotTo(HaveOccurred())
|
||||
_, err = clientInfo.Client.ResourceV1().ResourceSlices().Create(context.TODO(), resourceSlice, metav1.CreateOptions{})
|
||||
Expect(err).NotTo(HaveOccurred())
|
||||
|
||||
// Call getNetDelegate
|
||||
resourceMap := make(map[string]*types.ResourceInfo)
|
||||
delegate, updatedResourceMap, err := getNetDelegate(clientInfo, fakePod, "netdelegate-network", tmpDir, fakeNamespace, resourceMap)
|
||||
Expect(err).NotTo(HaveOccurred())
|
||||
Expect(delegate).NotTo(BeNil())
|
||||
Expect(updatedResourceMap).NotTo(BeNil())
|
||||
})
|
||||
})
|
||||
|
||||
Context("when pod has both device-plugin resources and a DRA ResourceClaim", func() {
|
||||
// Both tests override getResourceClientFunc so the kubelet stub returns a pre-seeded
|
||||
// SR-IOV device-plugin allocation, letting us exercise the combined kubelet+DRA path.
|
||||
const sriovResourceName = "openshift/sriov"
|
||||
const sriovDeviceIDKubelet = "0000:01:00.4"
|
||||
var origGetResourceClient func(string) (types.ResourceClient, error)
|
||||
|
||||
BeforeEach(func() {
|
||||
origGetResourceClient = getResourceClientFunc
|
||||
getResourceClientFunc = func(string) (types.ResourceClient, error) {
|
||||
return &fakeResourceClient{
|
||||
resourceMap: map[string]*types.ResourceInfo{
|
||||
sriovResourceName: {DeviceIDs: []string{sriovDeviceIDKubelet}},
|
||||
},
|
||||
}, nil
|
||||
}
|
||||
})
|
||||
|
||||
AfterEach(func() {
|
||||
getResourceClientFunc = origGetResourceClient
|
||||
})
|
||||
|
||||
It("should merge kubelet device-plugin entry with DRA SR-IOV claim that has full CNI attributes", func() {
|
||||
const fakePodName = "hybrid-sriov-pod"
|
||||
const fakeNamespace = "default"
|
||||
const sriovDeviceIDDRA = "0000:02:00.4"
|
||||
|
||||
netAttachDef := `{"name":"sriov-net","type":"sriov","cniVersion":"0.3.1"}`
|
||||
claimName := "sriov-dra-claim"
|
||||
driverName := "sriovnetwork.k8snetworkplumbingwg.io"
|
||||
poolName := "sriov-pool"
|
||||
deviceName := "vf-0"
|
||||
|
||||
claimNamePtr := claimName
|
||||
fakePod := testutils.NewFakePod(fakePodName, "sriov-net", "")
|
||||
fakePod.Namespace = fakeNamespace
|
||||
fakePod.Status.ResourceClaimStatuses = []v1.PodResourceClaimStatus{
|
||||
{Name: claimName, ResourceClaimName: &claimNamePtr},
|
||||
}
|
||||
|
||||
clientInfo := NewFakeClientInfo()
|
||||
_, err = clientInfo.AddPod(fakePod)
|
||||
Expect(err).NotTo(HaveOccurred())
|
||||
|
||||
// NAD carries the resourceName annotation → triggers both kubelet and DRA paths
|
||||
nad := testutils.NewFakeNetAttachDef(fakeNamespace, "sriov-net", netAttachDef)
|
||||
nad.Annotations = map[string]string{resourceNameAnnot: sriovResourceName}
|
||||
_, err = clientInfo.AddNetAttachDef(nad)
|
||||
Expect(err).NotTo(HaveOccurred())
|
||||
|
||||
// DRA device has both CNI attributes → appended to kubelet entry in resourceMap
|
||||
draDeviceID := sriovDeviceIDDRA
|
||||
draResName := sriovResourceName
|
||||
resourceSlice := &resourcev1api.ResourceSlice{
|
||||
ObjectMeta: metav1.ObjectMeta{Name: "sriov-dra-slice"},
|
||||
Spec: resourcev1api.ResourceSliceSpec{
|
||||
Driver: driverName,
|
||||
Pool: resourcev1api.ResourcePool{Name: poolName, ResourceSliceCount: 1},
|
||||
Devices: []resourcev1api.Device{
|
||||
{Name: deviceName, Attributes: map[resourcev1api.QualifiedName]resourcev1api.DeviceAttribute{
|
||||
"k8s.cni.cncf.io/deviceID": {StringValue: &draDeviceID},
|
||||
"k8s.cni.cncf.io/resourceName": {StringValue: &draResName},
|
||||
}},
|
||||
},
|
||||
},
|
||||
}
|
||||
resourceClaim := &resourcev1api.ResourceClaim{
|
||||
ObjectMeta: metav1.ObjectMeta{Name: claimName, Namespace: fakeNamespace},
|
||||
Status: resourcev1api.ResourceClaimStatus{Allocation: &resourcev1api.AllocationResult{
|
||||
Devices: resourcev1api.DeviceAllocationResult{Results: []resourcev1api.DeviceRequestAllocationResult{
|
||||
{Request: "sriov", Driver: driverName, Pool: poolName, Device: deviceName},
|
||||
}},
|
||||
}},
|
||||
}
|
||||
|
||||
_, err = clientInfo.Client.ResourceV1().ResourceSlices().Create(context.TODO(), resourceSlice, metav1.CreateOptions{})
|
||||
Expect(err).NotTo(HaveOccurred())
|
||||
_, err = clientInfo.Client.ResourceV1().ResourceClaims(fakeNamespace).Create(context.TODO(), resourceClaim, metav1.CreateOptions{})
|
||||
Expect(err).NotTo(HaveOccurred())
|
||||
|
||||
net := &types.NetworkSelectionElement{Name: "sriov-net", Namespace: fakeNamespace}
|
||||
delegate, resourceMap, err := getKubernetesDelegate(clientInfo, net, tmpDir, fakePod, nil)
|
||||
Expect(err).NotTo(HaveOccurred())
|
||||
Expect(delegate).NotTo(BeNil())
|
||||
|
||||
// Both kubelet and DRA device IDs must be present under the same resource name key
|
||||
Expect(resourceMap).To(HaveKey(sriovResourceName))
|
||||
Expect(resourceMap[sriovResourceName].DeviceIDs).To(ConsistOf(sriovDeviceIDKubelet, sriovDeviceIDDRA))
|
||||
})
|
||||
|
||||
It("should silently skip a GPU DRA claim that has no CNI attributes and still resolve the device-plugin SR-IOV entry", func() {
|
||||
const fakePodName = "hybrid-gpu-pod"
|
||||
const fakeNamespace = "default"
|
||||
|
||||
netAttachDef := `{"name":"sriov-net","type":"sriov","cniVersion":"0.3.1"}`
|
||||
claimName := "gpu-dra-claim"
|
||||
driverName := "gpu.nvidia.com"
|
||||
poolName := "gpu-pool"
|
||||
deviceName := "gpu-0"
|
||||
|
||||
claimNamePtr := claimName
|
||||
fakePod := testutils.NewFakePod(fakePodName, "sriov-net", "")
|
||||
fakePod.Namespace = fakeNamespace
|
||||
fakePod.Status.ResourceClaimStatuses = []v1.PodResourceClaimStatus{
|
||||
{Name: claimName, ResourceClaimName: &claimNamePtr},
|
||||
}
|
||||
|
||||
clientInfo := NewFakeClientInfo()
|
||||
_, err = clientInfo.AddPod(fakePod)
|
||||
Expect(err).NotTo(HaveOccurred())
|
||||
|
||||
nad := testutils.NewFakeNetAttachDef(fakeNamespace, "sriov-net", netAttachDef)
|
||||
nad.Annotations = map[string]string{resourceNameAnnot: sriovResourceName}
|
||||
_, err = clientInfo.AddNetAttachDef(nad)
|
||||
Expect(err).NotTo(HaveOccurred())
|
||||
|
||||
// GPU ResourceSlice intentionally has no k8s.cni.cncf.io/deviceID attribute
|
||||
gpuAttrVal := "gpu-model-a100"
|
||||
resourceSlice := &resourcev1api.ResourceSlice{
|
||||
ObjectMeta: metav1.ObjectMeta{Name: "gpu-dra-slice"},
|
||||
Spec: resourcev1api.ResourceSliceSpec{
|
||||
Driver: driverName,
|
||||
Pool: resourcev1api.ResourcePool{Name: poolName, ResourceSliceCount: 1},
|
||||
Devices: []resourcev1api.Device{
|
||||
{Name: deviceName, Attributes: map[resourcev1api.QualifiedName]resourcev1api.DeviceAttribute{
|
||||
"gpu.nvidia.com/model": {StringValue: &gpuAttrVal},
|
||||
}},
|
||||
},
|
||||
},
|
||||
}
|
||||
resourceClaim := &resourcev1api.ResourceClaim{
|
||||
ObjectMeta: metav1.ObjectMeta{Name: claimName, Namespace: fakeNamespace},
|
||||
Status: resourcev1api.ResourceClaimStatus{Allocation: &resourcev1api.AllocationResult{
|
||||
Devices: resourcev1api.DeviceAllocationResult{Results: []resourcev1api.DeviceRequestAllocationResult{
|
||||
{Request: "gpu", Driver: driverName, Pool: poolName, Device: deviceName},
|
||||
}},
|
||||
}},
|
||||
}
|
||||
|
||||
_, err = clientInfo.Client.ResourceV1().ResourceSlices().Create(context.TODO(), resourceSlice, metav1.CreateOptions{})
|
||||
Expect(err).NotTo(HaveOccurred())
|
||||
_, err = clientInfo.Client.ResourceV1().ResourceClaims(fakeNamespace).Create(context.TODO(), resourceClaim, metav1.CreateOptions{})
|
||||
Expect(err).NotTo(HaveOccurred())
|
||||
|
||||
net := &types.NetworkSelectionElement{Name: "sriov-net", Namespace: fakeNamespace}
|
||||
delegate, resourceMap, err := getKubernetesDelegate(clientInfo, net, tmpDir, fakePod, nil)
|
||||
Expect(err).NotTo(HaveOccurred())
|
||||
Expect(delegate).NotTo(BeNil())
|
||||
|
||||
// GPU claim contributed nothing; only the kubelet device-plugin entry is present
|
||||
Expect(resourceMap).To(HaveKey(sriovResourceName))
|
||||
Expect(resourceMap[sriovResourceName].DeviceIDs).To(Equal([]string{sriovDeviceIDKubelet}))
|
||||
})
|
||||
})
|
||||
})
|
||||
})
|
||||
|
||||
@@ -21,7 +21,6 @@ import (
|
||||
"net/url"
|
||||
"os"
|
||||
"path/filepath"
|
||||
"strings"
|
||||
"time"
|
||||
|
||||
"golang.org/x/net/context"
|
||||
@@ -111,7 +110,6 @@ type kubeletClient struct {
|
||||
}
|
||||
|
||||
func (rc *kubeletClient) getPodResources(client podresourcesapi.PodResourcesListerClient) error {
|
||||
|
||||
ctx, cancel := context.WithTimeout(context.Background(), 10*time.Second)
|
||||
defer cancel()
|
||||
|
||||
@@ -139,7 +137,6 @@ func (rc *kubeletClient) GetPodResourceMap(pod *v1.Pod) (map[string]*types.Resou
|
||||
if pr.Name == name && pr.Namespace == ns {
|
||||
for _, cnt := range pr.Containers {
|
||||
rc.getDevicePluginResources(cnt.Devices, resourceMap)
|
||||
rc.getDRAResources(cnt.DynamicResources, resourceMap)
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -157,27 +154,6 @@ func (rc *kubeletClient) getDevicePluginResources(devices []*podresourcesapi.Con
|
||||
}
|
||||
}
|
||||
|
||||
func (rc *kubeletClient) getDRAResources(dynamicResources []*podresourcesapi.DynamicResource, resourceMap map[string]*types.ResourceInfo) {
|
||||
for _, dynamicResource := range dynamicResources {
|
||||
var deviceIDs []string
|
||||
for _, claimResource := range dynamicResource.ClaimResources {
|
||||
for _, cdiDevice := range claimResource.CdiDevices {
|
||||
res := strings.Split(cdiDevice.Name, "=")
|
||||
if len(res) == 2 {
|
||||
deviceIDs = append(deviceIDs, res[1])
|
||||
} else {
|
||||
logging.Errorf("GetPodResourceMap: Invalid CDI format")
|
||||
}
|
||||
}
|
||||
}
|
||||
if rInfo, ok := resourceMap[dynamicResource.ClaimName]; ok {
|
||||
rInfo.DeviceIDs = append(rInfo.DeviceIDs, deviceIDs...)
|
||||
} else {
|
||||
resourceMap[dynamicResource.ClaimName] = &types.ResourceInfo{DeviceIDs: deviceIDs}
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
func hasKubeletAPIEndpoint(url *url.URL) bool {
|
||||
// Check for kubelet resource API socket file
|
||||
if _, err := os.Stat(url.Path); err != nil {
|
||||
|
||||
@@ -46,8 +46,8 @@ var (
|
||||
)
|
||||
|
||||
type fakeResourceServer struct {
|
||||
server *grpc.Server
|
||||
podresourcesapi.UnimplementedPodResourcesListerServer
|
||||
server *grpc.Server
|
||||
}
|
||||
|
||||
// TODO: This is stub code for test, but we may need to change for the testing we use this API in the future...
|
||||
@@ -68,32 +68,6 @@ func (m *fakeResourceServer) List(_ context.Context, _ *podresourcesapi.ListPodR
|
||||
},
|
||||
}
|
||||
|
||||
cdiDevices := []*podresourcesapi.CDIDevice{
|
||||
{
|
||||
Name: "cdi-kind=cdi-resource",
|
||||
},
|
||||
}
|
||||
draDriverName := "dra.example.com"
|
||||
poolName := "worker-1-pool"
|
||||
deviceName := "gpu-1"
|
||||
|
||||
claimsResource := []*podresourcesapi.ClaimResource{
|
||||
{
|
||||
CdiDevices: cdiDevices,
|
||||
DriverName: draDriverName,
|
||||
PoolName: poolName,
|
||||
DeviceName: deviceName,
|
||||
},
|
||||
}
|
||||
|
||||
dynamicResources := []*podresourcesapi.DynamicResource{
|
||||
{
|
||||
ClaimName: "resource-claim",
|
||||
ClaimNamespace: "dynamic-resource-pod-namespace",
|
||||
ClaimResources: claimsResource,
|
||||
},
|
||||
}
|
||||
|
||||
resp := &podresourcesapi.ListPodResourcesResponse{
|
||||
PodResources: []*podresourcesapi.PodResources{
|
||||
{
|
||||
@@ -106,16 +80,6 @@ func (m *fakeResourceServer) List(_ context.Context, _ *podresourcesapi.ListPodR
|
||||
},
|
||||
},
|
||||
},
|
||||
{
|
||||
Name: "dynamic-resource-pod-name",
|
||||
Namespace: "dynamic-resource-pod-namespace",
|
||||
Containers: []*podresourcesapi.ContainerResources{
|
||||
{
|
||||
Name: "dynamic-resource-container-name",
|
||||
DynamicResources: dynamicResources,
|
||||
},
|
||||
},
|
||||
},
|
||||
},
|
||||
}
|
||||
return resp, nil
|
||||
@@ -249,34 +213,6 @@ var _ = Describe("Kubelet resource endpoint data read operations", func() {
|
||||
Expect(resourceMap).To(Equal(outputRMap))
|
||||
})
|
||||
|
||||
It("should return no error with dynamic resource", func() {
|
||||
podUID := k8sTypes.UID("9f94e27b-4233-43d6-bd10-f73b4de6f456")
|
||||
fakePod := &v1.Pod{
|
||||
ObjectMeta: metav1.ObjectMeta{
|
||||
Name: "dynamic-resource-pod-name",
|
||||
Namespace: "dynamic-resource-pod-namespace",
|
||||
UID: podUID,
|
||||
},
|
||||
Spec: v1.PodSpec{
|
||||
Containers: []v1.Container{
|
||||
{
|
||||
Name: "dynamic-resource-container-name",
|
||||
},
|
||||
},
|
||||
},
|
||||
}
|
||||
client, err := getKubeletClient(testKubeletSocket)
|
||||
Expect(err).NotTo(HaveOccurred())
|
||||
|
||||
outputRMap := map[string]*mtypes.ResourceInfo{
|
||||
"resource-claim": {DeviceIDs: []string{"cdi-resource"}},
|
||||
}
|
||||
resourceMap, err := client.GetPodResourceMap(fakePod)
|
||||
Expect(err).NotTo(HaveOccurred())
|
||||
Expect(resourceMap).ShouldNot(BeNil())
|
||||
Expect(resourceMap).To(Equal(outputRMap))
|
||||
})
|
||||
|
||||
It("should return an error with garbage socket value", func() {
|
||||
u, err := url.Parse("/badfilepath!?//")
|
||||
Expect(err).NotTo(HaveOccurred())
|
||||
|
||||
@@ -138,6 +138,11 @@ func Errorf(format string, a ...interface{}) error {
|
||||
return fmt.Errorf(format, a...)
|
||||
}
|
||||
|
||||
// Warningf prints a warning at the same visibility threshold as Errorf (prefix "warning:").
|
||||
func Warningf(format string, a ...interface{}) {
|
||||
printf(ErrorLevel, "warning: "+format, a...)
|
||||
}
|
||||
|
||||
// Panicf prints logging plus stack trace. This should be used only for unrecoverable error
|
||||
func Panicf(format string, a ...interface{}) {
|
||||
printf(PanicLevel, format, a...)
|
||||
|
||||
@@ -622,7 +622,7 @@ func delPlugins(exec invoke.Exec, pod *v1.Pod, args *skel.CmdArgs, k8sArgs *type
|
||||
|
||||
// Check if we had any errors, and send them all back.
|
||||
if len(errorstrings) > 0 {
|
||||
return fmt.Errorf("%s", strings.Join(errorstrings, " / "))
|
||||
return fmt.Errorf("errors: %s", strings.Join(errorstrings, " / "))
|
||||
}
|
||||
|
||||
return nil
|
||||
|
||||
@@ -51,7 +51,6 @@ import (
|
||||
metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
|
||||
"k8s.io/apimachinery/pkg/fields"
|
||||
utilruntime "k8s.io/apimachinery/pkg/util/runtime"
|
||||
"k8s.io/apimachinery/pkg/util/sets"
|
||||
utilwait "k8s.io/apimachinery/pkg/util/wait"
|
||||
informerfactory "k8s.io/client-go/informers"
|
||||
v1coreinformers "k8s.io/client-go/informers/core/v1"
|
||||
@@ -479,9 +478,16 @@ func (s *Server) handleDelegateRequest(r *http.Request) ([]byte, error) {
|
||||
}
|
||||
|
||||
func overrideCNIConfigWithServerConfig(cniConf []byte, overrideConf []byte, ignoreReadinessIndicator bool) ([]byte, error) {
|
||||
if len(overrideConf) == 0 {
|
||||
// If there is no server-side override config AND we don't need to strip any keys,
|
||||
// return the client config unchanged.
|
||||
if len(overrideConf) == 0 && !ignoreReadinessIndicator {
|
||||
return cniConf, nil
|
||||
}
|
||||
// Treat a missing server config as an empty object so the key-stripping logic below
|
||||
// still runs when ignoreReadinessIndicator is true.
|
||||
if len(overrideConf) == 0 {
|
||||
overrideConf = []byte("{}")
|
||||
}
|
||||
|
||||
var cni map[string]interface{}
|
||||
if err := json.Unmarshal(cniConf, &cni); err != nil {
|
||||
@@ -493,14 +499,13 @@ func overrideCNIConfigWithServerConfig(cniConf []byte, overrideConf []byte, igno
|
||||
return nil, fmt.Errorf("failed to unmarshall CNI override config: %w", err)
|
||||
}
|
||||
|
||||
// Copy each key of the override config into the CNI config except for
|
||||
// a few specific keys
|
||||
ignoreKeys := sets.NewString()
|
||||
// Remove keys from the client config that the server wants to ignore, then
|
||||
// overlay the server-side overrides (also skipping those same keys).
|
||||
if ignoreReadinessIndicator {
|
||||
ignoreKeys.Insert("readinessindicatorfile")
|
||||
delete(cni, "readinessindicatorfile")
|
||||
}
|
||||
for overrideKey, overrideVal := range override {
|
||||
if !ignoreKeys.Has(overrideKey) {
|
||||
if !ignoreReadinessIndicator || overrideKey != "readinessindicatorfile" {
|
||||
cni[overrideKey] = overrideVal
|
||||
}
|
||||
}
|
||||
|
||||
Reference in New Issue
Block a user