github/multus-cni
1
0
Fork 1
mirror of https://github.com/k8snetworkplumbingwg/multus-cni.git synced 2025-09-17 15:07:14 +00:00
Code Issues Projects Releases Wiki Activity

Updating the Readme file to reflect K8s NPWG CRD changes (#80)

Browse Source 
update readme.md - tpr deprecated in k8s 1.7
This commit is contained in:
Kuralamudhan Ramakrishnan
2018-05-17 21:57:10 +01:00
committed by GitHub
parent 8dda8a5984
commit 9dbd585d04
2 changed files with 269 additions and 236 deletions
Download Patch File Download Diff File Expand all files Collapse all files

459
README.md
View File

@@ -1,34 +1,49 @@
# Attention !! this is WIP for Network Plumbing WG POC & Exp works, not part of Multus Implementation ![multus-cni Logo](https://github.com/intel/multus-cni/blob/master/doc/images/Multus.png)
* [Network Plumbing Working Group Agenda](https://docs.google.com/document/d/1oE93V3SgOGWJ4O1zeD1UmpeToa0ZiiO6LqRAmZBPFWM/edit) * [MULTUS CNI plugin](#multus-cni-plugin)
* [Dan's CRD proposal](https://docs.google.com/document/d/1Ny03h6IDVy_e_vmElOqR7UdTPAG_RNydhVE1Kx54kFQ/edit)
Table of Contents
=================
* [Multi network CNI plugin](#multi-network-cni-plugin)
* [Multi-Homed pod](#multi-homed-pod) * [Multi-Homed pod](#multi-homed-pod)
* [Build](#build) * [Build](#build)
* [Work flow](#work-flow) * [Work flow](#work-flow)
* [Usage with Kubernetes CRD/TPR based Network Objects](#usage-with-kubernetes-crdtpr-based-network-objects) * [Usage with Kubernetes CRD based network objects](#usage-with-kubernetes-crd-based-network-objects)
* [Creating “Network” third party resource in kubernetes](#creating-network-third-party-resource-in-kubernetes) * [Creating "Network" resources in Kubernetes](#creating-network-resources-in-kubernetes)
* [CRD based Network objects](#crd-based-network-objects) * [<strong>CRD based Network objects</strong>](#crd-based-network-objects)
* [TPR based Network objects](#tpr-based-network-objects) * [Creating network resources in Kubernetes](#creating-network-resources-in-kubernetes-1)
* [Creating “Custom Network objects” third party resource in kubernetes](#creating-custom-network-objects-third-party-resource-in-kubernetes)
* [Configuring Multus to use the kubeconfig](#configuring-multus-to-use-the-kubeconfig) * [Configuring Multus to use the kubeconfig](#configuring-multus-to-use-the-kubeconfig)
* [Configuring Multus to use the kubeconfig and also default networks](#configuring-multus-to-use-the-kubeconfig-and-also-default-networks) * [Configuring Multus to use kubeconfig and a default network](#configuring-multus-to-use-kubeconfig-and-a-default-network)
* [Configuring Pod to use the TPR Network objects](#configuring-pod-to-use-the-tpr-network-objects) * [Configuring Pod to use the CRD network objects](#configuring-pod-to-use-the-crd-network-objects)
* [Verifying Pod network](#verifying-pod-network) * [Verifying Pod network interfaces](#verifying-pod-network-interfaces)
* [Using Multus Conf file](#using-multus-conf-file) * [Using with Multus conf file](#using-with-multus-conf-file)
* [Testing the Multus CNI](#testing-the-multus-cni) * [Testing Multus CNI](#testing-multus-cni)
* [Multiple Flannel Network](#multiple-flannel-network) * [Multiple flannel networks](#multiple-flannel-networks)
* [docker](#docker) * [Configure Kubernetes with CNI](#configure-kubernetes-with-cni)
* [Kubernetes](#kubernetes)
* [Launching workloads in Kubernetes](#launching-workloads-in-kubernetes) * [Launching workloads in Kubernetes](#launching-workloads-in-kubernetes)
* [Multus additional plugins](#multus-additional-plugins)
* [NFV based networking in Kubernetes](#nfv-based-networking-in-kubernetes)
* [Need help](#need-help)
* [Contacts](#contacts) * [Contacts](#contacts)
# MULTUS CNI plugin
- _Multus_ is a latin word for &quot;Multi&quot;
- As the name suggests, it acts as a Multi plugin in Kubernetes and provides the multiple network interface support in a pod
- Multus supports all [reference plugins](https://github.com/containernetworking/plugins) (eg. [Flannel](https://github.com/containernetworking/plugins/tree/master/plugins/meta/flannel), [DHCP](https://github.com/containernetworking/plugins/tree/master/plugins/ipam/dhcp), [Macvlan](https://github.com/containernetworking/plugins/tree/master/plugins/main/macvlan)) that implement the CNI specification and all 3rd party plugins (eg. [Calico](https://github.com/projectcalico/cni-plugin), [Weave](https://github.com/weaveworks/weave), [Cilium](https://github.com/cilium/cilium), [Contiv](https://github.com/contiv/netplugin)). In addition to it, Multus supports [SRIOV](https://github.com/hustcat/sriov-cni), [SRIOV-DPDK](https://github.com/Intel-Corp/sriov-cni), [OVS-DPDK &amp; VPP](https://github.com/intel/vhost-user-net-plugin) workloads in Kubernetes with both cloud native and NFV based applications in Kubernetes
- It is a contact between the container runtime and other plugins, and it doesn&#39;t have any of its own net configuration, it calls other plugins like flannel/calico to do the real net conf job.
- Multus reuses the concept of invoking delegates as used in flannel by grouping multiple plugins into delegates and invoking them in the sequential order of the CNI configuration file provided in json format
- The default network gets "eth0" and additional network Pod interface name as “net0”, “net1”,… “netX and so on. Multus also support interface names from the user.
- Multus is one of the projects in the [Baremetal Container Experience kit](https://networkbuilders.intel.com/network-technologies/container-experience-kits).
# Multi network CNI plugin Please check the [CNI](https://github.com/containernetworking/cni) documentation for more information on container networking.
# Kubernetes Network Custom Resource Definition De-facto Standard - Reference implementation
* This project is a reference implementation for Kubernetes Network Custom Resource Definition De-facto Standard. For more information refer [Network Plumbing Working Group Agenda](https://docs.google.com/document/d/1oE93V3SgOGWJ4O1zeD1UmpeToa0ZiiO6LqRAmZBPFWM/edit)
* Kubernetes Network Custom Resource Definition De-facto Standard [documentation link](https://docs.google.com/document/d/1Ny03h6IDVy_e_vmElOqR7UdTPAG_RNydhVE1Kx54kFQ/edit)
* Reference implementation support following modes
* CNI config JSON in network object
* Not using CNI config (“thick” plugin usecase)
* CNI configuration stored in on-disk file
> refer the section 3.2 Network Object Definition for more details in Kubernetes Network Custom Resource Definition De-facto Standard
* Refer the reference implemenation presentation and demo details - [link](https://docs.google.com/presentation/d/1dbCin6MnhK-BjjcVun5YiPTL99VA2uSiyWAtWAPNlIc/edit?usp=sharing)
* Release version from v2.0 is not compatible with v1.1 and v1.2 network CRD specifications.
## Multi-Homed pod ## Multi-Homed pod
<p align="center"> <p align="center">
@@ -39,8 +54,7 @@ Table of Contents
**This plugin requires Go 1.8 to build.** **This plugin requires Go 1.8 to build.**
Go 1.5 users will need to set `GO15VENDOREXPERIMENT=1` to get vendored dependencies. This flag is set by default in 1.6. Go 1.5 users will need to set GO15VENDOREXPERIMENT=1 to get vendored dependencies. This flag is set by default in 1.6.
``` ```
#./build #./build
``` ```
@@ -50,40 +64,36 @@ Go 1.5 users will need to set `GO15VENDOREXPERIMENT=1` to get vendored dependenc
</p> </p>
## Network configuration reference ## Network configuration reference
* `name` (string, required): the name of the network - name (string, required): the name of the network
* `type` (string, required): "multus" - type (string, required): &quot;multus&quot;
* `kubeconfig` (string, optional): kubeconfig file for the out of cluster communication with kube-apiserver, Refer the doc - kubeconfig (string, optional): kubeconfig file for the out of cluster communication with kube-apiserver. See the example [kubeconfig](https://github.com/intel/multus-cni/blob/master/doc/node-kubeconfig.yaml)
* `delegates` (([]map,required): number of delegate details in the Multus, ignored in case kubeconfig is added. - delegates (([]map,required): number of delegate details in the Multus
## Usage with Kubernetes CRD/TPR based Network Objects ## Usage with Kubernetes CRD based network objects
Kubelet is responsible for establishing the network interfaces for each pod; it does this by invoking its configured CNI plugin. When Multus is invoked, it recovers pod annotations related to Multus, in turn, then it uses these annotations to recover a Kubernetes custom resource definition (CRD), which is an object that informs which plugins to invoke and the configuration needing to be passed to them. The order of plugin invocation is important as is the identity of the primary plugin. Kubelet is responsible for establishing network interfaces for pods; it does this by invoking its configured CNI plugin. When Multus is invoked it retrieves network references from Pod annotation. Multus then uses these network references to get network configurations. Network configurations are defined as Kubernetes Custom Resource Object (CRD). These configurations describe which CNI plugins to invoke and what their configurations are. The order of plugin invocation is important as it identifies the primary plugin. This order is taken from network object references given in a Pod spec.
Please refer the Kubernetes Network SIG - Multiple Network PoC proposal for more details refer the link - [K8s Multiple Network proposal](https://docs.google.com/document/d/1TW3P4c8auWwYy-w_5afIPDcGNLK3LZf0m14943eVfVg/edit)
Example configurations for CRD are also available in the [examples directory](examples/README.md).
<p align="center"> <p align="center">
<img src="doc/images/multus_crd_usage_diagram.JPG" width="1008" /> <img src="doc/images/multus_crd_usage_diagram.JPG" width="1008" />
</p> </p>
### Creating “Network” third party resource in kubernetes ### Creating &quot;Network&quot; resources in Kubernetes
Multus is compatible to work with both CRD/TPR. Both CRD/TPR based network object api self link is same. Multus is compatible to work with both CRD and TPR(deprecated in K8s 1.7).
#### CRD based Network objects ##### **CRD based Network objects**
1. Create a Third party resource “crdnetwork.yaml” for the network object as shown below 1. Create a Custom Resource Definition &quot;crdnetwork.yaml&quot; for the network object as shown below:
``` ```
apiVersion: apiextensions.k8s.io/v1beta1 apiVersion: apiextensions.k8s.io/v1beta1
kind: CustomResourceDefinition kind: CustomResourceDefinition
metadata: metadata:
# name must match the spec fields below, and be in the form: <plural>.<group> # name must match the spec fields below, and be in the form: <plural>.<group>
name: networks.kubernetes-network.cni.cncf.io name: networks.kubernetes.cni.cncf.io
spec: spec:
# group name to use for REST API: /apis/<group>/<version> # group name to use for REST API: /apis/<group>/<version>
group: kubernetes-network.cni.cncf.io group: kubernetes.cni.cncf.io
# version name to use for REST API: /apis/<group>/<version> # version name to use for REST API: /apis/<group>/<version>
version: v1 version: v1
# either Namespaced or Cluster # either Namespaced or Cluster
@@ -99,148 +109,110 @@ spec:
shortNames: shortNames:
- net - net
``` ```
2. Run kubectl create command for the Custom Resource Definition
2. kubectl create command for the Custom Resource Definition
``` ```
# kubectl create -f ./crdnetwork.yaml # kubectl create -f ./crdnetwork.yaml
customresourcedefinition "networks.kubernetes-network.cni.cncf.io" created customresourcedefinition "networks.kubernetes.cni.cncf.io" created
``` ```
3. Run kubectl get command to check the Network CRD creation
3. kubectl get command to check the Network CRD creation
``` ```
# kubectl get CustomResourceDefinition # kubectl get CustomResourceDefinition
NAME KIND NAME KIND
networks.kubernetes.com CustomResourceDefinition.v1beta1.apiextensions.k8s.io networks.kubernetes.cni.cncf.io CustomResourceDefinition.v1beta1.apiextensions.k8s.io
``` ```
For Kubernetes v1.7 and above use CRD to create network object. For version older than 1.7 use TPR based objects as shown below:
4. Save the below following YAML to flannel-network.yaml Note: Both TPR and CRD will have same selfLink :
*/apis/kubernetes.cni.cncf.io/v1/namespaces/default/networks/*
#### TPR based Network objects
1. Create a Third Party Resource &quot;tprnetwork.yaml&quot; for the network object as shown below:
``` ```
apiVersion: "kubernetes-network.cni.cncf.io/v1" apiVersion: extensions/v1beta1
kind: ThirdPartyResource
metadata:
name: network.kubernetes.cni.cncf.io
description: "A specification of a Network obj in the kubernetes"
versions:
- name: v1
```
2. Run kubectl create command for the Third Party Resource
```
# kubectl create -f ./tprnetwork.yaml
thirdpartyresource "network.kubernetes.cni.cncf.io" created
```
3. Run kubectl get command to check the Network TPR creation
```
# kubectl get thirdpartyresource
NAME DESCRIPTION VERSION(S)
network.kubernetes.cni.cncf.io A specification of a Network obj in the kubernetes v1
```
### Creating network resources in Kubernetes
1. After creating CRD network object you can create network resources in Kubernetes. These network resources may contain additional underlying CNI plugin parameters given in JSON format. In the following example shown below the args field contains parameters that will be passed into “flannel” plugin.
2. Save the following YAML to flannel-network.yaml
```
apiVersion: "kubernetes.cni.cncf.io/v1"
kind: Network kind: Network
metadata: metadata:
name: flannel-networkobj name: flannel-networkobj
plugin: flannel spec:
args: '[ config: '{
{ "cniVersion": "0.3.0",
"type": "flannel",
"delegate": { "delegate": {
"isDefaultGateway": true "isDefaultGateway": true
} }
} }'
]'
``` ```
5. create the custom resource definition
3. Create the custom resource definition
``` ```
# kubectl create -f customCRD/flannel-network.yaml # kubectl create -f customCRD/flannel-network.yaml
network "flannel-networkobj" created network "flannel-networkobj" created
``` ```
``` ```
# kubectl get network # kubectl get network
NAME KIND ARGS PLUGIN NAME AGE
flannel-networkobj Network.v1.kubernetes-network.cni.cncf.io [ { "delegate": { "isDefaultGateway": true } } ] flannel flannel-networkobj 26s
``` ```
6. Get the custom network object details 4. Get the custom network object details
``` ```
# kubectl get network flannel-networkobj -o yaml apiVersion: kubernetes.cni.cncf.io/v1
apiVersion: kubernetes.com/v1
args: '[ { "delegate": { "isDefaultGateway": true } } ]'
kind: Network kind: Network
metadata: metadata:
clusterName: "" clusterName: ""
creationTimestamp: 2017-07-11T21:46:52Z creationTimestamp: 2018-05-17T09:13:20Z
deletionGracePeriodSeconds: null deletionGracePeriodSeconds: null
deletionTimestamp: null deletionTimestamp: null
initializers: null
name: flannel-networkobj name: flannel-networkobj
namespace: default namespace: default
resourceVersion: "6848829" resourceVersion: "21176114"
selfLink: /apis/kubernetes-network.cni.cncf.io/v1/namespaces/default/networks/flannel-networkobj selfLink: /apis/kubernetes.cni.cncf.io/v1/namespaces/default/networks/flannel-networkobj
uid: 7311c965-6682-11e7-b0b9-408d5c537d27 uid: 8ac8f873-59b2-11e8-8308-a4bf01024e6f
plugin: flannel spec:
config: '{ "cniVersion": "0.3.0", "type": "flannel", "delegate": { "isDefaultGateway":
true } }'
``` ```
5. Save the following YAML to sriov-network.yaml to creating sriov network object. ( Refer to [Intel - SR-IOV CNI](https://github.com/Intel-Corp/sriov-cni) or contact @kural in [Intel-Corp Slack](https://intel-corp.herokuapp.com/) for running the DPDK based workloads in Kubernetes)
Both TPR and CRD will have same selfLink : **/apis/kubernetes-network.cni.cncf.io/v1/namespaces/default/networks/<netobjname>**
if you are using 1.7 or planning to use 1.8 kubernetes, you can use CRD itself. There is no need to change any thing in Multus. For Kubernetes user < 1.7 use TPR based network objects as follows
#### TPR based Network objects
1. Create a Third party resource “tprnetwork.yaml” for the network object as shown below
``` ```
apiVersion: extensions/v1beta1 apiVersion: "kubernetes.cni.cncf.io/v1"
kind: ThirdPartyResource
metadata:
name: network.kubernetes-network.cni.cncf.io
description: "A specification of a Network obj in the kubernetes"
versions:
- name: v1
```
2. Run kubectl create command for the Third Party Resource
```
# kubectl create -f ./tprnetwork.yaml
thirdpartyresource "network.kubernetes-network.cni.cncf.io" created
```
3. Run kubectl get command to check the Network TPR creation
```
# kubectl get thirdpartyresource
NAME DESCRIPTION VERSION(S)
network.kubernetes-network.cni.cncf.io A specification of a Network obj in the kubernetes v1
```
##### Creating “Custom Network objects” third party resource in kubernetes
1. After the ThirdPartyResource object has been created you can create network objects. Network objects should contain network fields. These fields are in JSON format. In the following example, a plugin and args fields are set to the object of kind Network. The kind Network is derived from the metadata.name of the ThirdPartyResource object we created above.
2. Save the below following YAML to flannel-network.yaml
```
apiVersion: "kubernetes-network.cni.cncf.io/v1"
kind: Network
metadata:
name: flannel-conf
plugin: flannel
args: '[
{
"delegate": {
"isDefaultGateway": true
}
}
]'
```
2. Run kubectl create command for the TPR - Network object
```
# kubectl create -f ./flannel-network.yaml
network "flannel-conf" created
```
3. Manage the Network objects using kubectl.
```
# kubectl get network
NAME KIND
flannel-conf Network.v1.kubernetes-network.cni.cncf.io
```
4. You can also view the raw JSON data. Here you can see that it contains the custom plugin and args fields from the yaml you used to create it:
```
# kubectl get network flannel-conf -o yaml
apiVersion: kubernetes-network.cni.cncf.io/v1
args: '[ { "delegate": { "isDefaultGateway": true } } ]'
kind: Network
metadata:
creationTimestamp: 2017-06-28T14:20:52Z
name: flannel-conf
namespace: default
resourceVersion: "5422876"
selfLink: /apis/kubernetes-network.cni.cncf.io/v1/namespaces/default/networks/flannel-conf
uid: fdcb94a2-5c0c-11e7-bbeb-408d5c537d27
plugin: flannel
```
4. The plugin field should be the name of the CNI plugin and args should have the flannel args, it should be in the JSON format as shown above. **User can create network objects for Calico, Weave, Romana, & Cilium and test the multus.**
5. Save the below following YAML to sriov-network.yaml. Refer [Intel - SR-IOV CNI](https://github.com/Intel-Corp/sriov-cni) or contact @kural in [Intel-Corp Slack](https://intel-corp.herokuapp.com/) for running the DPDK based workloads in Kubernetes
```
apiVersion: "kubernetes-network.cni.cncf.io/v1"
kind: Network kind: Network
metadata: metadata:
name: sriov-conf name: sriov-conf
plugin: sriov spec:
args: '[ config: '{
{ "type": "sriov",
"if0": "enp12s0f1", "if0": "enp12s0f1",
"ipam": { "ipam": {
"type": "host-local", "type": "host-local",
@@ -252,36 +224,35 @@ args: '[
], ],
"gateway": "10.56.217.1" "gateway": "10.56.217.1"
} }
} }'
]'
``` ```
6. Save the below following YAML to sriov-vlanid-l2enable-network.yaml 6. Likewise save the following YAML to sriov-vlanid-l2enable-network.yaml to create another sriov based network object:
``` ```
apiVersion: "kubernetes-network.cni.cncf.io/v1" apiVersion: "kubernetes.cni.cncf.io/v1"
kind: Network kind: Network
metadata: metadata:
name: sriov-vlanid-l2enable-conf name: sriov-vlanid-l2enable-conf
plugin: sriov spec:
args: '[ config: '{
{ "type": "sriov",
"if0": "enp2s0", "if0": "enp2s0",
"vlan": 210, "vlan": 210,
"if0name": "north",
"l2enable": true "l2enable": true
} }'
]'
``` ```
7. Follows the step 2 to create the network object “sriov-vlanid-l2enable-conf” and “sriov-conf” 7. Follow step 3 above to create &quot;sriov-vlanid-l2enable-conf&quot; and &quot;sriov-conf&quot; network objects
8. Manage the Network objects using kubectl. 8. View network objects using kubectl
``` ```
# kubectl get network # kubectl get network
NAME KIND NAME AGE
flannel-conf Network.v1.kubernetes-network.cni.cncf.io flannel-networkobj 29m
sriov-vlanid-l2enable-conf Network.v1.kubernetes-network.cni.cncf.io sriov-conf 6m
sriov-conf Network.v1.kubernetes-network.cni.cncf.io sriov-vlanid-l2enable-conf 2m
``` ```
### Configuring Multus to use the kubeconfig ### Configuring Multus to use the kubeconfig
1. Create Multus CNI configuration file /etc/cni/net.d/multus-cni.conf with below content on the nodes. Use only the absolute path to point to the kubeconfig file (it may change depending upon your cluster env) and make sure all CNI binary files are in `\opt\cni\bin` dir 1. Create a Mutlus CNI configuration file on each Kubernetes node. This file should be created in: /etc/cni/net.d/multus-cni.conf with the content shown below. Use only the absolute path to point to the kubeconfig file (as it may change depending upon your cluster env). We are assuming all CNI plugin binaries are default location (`\opt\cni\bin dir`)
``` ```
{ {
"name": "node-cni-network", "name": "node-cni-network",
@@ -293,8 +264,10 @@ sriov-conf Network.v1.kubernetes-network.cni.cncf.io
``` ```
# systemctl restart kubelet # systemctl restart kubelet
``` ```
### Configuring Multus to use the kubeconfig and also default networks ### Configuring Multus to use kubeconfig and a default network
1. Many user want default networking feature along with Network object. Refer [#14](https://github.com/Intel-Corp/multus-cni/issues/14) & [#17](https://github.com/Intel-Corp/multus-cni/issues/17) issues for more information. In this following config, Weave act as the default network in the absence of network field in the pod metadata annotation.
1. Many users want Kubernetes default networking feature along with network objects. Refer to issues [#14](https://github.com/intel/multus-cni/issues/14) &amp; [#17](https://github.com/intel/multus-cni/issues/17) for more information. In the following Multus configuration, Weave act as the default network in the absence of network field in the pod metadata annotation.
``` ```
{ {
"name": "node-cni-network", "name": "node-cni-network",
@@ -302,18 +275,20 @@ sriov-conf Network.v1.kubernetes-network.cni.cncf.io
"kubeconfig": "/etc/kubernetes/node-kubeconfig.yaml", "kubeconfig": "/etc/kubernetes/node-kubeconfig.yaml",
"delegates": [{ "delegates": [{
"type": "weave-net", "type": "weave-net",
"hairpinMode": true "hairpinMode": true,
"masterplugin": true
}] }]
} }
``` ```
2. Restart kubelet service 2. Restart kubelet service
``` ```
# systemctl restart kubelet # systemctl restart kubelet
``` ```
### Configuring Pod to use the CRD network objects
### Configuring Pod to use the TPR Network objects 1. Save the following YAML to pod-multi-network.yaml. In this case flannel-conf network object acts as the primary network.
1. Save the below following YAML to pod-multi-network.yaml. In this case flannel-conf network object act as the primary network.
``` ```
# cat pod-multi-network.yaml # cat pod-multi-network.yaml
apiVersion: v1 apiVersion: v1
@@ -321,10 +296,11 @@ kind: Pod
metadata: metadata:
name: multus-multi-net-poc name: multus-multi-net-poc
annotations: annotations:
networks: '[ kubernetes.v1.cni.cncf.io/networks: '[
{ "name": "flannel-conf" }, { "name": "flannel-conf" },
{ "name": "sriov-conf"}, { "name": "sriov-conf" },
{ "name": "sriov-vlanid-l2enable-conf" } { "name": "sriov-vlanid-l2enable-conf",
"interfaceRequest": "north" }
]' ]'
spec: # specification of the pod's contents spec: # specification of the pod's contents
containers: containers:
@@ -334,29 +310,34 @@ spec: # specification of the pod's contents
stdin: true stdin: true
tty: true tty: true
``` ```
3. Create Multiple network based pod from the master node
2. Create Multiple network based pod from the master node
``` ```
# kubectl create -f ./pod-multi-network.yaml # kubectl create -f ./pod-multi-network.yaml
pod "multus-multi-net-poc" created pod "multus-multi-net-poc" created
``` ```
4. Get the details of the running pod from the master
3. Get the details of the running pod from the master
``` ```
# kubectl get pods # kubectl get pods
NAME READY STATUS RESTARTS AGE NAME READY STATUS RESTARTS AGE
multus-multi-net-poc 1/1 Running 0 30s multus-multi-net-poc 1/1 Running 0 30s
``` ```
### Verifying Pod network
1. Run “ifconfig” command inside the container: ### Verifying Pod network interfaces
1. Run &quot;ifconfig&quot; command in Pod:
``` ```
# kubectl exec -it multus-multi-net-poc -- ifconfig # kubectl exec -it multus-multi-net-poc -- ifconfig
eth0 Link encap:Ethernet HWaddr 06:21:91:2D:74:B9 eth0 Link encap:Ethernet HWaddr C6:43:7C:09:B4:9C
inet addr:192.168.42.3 Bcast:0.0.0.0 Mask:255.255.255.0 inet addr:10.128.0.4 Bcast:0.0.0.0 Mask:255.255.255.0
inet6 addr: fe80::421:91ff:fe2d:74b9/64 Scope:Link
UP BROADCAST RUNNING MULTICAST MTU:1450 Metric:1 UP BROADCAST RUNNING MULTICAST MTU:1450 Metric:1
RX packets:0 errors:0 dropped:0 overruns:0 frame:0 RX packets:8 errors:0 dropped:0 overruns:0 frame:0
TX packets:8 errors:0 dropped:0 overruns:0 carrier:0 TX packets:1 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:0 collisions:0 txqueuelen:0
RX bytes:0 (0.0 B) TX bytes:648 (648.0 B) RX bytes:648 (648.0 B) TX bytes:42 (42.0 B)
lo Link encap:Local Loopback lo Link encap:Local Loopback
inet addr:127.0.0.1 Mask:255.0.0.0 inet addr:127.0.0.1 Mask:255.0.0.0
@@ -367,7 +348,16 @@ lo Link encap:Local Loopback
collisions:0 txqueuelen:1 collisions:0 txqueuelen:1
RX bytes:0 (0.0 B) TX bytes:0 (0.0 B) RX bytes:0 (0.0 B) TX bytes:0 (0.0 B)
net0 Link encap:Ethernet HWaddr D2:94:98:82:00:00 net0 Link encap:Ethernet HWaddr 06:21:91:2D:74:B9
inet addr:192.168.42.3 Bcast:0.0.0.0 Mask:255.255.255.0
inet6 addr: fe80::421:91ff:fe2d:74b9/64 Scope:Link
UP BROADCAST RUNNING MULTICAST MTU:1450 Metric:1
RX packets:0 errors:0 dropped:0 overruns:0 frame:0
TX packets:8 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:0
RX bytes:0 (0.0 B) TX bytes:648 (648.0 B)
net1 Link encap:Ethernet HWaddr D2:94:98:82:00:00
inet addr:10.56.217.171 Bcast:0.0.0.0 Mask:255.255.255.0 inet addr:10.56.217.171 Bcast:0.0.0.0 Mask:255.255.255.0
inet6 addr: fe80::d094:98ff:fe82:0/64 Scope:Link inet6 addr: fe80::d094:98ff:fe82:0/64 Scope:Link
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1 UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
@@ -384,13 +374,16 @@ north Link encap:Ethernet HWaddr BE:F2:48:42:83:12
collisions:0 txqueuelen:1000 collisions:0 txqueuelen:1000
RX bytes:95956 (93.7 KiB) TX bytes:82200 (80.2 KiB) RX bytes:95956 (93.7 KiB) TX bytes:82200 (80.2 KiB)
``` ```
Interface name | Description | Interface name | Description |
------------ | ------------- | --- | --- |
lo | loopback | lo | loopback |
eth0@if41 | Flannel network tap interface | eth0 | weave network interface |
net0 | VF0 of NIC 1 assigned to the container by [Intel - SR-IOV CNI](https://github.com/Intel-Corp/sriov-cni) plugin | net0 | Flannel network tap interface |
north | VF0 of NIC 2 assigned with VLAN ID 210 to the container by SR-IOV CNI plugin | net1 | VF0 of NIC 1 assigned to the container by [Intel - SR-IOV CNI](https://github.com/intel/sriov-cni) plugin |
| north | VF0 of NIC 2 assigned with VLAN ID 210 to the container by SR-IOV CNI plugin |
2. Check the vlan ID of the NIC 2 VFs 2. Check the vlan ID of the NIC 2 VFs
``` ```
# ip link show enp2s0 # ip link show enp2s0
20: enp2s0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP mode DEFAULT group default qlen 1000 20: enp2s0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP mode DEFAULT group default qlen 1000
@@ -401,7 +394,7 @@ north | VF0 of NIC 2 assigned with VLAN ID 210 to the container by SR-IOV CNI p
vf 3 MAC 00:00:00:00:00:00, vlan 4095, spoof checking off, link-state auto vf 3 MAC 00:00:00:00:00:00, vlan 4095, spoof checking off, link-state auto
``` ```
## Using Multus Conf file ## Using with Multus conf file
Given the following network configuration: Given the following network configuration:
@@ -441,6 +434,7 @@ Given the following network configuration:
}, },
{ {
"type": "flannel", "type": "flannel",
"masterplugin": true,
"delegate": { "delegate": {
"isDefaultGateway": true "isDefaultGateway": true
} }
@@ -448,39 +442,42 @@ Given the following network configuration:
] ]
} }
EOF EOF
``` ```
## Testing the Multus CNI ## ## Testing Multus CNI
### Multiple Flannel Network
Github user [YYGCui](https://github.com/YYGCui) has used Multiple flannel network to work with Multus CNI plugin. Please refer this [closed issue](https://github.com/Intel-Corp/multus-cni/issues/7) for Multiple overlay network support with Multus CNI.
### docker ### Multiple flannel networks
Make sure that the multus, [sriov](https://github.com/Intel-Corp/sriov-cni), [flannel](https://github.com/containernetworking/cni/blob/master/Documentation/flannel.md), and [ptp](https://github.com/containernetworking/cni/blob/master/Documentation/ptp.md) binaries are in the `/opt/cni/bin` directories and follow the steps as mention in the [CNI](https://github.com/containernetworking/cni/#running-a-docker-container-with-network-namespace-set-up-by-cni-plugins)
### Kubernetes Github user [YYGCui](https://github.com/YYGCui) has used multiple flannel network to work with Multus CNI plugin. Please refer to this [closed issue](https://github.com/intel/multus-cni/issues/7) for ,multiple overlay network support with Multus CNI.
Refer the Kubernetes User Guide and network plugin
* [Single Node](https://kubernetes.io/docs/getting-started-guides/fedora/fedora_manual_config/) Make sure that the multus, [sriov](https://github.com/Intel-Corp/sriov-cni), [flannel](https://github.com/containernetworking/cni/blob/master/Documentation/flannel.md), and [ptp](https://github.com/containernetworking/cni/blob/master/Documentation/ptp.md) binaries are in the /opt/cni/bin directories and follow the steps as mentioned in the [CNI](https://github.com/containernetworking/cni/#running-a-docker-container-with-network-namespace-set-up-by-cni-plugins)
* [Multi Node](https://kubernetes.io/docs/getting-started-guides/fedora/flannel_multi_node_cluster/)
* [Network Plugin](https://kubernetes.io/docs/admin/network-plugins/) #### Configure Kubernetes with CNI
Kubelet must be configured to run with the CNI network plugin. Edit /etc/kubernetes/kubelet file and add "--network-plugin=cni" flags in KUBELET\_OPTS as shown below:
Kubelet must be configured to run with the CNI `--network-plugin`, with the following configuration information.
Edit `/etc/default/kubelet` file and add `KUBELET_OPTS`:
``` ```
KUBELET_OPTS="... KUBELET_OPTS="...
--network-plugin-dir=/etc/cni/net.d --network-plugin-dir=/etc/cni/net.d
--network-plugin=cni --network-plugin=cni
" "
``` ```
Restart the kubelet Refer to the Kubernetes User Guide and network plugin for more information.
- [Single Node](https://kubernetes.io/docs/getting-started-guides/fedora/fedora_manual_config/)
- [Multi Node](https://kubernetes.io/docs/getting-started-guides/fedora/flannel_multi_node_cluster/)
- [Network plugin](https://kubernetes.io/docs/admin/network-plugins/)
Restart kubelet:
``` ```
# systemctl restart kubelet.service # systemctl restart kubelet.service
``` ```
#### Launching workloads in Kubernetes #### Launching workloads in Kubernetes
Launch the workload using yaml file in the kubernetes master, with above configuration in the multus CNI, each pod should have multiple interfaces.
> Note: To verify whether Multus CNI plugin is working fine create a pod containing one “busybox” container and execute “ip link” command to check if interfaces management follows configuration.
1. Create “multus-test.yaml” file containing below configuration. Created pod will consist of one “busybox” container running “top” command. With Multus CNI configured as described in sections above each workload launched via a Kubernetes Pod will have multiple network interfacesLaunch the workload using yaml file in the kubernetes master, with above configuration in the multus CNI, each pod should have multiple interfaces.
Note: To verify whether Multus CNI plugin is working correctly, create a pod containing one &quot;busybox&quot; container and execute &quot;ip link&quot; command to check if interfaces management follows configuration.
1. Create &quot;multus-test.yaml&quot; file containing below configuration. Created pod will consist of one &quot;busybox&quot; container running &quot;top&quot; command.
``` ```
apiVersion: v1 apiVersion: v1
kind: Pod kind: Pod
@@ -497,11 +494,13 @@ spec: # specification of the pod's contents
``` ```
2. Create pod using command: 2. Create pod using command:
``` ```
# kubectl create -f multus-test.yaml # kubectl create -f multus-test.yaml
pod "multus-test" created pod "multus-test" created
``` ```
3. Run “ip link” command inside the container: 3. Run &quot;ip link&quot; command inside the container:
``` ```
# 1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue qlen 1 # 1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue qlen 1
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
@@ -513,13 +512,47 @@ pod "multus-test" created
link/ether 76:13:b1:60:00:00 brd ff:ff:ff:ff:ff:ff link/ether 76:13:b1:60:00:00 brd ff:ff:ff:ff:ff:ff
``` ```
Interface name | Description | Interface name | Description |
------------ | ------------- | --- | --- |
lo | loopback | lo | loopback |
eth0@if41 | Flannel network tap interface | eth0@if41 | Flannel network tap interface |
net0 | VF assigned to the container by [SR-IOV CNI](https://github.com/Intel-Corp/sriov-cni) plugin | net0 | VF assigned to the container by [SR-IOV CNI](https://github.com/Intel-Corp/sriov-cni) plugin |
net1 | ptp localhost interface | net1 | ptp localhost interface |
## Multus additional plugins
- [DPDK-SRIOV CNI](https://github.com/Intel-Corp/sriov-cni)
- [Vhostuser CNI](https://github.com/intel/vhost-user-net-plugin) - a Dataplane network plugin - Supports OVS-DPDK &amp; VPP
- [Bond CNI](https://github.com/Intel-Corp/bond-cni) - For fail-over and high availability of networking
## NFV based networking in Kubernetes
- KubeCon workshop on [&quot;Enabling NFV features in Kubernetes&quot;](https://kccncna17.sched.com/event/Cvnw/enabling-nfv-features-in-kubernetes-hosted-by-kuralamudhan-ramakrishnan-ivan-coughlan-intel) presentation [slide deck](https://www.slideshare.net/KuralamudhanRamakris/enabling-nfv-features-in-kubernetes-83923352)
- Feature brief
- [Multiple Network Interface Support in Kubernetes](https://builders.intel.com/docs/networkbuilders/multiple-network-interfaces-support-in-kubernetes-feature-brief.pdf)
- [Enhanced Platform Awareness in Kubernetes](https://builders.intel.com/docs/networkbuilders/enhanced-platform-awareness-feature-brief.pdf)
- Application note
- [Multiple Network Interfaces in Kubernetes and Container Bare Metal](https://builders.intel.com/docs/networkbuilders/multiple-network-interfaces-in-kubernetes-application-note.pdf)
- [Enhanced Platform Awareness Features in Kubernetes](https://builders.intel.com/docs/networkbuilders/enhanced-platform-awareness-in-kubernetes-application-note.pdf)
- White paper
- [Enabling New Features with Kubernetes for NFV](https://builders.intel.com/docs/networkbuilders/enabling_new_features_in_kubernetes_for_NFV.pdf)
- Multus&#39;s related project github pages
- [Multus](https://github.com/Intel-Corp/multus-cni)
- [SRIOV - DPDK CNI](https://github.com/Intel-Corp/sriov-cni)
- [Vhostuser - VPP &amp; OVS - DPDK CNI](https://github.com/intel/vhost-user-net-plugin)
- [Bond CNI](https://github.com/Intel-Corp/bond-cni)
- [Node Feature Discovery](https://github.com/kubernetes-incubator/node-feature-discovery)
- [CPU Manager for Kubernetes](https://github.com/Intel-Corp/CPU-Manager-for-Kubernetes)
## Need help
- Read [Containers Experience Kits](https://networkbuilders.intel.com/network-technologies/container-experience-kits)
- Try our container exp kit demo - KubeCon workshop on [Enabling NFV Features in Kubernetes](https://github.com/intel/container-experience-kits-demo-area/)
- Join us on [#intel-sddsg-slack](https://intel-corp.herokuapp.com/) slack channel and ask question in [#general-discussion](https://intel-corp-team.slack.com/messages/C4C5RSEER)
- You can also [email](mailto:kuralamudhan.ramakrishnan@intel.com) us
- Feel free to [submit](https://github.com/Intel-Corp/multus-cni/issues/new) an issue
Please fill in the Questions/feedback - [google-form](https://goo.gl/forms/upBWyGs8Wmq69IEi2)!
## Contacts ## Contacts
For any questions about Multus CNI, please reach out on github issue or feel free to contact the developer @kural in our [Intel-Corp Slack](https://intel-corp.herokuapp.com/) For any questions about Multus CNI, please reach out on github issue or feel free to contact the developer @kural in our [Intel-Corp Slack](https://intel-corp.herokuapp.com/)

BIN
doc/images/multus_crd_usage_diagram.JPG
View File

Binary file not shown.
Side by Side Swipe Overlay

Before

Width:  |  Height:  |  Size: 183 KiB

After

Width:  |  Height:  |  Size: 197 KiB