Fix grammar in README

Update README.md
Change-Id: I7b48d923def4998106f2d537a5c1f4412179f372
2025-09-02 17:35:49 +00:00 · 2018-04-28 16:34:18 +01:00 · 2018-03-21 13:47:07 +00:00 · 2018-02-28 11:58:58 +00:00 · 2017-12-20 11:08:54 +00:00 · 2017-12-20 10:03:14 +00:00
4 changed files with 396 additions and 107 deletions
--- a/CONTRIBUTING.md
+++ b/CONTRIBUTING.md
@@ -0,0 +1,57 @@
+# How to Contribute
+
+Multus CNI is [Apache 2.0 licensed](LICENSE) and accepts contributions via GitHub
+pull requests. This document outlines some of the conventions on development
+workflow, commit message formatting, contact points and other resources to make
+it easier to get your contribution accepted.
+
+## Coding Style
+
+Please follows the standard formatting recommendations and language idioms set out
+in [Effective Go](https://golang.org/doc/effective_go.html) and in the
+[Go Code Review Comments wiki](https://github.com/golang/go/wiki/CodeReviewComments).
+
+## Certificate of Origin
+
+In order to get a clear contribution chain of trust we use the [signed-off-by language](https://01.org/community/signed-process)
+used by the Linux kernel project.
+
+## Format of the patch
+
+Beside the signed-off-by footer, we expect each patch to comply with the following format:
+
+```
+Change summary
+
+More detailed explanation of your changes: Why and how.
+Wrap it to 72 characters.
+See [here] (http://chris.beams.io/posts/git-commit/)
+for some more good advices.
+
+Fixes #NUMBER (or URL to the issue)
+
+Signed-off-by: <contributor@foo.com>
+```
+
+For example:
+
+```
+Fix poorly named identifiers
+  
+One identifier, fnname, in func.go was poorly named.  It has been renamed
+to fnName.  Another identifier retval was not needed and has been removed
+entirely.
+
+Fixes #1
+    
+Signed-off-by: Abc Xyz <abc.xyz@intel.com>
+```
+
+## Pull requests
+
+We accept github pull requests.
+
+## Email and Chat
+
+The project uses the Slack chat:
+- Slack: #[Intel-Corp](https://intel-corp.herokuapp.com/) channel on slack
--- a/README.md
+++ b/README.md
@@ -1,23 +1,42 @@
 ![multus-cni Logo](https://github.com/Intel-Corp/multus-cni/blob/master/doc/images/Multus.png)
+
+   * [MULTUS CNI plugin](#multus-cni-plugin)
+      * [Multi-Homed pod](#multi-homed-pod)
+      * [Build](#build)
+      * [Work flow](#work-flow)
+      * [Usage with Kubernetes CRD/TPR based network objects](#usage-with-kubernetes-crdtpr-based-network-objects)
+         * [Creating "Network" resources in Kubernetes](#creating-network-resources-in-kubernetes)
+               * [<strong>CRD based Network objects</strong>](#crd-based-network-objects)
+            * [TPR based Network objects](#tpr-based-network-objects)
+         * [Creating network resources in Kubernetes](#creating-network-resources-in-kubernetes-1)
+         * [Configuring Multus to use the kubeconfig](#configuring-multus-to-use-the-kubeconfig)
+         * [Configuring Multus to use kubeconfig and a default network](#configuring-multus-to-use-kubeconfig-and-a-default-network)
+         * [Configuring Pod to use the CRD/TPR network objects](#configuring-pod-to-use-the-crdtpr-network-objects)
+         * [Verifying Pod network interfaces](#verifying-pod-network-interfaces)
+      * [Using with Multus conf file](#using-with-multus-conf-file)
+      * [Testing Multus CNI](#testing-multus-cni)
+         * [Multiple flannel networks](#multiple-flannel-networks)
+            * [Configure Kubernetes with CNI](#configure-kubernetes-with-cni)
+            * [Launching workloads in Kubernetes](#launching-workloads-in-kubernetes)
+      * [Multus additional plugins](#multus-additional-plugins)
+      * [NFV based networking in Kubernetes](#nfv-based-networking-in-kubernetes)
+      * [Need help](#need-help)
+      * [Contacts](#contacts)
+
 # MULTUS CNI plugin
+- _Multus_ is a latin word for &quot;Multi&quot;
+- As the name suggests, it acts as a Multi plugin in Kubernetes and provides the multiple network interface support in a pod
+- Multus supports all [reference plugins](https://github.com/containernetworking/plugins) (eg. [Flannel](https://github.com/containernetworking/plugins/tree/master/plugins/meta/flannel), [DHCP](https://github.com/containernetworking/plugins/tree/master/plugins/ipam/dhcp), [Macvlan](https://github.com/containernetworking/plugins/tree/master/plugins/main/macvlan)) that implement the CNI specification and all 3rd party plugins (eg. [Calico](https://github.com/projectcalico/cni-plugin), [Weave](https://github.com/weaveworks/weave), [Cilium](https://github.com/cilium/cilium), [Contiv](https://github.com/contiv/netplugin)). In addition to it, Multus supports [SRIOV](https://github.com/hustcat/sriov-cni), [SRIOV-DPDK](https://github.com/Intel-Corp/sriov-cni), [OVS-DPDK &amp; VPP](https://github.com/intel/vhost-user-net-plugin) workloads in Kubernetes with both cloud native and NFV based applications in Kubernetes
+- It is a contact between the container runtime and other plugins, and it doesn&#39;t have any of its own net configuration, it calls other plugins like flannel/calico to do the real net conf job.
+- Multus reuses the concept of invoking delegates as used in flannel by grouping multiple plugins into delegates and invoking them in the sequential order of the CNI configuration file provided in json format
+- Number of plugins supported is depends on the number of delegates in the configuration file.
+- The "masterplugin" is the only net conf option of multus cni, it identifies the primary network. The default route will point to the primary network
+- One of the plugin acts as a “Master” plugin and responsible for configuring k8s network with Pod interface “eth0”
+- The “Master” plugin also responsible to set the default route for the Pod
+- Any subsequent plugin gets Pod interface name as “net0”, “net1”,… “netX and so on
+- Multus is one of the projects in the [Baremetal Container Experience kit](https://networkbuilders.intel.com/network-technologies/container-experience-kits).

- *Multus* is the latin word for “Multi”
-
- As the name suggests, it acts as the Multi plugin in Kubernetes and provides the Multi interface support in a pod
-
- It is generic to run with any plugins like Calico, Weave, SRIOV, Ciliuim, Canal and Flannel, with different IPAM and networks. 
-
- It is a contact between the container runtime and other plugins, and it doesn't have any of its own net configuration, it calls other plugins like flannel/calico to do the real net conf job. 
-
- Multus reuses the concept of invoking the delegates in flannel, it groups the multi plugins into delegates and invoke each other in sequential order, according to the JSON scheme in the cni configuration.
-
- No. of plugins supported is dependent upon the number of delegates in the conf file.
-
- Master plugin invokes "eth0" interface in the pod, rest of plugins(Mininon plugins eg: sriov,ipam) invoke interfaces as "net0", "net1".. "netn"
-
- The "masterplugin" is the only net conf option of multus cni, it identifies the primary network. The default route will point to the primary network 
-
-Please read [CNI](https://github.com/containernetworking/cni) for more information on container networking.
+Please check the [CNI](https://github.com/containernetworking/cni) documentation for more information on container networking.

 ## Multi-Homed pod
 <p align="center">
@@ -26,10 +45,9 @@ Please read [CNI](https://github.com/containernetworking/cni) for more informati

 ## Build

-This plugin requires Go 1.7 to build.
-
-Go 1.5 users will need to set `GO15VENDOREXPERIMENT=1` to get vendored dependencies. This flag is set by default in 1.6.
+**This plugin requires Go 1.8 to build.**

+Go 1.5 users will need to set GO15VENDOREXPERIMENT=1 to get vendored dependencies. This flag is set by default in 1.6.
 ```
 #./build
 ```
@@ -39,18 +57,124 @@ Go 1.5 users will need to set `GO15VENDOREXPERIMENT=1` to get vendored dependenc
 </p>
 ## Network configuration reference

-* `name` (string, required): the name of the network
-* `type` (string, required): "multus"
-* `kubeconfig` (string, optional): kubeconfig file for the out of cluster communication with kube-apiserver, Refer the doc
-* `delegates` (([]map,required): number of delegate details in the Multus, ignored in case kubeconfig is added.
-* `masterplugin` (bool,required): master plugin to report back the IP address and DNS to the container
+- name (string, required): the name of the network
+- type (string, required): &quot;multus&quot;
+- kubeconfig (string, optional): kubeconfig file for the out of cluster communication with kube-apiserver. See the example [kubeconfig](https://github.com/Intel-Corp/multus-cni/blob/master/doc/node-kubeconfig.yaml)
+- delegates (([]map,required): number of delegate details in the Multus, ignored in case kubeconfig is added.
+- masterplugin (bool,required): master plugin to report back the IP address and DNS to the container

-## Usage with Kubernetes TPR based Network Objects
+## Usage with Kubernetes CRD/TPR based network objects

-Please refer the Kubernetes Network SIG - Multiple Network PoC proposal for more details refer the link - [K8s Multiple Network proposal](https://docs.google.com/document/d/1TW3P4c8auWwYy-w_5afIPDcGNLK3LZf0m14943eVfVg/edit)
+Kubelet is responsible for establishing network interfaces for pods; it does this by invoking its configured CNI plugin. When Multus is invoked it retrieves network references from Pod annotation. Multus then uses these network references to get network configurations. Network configurations are defined as Kubernetes Custom Resource Object (CRD). These configurations describe which CNI plugins to invoke and what their configurations are. The order of plugin invocation is important as it identifies the primary plugin. This order is taken from network object references given in a Pod spec.
+
+<p align="center">
+   <img src="doc/images/multus_crd_usage_diagram.JPG" width="1008" />
+</p>
+
+### Creating &quot;Network&quot; resources in Kubernetes
+
+Multus is compatible to work with both CRD/TPR. Both CRD/TPR based network object api self link is same.
+
+##### **CRD based Network objects**
+
+1. Create a Custom Resource Definition &quot;crdnetwork.yaml&quot; for the network object as shown below:
+
+```
+apiVersion: apiextensions.k8s.io/v1beta1
+kind: CustomResourceDefinition
+metadata:
+  # name must match the spec fields below, and be in the form: <plural>.<group>
+  name: networks.kubernetes.com
+spec:
+  # group name to use for REST API: /apis/<group>/<version>
+  group: kubernetes.com
+  # version name to use for REST API: /apis/<group>/<version>
+  version: v1
+  # either Namespaced or Cluster
+  scope: Namespaced
+  names:
+    # plural name to be used in the URL: /apis/<group>/<version>/<plural>
+    plural: networks
+    # singular name to be used as an alias on the CLI and for display
+    singular: network
+    # kind is normally the CamelCased singular type. Your resource manifests use this.
+    kind: Network
+    # shortNames allow shorter string to match your resource on the CLI
+    shortNames:
+    - net
+```
+2. Run kubectl create command for the Custom Resource Definition
+```
+# kubectl create -f ./crdnetwork.yaml
+customresourcedefinition "network.kubernetes.com" created
+```
+3. Run kubectl get command to check the Network CRD creation
+```
+# kubectl get CustomResourceDefinition
+NAME                      KIND
+networks.kubernetes.com   CustomResourceDefinition.v1beta1.apiextensions.k8s.io
+```
+
+4. Save the following YAML to flannel-network.yaml
+
+```
+apiVersion: "kubernetes.com/v1"
+kind: Network
+metadata:
+  name: flannel-networkobj
+plugin: flannel
+args: '[
+        {
+                "delegate": {
+                        "isDefaultGateway": true
+                }
+        }
+]'
+```
+
+5. Create the custom resource definition
+
+```
+# kubectl create -f customCRD/flannel-network.yaml
+network "flannel-networkobj" created
+```
+```
+# kubectl get network
+NAME                 KIND                        ARGS                                               PLUGIN
+flannel-networkobj   Network.v1.kubernetes.com   [ { "delegate": { "isDefaultGateway": true } } ]   flannel
+```
+
+
+6. Get the custom network object details
+```
+# kubectl get network flannel-networkobj -o yaml
+apiVersion: kubernetes.com/v1
+args: '[ { "delegate": { "isDefaultGateway": true } } ]'
+kind: Network
+metadata:
+  clusterName: ""
+  creationTimestamp: 2017-07-11T21:46:52Z
+  deletionGracePeriodSeconds: null
+  deletionTimestamp: null
+  name: flannel-networkobj
+  namespace: default
+  resourceVersion: "6848829"
+  selfLink: /apis/kubernetes.com/v1/namespaces/default/networks/flannel-networkobj
+  uid: 7311c965-6682-11e7-b0b9-408d5c537d27
+plugin: flannel
+```
+
+For Kubernetes v1.7 and above use CRD to create network object. For version older than 1.7 use TPR based objects as shown below: 
+
+Note: Both TPR and CRD will have same selfLink : 
+
+*/apis/kubernetes.com/v1/namespaces/default/networks/*
+
+
+#### TPR based Network objects
+
+1. Create a Third Party Resource &quot;tprnetwork.yaml&quot; for the network object as shown below:

-### Creating “Network” third party resource in kubernetes
-1. Create a Third party resource “tprnetwork.yaml” for the network object as shown below
 ```
 apiVersion: extensions/v1beta1
 kind: ThirdPartyResource
@@ -60,7 +184,9 @@ description: "A specification of a Network obj in the kubernetes"
 versions:
 - name: v1
 ```
+
 2. Run kubectl create command for the Third Party Resource
+
 ```
 # kubectl create -f ./tprnetwork.yaml
 thirdpartyresource "network.kubernetes.com" created
@@ -71,10 +197,12 @@ thirdpartyresource "network.kubernetes.com" created
 NAME                     DESCRIPTION                                          VERSION(S)
 network.kubernetes.com   A specification of a Network obj in the kubernetes   v1
 ```
-### Creating “Custom Network objects” third party resource in kubernetes
-1. After the ThirdPartyResource object has been created you can create network objects. Network objects should contain network fields. These fields are in JSON format. In the following example, a plugin and args fields are set to the object of kind Network. The kind Network is derived from the metadata.name of the ThirdPartyResource object we created above.
+### Creating network resources in Kubernetes
+
+1. After creating CRD/TPR network object you can create network resources in Kubernetes. These network resources may contain additional underlying CNI plugin parameters given in JSON format. In the following example shown below the args field contains parameters that will be passed into “flannel” plugin.
+
+2. Save the following YAML to flannel-network.yaml

-2. Save the below following YAML to flannel-network.yaml
 ```
 apiVersion: "kubernetes.com/v1"
 kind: Network
@@ -89,18 +217,19 @@ args: '[
        }
 ]'
 ```
-2. Run kubectl create command for the TPR - Network object
+3. Run kubectl create command to create network object
 ```
 # kubectl create -f ./flannel-network.yaml 
 network "flannel-conf" created
 ```
-3. Manage the Network objects using kubectl.
+4. Show network objects using kubectl:
 ```
 # kubectl get network
 NAME                         KIND
 flannel-conf                 Network.v1.kubernetes.com
 ```
-4. You can also view the raw JSON data. Here you can see that it contains the custom plugin and args fields from the yaml you used to create it:
+5. Show details of the network object:
+
 ```
 # kubectl get network flannel-conf -o yaml
 apiVersion: kubernetes.com/v1
@@ -115,8 +244,7 @@ metadata:
  uid: fdcb94a2-5c0c-11e7-bbeb-408d5c537d27
 plugin: flannel
 ```
-4. The plugin field should be the name of the CNI plugin and args should have the flannel args, it should be in the the JSON format as shown above. **User can create network objects for Calico, Weave, Romana, & Cilium and test the multus.** 
-5. Save the below following YAML to sriov-network.yaml. Refer [Intel - SR-IOV CNI](https://github.com/Intel-Corp/sriov-cni) or contact @kural in [Intel-Corp Slack](https://intel-corp.herokuapp.com/) for running the DPDK based workloads in Kubernetes
+6. Save the following YAML to sriov-network.yaml to creating sriov network object. ( Refer to [Intel - SR-IOV CNI](https://github.com/Intel-Corp/sriov-cni) or contact @kural in [Intel-Corp Slack](https://intel-corp.herokuapp.com/) for running the DPDK based workloads in Kubernetes)
 ```
 apiVersion: "kubernetes.com/v1"
 kind: Network
@@ -139,7 +267,8 @@ args: '[
        }
 ]'
 ```
-6. Save the below following YAML to sriov-vlanid-l2enable-network.yaml
+7. Likewise save the following YAML to sriov-vlanid-l2enable-network.yaml to create another sriov based network object:
+
 ```
 apiVersion: "kubernetes.com/v1"
 kind: Network
@@ -155,8 +284,8 @@ args: '[
        }
 ]'
 ```
-7. Follows the step 2 to create the network object “sriov-vlanid-l2enable-conf” and “sriov-conf”
-8.	Manage the Network objects using kubectl.
+8. Follow step 3 above to create &quot;sriov-vlanid-l2enable-conf&quot; and &quot;sriov-conf&quot; network objects
+9. View network objects using kubectl
 ```
 # kubectl get network
 NAME                         KIND
@@ -165,20 +294,43 @@ sriov-vlanid-l2enable-conf   Network.v1.kubernetes.com
 sriov-conf                   Network.v1.kubernetes.com
 ```
 ### Configuring Multus to use the kubeconfig
-1.	Create Multus CNI configuration file /etc/cni/net.d/multus-cni.conf with below content in minions. Use only the absolute path to point to the kubeconfig file (it may change depending upon your cluster env) and make sure all CNI binary files are in `\opt\cni\bin` dir
+1. Create a Mutlus CNI configuration file on each Kubernetes node. This file should be created in: /etc/cni/net.d/multus-cni.conf with the content shown below. Use only the absolute path to point to the kubeconfig file (as it may change depending upon your cluster env). We are assuming all CNI plugin binaries are default location (`\opt\cni\bin dir`)
 ```
 {
-    "name": "minion-cni-network",
+    "name": "node-cni-network",
    "type": "multus",
    "kubeconfig": "/etc/kubernetes/node-kubeconfig.yaml"
 }
 ```
-2.	Restart kubelet service
+2. Restart kubelet service
 ```
 # systemctl restart kubelet
 ```
-### Configuring Pod to use the TPR Network objects
-1. 	Save the below following YAML to pod-multi-network.yaml. In this case flannel-conf network object act as the primary network. 
+### Configuring Multus to use kubeconfig and a default network
+
+1. Many users want Kubernetes default networking feature along with network objects. Refer to issues [#14](https://github.com/Intel-Corp/multus-cni/issues/14) &amp; [#17](https://github.com/Intel-Corp/multus-cni/issues/17) for more information. In the following Multus configuration, Weave act as the default network in the absence of network field in the pod metadata annotation.
+
+```
+{
+    "name": "node-cni-network",
+    "type": "multus",
+    "kubeconfig": "/etc/kubernetes/node-kubeconfig.yaml",
+    "delegates": [{
+        "type": "weave-net",
+        "hairpinMode": true,
+        "masterplugin": true
+    }]
+}
+```
+2. Restart kubelet service
+
+```
+# systemctl restart kubelet
+```
+
+### Configuring Pod to use the CRD/TPR network objects
+
+1. Save the following YAML to pod-multi-network.yaml. In this case flannel-conf network object acts as the primary network.
 ```
 # cat pod-multi-network.yaml 
 apiVersion: v1
@@ -199,19 +351,25 @@ spec:  # specification of the pod's contents
    stdin: true
    tty: true
 ```
-3.	Create Multiple network based pod from the master node
+
+2. Create Multiple network based pod from the master node
+
 ```
 # kubectl create -f ./pod-multi-network.yaml
 pod "multus-multi-net-poc" created
 ```
-4.	Get the details of the running pod from the master
+
+3. Get the details of the running pod from the master
+
 ```
 # kubectl get pods
 NAME                   READY     STATUS    RESTARTS   AGE
 multus-multi-net-poc   1/1       Running   0          30s
 ```
-### Verifying Pod network
-1.	Run “ifconfig” command inside the container:
+
+### Verifying Pod network interfaces
+
+1. Run &quot;ifconfig&quot; command in Pod:
 ```
 # kubectl exec -it multus-multi-net-poc -- ifconfig
 eth0      Link encap:Ethernet  HWaddr 06:21:91:2D:74:B9  
@@ -249,13 +407,15 @@ north     Link encap:Ethernet  HWaddr BE:F2:48:42:83:12
          collisions:0 txqueuelen:1000 
          RX bytes:95956 (93.7 KiB)  TX bytes:82200 (80.2 KiB)
 ```
-Interface name | Description
------------ | -------------
-lo | loopback
-eth0@if41 | Flannel network tap interface
-net0 | VF0 of NIC 1 assigned to the container by [Intel - SR-IOV CNI](https://github.com/Intel-Corp/sriov-cni) plugin
-north | 	VF0 of NIC 2 assigned with VLAN ID 210 to the container by SR-IOV CNI plugin
-2.	Check the vlan ID of the NIC 2 VFs
+| Interface name | Description |
+| --- | --- |
+| lo | loopback |
+| eth0@if41 | Flannel network tap interface |
+| net0 | VF0 of NIC 1 assigned to the container by [Intel - SR-IOV CNI](https://github.com/Intel-Corp/sriov-cni) plugin |
+| north | VF0 of NIC 2 assigned with VLAN ID 210 to the container by SR-IOV CNI plugin |
+
+2. Check the vlan ID of the NIC 2 VFs
+
 ```
 # ip link show enp2s0
 20: enp2s0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP mode DEFAULT group default qlen 1000
@@ -266,7 +426,7 @@ north | 	VF0 of NIC 2 assigned with VLAN ID 210 to the container by SR-IOV CNI p
    vf 3 MAC 00:00:00:00:00:00, vlan 4095, spoof checking off, link-state auto
 ```

-## Using Multus Conf file
+## Using with Multus conf file

 Given the following network configuration:

@@ -314,37 +474,42 @@ Given the following network configuration:
    ]
 }
 EOF
-
 ```
-## Testing the Multus CNI with Multiple Flannel Network
-Github user [YYGCui](https://github.com/YYGCui) has used Multiple flannel network to work with Multus CNI plugin. Please refer this [closed issue](https://github.com/Intel-Corp/multus-cni/issues/7) for Multiple overlay network support with Multus CNI.

-## Testing the Multus CNI with docker
-Make sure that the multus, [sriov](https://github.com/Intel-Corp/sriov-cni), [flannel](https://github.com/containernetworking/cni/blob/master/Documentation/flannel.md), and [ptp](https://github.com/containernetworking/cni/blob/master/Documentation/ptp.md) binaries are in the `/opt/cni/bin` directories and follow the steps as mention in the [CNI](https://github.com/containernetworking/cni/#running-a-docker-container-with-network-namespace-set-up-by-cni-plugins)
+## Testing Multus CNI

-## Testing the Multus CNI with Kubernetes
-Refer the Kubernetes User Guide and network plugin
-* [Single Node](https://kubernetes.io/docs/getting-started-guides/fedora/fedora_manual_config/)
-* [Multi Node](https://kubernetes.io/docs/getting-started-guides/fedora/flannel_multi_node_cluster/)
-* [Network Plugin](https://kubernetes.io/docs/admin/network-plugins/)
+### Multiple flannel networks
+
+Github user [YYGCui](https://github.com/YYGCui) has used multiple flannel network to work with Multus CNI plugin. Please refer to this [closed issue](https://github.com/Intel-Corp/multus-cni/issues/7) for ,multiple overlay network support with Multus CNI.
+
+Make sure that the multus, [sriov](https://github.com/Intel-Corp/sriov-cni), [flannel](https://github.com/containernetworking/cni/blob/master/Documentation/flannel.md), and [ptp](https://github.com/containernetworking/cni/blob/master/Documentation/ptp.md) binaries are in the /opt/cni/bin directories and follow the steps as mentioned in the [CNI](https://github.com/containernetworking/cni/#running-a-docker-container-with-network-namespace-set-up-by-cni-plugins)
+
+#### Configure Kubernetes with CNI
+Kubelet must be configured to run with the CNI network plugin. Edit /etc/kubernetes/kubelet file and add "--network-plugin=cni" flags in KUBELET\_OPTS as shown below:

-Kubelet must be configured to run with the CNI `--network-plugin`, with the following configuration information. 
-Edit `/etc/default/kubelet` file and add `KUBELET_OPTS`:
 ```
 KUBELET_OPTS="...
 --network-plugin-dir=/etc/cni/net.d
 --network-plugin=cni
 "
 ```
-Restart the kubelet
+Refer to the Kubernetes User Guide and network plugin for more information.
+- [Single Node](https://kubernetes.io/docs/getting-started-guides/fedora/fedora_manual_config/)
+- [Multi Node](https://kubernetes.io/docs/getting-started-guides/fedora/flannel_multi_node_cluster/)
+- [Network plugin](https://kubernetes.io/docs/admin/network-plugins/)
+
+Restart kubelet:
 ```
 # systemctl restart kubelet.service
 ```
-### Launching workloads in Kubernetes
-Launch the workload using yaml file in the kubernetes master, with above configuration in the multus CNI, each pod should have multiple interfaces.
-> Note:	To verify whether Multus CNI plugin is working fine create a pod containing one “busybox” container and execute “ip link” command to check if interfaces management follows configuration.
+#### Launching workloads in Kubernetes
+
+With Multus CNI configured as described in sections above each workload launched via a Kubernetes Pod will have multiple network interfacesLaunch the workload using yaml file in the kubernetes master, with above configuration in the multus CNI, each pod should have multiple interfaces.
+
+Note: To verify whether Multus CNI plugin is working correctly, create a pod containing one &quot;busybox&quot; container and execute &quot;ip link&quot; command to check if interfaces management follows configuration.
+
+1. Create &quot;multus-test.yaml&quot; file containing below configuration. Created pod will consist of one &quot;busybox&quot; container running &quot;top&quot; command.

-1. Create “multus-test.yaml” file containing below configuration. Created pod will consist of one “busybox” container running “top” command.
 ```
 apiVersion: v1
 kind: Pod
@@ -361,11 +526,13 @@ spec:  # specification of the pod's contents

 ```
 2. Create pod using command:
+
 ```
 # kubectl create -f multus-test.yaml
 pod "multus-test" created
 ```
-3. Run “ip link” command inside the container:
+3. Run &quot;ip link&quot; command inside the container:
+
 ```
 # 1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue qlen 1
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
@@ -377,13 +544,47 @@ pod "multus-test" created
    link/ether 76:13:b1:60:00:00 brd ff:ff:ff:ff:ff:ff 
 ```

-Interface name | Description
------------ | -------------
-lo | loopback
-eth0@if41 | Flannel network tap interface
-net0 | VF assigned to the container by [SR-IOV CNI](https://github.com/Intel-Corp/sriov-cni) plugin
-net1 | ptp localhost interface
+| Interface name | Description |
+| --- | --- |
+| lo | loopback |
+| eth0@if41 | Flannel network tap interface |
+| net0 | VF assigned to the container by [SR-IOV CNI](https://github.com/Intel-Corp/sriov-cni) plugin |
+| net1 | ptp localhost interface |

-### Contacts
+## Multus additional plugins
+
+- [DPDK-SRIOV CNI](https://github.com/Intel-Corp/sriov-cni)
+- [Vhostuser CNI](https://github.com/intel/vhost-user-net-plugin) - a Dataplane network plugin - Supports OVS-DPDK &amp; VPP
+- [Bond CNI](https://github.com/Intel-Corp/bond-cni) - For fail-over and high availability of networking
+
+## NFV based networking in Kubernetes
+
+- KubeCon workshop on [&quot;Enabling NFV features in Kubernetes&quot;](https://kccncna17.sched.com/event/Cvnw/enabling-nfv-features-in-kubernetes-hosted-by-kuralamudhan-ramakrishnan-ivan-coughlan-intel) presentation [slide deck](https://www.slideshare.net/KuralamudhanRamakris/enabling-nfv-features-in-kubernetes-83923352)
+- Feature brief
+  - [Multiple Network Interface Support in Kubernetes](https://builders.intel.com/docs/networkbuilders/multiple-network-interfaces-support-in-kubernetes-feature-brief.pdf)
+  - [Enhanced Platform Awareness in Kubernetes](https://builders.intel.com/docs/networkbuilders/enhanced-platform-awareness-feature-brief.pdf)
+- Application note
+  - [Multiple Network Interfaces in Kubernetes and Container Bare Metal](https://builders.intel.com/docs/networkbuilders/multiple-network-interfaces-in-kubernetes-application-note.pdf)
+  - [Enhanced Platform Awareness Features in Kubernetes](https://builders.intel.com/docs/networkbuilders/enhanced-platform-awareness-in-kubernetes-application-note.pdf)
+- White paper
+  - [Enabling New Features with Kubernetes for NFV](https://builders.intel.com/docs/networkbuilders/enabling_new_features_in_kubernetes_for_NFV.pdf)
+- Multus&#39;s related project github pages
+  - [Multus](https://github.com/Intel-Corp/multus-cni)
+  - [SRIOV - DPDK CNI](https://github.com/Intel-Corp/sriov-cni)
+  - [Vhostuser - VPP &amp; OVS - DPDK CNI](https://github.com/intel/vhost-user-net-plugin)
+  - [Bond CNI](https://github.com/Intel-Corp/bond-cni)
+  - [Node Feature Discovery](https://github.com/kubernetes-incubator/node-feature-discovery)
+  - [CPU Manager for Kubernetes](https://github.com/Intel-Corp/CPU-Manager-for-Kubernetes)
+
+
+## Need help
+
+- Read [Containers Experience Kits](https://networkbuilders.intel.com/network-technologies/container-experience-kits)
+- Try our container exp kit demo - KubeCon workshop on [Enabling NFV Features in Kubernetes](https://github.com/intel/container-experience-kits-demo-area/)
+- Join us on [#intel-sddsg-slack](https://intel-corp.herokuapp.com/) slack channel and ask question in [#general-discussion](https://intel-corp-team.slack.com/messages/C4C5RSEER)
+- You can also [email](mailto:kuralamudhan.ramakrishnan@intel.com) us
+- Feel free to [submit](https://github.com/Intel-Corp/multus-cni/issues/new) an issue
+
+Please fill in the Questions/feedback - [google-form](https://goo.gl/forms/upBWyGs8Wmq69IEi2)!
+## Contacts
 For any questions about Multus CNI, please reach out on github issue or feel free to contact the developer @kural in our [Intel-Corp Slack](https://intel-corp.herokuapp.com/)
-
--- a/doc/images/multus_crd_usage_diagram.JPG
+++ b/doc/images/multus_crd_usage_diagram.JPG
--- a/multus/multus.go
+++ b/multus/multus.go
@@ -1,4 +1,4 @@
-// Copyright 2015 CNI authors
+// Copyright (c) 2017 Intel Corporation
 //
 // Licensed under the Apache License, Version 2.0 (the "License");
 // you may not use this file except in compliance with the License.
@@ -12,7 +12,7 @@
 // See the License for the specific language governing permissions and
 // limitations under the License.

-// This is a "Multi-plugin".It is a fork of flannel CNI
+// This is a "Multi-plugin".The delegate concept refered from CNI project
 // It reads other plugin netconf, and then invoke them, e.g.
 // flannel or sriov plugin.

@@ -41,6 +41,7 @@ import (
 const defaultCNIDir = "/var/lib/cni/multus"

 var masterpluginEnabled bool
+var defaultcninetwork bool

 type NetConf struct {
 	types.NetConf
@@ -86,12 +87,16 @@ func loadNetConf(bytes []byte) (*NetConf, error) {
 		return nil, fmt.Errorf("failed to load netconf: %v", err)
 	}

-	if netconf.Kubeconfig != "" {
+	if netconf.Kubeconfig != "" && netconf.Delegates != nil {
+		defaultcninetwork = true
+	}
+
+	if netconf.Kubeconfig != "" && !defaultcninetwork {
 		return netconf, nil
 	}

-	if netconf.Delegates == nil {
-		return nil, fmt.Errorf(`"delegates" is must, refer README.md`)
+	if len(netconf.Delegates) == 0 && !defaultcninetwork {
+		return nil, fmt.Errorf(`delegates or kubeconfig option is must, refer README.md`)
 	}

 	if netconf.CNIDir == "" {
@@ -120,12 +125,7 @@ func consumeScratchNetConf(containerID, dataDir string) ([]byte, error) {
 	path := filepath.Join(dataDir, containerID)
 	defer os.Remove(path)

-	data, err := ioutil.ReadFile(path)
-	if err != nil {
-		return nil, fmt.Errorf("failed to read container data in the path(%q): %v", path, err)
-	}
-
-	return data, err
+	return ioutil.ReadFile(path)
 }

 func getifname() (f func() string) {
@@ -405,6 +405,9 @@ func getMultusDelegates(delegate string) ([]map[string]interface{}, error) {
 	return tmpNetconf.Delegates, nil
 }

+type NoK8sNetworkError string
+func (e NoK8sNetworkError) Error() string  { return string(e) }
+
 func getK8sNetwork(args *skel.CmdArgs, kubeconfig string) ([]map[string]interface{}, error) {
 	k8sArgs := K8sArgs{}
 	var podNet []map[string]interface{}
@@ -424,6 +427,10 @@ func getK8sNetwork(args *skel.CmdArgs, kubeconfig string) ([]map[string]interfac
 		return podNet, err
 	}

+	if len(netAnnot) == 0 {
+		return podNet, NoK8sNetworkError("no kubernetes network found")
+	}
+
 	netObjs, err := parsePodNetworkObject(netAnnot)
 	if err != nil {
 		return podNet, err
@@ -444,18 +451,28 @@ func getK8sNetwork(args *skel.CmdArgs, kubeconfig string) ([]map[string]interfac

 func cmdAdd(args *skel.CmdArgs) error {
 	var result error
+	var nopodnet bool
 	n, err := loadNetConf(args.StdinData)
 	if err != nil {
-		return err
+		return fmt.Errorf("err in loading netconf: %v", err)
 	}

 	if n.Kubeconfig != "" {
-		podDelegate, r := getK8sNetwork(args, n.Kubeconfig)
-		if r != nil {
-			return fmt.Errorf("Multus: Err in getting k8s network from pod: %v", r)
+		podDelegate, err := getK8sNetwork(args, n.Kubeconfig)
+		if err != nil {
+			if _, ok := err.(*NoK8sNetworkError); ok {
+				nopodnet = true
+				if !defaultcninetwork {
+					return fmt.Errorf("Multus: Err in getting k8s network from the pod spec annotation, check the pod spec or set delegate for the default network, Refer the README.md: %v", err)
+				}
+			} else if !defaultcninetwork {
+				return fmt.Errorf("Multus: Err in getting k8s network from pod: %v", err)
+			}
 		}

-		n.Delegates = podDelegate
+		if len(podDelegate) != 0 {
+			n.Delegates = podDelegate
+		}
 	}

 	for _, delegate := range n.Delegates {
@@ -464,7 +481,7 @@ func cmdAdd(args *skel.CmdArgs) error {
 		}
 	}

-	if n.Kubeconfig == "" {
+	if n.Kubeconfig == "" || nopodnet {
 		if err := saveDelegates(args.ContainerID, n.CNIDir, n.Delegates); err != nil {
 			return fmt.Errorf("Multus: Err in saving the delegates: %v", err)
 		}
@@ -500,7 +517,7 @@ func cmdAdd(args *skel.CmdArgs) error {

 func cmdDel(args *skel.CmdArgs) error {
 	var result error
-	var Delegates []map[string]interface{}
+	var nopodnet bool

 	in, err := loadNetConf(args.StdinData)
 	if err != nil {
@@ -509,25 +526,39 @@ func cmdDel(args *skel.CmdArgs) error {

 	if in.Kubeconfig != "" {
 		podDelegate, r := getK8sNetwork(args, in.Kubeconfig)
-		if r != nil {
-			return r
+		if r != nil && r.Error() == "nonet" {
+			nopodnet = true
+			if !defaultcninetwork {
+				return fmt.Errorf("Multus: Err in getting k8s network from the poc spec, check the pod spec or set delegate for the default network, Refer the README.md: %v", r)
+			}
 		}

-		Delegates = podDelegate
-	} else {
+		if r != nil && !defaultcninetwork {
+			return fmt.Errorf("Multus: Err in getting k8s network from pod: %v", r)
+		}

+		if len(podDelegate) != 0 {
+			in.Delegates = podDelegate
+		}
+	}
+
+	if in.Kubeconfig == "" || nopodnet {
 		netconfBytes, err := consumeScratchNetConf(args.ContainerID, in.CNIDir)
 		if err != nil {
+			if os.IsNotExist(err) {
+				// Per spec should ignore error if resources are missing / already removed
+				return nil
+			}
 			return fmt.Errorf("Multus: Err in  reading the delegates: %v", err)
 		}

-		if err := json.Unmarshal(netconfBytes, &Delegates); err != nil {
+		if err := json.Unmarshal(netconfBytes, &in.Delegates); err != nil {
 			return fmt.Errorf("Multus: failed to load netconf: %v", err)
 		}
 	}

 	podifName := getifname()
-	for _, delegate := range Delegates {
+	for _, delegate := range in.Delegates {
 		r := delegateDel(podifName, args.IfName, delegate)
 		if r != nil {
 			return r
Author	SHA1	Message	Date
Jason Murray	b9446232cd	Fix grammar in README	2018-04-28 16:34:18 +01:00
Abdul Halim	359dc16285	Update README.md Change-Id: I7b48d923def4998106f2d537a5c1f4412179f372	2018-03-21 13:47:07 +00:00
Dave	6f6a4c20e6	Remove referecne to minions The term "minion" was changed to "node" in 2014, The discussion and eventual deprcation is documented on https://github.com/kubernetes/kubernetes/issues/1111 "A node is a worker machine in Kubernetes, previously known as a minion." - https://kubernetes.io/docs/concepts/architecture/nodes/ updating the documentation to reflect correct terminology	2018-02-28 11:58:58 +00:00
Kuralamudhan Ramakrishnan	1ea64694f2	Update README.md	2017-12-20 11:08:54 +00:00
Kuralamudhan Ramakrishnan	81b0018b47	Update README.md	2017-12-20 10:03:14 +00:00
Kuralamudhan Ramakrishnan	5c72ca9026	Update README.md	2017-12-20 10:01:56 +00:00
Kuralamudhan Ramakrishnan	13f3efac2e	Update README.md	2017-12-12 11:08:22 +00:00
Dan Williams	7066439e38	Use typed error when kubernetes network is missing Matching error text is error-prone, so use a typed error instead.	2017-12-05 06:29:46 -06:00
Wade Lee	b2d67c8909	Fix typo in README (#32 )	2017-12-02 21:13:53 +00:00
Kuralamudhan Ramakrishnan	2ae1c3d58d	Update README.md	2017-11-20 19:07:40 +00:00
Kuralamudhan Ramakrishnan	91f7c7c02d	Merge and updated the PR#8 (#27 ) * Fix retrying DEL failure bug * bug fix for consuming net conf file doesn't exist Change-Id: I810dc308285437d167fa74ca1f63c71dad999f26	2017-11-09 17:55:37 +00:00
Kuralamudhan Ramakrishnan	a0c23d3ed9	Add files via upload	2017-11-03 17:11:24 +00:00
Kuralamudhan Ramakrishnan	2ef11d6eec	update typo errors	2017-11-03 17:07:09 +00:00
Kuralamudhan Ramakrishnan	4049d156e0	Update README.md	2017-11-02 16:37:21 +00:00
Kuralamudhan Ramakrishnan	a9a0e808ee	inc multus crd usage diagram	2017-11-02 16:35:00 +00:00
Kuralamudhan Ramakrishnan	e82778a267	Update README.md	2017-11-02 16:33:21 +00:00
Kuralamudhan Ramakrishnan	c57aca6037	Update README.md	2017-10-26 16:53:49 +01:00
Kuralamudhan Ramakrishnan	a2d350a495	Update README.md	2017-10-26 13:47:46 +01:00
Kuralamudhan Ramakrishnan	fe47519451	Merge pull request #24 from PrashantSunkari/master Updating README to improve readability	2017-10-25 09:26:52 +01:00
Prashant Sunkari	59c4f095c8	Merge pull request #2 from PrashantSunkari/PrashantSunkari-cleaningUpReadme Updating TOC and testing subsections	2017-10-24 13:17:27 -07:00
Prashant Sunkari	265589f846	Updating TOC in Readme Updating TOC and testing subsections	2017-10-24 13:12:25 -07:00
Prashant Sunkari	96ab1c18f7	Merge pull request #1 from PrashantSunkari/PrashantSunkari-cleaningUpReadme WIP: Grouping testing related information of README	2017-10-24 13:06:47 -07:00
Prashant Sunkari	b728e117be	WIP: Grouping testing related information of README TODO: - will need to fix TOC	2017-10-24 13:06:12 -07:00
Prashant Sunkari	1adbe17e82	Adding TOC to README Created by [gh-md-toc](https://github.com/ekalinin/github-markdown-toc)	2017-10-24 12:51:03 -07:00
Kuralamudhan Ramakrishnan	74e59eca55	Update README.md	2017-10-17 22:55:56 +01:00
Kuralamudhan Ramakrishnan	454daad880	Merge pull request #22 from s1061123/fix-readme Fix typo in README.md	2017-10-11 10:19:53 +01:00
Tomofumi Hayashi	59e3622d69	Fix typo in README.md The diff just fixes the typo in README.md.	2017-10-11 13:37:01 +09:00
Kuralamudhan Ramakrishnan	ef3c26fd20	Update README.md	2017-10-06 09:01:43 +01:00
Kuralamudhan Ramakrishnan	f1bf15fff5	Update README.md	2017-10-05 17:44:52 +01:00
Kuralamudhan Ramakrishnan	833a064001	Merge pull request #21 from Intel-Corp/pr-issue#19 fix the cmdDel interface bug	2017-10-05 15:56:24 +01:00
kuralamudhan ramakrishnan	fe50bb89ff	fix the cmdDel interface bug Change-Id: Ie266603cff714998dd29bb62e3aa6de531b46c23	2017-09-28 15:43:26 +01:00
Kuralamudhan Ramakrishnan	a4fcc2ef7f	Update CONTRIBUTING.md	2017-09-26 13:53:10 +01:00
Kuralamudhan Ramakrishnan	fa7c3d0a84	Update CONTRIBUTING.md	2017-09-26 13:52:39 +01:00
Kuralamudhan Ramakrishnan	8777509d60	Create CONTRIBUTING.md With references to the #18 issue	2017-09-26 13:50:20 +01:00
Kuralamudhan Ramakrishnan	0ad944aac9	Update README.md	2017-09-18 22:16:17 +01:00
Kuralamudhan Ramakrishnan	9db4183d64	Update README.md	2017-09-18 22:08:14 +01:00
Kuralamudhan Ramakrishnan	c316fb2790	Update README.md	2017-09-18 22:02:43 +01:00
swatisehgal	f387569a8b	Merge pull request #15 from Intel-Corp/dev/defaultcninet Bug fix to provide default CNI networking.	2017-08-11 17:31:17 +01:00
kuralamudhan ramakrishnan	088647ee41	provide default network for pod spec without network annotation	2017-08-11 17:24:43 +01:00
Kuralamudhan Ramakrishnan	e556ec685a	Update multus.go Copyrights updated with Intel - Apache License 2.0	2017-07-21 13:37:50 +01:00
Kuralamudhan Ramakrishnan	2e3ba8ad87	Update README.md	2017-07-06 14:08:25 +01:00