[docs][npwg] Examples directory & documentation

README.md

@@ -62,6 +62,8 @@ Kubelet is responsible for establishing the network interfaces for each pod; it
 
 Please refer the Kubernetes Network SIG - Multiple Network PoC proposal for more details refer the link - [K8s Multiple Network proposal](https://docs.google.com/document/d/1TW3P4c8auWwYy-w_5afIPDcGNLK3LZf0m14943eVfVg/edit)
 
+Example configurations for CRD are also available in the [examples directory](examples/README.md).
+
 <p align="center">
    <img src="doc/images/multus_crd_usage_diagram.JPG" width="1008" />
 </p>

examples/README.md

@@ -0,0 +1,62 @@
+# Example Configurations & Pod Specs
+
+In the `./examples` folder some example configurations are provided for using Multus, especially with CRDs, and doubly so in reference to their usage with the [defacto standard for CRDs](https://docs.google.com/document/d/1Ny03h6IDVy_e_vmElOqR7UdTPAG_RNydhVE1Kx54kFQ/edit) as proposed by the Network Plumbing Working Group.
+
+## Examples overview
+
+Generally, the examples here show a setup using Multus with CRD support. The examples here demonstrate a setup with Multus as the meta-plugin used by Kubernetes, and delgating to either Flannel (which will be the default pod network), or to macvlan. The CRDs are intended to be alignment with the defacto standard.
+
+It is expected that aspects of your own setup will vary, at least in part, from some of what's demonstrated here. Namely, the IP address spaces, and likely the host ethernet interface names used in the macvlan part of the configuration.
+
+More specifically, these examples show:
+
+* Multus configured, using CNI a `.conf` file, with CRD support, specifying that we will use a "default network" with the `always_use_default` option set.
+* A resource definition with a daemonset that places the `.conf` on each node in the cluster.
+* A CRD definining the "networks" @ `networks.kubernetes.cni.cncf.io` 
+* CRD objects containing the configuration for both Flannel & macvlan.
+
+## Quick-start instructions
+
+* Compile Multus and place binaries into (typically) `/opt/cni/bin/`
+    - Refer to the primary README.md for more details on compilation.
+* Allow `system:node` access to enable Multus to pull CRD objects.
+    - See "RBAC configuration section below for details."
+* Create the Flannel + Multus setup with the daemonset provided
+    - As in: `kubectl create -f multus-with-flannel.yml`
+    - Optionally, verify that the `/etc/cni/net.d/*.conf` exists on each node.
+* Create the CRDs
+    - Create the CRD itself, `kubectl create -f crd.yml`
+    - Create the network attachment configurations (i.e. CNI configurations packed into CRD objects)
+        + `kubectl create -f flannel-conf.yml`
+        + `kubectl create -f macvlan-conf.yml`
+        + Verify the CRD objects are created with: `kubectl get networks`
+* Spin up an sample pod
+    - `kubectl create -f sample-pod.yml`
+    - Verify that it has multiple interfaces with:
+        + `kubectl exec -it samplepod -- ip a`
+
+## RBAC configuration
+
+You'll need to abnel the `system:node` users access to the API endpoints that will deliver the CRD objects to Multus. 
+
+Using these examples, you'll first create a cluster role with the provided sample:
+
+```
+kubectl create -f clusterrole.yml
+```
+
+You will then create a `clusterrolebinding` for each hostname in the Kubernetes cluster. Replace `HOSTNAME` below with the host name of a node, and then repeat for all hostnames in the cluster.
+
+```
+kubectl create clusterrolebinding multus-node-HOSTNAME \
+    --clusterrole=multus-crd-overpowered \
+    --user=system:node:HOSTNAME
+```
+
+## CNI Configuration
+
+A sample `cni-configuration.conf` is provided, typically this file is placed in `/etc/cni/net.d/`. It must be the first file alphabetically in this folder in order for the Kubelet to honor its use. However, if you opt to use the provided Flannel + Multus YAML file, this will deploy a configuration (packed inside a daemonset therein) on each node in your Kubernetes cluster.
+
+## Other considerations
+
+Primarily in this setup one thing that one should consider are the aspects of the `macvlan-conf.yml`, which is likely specific to the configuration of the node on which this resides.

examples/clusterrole.yml

@@ -0,0 +1,16 @@
+---
+apiVersion: rbac.authorization.k8s.io/v1
+kind: ClusterRole
+metadata:
+  name: multus-crd-overpowered
+rules:
+- apiGroups:
+  - '*'
+  resources:
+  - '*'
+  verbs:
+  - '*'
+- nonResourceURLs:
+  - '*'
+  verbs:
+  - '*'

examples/cni-configuration.conf

@@ -0,0 +1,15 @@
+{
+  "name": "multus-cni-network",
+  "type": "multus",
+  "delegates": [
+    {
+      "type": "flannel",
+      "masterplugin": true,
+      "delegate": {
+        "isDefaultGateway": true
+      }
+    }
+  ],
+  "always_use_default": true,
+  "kubeconfig": "/etc/kubernetes/kubelet.conf"
+}

examples/crd.yml

@@ -0,0 +1,22 @@
+apiVersion: apiextensions.k8s.io/v1beta1
+kind: CustomResourceDefinition
+metadata:
+  # name must match the spec fields below, and be in the form: <plural>.<group>
+  name: networks.kubernetes.cni.cncf.io
+spec:
+  # group name to use for REST API: /apis/<group>/<version>
+  group: kubernetes.cni.cncf.io
+  # version name to use for REST API: /apis/<group>/<version>
+  version: v1
+  # either Namespaced or Cluster
+  scope: Namespaced
+  names:
+    # plural name to be used in the URL: /apis/<group>/<version>/<plural>
+    plural: networks
+    # singular name to be used as an alias on the CLI and for display
+    singular: network
+    # kind is normally the CamelCased singular type. Your resource manifests use this.
+    kind: Network
+    # shortNames allow shorter string to match your resource on the CLI
+    shortNames:
+    - net

examples/flannel-conf.yml

@@ -0,0 +1,12 @@
+apiVersion: "kubernetes.cni.cncf.io/v1"
+kind: Network
+metadata:
+  name: flannel-conf
+plugin: flannel
+args: '[
+        {
+                "delegate": {
+                        "isDefaultGateway": true
+                }
+        }
+]'

examples/macvlan-conf.yml

@@ -0,0 +1,21 @@
+apiVersion: "kubernetes.cni.cncf.io/v1"
+kind: Network
+metadata:
+  name: macvlan-conf
+plugin: macvlan
+args: '[
+        {
+            "master": "eth0",
+            "mode": "bridge",
+            "ipam": {
+                "type": "host-local",
+                "subnet": "192.168.1.0/24",
+                "rangeStart": "192.168.1.200",
+                "rangeEnd": "192.168.1.216",
+                "routes": [
+                    { "dst": "0.0.0.0/0" }
+                ],
+                "gateway": "192.168.1.1"
+            }
+        }
+]'

examples/multus-with-flannel.yml

@@ -0,0 +1,164 @@
+# -----------------------------------------------
+# - Example Configuration Deployment
+# -----------------------------------------------
+# - Deploys a .conf file on each node
+# - Configured for Multus + Flannel.
+# - As well as assets for Flannel
+# - Based on https://github.com/coreos/flannel/blob/master/Documentation/kube-flannel.yml
+# -----------------------------------------------
+---
+kind: ClusterRole
+apiVersion: rbac.authorization.k8s.io/v1beta1
+metadata:
+  name: flannel
+rules:
+  - apiGroups:
+      - ""
+    resources:
+      - pods
+    verbs:
+      - get
+  - apiGroups:
+      - ""
+    resources:
+      - nodes
+    verbs:
+      - list
+      - watch
+  - apiGroups:
+      - ""
+    resources:
+      - nodes/status
+    verbs:
+      - patch
+---
+kind: ClusterRoleBinding
+apiVersion: rbac.authorization.k8s.io/v1beta1
+metadata:
+  name: flannel
+roleRef:
+  apiGroup: rbac.authorization.k8s.io
+  kind: ClusterRole
+  name: flannel
+subjects:
+- kind: ServiceAccount
+  name: flannel
+  namespace: kube-system
+---
+apiVersion: v1
+kind: ServiceAccount
+metadata:
+  name: flannel
+  namespace: kube-system
+---
+kind: ConfigMap
+apiVersion: v1
+metadata:
+  name: kube-multus-cfg
+  namespace: kube-system
+  labels:
+    tier: node
+    app: multus
+data:
+  cni-conf.json: |
+    {
+      "name": "multus-cni-network",
+      "type": "multus",
+      "delegates": [
+        {
+          "type": "flannel",
+          "masterplugin": true,
+          "delegate": {
+            "isDefaultGateway": true
+          }
+        }
+      ],
+      "always_use_default": true,
+      "kubeconfig": "/etc/kubernetes/kubelet.conf"
+    }
+  net-conf.json: |
+    {
+      "Network": "10.244.0.0/16",
+      "Backend": {
+        "Type": "vxlan"
+      }
+    }
+---
+apiVersion: extensions/v1beta1
+kind: DaemonSet
+metadata:
+  name: kube-multus-ds
+  namespace: kube-system
+  labels:
+    tier: node
+    app: multus
+spec:
+  template:
+    metadata:
+      labels:
+        tier: node
+        app: multus
+    spec:
+      hostNetwork: true
+      nodeSelector:
+        beta.kubernetes.io/arch: amd64
+      tolerations:
+      - key: node-role.kubernetes.io/master
+        operator: Exists
+        effect: NoSchedule
+      serviceAccountName: flannel
+      initContainers:
+      - name: install-cni
+        image: quay.io/coreos/flannel:v0.10.0-amd64
+        command:
+        - cp
+        args:
+        - -f
+        - /etc/kube-flannel/cni-conf.json
+        - /etc/cni/net.d/10-multus-with-flannel.conf
+        volumeMounts:
+        - name: cni
+          mountPath: /etc/cni/net.d
+        - name: multus-cfg
+          mountPath: /etc/kube-flannel/
+      containers:
+      - name: kube-flannel
+        image: quay.io/coreos/flannel:v0.10.0-amd64
+        command:
+        - /opt/bin/flanneld
+        args:
+        - --ip-masq
+        - --kube-subnet-mgr
+        resources:
+          requests:
+            cpu: "100m"
+            memory: "50Mi"
+          limits:
+            cpu: "100m"
+            memory: "50Mi"
+        securityContext:
+          privileged: true
+        env:
+        - name: POD_NAME
+          valueFrom:
+            fieldRef:
+              fieldPath: metadata.name
+        - name: POD_NAMESPACE
+          valueFrom:
+            fieldRef:
+              fieldPath: metadata.namespace
+        volumeMounts:
+        - name: run
+          mountPath: /run
+        - name: multus-cfg
+          mountPath: /etc/kube-flannel/
+      volumes:
+        - name: run
+          hostPath:
+            path: /run
+        - name: cni
+          hostPath:
+            path: /etc/cni/net.d
+        - name: multus-cfg
+          configMap:
+            name: kube-multus-cfg

examples/sample-pod.yml

@@ -0,0 +1,14 @@
+---
+apiVersion: v1
+kind: Pod
+metadata:
+  name: samplepod
+  annotations:
+    kubernetes.v1.cni.cncf.io/networks: macvlan-conf
+spec:
+  containers:
+  - name: samplepod
+    command: ["/bin/bash", "-c", "sleep 2000000000000"]
+    image: dougbtv/centos-network
+    ports:
+    - containerPort: 80