Merge pull request #536 from s1061123/fix/update-vendor

Update vendors

.github/ISSUE_TEMPLATE/bug-report.md

@@ -0,0 +1,28 @@
+---
+name: Bug Report
+about: Report a bug encountered
+
+---
+<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks!-->
+
+
+**What happend**:
+
+**What you expected to happen**:
+
+**How to reproduce it (as minimally and precisely as possible)**:
+
+**Anything else we need to know?**:
+
+**Environment**:
+
+- Multus version 
+  image path and image ID (from 'docker images')
+- Kubernetes version (use `kubectl version`):
+- Primary CNI for Kubernetes cluster:
+- OS (e.g. from /etc/os-release):
+- File of '/etc/cni/net.d/'
+- File of '/etc/cni/multus/net.d'
+- NetworkAttachment info (use `kubectl get net-attach-def -o yaml`)
+- Target pod yaml info (with annotation, use `kubectl get pod <podname> -o yaml`)
+- Other log outputs (if you use multus logging)

.github/ISSUE_TEMPLATE/enhancement.md

@@ -0,0 +1,10 @@
+---
+name: Enhancement Request
+about: Suggest an enhancement to multus
+
+---
+<!-- Please only use this template for submitting enhancement requests -->
+
+**What would you like to be added**:
+
+**Why is this needed**:

.github/ISSUE_TEMPLATE/support.md

@@ -0,0 +1,5 @@
+---
+name: Support Request
+about: Support request or question relating to multus-cni
+
+---

.github/workflows/docker-build-push-master.yml

@@ -0,0 +1,64 @@
+name: Docker-build-push-latest
+on: 
+  push:
+    branches:
+      - master
+jobs:
+  build:
+    name: Docker build
+    runs-on: ubuntu-latest
+    env:
+      DOCKER_CLI_EXPERIMENTAL: enabled
+      GO111MODULE: on
+      TARGET: amd64
+      REPOSITORY: nfvpe/multus
+      REPOSITORY_USER: nfvperobot
+    steps:
+      - name: Check out code into the Go module directory
+        uses: actions/checkout@v2
+  
+      - name: Build latest-amd64
+        run: docker build -t ${REPOSITORY}:latest-amd64 .
+  
+      - name: Build latest-ppc64le
+        run: docker build -t ${REPOSITORY}:latest-ppc64le -f Dockerfile.ppc64le .
+  
+      - name: Build latest-arm64v8
+        run: docker build -t ${REPOSITORY}:latest-arm64v8 -f Dockerfile.arm64 .
+  
+      - name: Build latest-origin
+        run: docker build -t ${REPOSITORY}:latest-origin -f Dockerfile.openshift .
+  
+      - name: Tag snapshot
+        run: |
+          docker tag ${REPOSITORY}:latest-amd64 ${REPOSITORY}:snapshot-amd64
+          docker tag ${REPOSITORY}:latest-ppc64le ${REPOSITORY}:snapshot-ppc64le
+          docker tag ${REPOSITORY}:latest-arm64v8 ${REPOSITORY}:snapshot-arm64v8
+
+      - name: Login to registry
+        if: github.repository == 'intel/multus-cni'
+        run: docker login -u ${REPOSITORY_USER} -p ${{ secrets.REPOSITORY_PASS }}
+  
+      - name: Push latest/snapshot images
+        if: github.repository == 'intel/multus-cni'
+        run: |
+          docker push ${REPOSITORY}:latest-amd64
+          docker push ${REPOSITORY}:latest-ppc64le
+          docker push ${REPOSITORY}:latest-arm64v8
+          docker push ${REPOSITORY}:snapshot-amd64
+          docker push ${REPOSITORY}:snapshot-ppc64le
+          docker push ${REPOSITORY}:snapshot-arm64v8
+  
+      - name: Create manifest for multi-arch images
+        if: github.repository == 'intel/multus-cni'
+        run: |
+          docker manifest create ${REPOSITORY}:snapshot ${REPOSITORY}:snapshot-amd64 ${REPOSITORY}:snapshot-ppc64le ${REPOSITORY}:snapshot-arm64v8
+          docker manifest annotate ${REPOSITORY}:snapshot ${REPOSITORY}:snapshot-amd64 --arch amd64
+          docker manifest annotate ${REPOSITORY}:snapshot ${REPOSITORY}:snapshot-ppc64le --arch ppc64le
+          docker manifest annotate ${REPOSITORY}:snapshot ${REPOSITORY}:snapshot-arm64v8 --arch arm64
+          docker manifest push ${REPOSITORY}:snapshot
+          docker manifest create ${REPOSITORY}:latest ${REPOSITORY}:latest-amd64 ${REPOSITORY}:latest-ppc64le ${REPOSITORY}:latest-arm64v8
+          docker manifest annotate ${REPOSITORY}:latest ${REPOSITORY}:latest-amd64 --arch amd64
+          docker manifest annotate ${REPOSITORY}:latest ${REPOSITORY}:latest-ppc64le --arch ppc64le
+          docker manifest annotate ${REPOSITORY}:latest ${REPOSITORY}:latest-arm64v8 --arch arm64
+          docker manifest push ${REPOSITORY}:latest

.github/workflows/docker-build-push-release.yml

@@ -0,0 +1,75 @@
+name: Docker-build-push-stable
+on: 
+  push:
+    tags:
+      - v*
+jobs:
+  build:
+    name: Docker build
+    runs-on: ubuntu-latest
+    env:
+      DOCKER_CLI_EXPERIMENTAL: enabled
+      GO111MODULE: on
+      TARGET: amd64
+      REPOSITORY: nfvpe/multus
+      REPOSITORY_USER: nfvperobot
+    steps:
+      - name: Check out code into the Go module directory
+        uses: actions/checkout@v2
+  
+      - name: Build latest-amd64
+        run: docker build -t ${REPOSITORY}:latest-amd64 .
+  
+      - name: Build latest-ppc64le
+        run: docker build -t ${REPOSITORY}:latest-ppc64le -f Dockerfile.ppc64le .
+  
+      - name: Build latest-arm64v8
+        run: docker build -t ${REPOSITORY}:latest-arm64v8 -f Dockerfile.arm64 .
+  
+      - name: Build latest-origin
+        run: docker build -t ${REPOSITORY}:latest-origin -f Dockerfile.openshift .
+  
+      - name: Tag stable
+        run: |
+          docker tag ${REPOSITORY}:latest-amd64 ${REPOSITORY}:stable-amd64
+          docker tag ${REPOSITORY}:latest-ppc64le ${REPOSITORY}:stable-ppc64le
+          docker tag ${REPOSITORY}:latest-arm64v8 ${REPOSITORY}:stable-arm64v8
+          docker tag ${REPOSITORY}:latest-amd64 ${REPOSITORY}:${GITHUB_REF##*/}-amd64
+          docker tag ${REPOSITORY}:latest-ppc64le ${REPOSITORY}:${GITHUB_REF##*/}-ppc64le
+          docker tag ${REPOSITORY}:latest-arm64v8 ${REPOSITORY}:${GITHUB_REF##*/}-arm64v8
+  
+      - name: Login to registry
+        if: github.repository == 'intel/multus-cni'
+        run: docker login -u ${REPOSITORY_USER} -p ${{ secrets. REPOSITORY_PASS }}
+  
+      - name: Push latest/snapshot images
+        if: github.repository == 'intel/multus-cni'
+        run: |
+          docker push ${REPOSITORY}:latest-amd64
+          docker push ${REPOSITORY}:latest-ppc64le
+          docker push ${REPOSITORY}:latest-arm64v8
+          docker push ${REPOSITORY}:stable-amd64
+          docker push ${REPOSITORY}:stable-ppc64le
+          docker push ${REPOSITORY}:stable-arm64v8
+          docker push ${REPOSITORY}:${GITHUB_REF##*/}-amd64
+          docker push ${REPOSITORY}:${GITHUB_REF##*/}-ppc64le
+          docker push ${REPOSITORY}:${GITHUB_REF##*/}-arm64v8
+  
+      - name: Create manifest for multi-arch images
+        if: github.repository == 'intel/multus-cni'
+        run: |
+          docker manifest create ${REPOSITORY}:stable ${REPOSITORY}:stable-amd64 ${REPOSITORY}:stable-ppc64le ${REPOSITORY}:stable-arm64v8
+          docker manifest annotate ${REPOSITORY}:stable ${REPOSITORY}:stable-amd64 --arch amd64
+          docker manifest annotate ${REPOSITORY}:stable ${REPOSITORY}:stable-ppc64le --arch ppc64le
+          docker manifest annotate ${REPOSITORY}:stable ${REPOSITORY}:stable-arm64v8 --arch arm64
+          docker manifest push ${REPOSITORY}:stable
+          docker manifest create ${REPOSITORY}:latest ${REPOSITORY}:latest-amd64 ${REPOSITORY}:latest-ppc64le ${REPOSITORY}:latest-arm64v8
+          docker manifest annotate ${REPOSITORY}:latest ${REPOSITORY}:latest-amd64 --arch amd64
+          docker manifest annotate ${REPOSITORY}:latest ${REPOSITORY}:latest-ppc64le --arch ppc64le
+          docker manifest annotate ${REPOSITORY}:latest ${REPOSITORY}:latest-arm64v8 --arch arm64
+          docker manifest push ${REPOSITORY}:latest
+          docker manifest create ${REPOSITORY}:${GITHUB_REF##*/} ${REPOSITORY}:${GITHUB_REF##*/}-amd64 ${REPOSITORY}:${GITHUB_REF##*/}-ppc64le ${REPOSITORY}:${GITHUB_REF##*/}-arm64v8
+          docker manifest annotate ${REPOSITORY}:${GITHUB_REF##*/} ${REPOSITORY}:${GITHUB_REF##*/}-amd64 --arch amd64
+          docker manifest annotate ${REPOSITORY}:${GITHUB_REF##*/} ${REPOSITORY}:${GITHUB_REF##*/}-ppc64le --arch ppc64le
+          docker manifest annotate ${REPOSITORY}:${GITHUB_REF##*/} ${REPOSITORY}:${GITHUB_REF##*/}-arm64v8 --arch arm64
+          docker manifest push ${REPOSITORY}:${GITHUB_REF##*/}

.github/workflows/docker-build.yml

@@ -0,0 +1,22 @@
+name: Docker-build
+on: [pull_request]
+jobs:
+  build:
+    name: Docker build
+    runs-on: ubuntu-latest
+    env:
+      DOCKER_CLI_EXPERIMENTAL: enabled
+      GO111MODULE: on
+      REPOSITORY: nfvpe/multus
+    steps:
+      - name: Check out code into the Go module directory
+        uses: actions/checkout@v2
+  
+      - name: Build latest-amd64
+        run: docker build -t ${REPOSITORY}:latest-amd64 .
+  
+      - name: Build latest-ppc64le
+        run: docker build -t ${REPOSITORY}:latest-ppc64le -f Dockerfile.ppc64le .
+  
+      - name: Build latest-origin
+        run: docker build -t ${REPOSITORY}:latest-origin -f Dockerfile.openshift .

.github/workflows/go-build-arm64.yml

@@ -0,0 +1,25 @@
+name: Go-build-ppc64le
+on: [push, pull_request]
+jobs:
+
+  build:
+    name: Build and test
+    runs-on: ubuntu-latest
+    if: >
+      (( github.event.pull_request.head.repo.owner.login != github.event.pull_request.base.repo.owner.login ) &&
+        github.event_name == 'pull_request' ) || (github.event_name == 'push' && github.event.commits != '[]' )
+    env:
+      GO111MODULE: on
+      TARGET: arm64
+    steps:
+      - name: Set up Go 1.13
+        uses: actions/setup-go@v1
+        with:
+          go-version: 1.13
+        id: go
+  
+      - name: Check out code into the Go module directory
+        uses: actions/checkout@v2
+  
+      - name: Build
+        run: GOARCH="${TARGET}" ./build

.github/workflows/go-build-ppc64.yml

@@ -0,0 +1,25 @@
+name: Go-build-ppc64le
+on: [push, pull_request]
+jobs:
+
+  build:
+    name: Build and test
+    runs-on: ubuntu-latest
+    if: >
+      (( github.event.pull_request.head.repo.owner.login != github.event.pull_request.base.repo.owner.login ) &&
+        github.event_name == 'pull_request' ) || (github.event_name == 'push' && github.event.commits != '[]' )
+    env:
+      GO111MODULE: on
+      TARGET: ppc64le
+    steps:
+      - name: Set up Go 1.13
+        uses: actions/setup-go@v1
+        with:
+          go-version: 1.13
+        id: go
+  
+      - name: Check out code into the Go module directory
+        uses: actions/checkout@v2
+  
+      - name: Build
+        run: GOARCH="${TARGET}" ./build

.github/workflows/go-build-test-amd64.yml

@@ -0,0 +1,48 @@
+name: Go-build-and-test-amd64
+on: [push, pull_request]
+jobs:
+
+  build:
+    name: Build and test
+    runs-on: ubuntu-latest
+    if: >
+      (( github.event.pull_request.head.repo.owner.login != github.event.pull_request.base.repo.owner.login ) &&
+        github.event_name == 'pull_request' ) || (github.event_name == 'push' && github.event.commits != '[]' )
+    env:
+      GO111MODULE: on
+      TARGET: amd64
+    steps:
+      - name: Set up Go 1.13
+        uses: actions/setup-go@v1
+        with:
+          go-version: 1.13
+        id: go
+  
+      - name: Check out code into the Go module directory
+        uses: actions/checkout@v2
+  
+      - name: install goberalls
+        run: go get github.com/mattn/goveralls
+  
+      - name: install golint
+        run: go get -u golang.org/x/lint/golint
+  
+      - name: golint
+        run: golint ./... | grep -v vendor | grep -v ALL_CAPS | xargs -r false
+  
+      - name: gofmt
+        run: go fmt ./...
+  
+      - name: go vet
+        run: go vet ./...
+  
+      - name: Build
+        run: GOARCH="${TARGET}" ./build
+  
+      - name: Go test
+        run: sudo ./test.sh
+  
+      - name: goveralls
+        uses: shogo82148/actions-goveralls@v1
+        with:
+          path-to-profile: coverage.out

.github/workflows/kind-e2e.yml

@@ -0,0 +1,40 @@
+name: e2e-kind
+on: [push, pull_request]
+jobs:
+
+  build:
+    name: Build and test
+    runs-on: ubuntu-latest
+    if: >
+      (( github.event.pull_request.head.repo.owner.login != github.event.pull_request.base.repo.owner.login ) &&
+        github.event_name == 'pull_request' ) || (github.event_name == 'push' && github.event.commits != '[]' )
+    steps:
+      - name: Check out code into the Go module directory
+        uses: actions/checkout@v2
+
+      - name: Setup registry
+        run: docker run -d --restart=always -p "5000:5000" --name "kind-registry" registry:2
+  
+      - name: Build latest-amd64
+        run: docker build -t localhost:5000/multus:e2e .
+ 
+      - name: Push to local registry
+        run: docker push localhost:5000/multus:e2e
+
+      - name: Get kind/kubectl/koko
+        working-directory: ./e2e
+        run: ./get_tools.sh
+
+      - name: Setup cluster
+        working-directory: ./e2e
+        run: ./setup_cluster.sh
+
+      - name: Test macvlan1
+        working-directory: ./e2e
+        run: ./test-simple-macvlan1.sh
+
+      - name: cleanup cluster and registry
+        run: |
+          kind delete cluster
+          docker kill kind-registry
+          docker rm kind-registry

.github/workflows/release-binary.yml

@@ -0,0 +1,29 @@
+name: Release-binaries
+
+on:
+  push:
+    tags:
+      - v*
+
+jobs:
+  goreleaser:
+    runs-on: ubuntu-latest
+    steps:
+      - name: Checkout
+        uses: actions/checkout@v2
+
+      - name: Unshallow
+        run: git fetch --prune --unshallow
+
+      - name: Set up Go
+        uses: actions/setup-go@v1
+        with:
+          go-version: 1.13.x
+
+      - name: Run GoReleaser
+        uses: goreleaser/goreleaser-action@v1
+        with:
+          version: latest
+          args: release --rm-dist
+        env:
+          GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}

.gitignore

@@ -0,0 +1,14 @@
+# Binary output dir
+bin/
+
+# GOPATH created by the build script
+gopath/
+
+# Editor paths
+.swp*
+.swo*
+.idea*
+
+# Test outputs
+*.out
+*.test

.goreleaser.yml

@@ -0,0 +1,29 @@
+# This is an example goreleaser.yaml file with some sane defaults.
+# Make sure to check the documentation at http://goreleaser.com
+env:
+  - GO111MODULE=on
+before:
+  hooks:
+    - go mod download
+builds:
+-  
+  env:
+  - CGO_ENABLED=0
+  main: ./multus/
+  goos:
+  - linux
+  goarch:
+  - 386
+  - amd64
+  - arm
+  - arm64
+archives:
+  - wrap_in_directory: true
+checksum:
+  name_template: 'checksums.txt'
+snapshot:
+  name_template: "{{ .Tag }}-snapshot"
+#release:
+#  draft: true
+changelog:
+  skip: true

.travis.yml

@@ -0,0 +1,116 @@
+os: linux
+language: go
+# see https://docs.travis-ci.com/user/reference/overview/#Virtualization-environments
+# for the detail
+# sudo: requried
+dist: bionic
+
+services:
+  - docker
+
+go:
+  - 1.13.x
+
+env:
+  global:
+    - GO111MODULE=on
+    - REGISTRY_USER=${REGISTRY_USER:-nfvpe}
+    - REGISTRY_PASS=${REGISTRY_PASS}
+    - REPOSITORY_NAME=${REPOSITORY_NAME}
+    - REPOSITORY_USER=${REPOSITORY_USER}
+    - DOCKER_CLI_EXPERIMENTAL="enabled"
+    - secure: "${REGISTRY_SECURE}"
+  jobs:
+    - TARGET=amd64
+    - TARGET=ppc64le
+
+before_install:
+  - if [ "${REPOSITORY_NAME}" = "" ]; then export REPOSITORY_NAME=multus; fi
+  - sudo apt-get update -qq
+  - go get github.com/mattn/goveralls
+
+install:
+  - go get -u golang.org/x/lint/golint
+
+before_script:
+  # Make gopath... to run golint/go fmt/go vet
+  # Suppress golint for fixing lint later.
+  - golint ./... | grep -v vendor | grep -v ALL_CAPS | xargs -r false
+  - go fmt ./...
+  - go vet ./...
+#  - gocyclo -over 15 ./multus
+
+script:
+  - GOARCH="${TARGET}" ./build
+  - |
+    if [ "${TARGET}" == "amd64" ]; then
+      sudo env PATH=${PATH} ./test.sh
+      goveralls -coverprofile=coverage.out -service=travis-ci
+      docker build -t ${REPOSITORY_USER}/${REPOSITORY_NAME}:latest-amd64 .
+      docker build -t ${REPOSITORY_USER}/${REPOSITORY_NAME}:latest-ppc64le -f Dockerfile.ppc64le .
+      docker build -t ${REPOSITORY_USER}/${REPOSITORY_NAME}-origin:latest -f Dockerfile.openshift .
+    fi
+
+deploy:
+  # Release on versioned tag (e.g. v1.0)
+  - provider: script
+    #cleanup: false
+    script: curl -sL https://git.io/goreleaser
+    on:
+      tags: true
+      all_branches: true
+      condition: "$TARGET = amd64 && $TRAVIS_TAG =~ ^v[0-9].*$ && ! -z $GITHUB_TOKEN && $TRAVIS_OS_NAME = linux"
+  # Push images to Dockerhub on tag
+  - provider: script
+    cleanup: false
+    script: >
+      bash -c '
+      docker tag ${REPOSITORY_USER}/${REPOSITORY_NAME}:latest-amd64 ${REPOSITORY_USER}/${REPOSITORY_NAME}:latest;
+      docker tag ${REPOSITORY_USER}/${REPOSITORY_NAME}:latest-amd64 ${REPOSITORY_USER}/${REPOSITORY_NAME}:stable;
+      docker tag ${REPOSITORY_USER}/${REPOSITORY_NAME}:latest-amd64 ${REPOSITORY_USER}/${REPOSITORY_NAME}:stable-amd64;
+      docker tag ${REPOSITORY_USER}/${REPOSITORY_NAME}:latest-amd64 ${REPOSITORY_USER}/${REPOSITORY_NAME}:$TRAVIS_TAG;
+      docker tag ${REPOSITORY_USER}/${REPOSITORY_NAME}:latest-ppc64le ${REPOSITORY_USER}/${REPOSITORY_NAME}:stable-ppc64le;
+      docker login -u "$REGISTRY_USER" -p "$REGISTRY_PASS"; 
+      docker push ${REPOSITORY_USER}/${REPOSITORY_NAME}:latest-amd64;
+      docker push ${REPOSITORY_USER}/${REPOSITORY_NAME}:latest-ppc64le;
+      docker push ${REPOSITORY_USER}/${REPOSITORY_NAME}:stable-amd64;
+      docker push ${REPOSITORY_USER}/${REPOSITORY_NAME}:stable-ppc64le;
+      docker push ${REPOSITORY_USER}/${REPOSITORY_NAME}:$TRAVIS_TAG;
+      export DOCKER_CLI_EXPERIMENTAL="enabled";
+      docker manifest create ${REPOSITORY_USER}/${REPOSITORY_NAME}:latest ${REPOSITORY_USER}/${REPOSITORY_NAME}:latest-amd64 ${REPOSITORY_USER}/${REPOSITORY_NAME}:latest-ppc64le;
+      docker manifest annotate ${REPOSITORY_USER}/${REPOSITORY_NAME}:latest ${REPOSITORY_USER}/${REPOSITORY_NAME}:latest-amd64 --arch amd64;
+      docker manifest annotate ${REPOSITORY_USER}/${REPOSITORY_NAME}:latest ${REPOSITORY_USER}/${REPOSITORY_NAME}:latest-ppc64le --arch ppc64le;
+      docker manifest push ${REPOSITORY_USER}/${REPOSITORY_NAME}:latest;
+      docker manifest create ${REPOSITORY_USER}/${REPOSITORY_NAME}:stable ${REPOSITORY_USER}/${REPOSITORY_NAME}:stable-amd64 ${REPOSITORY_USER}/${REPOSITORY_NAME}:stable-ppc64le;
+      docker manifest annotate ${REPOSITORY_USER}/${REPOSITORY_NAME}:stable ${REPOSITORY_USER}/${REPOSITORY_NAME}:stable-amd64 --arch amd64;
+      docker manifest annotate ${REPOSITORY_USER}/${REPOSITORY_NAME}:stable ${REPOSITORY_USER}/${REPOSITORY_NAME}:stable-ppc64le --arch ppc64le;
+      docker manifest push ${REPOSITORY_USER}/${REPOSITORY_NAME}:stable;
+      echo done'
+    on:
+      tags: true
+      all_branches: true
+      condition: "$TRAVIS_TAG =~ ^v[0-9].*$ && -n $REGISTRY_USER && -n $REGISTRY_PASS && -n $REPOSITORY_NAME && -n $REPOSITORY_USER"
+  # Push images to Dockerhub on merge to master
+  - provider: script
+    on:
+      branch: master
+      condition: "-n $REGISTRY_USER && -n $REGISTRY_PASS && -n $REPOSITORY_NAME && -n $REPOSITORY_USER"
+    script: >
+      bash -c '
+      docker tag ${REPOSITORY_USER}/:latest-amd64 ${REPOSITORY_USER}/${REPOSITORY_NAME}:snapshot;
+      docker tag ${REPOSITORY_USER}/${REPOSITORY_NAME}:latest-amd64 ${REPOSITORY_USER}/${REPOSITORY_NAME}:snapshot-amd64;
+      docker tag ${REPOSITORY_USER}/${REPOSITORY_NAME}:latest-ppc64le ${REPOSITORY_USER}/${REPOSITORY_NAME}:snapshot-ppc64le;
+      docker login -u "$REGISTRY_USER" -p "$REGISTRY_PASS";
+      docker push ${REPOSITORY_USER}/${REPOSITORY_NAME}:snapshot-amd64;
+      docker push ${REPOSITORY_USER}/${REPOSITORY_NAME}:snapshot-ppc64le;
+      docker push ${REPOSITORY_USER}/${REPOSITORY_NAME}:latest-amd64;
+      docker push ${REPOSITORY_USER}/${REPOSITORY_NAME}:latest-ppc64le;
+      docker manifest create ${REPOSITORY_USER}/${REPOSITORY_NAME}:snapshot ${REPOSITORY_USER}/${REPOSITORY_NAME}:snapshot-amd64 ${REPOSITORY_USER}/${REPOSITORY_NAME}:snapshot-ppc64le;
+      docker manifest annotate ${REPOSITORY_USER}/${REPOSITORY_NAME}:snapshot ${REPOSITORY_USER}/${REPOSITORY_NAME}:snapshot-amd64 --arch amd64;
+      docker manifest annotate ${REPOSITORY_USER}/${REPOSITORY_NAME}:snapshot ${REPOSITORY_USER}/${REPOSITORY_NAME}:snapshot-ppc64le --arch ppc64le;
+      docker manifest push ${REPOSITORY_USER}/${REPOSITORY_NAME}:snapshot;
+      docker manifest create ${REPOSITORY_USER}/${REPOSITORY_NAME}:latest ${REPOSITORY_USER}/${REPOSITORY_NAME}:latest-amd64 ${REPOSITORY_USER}/${REPOSITORY_NAME}:latest-ppc64le;
+      docker manifest annotate ${REPOSITORY_USER}/${REPOSITORY_NAME}:latest ${REPOSITORY_USER}/${REPOSITORY_NAME}:latest-amd64 --arch amd64;
+      docker manifest annotate ${REPOSITORY_USER}/${REPOSITORY_NAME}:latest ${REPOSITORY_USER}/${REPOSITORY_NAME}:latest-ppc64le --arch ppc64le;
+      docker manifest push ${REPOSITORY_USER}/${REPOSITORY_NAME}:latest;
+      echo done'

CODE_OF_CONDUCT.md

@@ -0,0 +1,130 @@
+# Multus CNI Contributor Covenant Code of Conduct
+
+## Our Pledge
+
+We as members, contributors, and leaders pledge to make participation in our
+community a harassment-free experience for everyone, regardless of age, body
+size, visible or invisible disability, ethnicity, sex characteristics, gender
+identity and expression, level of experience, education, socio-economic status,
+nationality, personal appearance, race, religion, or sexual identity
+and orientation.
+
+We pledge to act and interact in ways that contribute to an open, welcoming,
+diverse, inclusive, and healthy community.
+
+## Our Standards
+
+Examples of behavior that contributes to a positive environment for our
+community include:
+
+* Demonstrating empathy and kindness toward other people
+* Being respectful of differing opinions, viewpoints, and experiences
+* Giving and gracefully accepting constructive feedback
+* Accepting responsibility and apologizing to those affected by our mistakes,
+  and learning from the experience
+* Focusing on what is best not just for us as individuals, but for the
+  overall community
+
+Examples of unacceptable behavior include:
+
+* The use of sexualized language or imagery, and sexual attention or
+  advances of any kind
+* Trolling, insulting or derogatory comments, and personal or political attacks
+* Public or private harassment
+* Publishing others' private information, such as a physical or email
+  address, without their explicit permission
+* Other conduct which could reasonably be considered inappropriate in a
+  professional setting
+
+## Enforcement Responsibilities
+
+Community leaders are responsible for clarifying and enforcing our standards of
+acceptable behavior and will take appropriate and fair corrective action in
+response to any behavior that they deem inappropriate, threatening, offensive,
+or harmful.
+
+Community leaders have the right and responsibility to remove, edit, or reject
+comments, commits, code, wiki edits, issues, and other contributions that are
+not aligned to this Code of Conduct, and will communicate reasons for moderation
+decisions when appropriate.
+
+## Scope
+
+This Code of Conduct applies within all community spaces, and also applies when
+an individual is officially representing the community in public spaces.
+Examples of representing our community include using an official e-mail address,
+posting via an official social media account, or acting as an appointed
+representative at an online or offline event.
+
+## Enforcement
+
+Instances of abusive, harassing, or otherwise unacceptable behavior may be
+reported to the community leaders responsible for enforcement at
+[The Multus Slack Page](https://intel-corp.herokuapp.com/).
+All complaints will be reviewed and investigated promptly and fairly. Or you
+may specifically contact Doug Smith (dosmith@redhat.com) via email.
+
+All community leaders are obligated to respect the privacy and security of the
+reporter of any incident.
+
+## Enforcement Guidelines
+
+Community leaders will follow these Community Impact Guidelines in determining
+the consequences for any action they deem in violation of this Code of Conduct:
+
+### 1. Correction
+
+**Community Impact**: Use of inappropriate language or other behavior deemed
+unprofessional or unwelcome in the community.
+
+**Consequence**: A private, written warning from community leaders, providing
+clarity around the nature of the violation and an explanation of why the
+behavior was inappropriate. A public apology may be requested.
+
+### 2. Warning
+
+**Community Impact**: A violation through a single incident or series
+of actions.
+
+**Consequence**: A warning with consequences for continued behavior. No
+interaction with the people involved, including unsolicited interaction with
+those enforcing the Code of Conduct, for a specified period of time. This
+includes avoiding interactions in community spaces as well as external channels
+like social media. Violating these terms may lead to a temporary or
+permanent ban.
+
+### 3. Temporary Ban
+
+**Community Impact**: A serious violation of community standards, including
+sustained inappropriate behavior.
+
+**Consequence**: A temporary ban from any sort of interaction or public
+communication with the community for a specified period of time. No public or
+private interaction with the people involved, including unsolicited interaction
+with those enforcing the Code of Conduct, is allowed during this period.
+Violating these terms may lead to a permanent ban.
+
+### 4. Permanent Ban
+
+**Community Impact**: Demonstrating a pattern of violation of community
+standards, including sustained inappropriate behavior,  harassment of an
+individual, or aggression toward or disparagement of classes of individuals.
+
+**Consequence**: A permanent ban from any sort of public interaction within
+the community.
+
+## Attribution
+
+This Code of Conduct is adapted from the [Contributor Covenant][homepage],
+version 2.0, available at
+https://www.contributor-covenant.org/version/2/0/code_of_conduct.html.
+
+Community Impact Guidelines were inspired by [Mozilla's code of conduct
+enforcement ladder](https://github.com/mozilla/diversity).
+
+[homepage]: https://www.contributor-covenant.org
+
+For answers to common questions about this code of conduct, see the FAQ at
+https://www.contributor-covenant.org/faq. Translations are available at
+https://www.contributor-covenant.org/translations.
+

CONTRIBUTING.md

@@ -0,0 +1,57 @@
+# How to Contribute
+
+Multus CNI is [Apache 2.0 licensed](LICENSE) and accepts contributions via GitHub
+pull requests. This document outlines some of the conventions on development
+workflow, commit message formatting, contact points and other resources to make
+it easier to get your contribution accepted.
+
+## Coding Style
+
+Please follows the standard formatting recommendations and language idioms set out
+in [Effective Go](https://golang.org/doc/effective_go.html) and in the
+[Go Code Review Comments wiki](https://github.com/golang/go/wiki/CodeReviewComments).
+
+## Certificate of Origin
+
+In order to get a clear contribution chain of trust we use the [signed-off-by language](https://01.org/community/signed-process)
+used by the Linux kernel project.
+
+## Format of the patch
+
+Beside the signed-off-by footer, we expect each patch to comply with the following format:
+
+```
+Change summary
+
+More detailed explanation of your changes: Why and how.
+Wrap it to 72 characters.
+See [here] (http://chris.beams.io/posts/git-commit/)
+for some more good advices.
+
+Fixes #NUMBER (or URL to the issue)
+
+Signed-off-by: <contributor@foo.com>
+```
+
+For example:
+
+```
+Fix poorly named identifiers
+  
+One identifier, fnname, in func.go was poorly named.  It has been renamed
+to fnName.  Another identifier retval was not needed and has been removed
+entirely.
+
+Fixes #1
+    
+Signed-off-by: Abc Xyz <abc.xyz@intel.com>
+```
+
+## Pull requests
+
+We accept github pull requests.
+
+## Email and Chat
+
+The project uses the Slack chat:
+- Slack: #[Intel-Corp](https://intel-corp.herokuapp.com/) channel on slack

Dockerfile

@@ -0,0 +1,20 @@
+# This Dockerfile is used to build the image available on DockerHub
+FROM centos:centos7 as build
+
+# Add everything
+ADD . /usr/src/multus-cni
+
+ENV INSTALL_PKGS "git golang-1.13.10-0.el7.x86_64"
+RUN rpm --import https://mirror.go-repo.io/centos/RPM-GPG-KEY-GO-REPO && \
+    curl -s https://mirror.go-repo.io/centos/go-repo.repo | tee /etc/yum.repos.d/go-repo.repo && \
+    yum install -y $INSTALL_PKGS && \
+    rpm -V $INSTALL_PKGS && \
+    cd /usr/src/multus-cni && \
+    ./build
+
+FROM centos:centos7
+COPY --from=build /usr/src/multus-cni /usr/src/multus-cni
+WORKDIR /
+
+ADD ./images/entrypoint.sh /
+ENTRYPOINT ["/entrypoint.sh"]

Dockerfile.arm64

@@ -0,0 +1,20 @@
+# This Dockerfile is used to build the image available on DockerHub
+FROM golang:1.13.4 as build
+
+# Add everything
+ADD . /usr/src/multus-cni
+
+ENV GOARCH "arm64"
+ENV GOOS "linux"
+
+RUN  cd /usr/src/multus-cni && \
+     ./build
+
+# build arm64 container
+FROM arm64v8/centos:7
+COPY --from=build /usr/src/multus-cni /usr/src/multus-cni
+
+WORKDIR /
+ADD ./images/entrypoint.sh /
+
+ENTRYPOINT ["/entrypoint.sh"]

Dockerfile.openshift

@@ -0,0 +1,20 @@
+# This dockerfile is specific to building Multus for OpenShift
+FROM openshift/origin-release:golang-1.10 as builder
+
+ADD . /usr/src/multus-cni
+
+WORKDIR /usr/src/multus-cni
+ENV GO111MODULE=off
+RUN ./build
+
+FROM openshift/origin-base
+RUN mkdir -p /usr/src/multus-cni/images && mkdir -p /usr/src/multus-cni/bin
+COPY --from=builder /usr/src/multus-cni/bin/multus /usr/src/multus-cni/bin
+ADD ./images/entrypoint.sh /
+
+LABEL io.k8s.display-name="Multus CNI" \
+      io.k8s.description="This is a component of OpenShift Container Platform and provides a meta CNI plugin." \
+      io.openshift.tags="openshift" \
+      maintainer="Doug Smith <dosmith@redhat.com>"
+
+ENTRYPOINT ["/entrypoint.sh"]

Dockerfile.ppc64le

@@ -0,0 +1,25 @@
+# This Dockerfile is used to build the image available on DockerHub
+FROM centos:centos7 as build
+
+# Add everything
+ADD . /usr/src/multus-cni
+
+ENV GOARCH "ppc64le"
+ENV GOOS "linux"
+
+ENV INSTALL_PKGS "git golang"
+RUN rpm --import https://mirror.go-repo.io/centos/RPM-GPG-KEY-GO-REPO && \
+    curl -s https://mirror.go-repo.io/centos/go-repo.repo | tee /etc/yum.repos.d/go-repo.repo && \
+    yum install -y $INSTALL_PKGS && \
+    rpm -V $INSTALL_PKGS && \
+    cd /usr/src/multus-cni && \
+    ./build
+
+# build ppc container
+FROM ppc64le/centos:latest
+COPY --from=build /usr/src/multus-cni /usr/src/multus-cni
+
+WORKDIR /
+ADD ./images/entrypoint.sh /
+
+ENTRYPOINT ["/entrypoint.sh"]

Godeps/Godeps.json

@@ -1,200 +0,0 @@
-{
-	"ImportPath": "github.com/Intel-Corp/multus-cni",
-	"GoVersion": "go1.6",
-	"GodepVersion": "v74",
-	"Packages": [
-		"./..."
-	],
-	"Deps": [
-		{
-			"ImportPath": "github.com/containernetworking/cni/pkg/invoke",
-			"Comment": "v0.3.0-19-gfbe1a87",
-			"Rev": "fbe1a87250aed33da9a57abbd76b08eb85ec85c1"
-		},
-		{
-			"ImportPath": "github.com/containernetworking/cni/pkg/ip",
-			"Comment": "v0.3.0-19-gfbe1a87",
-			"Rev": "fbe1a87250aed33da9a57abbd76b08eb85ec85c1"
-		},
-		{
-			"ImportPath": "github.com/containernetworking/cni/pkg/ipam",
-			"Comment": "v0.3.0-19-gfbe1a87",
-			"Rev": "fbe1a87250aed33da9a57abbd76b08eb85ec85c1"
-		},
-		{
-			"ImportPath": "github.com/containernetworking/cni/pkg/ns",
-			"Comment": "v0.3.0-19-gfbe1a87",
-			"Rev": "fbe1a87250aed33da9a57abbd76b08eb85ec85c1"
-		},
-		{
-			"ImportPath": "github.com/containernetworking/cni/pkg/skel",
-			"Comment": "v0.3.0-19-gfbe1a87",
-			"Rev": "fbe1a87250aed33da9a57abbd76b08eb85ec85c1"
-		},
-		{
-			"ImportPath": "github.com/containernetworking/cni/pkg/testutils",
-			"Comment": "v0.3.0-19-gfbe1a87",
-			"Rev": "fbe1a87250aed33da9a57abbd76b08eb85ec85c1"
-		},
-		{
-			"ImportPath": "github.com/containernetworking/cni/pkg/types",
-			"Comment": "v0.3.0-19-gfbe1a87",
-			"Rev": "fbe1a87250aed33da9a57abbd76b08eb85ec85c1"
-		},
-		{
-			"ImportPath": "github.com/coreos/go-iptables/iptables",
-			"Rev": "90456be57fcb8185b264b77ce42a9539df42df25"
-		},
-		{
-			"ImportPath": "github.com/onsi/ginkgo",
-			"Comment": "v1.2.0-29-g7f8ab55",
-			"Rev": "7f8ab55aaf3b86885aa55b762e803744d1674700"
-		},
-		{
-			"ImportPath": "github.com/onsi/ginkgo/config",
-			"Comment": "v1.2.0-29-g7f8ab55",
-			"Rev": "7f8ab55aaf3b86885aa55b762e803744d1674700"
-		},
-		{
-			"ImportPath": "github.com/onsi/ginkgo/internal/codelocation",
-			"Comment": "v1.2.0-29-g7f8ab55",
-			"Rev": "7f8ab55aaf3b86885aa55b762e803744d1674700"
-		},
-		{
-			"ImportPath": "github.com/onsi/ginkgo/internal/containernode",
-			"Comment": "v1.2.0-29-g7f8ab55",
-			"Rev": "7f8ab55aaf3b86885aa55b762e803744d1674700"
-		},
-		{
-			"ImportPath": "github.com/onsi/ginkgo/internal/failer",
-			"Comment": "v1.2.0-29-g7f8ab55",
-			"Rev": "7f8ab55aaf3b86885aa55b762e803744d1674700"
-		},
-		{
-			"ImportPath": "github.com/onsi/ginkgo/internal/leafnodes",
-			"Comment": "v1.2.0-29-g7f8ab55",
-			"Rev": "7f8ab55aaf3b86885aa55b762e803744d1674700"
-		},
-		{
-			"ImportPath": "github.com/onsi/ginkgo/internal/remote",
-			"Comment": "v1.2.0-29-g7f8ab55",
-			"Rev": "7f8ab55aaf3b86885aa55b762e803744d1674700"
-		},
-		{
-			"ImportPath": "github.com/onsi/ginkgo/internal/spec",
-			"Comment": "v1.2.0-29-g7f8ab55",
-			"Rev": "7f8ab55aaf3b86885aa55b762e803744d1674700"
-		},
-		{
-			"ImportPath": "github.com/onsi/ginkgo/internal/specrunner",
-			"Comment": "v1.2.0-29-g7f8ab55",
-			"Rev": "7f8ab55aaf3b86885aa55b762e803744d1674700"
-		},
-		{
-			"ImportPath": "github.com/onsi/ginkgo/internal/suite",
-			"Comment": "v1.2.0-29-g7f8ab55",
-			"Rev": "7f8ab55aaf3b86885aa55b762e803744d1674700"
-		},
-		{
-			"ImportPath": "github.com/onsi/ginkgo/internal/testingtproxy",
-			"Comment": "v1.2.0-29-g7f8ab55",
-			"Rev": "7f8ab55aaf3b86885aa55b762e803744d1674700"
-		},
-		{
-			"ImportPath": "github.com/onsi/ginkgo/internal/writer",
-			"Comment": "v1.2.0-29-g7f8ab55",
-			"Rev": "7f8ab55aaf3b86885aa55b762e803744d1674700"
-		},
-		{
-			"ImportPath": "github.com/onsi/ginkgo/reporters",
-			"Comment": "v1.2.0-29-g7f8ab55",
-			"Rev": "7f8ab55aaf3b86885aa55b762e803744d1674700"
-		},
-		{
-			"ImportPath": "github.com/onsi/ginkgo/reporters/stenographer",
-			"Comment": "v1.2.0-29-g7f8ab55",
-			"Rev": "7f8ab55aaf3b86885aa55b762e803744d1674700"
-		},
-		{
-			"ImportPath": "github.com/onsi/ginkgo/types",
-			"Comment": "v1.2.0-29-g7f8ab55",
-			"Rev": "7f8ab55aaf3b86885aa55b762e803744d1674700"
-		},
-		{
-			"ImportPath": "github.com/onsi/gomega",
-			"Comment": "v1.0-71-g2152b45",
-			"Rev": "2152b45fa28a361beba9aab0885972323a444e28"
-		},
-		{
-			"ImportPath": "github.com/onsi/gomega/format",
-			"Comment": "v1.0-71-g2152b45",
-			"Rev": "2152b45fa28a361beba9aab0885972323a444e28"
-		},
-		{
-			"ImportPath": "github.com/onsi/gomega/internal/assertion",
-			"Comment": "v1.0-71-g2152b45",
-			"Rev": "2152b45fa28a361beba9aab0885972323a444e28"
-		},
-		{
-			"ImportPath": "github.com/onsi/gomega/internal/asyncassertion",
-			"Comment": "v1.0-71-g2152b45",
-			"Rev": "2152b45fa28a361beba9aab0885972323a444e28"
-		},
-		{
-			"ImportPath": "github.com/onsi/gomega/internal/oraclematcher",
-			"Comment": "v1.0-71-g2152b45",
-			"Rev": "2152b45fa28a361beba9aab0885972323a444e28"
-		},
-		{
-			"ImportPath": "github.com/onsi/gomega/internal/testingtsupport",
-			"Comment": "v1.0-71-g2152b45",
-			"Rev": "2152b45fa28a361beba9aab0885972323a444e28"
-		},
-		{
-			"ImportPath": "github.com/onsi/gomega/matchers",
-			"Comment": "v1.0-71-g2152b45",
-			"Rev": "2152b45fa28a361beba9aab0885972323a444e28"
-		},
-		{
-			"ImportPath": "github.com/onsi/gomega/matchers/support/goraph/bipartitegraph",
-			"Comment": "v1.0-71-g2152b45",
-			"Rev": "2152b45fa28a361beba9aab0885972323a444e28"
-		},
-		{
-			"ImportPath": "github.com/onsi/gomega/matchers/support/goraph/edge",
-			"Comment": "v1.0-71-g2152b45",
-			"Rev": "2152b45fa28a361beba9aab0885972323a444e28"
-		},
-		{
-			"ImportPath": "github.com/onsi/gomega/matchers/support/goraph/node",
-			"Comment": "v1.0-71-g2152b45",
-			"Rev": "2152b45fa28a361beba9aab0885972323a444e28"
-		},
-		{
-			"ImportPath": "github.com/onsi/gomega/matchers/support/goraph/util",
-			"Comment": "v1.0-71-g2152b45",
-			"Rev": "2152b45fa28a361beba9aab0885972323a444e28"
-		},
-		{
-			"ImportPath": "github.com/onsi/gomega/types",
-			"Comment": "v1.0-71-g2152b45",
-			"Rev": "2152b45fa28a361beba9aab0885972323a444e28"
-		},
-		{
-			"ImportPath": "github.com/vishvananda/netlink",
-			"Rev": "77483a0e697ebcc2584461e9624b611767e7d203"
-		},
-		{
-			"ImportPath": "github.com/vishvananda/netlink/nl",
-			"Rev": "77483a0e697ebcc2584461e9624b611767e7d203"
-		},
-		{
-			"ImportPath": "github.com/vishvananda/netns",
-			"Rev": "8ba1072b58e0c2a240eb5f6120165c7776c3e7b8"
-		},
-		{
-			"ImportPath": "golang.org/x/sys/unix",
-			"Rev": "e11762ca30adc5b39fdbfd8c4250dabeb8e456d3"
-		}
-	]
-}

Godeps/Readme

@@ -1,5 +0,0 @@
-This directory tree is generated automatically by godep.
-
-Please do not edit.
-
-See https://github.com/tools/godep for more information.

README.md

@@ -1,164 +1,51 @@
-![multus-cni Logo](https://github.com/Intel-Corp/multus-cni/blob/master/doc/images/Multus.png)
-# MULTUS CNI plugin
+# Multus-CNI
 
-- *Multus* is the latin word for “Multi”
+![multus-cni Logo](https://github.com/intel/multus-cni/blob/master/doc/images/Multus.png)
 
-- As the name suggests, it acts as the Multi plugin in Kubernetes and provides the Multi interface support in a pod
+[![Travis CI](https://travis-ci.org/intel/multus-cni.svg?branch=master)](https://travis-ci.org/intel/multus-cni/builds)[![Go Report Card](https://goreportcard.com/badge/github.com/intel/multus-cni)](https://goreportcard.com/report/github.com/intel/multus-cni)[![Coverage Status](https://coveralls.io/repos/github/intel/multus-cni/badge.svg)](https://coveralls.io/github/intel/multus-cni)
 
-- It is generic to run with other plugins like ptp, local-host, calico and flannel, with different IPAM and networks. 
+Multus CNI enables attaching multiple network interfaces to pods in Kubernetes.
 
-- It is a contact between the container runtime and other plugins, and it doesn't have any of its own net configuration, it calls other plugins like flannel/calico to do the real net conf job. 
+## How it works
 
-- Multus reuses the concept of invoking the delegates in flannel, it groups the multi plugins into delegates and invoke each other in sequential order, according to the JSON scheme in the cni configuration.
+Multus CNI is a container network interface (CNI) plugin for Kubernetes that enables attaching multiple network interfaces to pods. Typically, in Kubernetes each pod only has one network interface (apart from a loopback) -- with Multus you can create a multi-homed pod that has multiple interfaces. This is accomplished by Multus acting as a "meta-plugin", a CNI plugin that can call multiple other CNI plugins.
 
-- No. of plugins supported is dependent upon the number of delegates in the conf file.
+Multus CNI follows the [Kubernetes Network Custom Resource Definition De-facto Standard](https://docs.google.com/document/d/1Ny03h6IDVy_e_vmElOqR7UdTPAG_RNydhVE1Kx54kFQ/edit) to provide a standardized method by which to specify the configurations for additional network interfaces. This standard is put forward by the Kubernetes [Network Plumbing Working Group](https://docs.google.com/document/d/1oE93V3SgOGWJ4O1zeD1UmpeToa0ZiiO6LqRAmZBPFWM/edit).
 
-- Master plugin invokes "eth0" interface in the pod, rest of plugins(Mininon plugins eg: sriov,ipam) invoke interfaces as "net0", "net1".. "netn"
+Multus is one of the projects in the [Baremetal Container Experience kit](https://networkbuilders.intel.com/network-technologies/container-experience-kits)
 
-- The "masterplugin" is the only net conf option of multus cni, it identifies the primary network. The default route will point to the primary network 
+### Multi-Homed pod
 
-Please read [CNI](https://github.com/containernetworking/cni) for more information on container networking.
+Here's an illustration of the network interfaces attached to a pod, as provisioned by Multus CNI. The diagram shows the pod with three interfaces: `eth0`, `net0` and `net1`. `eth0` connects kubernetes cluster network to connect with kubernetes server/services (e.g. kubernetes api-server, kubelet and so on). `net0` and `net1` are additional network attachments and connect to other networks by using [other CNI plugins](https://kubernetes.io/docs/concepts/extend-kubernetes/compute-storage-net/network-plugins/) (e.g. vlan/vxlan/ptp).
 
-## Multi-Homed pod
-<p align="center">
-   <img src="doc/images/multus_cni_pod.png" width="1008" />
-</p>
+![multus-pod-image](doc/images/multus-pod-image.svg)
 
-## Build
+## Quickstart Installation Guide
 
-This plugin requires Go 1.5+ to build.
+The quickstart installation method for Multus requires that you have first installed a Kubernetes CNI plugin to serve as your pod-to-pod network, which we refer to as your "default network" (a network interface that every pod will be creatd with). Each network attachment created by Multus will be in addition to this default network interface. For more detail on installing a default network CNI plugins, refer to our [quick-start guide](doc/quickstart.md).
 
-Go 1.5 users will need to set `GO15VENDOREXPERIMENT=1` to get vendored dependencies. This flag is set by default in 1.6.
+Clone this GitHub repository, we'll apply a daemonset which installs Multus using to `kubectl` from this repo. From the root directory of the clone, apply the daemonset YAML file:
 
 ```
-#./build
-```
-## Work flow
-<p align="center">
-   <img src="doc/images/workflow.png" width="1008" />
-</p>
-## Network configuration reference
-
-* `name` (string, required): the name of the network
-* `type` (string, required): "multus"
-* `delegates` (([]map,required): number of delegate details in the Multus
-* `masterplugin` (bool,required): master plugin to report back the IP address and DNS to the container
-
-## Usage
-
-Given the following network configuration:
-
-```
-# tee /etc/cni/net.d/multus-cni.conf <<-'EOF'
-{
-    "name": "multus-demo-network",
-    "type": "multus",
-    "delegates": [
-        {
-                "type": "sriov",
-                #part of sriov plugin conf
-                "if0": "enp12s0f0", 
-                "ipam": {
-                        "type": "host-local",
-                        "subnet": "10.56.217.0/24",
-                        "rangeStart": "10.56.217.131",
-                        "rangeEnd": "10.56.217.190",
-                        "routes": [
-                                { "dst": "0.0.0.0/0" }
-                        ],
-                        "gateway": "10.56.217.1"
-                }
-        },
-        {
-                "type": "ptp",
-                "ipam": {
-                        "type": "host-local",
-                        "subnet": "10.168.1.0/24",
-                        "rangeStart": "10.168.1.11",
-                        "rangeEnd": "10.168.1.20",
-                        "routes": [
-                                { "dst": "0.0.0.0/0" }
-                        ],
-                        "gateway": "10.168.1.1"
-                }
-        },
-        {
-                "type": "flannel",
-                "masterplugin": true,
-                "delegate": {
-                        "isDefaultGateway": true
-                }
-        }
-    ]
-}
-EOF
-
-```
-## Testing the Multus CNI with docker
-Make sure that the multus, [sriov](https://github.com/Intel-Corp/sriov-cni), [flannel](https://github.com/containernetworking/cni/blob/master/Documentation/flannel.md), and [ptp](https://github.com/containernetworking/cni/blob/master/Documentation/ptp.md) binaries are in the `/opt/cni/bin` directories and follow the steps as mention in the [CNI](https://github.com/containernetworking/cni/#running-a-docker-container-with-network-namespace-set-up-by-cni-plugins)
-
-## Testing the Multus CNI with Kubernetes
-Refer the Kubernetes User Guide and network plugin
-* [Single Node](https://kubernetes.io/docs/getting-started-guides/fedora/fedora_manual_config/)
-* [Multi Node](https://kubernetes.io/docs/getting-started-guides/fedora/flannel_multi_node_cluster/)
-* [Network Plugin](https://kubernetes.io/docs/admin/network-plugins/)
-
-Kubelet must be configured to run with the CNI `--network-plugin`, with the following configuration information. 
-Edit `/etc/default/kubelet` file and add `KUBELET_OPTS`:
-```
-KUBELET_OPTS="...
---network-plugin-dir=/etc/cni/net.d
---network-plugin=cni
-"
-```
-Restart the kubelet
-```
-# systemctl restart kubelet.service
-```
-### Launching workloads in Kubernetes
-Launch the workload using yaml file in the kubernetes master, with above configuration in the multus CNI, each pod should have multiple interfaces.
-> Note:	To verify whether Multus CNI plugin is working fine create a pod containing one “busybox” container and execute “ip link” command to check if interfaces management follows configuration.
-
-1. Create “multus-test.yaml” file containing below configuration. Created pod will consist of one “busybox” container running “top” command.
-```
-apiVersion: v1
-kind: Pod
-metadata:
-  name: multus-test
-spec:  # specification of the pod's contents
-  restartPolicy: Never
-  containers:
-  - name: test1
-    image: "busybox"
-    command: ["top"]
-    stdin: true
-    tty: true
-
-```
-2. Create pod using command:
-```
-# kubectl create -f multus-test.yaml
-pod "multus-test" created
-```
-3. Run “ip link” command inside the container:
-```
-# 1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue qlen 1
-    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
-3: eth0@if41: <BROADCAST,MULTICAST,UP,LOWER_UP,M-DOWN> mtu 1500 qdisc noqueue
-    link/ether 26:52:6b:d8:44:2d brd ff:ff:ff:ff:ff:ff
-20: net0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq qlen 1000
-    link/ether f6:fb:21:4f:1d:63 brd ff:ff:ff:ff:ff:ff
-21: net1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq qlen 1000
-    link/ether 76:13:b1:60:00:00 brd ff:ff:ff:ff:ff:ff 
+$ cat ./images/multus-daemonset.yml | kubectl apply -f -
 ```
 
-Interface name | Description
------------- | -------------
-lo | loopback
-eth0@if41 | Flannel network tap interface
-net0 | VF assigned to the container by [SR_IOV CNI](https://github.com/Intel-Corp/sriov-cni) plugin
-net1 | ptp localhost interface
+This will configure your systems to be ready to use Multus CNI, but, to get started with adding additional interfaces to your pods, refer to our complete [quick-start guide](doc/quickstart.md)
 
-### Contacts
-For any questions about Multus CNI, please reach out on github issue or feel free to contact the developer @kural and @dmzoneill in our [Intel-Corp Slack](https://intel-corp.herokuapp.com/)
+## Additional installation Options
 
+- Install via daemonset using the quick-start guide, above.
+- Download binaries from [release page](https://github.com/intel/multus-cni/releases)
+- By Docker image from [Docker Hub](https://hub.docker.com/r/nfvpe/multus/tags/)
+- Or, roll-your-own and build from source
+  - See [Development](doc/development.md)
+
+## Comprehensive Documentation
+
+- [How to use](doc/how-to-use.md)
+- [Configuration](doc/configuration.md)
+- [Development](doc/development.md)
+
+## Contact Us
+
+For any questions about Multus CNI, feel free to ask a question in #general in the [Intel-Corp Slack](https://intel-corp.herokuapp.com/), or open up a GitHub issue.

build

@@ -1,17 +1,54 @@
 #!/usr/bin/env bash
 set -e
 
-ORG_PATH="github.com/Intel-Corp"
-REPO_PATH="${ORG_PATH}/multus-cni"
+DEST_DIR="bin"
 
-if [ ! -h gopath/src/${REPO_PATH} ]; then
-	mkdir -p gopath/src/${ORG_PATH}
-	ln -s ../../../.. gopath/src/${REPO_PATH} || exit 255
+if [ ! -d ${DEST_DIR} ]; then
+	mkdir ${DEST_DIR}
 fi
 
-export GO15VENDOREXPERIMENT=1
-export GOBIN=${PWD}/bin
-export GOPATH=${PWD}/gopath
+# Add version/commit/date into binary
+# In case of TravisCI, need to check error code of 'git describe'.
+if [ -z "$VERSION" ]; then
+	set +e
+	git describe --tags --abbrev=0 > /dev/null 2>&1
+	if [ "$?" != "0" ]; then
+		VERSION="master"
+	else
+		VERSION=$(git describe --tags --abbrev=0)
+	fi
+	set -e
+fi
+DATE=$(date -u -d "@${SOURCE_DATE_EPOCH:-$(date +%s)}" --iso-8601=seconds)
+COMMIT=${COMMIT:-$(git rev-parse --verify HEAD)}
+LDFLAGS="-X main.version=${VERSION:-master} -X main.commit=${COMMIT} -X main.date=${DATE}"
+export CGO_ENABLED=0
 
-echo "Building plugins"
-go install "$@" ${REPO_PATH}/multus
+# this if... will be removed when gomodules goes default
+if [ "$GO111MODULE" == "off" ]; then
+	echo "Building plugin without go module"
+	echo "Warning: this will be deprecated in near future so please use go modules!"
+
+	ORG_PATH="gopkg.in/intel"
+	REPO_PATH="${ORG_PATH}/multus-cni.v3"
+
+	if [ ! -h gopath/src/${REPO_PATH} ]; then
+		mkdir -p gopath/src/${ORG_PATH}
+		ln -s ../../../.. gopath/src/${REPO_PATH} || exit 255
+	fi
+
+	export GO15VENDOREXPERIMENT=1
+	export GOBIN=${PWD}/bin
+	export GOPATH=${PWD}/gopath
+	go install -tags no_openssl -ldflags "${LDFLAGS}" "$@" ${REPO_PATH}/multus
+else
+	# build with go modules
+	export GO111MODULE=on
+	BUILD_ARGS=(-o ${DEST_DIR}/multus -tags no_openssl)
+	if [ -n "$MODMODE" ]; then
+		BUILD_ARGS+=(-mod "$MODMODE")
+	fi
+
+	echo "Building plugins"
+	go build ${BUILD_ARGS[*]} -ldflags "${LDFLAGS}" "$@" ./multus
+fi

checkpoint/checkpoint.go

@@ -0,0 +1,110 @@
+// Copyright (c) 2018 Intel Corporation
+//
+// Licensed under the Apache License, Version 2.0 (the "License");
+// you may not use this file except in compliance with the License.
+// You may obtain a copy of the License at
+//
+//     http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+//
+
+package checkpoint
+
+import (
+	"encoding/json"
+	"io/ioutil"
+
+	"gopkg.in/intel/multus-cni.v3/logging"
+	"gopkg.in/intel/multus-cni.v3/types"
+	v1 "k8s.io/api/core/v1"
+)
+
+const (
+	checkPointfile = "/var/lib/kubelet/device-plugins/kubelet_internal_checkpoint"
+)
+
+// PodDevicesEntry maps PodUID, resource name and allocated device id
+type PodDevicesEntry struct {
+	PodUID        string
+	ContainerName string
+	ResourceName  string
+	DeviceIDs     []string
+	AllocResp     []byte
+}
+
+type checkpointData struct {
+	PodDeviceEntries  []PodDevicesEntry
+	RegisteredDevices map[string][]string
+}
+
+type checkpointFileData struct {
+	Data     checkpointData
+	Checksum uint64
+}
+
+type checkpoint struct {
+	fileName   string
+	podEntires []PodDevicesEntry
+}
+
+// GetCheckpoint returns an instance of Checkpoint
+func GetCheckpoint() (types.ResourceClient, error) {
+	logging.Debugf("GetCheckpoint(): invoked")
+	return getCheckpoint(checkPointfile)
+}
+
+func getCheckpoint(filePath string) (types.ResourceClient, error) {
+	cp := &checkpoint{fileName: filePath}
+	err := cp.getPodEntries()
+	if err != nil {
+		return nil, err
+	}
+	logging.Debugf("getCheckpoint: created checkpoint instance with file: %s", filePath)
+	return cp, nil
+}
+
+// getPodEntries gets all Pod device allocation entries from checkpoint file
+func (cp *checkpoint) getPodEntries() error {
+
+	cpd := &checkpointFileData{}
+	rawBytes, err := ioutil.ReadFile(cp.fileName)
+	if err != nil {
+		return logging.Errorf("getPodEntries: error reading file %s\n%v\n", checkPointfile, err)
+	}
+
+	if err = json.Unmarshal(rawBytes, cpd); err != nil {
+		return logging.Errorf("getPodEntries: error unmarshalling raw bytes %v", err)
+	}
+
+	cp.podEntires = cpd.Data.PodDeviceEntries
+	logging.Debugf("getPodEntries: podEntires %+v", cp.podEntires)
+	return nil
+}
+
+// GetComputeDeviceMap returns an instance of a map of ResourceInfo
+func (cp *checkpoint) GetPodResourceMap(pod *v1.Pod) (map[string]*types.ResourceInfo, error) {
+	podID := string(pod.UID)
+	resourceMap := make(map[string]*types.ResourceInfo)
+
+	if podID == "" {
+		return nil, logging.Errorf("GetPodResourceMap: invalid Pod cannot be empty")
+	}
+	for _, pod := range cp.podEntires {
+		if pod.PodUID == podID {
+			entry, ok := resourceMap[pod.ResourceName]
+			if ok {
+				// already exists; append to it
+				entry.DeviceIDs = append(entry.DeviceIDs, pod.DeviceIDs...)
+			} else {
+				// new entry
+				resourceMap[pod.ResourceName] = &types.ResourceInfo{DeviceIDs: pod.DeviceIDs}
+			}
+		}
+	}
+	return resourceMap, nil
+}

checkpoint/checkpoint_test.go

@@ -0,0 +1,203 @@
+package checkpoint
+
+import (
+	"fmt"
+	"os"
+
+	. "github.com/onsi/ginkgo"
+	. "github.com/onsi/gomega"
+
+	"io/ioutil"
+	"testing"
+
+	"gopkg.in/intel/multus-cni.v3/types"
+	v1 "k8s.io/api/core/v1"
+	metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
+	k8sTypes "k8s.io/apimachinery/pkg/types"
+)
+
+const (
+	fakeTempFile = "/tmp/kubelet_internal_checkpoint"
+)
+
+type fakeCheckpoint struct {
+	fileName string
+}
+
+func (fc *fakeCheckpoint) WriteToFile(inBytes []byte) error {
+	return ioutil.WriteFile(fc.fileName, inBytes, 0600)
+}
+
+func (fc *fakeCheckpoint) DeleteFile() error {
+	return os.Remove(fc.fileName)
+}
+
+func TestCheckpoint(t *testing.T) {
+	RegisterFailHandler(Fail)
+	RunSpecs(t, "Checkpoint")
+}
+
+var _ = BeforeSuite(func() {
+	sampleData := `{
+		"Data": {
+			"PodDeviceEntries": [
+			{
+				"PodUID": "970a395d-bb3b-11e8-89df-408d5c537d23",
+				"ContainerName": "appcntr1",
+				"ResourceName": "intel.com/sriov_net_A",
+				"DeviceIDs": [
+				"0000:03:02.3",
+				"0000:03:02.0"
+				],
+				"AllocResp": "CikKC3NyaW92X25ldF9BEhogMDAwMDowMzowMi4zIDAwMDA6MDM6MDIuMA=="
+			}
+			],
+			"RegisteredDevices": {
+			"intel.com/sriov_net_A": [
+				"0000:03:02.1",
+				"0000:03:02.2",
+				"0000:03:02.3",
+				"0000:03:02.0"
+			],
+			"intel.com/sriov_net_B": [
+				"0000:03:06.3",
+				"0000:03:06.0",
+				"0000:03:06.1",
+				"0000:03:06.2"
+			]
+			}
+		},
+		"Checksum": 229855270
+		}`
+
+	fakeCheckpoint := &fakeCheckpoint{fileName: fakeTempFile}
+	err := fakeCheckpoint.WriteToFile([]byte(sampleData))
+	Expect(err).NotTo(HaveOccurred())
+})
+
+var _ = Describe("Kubelet checkpoint data read operations", func() {
+	Context("Using /tmp/kubelet_internal_checkpoint file", func() {
+		var (
+			cp            types.ResourceClient
+			err           error
+			resourceMap   map[string]*types.ResourceInfo
+			resourceInfo  *types.ResourceInfo
+			resourceAnnot = "intel.com/sriov_net_A"
+		)
+
+		It("should get a Checkpoint instance from file", func() {
+			cp, err = getCheckpoint(fakeTempFile)
+			Expect(err).NotTo(HaveOccurred())
+		})
+
+		It("should return a ResourceMap instance", func() {
+			podUID := k8sTypes.UID("970a395d-bb3b-11e8-89df-408d5c537d23")
+			fakePod := &v1.Pod{
+				ObjectMeta: metav1.ObjectMeta{
+					Name:      "fakePod",
+					Namespace: "podNamespace",
+					UID:       podUID,
+				},
+			}
+			rmap, err := cp.GetPodResourceMap(fakePod)
+			Expect(err).NotTo(HaveOccurred())
+			Expect(rmap).NotTo(BeEmpty())
+			resourceMap = rmap
+		})
+
+		It("resourceMap should have value for \"intel.com/sriov_net_A\"", func() {
+			rInfo, ok := resourceMap[resourceAnnot]
+			Expect(ok).To(BeTrue())
+			resourceInfo = rInfo
+		})
+
+		It("should have 2 deviceIDs", func() {
+			Expect(len(resourceInfo.DeviceIDs)).To(BeEquivalentTo(2))
+		})
+
+		It("should have \"0000:03:02.3\" in deviceIDs[0]", func() {
+			Expect(resourceInfo.DeviceIDs[0]).To(BeEquivalentTo("0000:03:02.3"))
+		})
+
+		It("should have \"0000:03:02.0\" in deviceIDs[1]", func() {
+			Expect(resourceInfo.DeviceIDs[1]).To(BeEquivalentTo("0000:03:02.0"))
+		})
+	})
+
+	Context("Using faulty or incompatible information", func() {
+		var (
+			cp  types.ResourceClient
+			err error
+		)
+
+		It("should not get a Checkpoint instance from file given bad filepath", func() {
+			_, err = getCheckpoint("invalid/file/path")
+			Expect(err).To(HaveOccurred())
+		})
+
+		It("should not get a Checkpoint instance from file given bad json", func() {
+			sampleData := `{
+				"Data": {
+					"PodDeviceEntries": [
+					{
+						"PodUID": "970a395d-bb3b-11e8-89df-408d5c537d23",
+						"ContainerName": "appcntr1",
+						"ResourceName": "intel.com/sriov_net_A",
+						"DeviceIDs": [
+						"0000:03:02.3",
+						"0000:03:02.0"
+						],
+						"AllocResp": "CikKC3NyaW92X25ldF9BEhogMDAwMDowMzowMi4zIDAwMDA6MDM6MDIuMA=="
+					}
+					],
+					"RegisteredDevices": {
+					"intel.com/sriov_net_A": [
+						"0000:03:02.1",
+						"0000:03:02.2",
+						"0000:03:02.3",
+						"0000:03:02.0"
+					],
+					"intel.com/sriov_net_B": [
+						"0000:03:06.3",
+						"0000:03:06.0",
+						"0000:03:06.1",
+						"0000:03:06.2"
+					]
+					}
+				},
+				"Checksum": 229855270
+				}`
+
+			//missing a close bracket
+			badSampleData := `BAD BAD DATA`
+
+			fakeCheckpoint := &fakeCheckpoint{fileName: fakeTempFile}
+			fakeCheckpoint.WriteToFile([]byte(badSampleData))
+			_, err = getCheckpoint(fakeTempFile)
+			Expect(err).To(HaveOccurred())
+			fakeCheckpoint.WriteToFile([]byte(sampleData))
+		})
+
+		It("should not return a ResourceMap instance", func() {
+			cp, err = getCheckpoint(fakeTempFile)
+			podUID := k8sTypes.UID("")
+			fakePod := &v1.Pod{
+				ObjectMeta: metav1.ObjectMeta{
+					Name:      "fakePod",
+					Namespace: "podNamespace",
+					UID:       podUID,
+				},
+			}
+			fmt.Println("fakePod-podID: ", fakePod.UID)
+			rmap, err := cp.GetPodResourceMap(fakePod)
+			Expect(err).To(HaveOccurred())
+			Expect(rmap).To(BeEmpty())
+		})
+	})
+})
+
+var _ = AfterSuite(func() {
+	fakeCheckpoint := &fakeCheckpoint{fileName: fakeTempFile}
+	err := fakeCheckpoint.DeleteFile()
+	Expect(err).NotTo(HaveOccurred())
+})

doc/configuration.md

@@ -0,0 +1,279 @@
+## Multus-cni Configuration Reference
+
+Following is the example of multus config file, in `/etc/cni/net.d/`.
+(`"Note1"` and `"Note2"` are just comments, so you can remove them at your configuration)
+
+```
+{
+    "name": "node-cni-network",
+    "type": "multus",
+    "kubeconfig": "/etc/kubernetes/node-kubeconfig.yaml",
+    "confDir": "/etc/cni/multus/net.d",
+    "cniDir": "/var/lib/cni/multus",
+    "binDir": "/opt/cni/bin",
+    "logFile": "/var/log/multus.log",
+    "logLevel": "debug",
+    "capabilities": {
+        "portMappings": true
+    },    
+    "readinessindicatorfile": "",
+    "namespaceIsolation": false,
+"Note1":"NOTE: you can set clusterNetwork+defaultNetworks OR delegates!!",
+    "clusterNetwork": "defaultCRD",
+    "defaultNetworks": ["sidecarCRD", "flannel"],
+    "systemNamespaces": ["kube-system", "admin"],
+    "multusNamespace": "kube-system",
+"Note2":"NOTE: If you use clusterNetwork/defaultNetworks, delegates is ignored",
+    "delegates": [{
+        "type": "weave-net",
+        "hairpinMode": true
+    }, {
+        "type": "macvlan",
+        ... (snip)
+    }]
+}
+```
+
+* `name` (string, required): the name of the network
+* `type` (string, required): "multus"
+* `confDir` (string, optional): directory for CNI config file that multus reads. default `/etc/cni/multus/net.d`
+* `cniDir` (string, optional): Multus CNI data directory, default `/var/lib/cni/multus`
+* `binDir` (string, optional): additional directory for CNI plugins which multus calls, in addition to the default (the default is typically set to `/opt/cni/bin`)
+* `kubeconfig` (string, optional): kubeconfig file for the out of cluster communication with kube-apiserver. See the example [kubeconfig](https://github.com/intel/multus-cni/blob/master/doc/node-kubeconfig.yaml). If you would like to use CRD (i.e. network attachment definition), this is required
+* `logFile` (string, optional): file path for log file. multus puts log in given file
+* `logLevel` (string, optional): logging level ("debug", "error", "verbose", or "panic")
+* `namespaceIsolation` (boolean, optional): Enables a security feature where pods are only allowed to access `NetworkAttachmentDefinitions` in the namespace where the pod resides. Defaults to false.
+* `capabilities` ({}list, optional): [capabilities](https://github.com/containernetworking/cni/blob/master/CONVENTIONS.md#dynamic-plugin-specific-fields-capabilities--runtime-configuration) supported by at least one of the delegates. (NOTE: Multus only supports portMappings capability for now). See the [example](https://github.com/intel/multus-cni/blob/master/examples/multus-ptp-portmap.conf).
+* `readinessindicatorfile`: The path to a file whose existance denotes that the default network is ready
+
+User should chose following parameters combination (`clusterNetwork`+`defaultNetworks` or `delegates`):
+
+* `clusterNetwork` (string, required): default CNI network for pods, used in kubernetes cluster (Pod IP and so on): name of network-attachment-definition, CNI json file name (without extention, .conf/.conflist) or directory for CNI config file
+* `defaultNetworks` ([]string, required): default CNI network attachment: name of network-attachment-definition, CNI json file name (without extention, .conf/.conflist) or directory for CNI config file
+* `systemNamespaces` ([]string, optional): list of namespaces for Kubernetes system (namespaces listed here will not have `defaultNetworks` added)
+* `multusNamespace` (string, optional): namespace for `clusterNetwork`/`defaultNetworks`
+* `delegates` ([]map,required): number of delegate details in the Multus
+
+### Network selection flow of clusterNetwork/defaultNetworks
+
+Multus will find network for clusterNetwork/defaultNetworks as following sequences:
+
+1. CRD object for given network name, in 'kube-system' namespace
+1. CNI json config file in `confDir`. Given name should be without extention, like .conf/.conflist. (e.g. "test" for "test.conf")
+1. Directory for CNI json config file. Multus will find alphabetically first file for the network
+1. Multus failed to find network. Multus raise error message
+
+## Miscellaneous config
+
+### Default Network Readiness Indicator
+
+You may wish for your "default network" (that is, the CNI plugin & its configuration you specify as your default delegate) to become ready before you attach networks with Multus. This is disabled by default and not used unless you add the readiness check option(s) to your CNI configuration file.
+
+For example, if you use Flannel as a default network, the recommended method for Flannel to be installed is via a daemonset that also drops a configuration file in `/etc/cni/net.d/`. This may apply to other plugins that place that configuration file upon their readiness, hence, Multus uses their configuration filename as a semaphore and optionally waits to attach networks to pods until that file exists.
+
+In this manner, you may prevent pods from crash looping, and instead wait for that default network to be ready.
+
+Only one option is necessary to configure this functionality:
+
+* `readinessindicatorfile`: The path to a file whose existance denotes that the default network is ready.
+
+*NOTE*: If `readinessindicatorfile` is unset, or is an empty string, this functionality will be disabled, and is disabled by default.
+
+
+### Logging
+
+You may wish to enable some enhanced logging for Multus, especially during the process where you're configuring Multus and need to understand what is or isn't working with your particular configuration.
+
+Multus will always log via `STDERR`, which is the standard method by which CNI plugins communicate errors, and these errors are logged by the Kubelet. This method is always enabled.
+
+#### Writing to a Log File
+
+Optionally, you may have Multus log to a file on the filesystem. This file will be written locally on each node where Multus is executed. You may configure this via the `LogFile` option in the CNI configuration. By default this additional logging to a flat file is disabled.
+
+For example in your CNI configuration, you may set:
+
+```
+    "LogFile": "/var/log/multus.log",
+```
+
+#### Logging Level
+
+The default logging level is set as `panic` -- this will log only the most critical errors, and is the least verbose logging level.
+
+The available logging level values, in decreasing order of verbosity are:
+
+* `debug`
+* `verbose`
+* `error`
+* `panic`
+
+You may configure the logging level by using the `LogLevel` option in your CNI configuration. For example:
+
+```
+    "LogLevel": "debug",
+```
+
+### Namespace Isolation
+
+The functionality provided by the `namespaceIsolation` configuration option enables a mode where Multus only allows pods to access custom resources (the `NetworkAttachmentDefinitions`) within the namespace where that pod resides. In other words, the `NetworkAttachmentDefinitions` are isolated to usage within the namespace in which they're created. 
+
+**NOTE**: The default namespace is special in this scenario. Even with namespace isolation enabled, any pod, in any namespace is allowed to refer to `NetworkAttachmentDefinitions` in the default namespace. This allows you to create commonly used unprivileged `NetworkAttachmentDefinitions` without having to put them in all namespaces. For example, if you had a `NetworkAttachmentDefinition` named `foo` the default namespace, you may reference it in an annotation with: `default/foo`.
+
+For example, if a pod is created in the namespace called `development`, Multus will not allow networks to be attached when defined by custom resources created in a different namespace, say in the `default` network.
+
+Consider the situation where you have a system that has users of different privilege levels -- as an example, a platform which has two administrators: a Senior Administrator and a Junior Administrator. The Senior Administrator may have access to all namespaces, and some network configurations as used by Multus are considered to be privileged in that they allow access to some protected resources available on the network. However, the Junior Administrator has access to only a subset of namespaces, and therefore it should be assumed that the Junior Administrator cannot create pods in their limited subset of namespaces. The `namespaceIsolation` feature provides for this isolation, allowing pods created in given namespaces to only access custom resources in the same namespace as the pod.
+
+Namespace Isolation is disabled by default.
+
+#### Configuration example
+
+```
+  "namespaceIsolation": true,
+```
+
+#### Usage example
+
+Let's setup an example where we:
+
+* Create a custom resource in a namespace called `privileged`
+* Create a pod in a namespace called `development`, and have annotations that reference a custom resource in the `privileged` namespace. The creation of this pod should be disallowed by Multus (as we'll have the use of the custom resources limited only to those custom resources created within the same namespace as the pod).
+
+Given the above scenario with a Junior & Senior Administrator. You may assume that the Senior Administrator has access to all namespaces, whereas the Junior Administrator has access only to the `development` namespace.
+
+Firstly, we show that we have a number of namespaces available:
+
+```
+# List the available namespaces
+[user@kube-master ~]$ kubectl get namespaces
+NAME          STATUS   AGE
+default       Active   7h27m
+development   Active   3h
+kube-public   Active   7h27m
+kube-system   Active   7h27m
+privileged    Active   4s
+```
+
+We'll create a `NetworkAttachmentDefinition` in the `privileged` namespace.
+
+```
+# Show the network attachment definition we're creating.
+[user@kube-master ~]$ cat cr.yml
+apiVersion: "k8s.cni.cncf.io/v1"
+kind: NetworkAttachmentDefinition
+metadata:
+  name: macvlan-conf
+spec:
+  config: '{
+      "cniVersion": "0.3.0",
+      "type": "macvlan",
+      "master": "eth0",
+      "mode": "bridge",
+      "ipam": {
+        "type": "host-local",
+        "subnet": "192.168.1.0/24",
+        "rangeStart": "192.168.1.200",
+        "rangeEnd": "192.168.1.216",
+        "routes": [
+          { "dst": "0.0.0.0/0" }
+        ],
+        "gateway": "192.168.1.1"
+      }
+    }'
+
+# Create that network attachment definition in the privileged namespace
+[user@kube-master ~]$ kubectl create -f cr.yml -n privileged
+networkattachmentdefinition.k8s.cni.cncf.io/macvlan-conf created
+
+# List the available network attachment definitions in the privileged namespace.
+[user@kube-master ~]$ kubectl get networkattachmentdefinition.k8s.cni.cncf.io -n privileged
+NAME           AGE
+macvlan-conf   11s
+```
+
+Next, we'll create a pod with an annotation that references the privileged namespace. Pay particular attention to the annotation that reads `k8s.v1.cni.cncf.io/networks: privileged/macvlan-conf` -- where it contains a reference to a `namespace/configuration-name` formatted network attachment name. In this case referring to the `macvlan-conf` in the namespace called `privileged`.
+
+```
+# Show the yaml for a pod.
+[user@kube-master ~]$ cat example.pod.yml
+apiVersion: v1
+kind: Pod
+metadata:
+  name: samplepod
+  annotations:
+    k8s.v1.cni.cncf.io/networks: privileged/macvlan-conf
+spec:
+  containers:
+  - name: samplepod
+    command: ["/bin/bash", "-c", "sleep 2000000000000"]
+    image: dougbtv/centos-network
+
+# Create that pod.
+[user@kube-master ~]$ kubectl create -f example.pod.yml -n development
+pod/samplepod created
+```
+
+You'll note that pod fails to spawn successfully. If you check the Multus logs, you'll see an entry such as:
+
+```
+2018-12-18T21:41:32Z [error] GetPodNetwork: namespace isolation violation: podnamespace: development / target namespace: privileged
+```
+
+This error expresses that the pod resides in the namespace named `development` but refers to a `NetworkAttachmentDefinition` outside of that namespace, in this case, the namespace named `privileged`.
+
+In a positive example, you'd instead create the `NetworkAttachmentDefinition` in the `development` namespace, and you'd have an annotation that either A. does not reference a namespace, or B. refers to the same annotation.
+
+A positive example may be:
+
+```
+# Create the same NetworkAttachmentDefinition as above, however in the development namespace
+[user@kube-master ~]$ kubectl create -f cr.yml -n development
+networkattachmentdefinition.k8s.cni.cncf.io/macvlan-conf created
+
+# Show the yaml for a sample pod which references macvlan-conf without a namspace/ format
+[user@kube-master ~]$ cat positive.example.pod
+apiVersion: v1
+kind: Pod
+metadata:
+  name: samplepod
+  annotations:
+    k8s.v1.cni.cncf.io/networks: macvlan-conf
+spec:
+  containers:
+  - name: samplepod
+    command: ["/bin/bash", "-c", "sleep 2000000000000"]
+    image: dougbtv/centos-network
+
+# Create that pod.
+[user@kube-master ~]$ kubectl create -f positive.example.pod -n development
+pod/samplepod created
+
+# We can see that this pod has been launched successfully.
+[user@kube-master ~]$ kubectl get pods -n development
+NAME        READY   STATUS    RESTARTS   AGE
+samplepod   1/1     Running   0          31s
+```
+
+### Specify default cluster network in Pod annotations
+
+Users may also specify the default network for any given pod (via annotation), for cases where there are multiple cluster networks available within a Kubernetes cluster.
+
+Example use cases may include:
+
+1. During a migration from one default network to another (e.g. from Flannel to Calico), it may be practical if both network solutions are able to operate in parallel. Users can then control which network a pod should attach to during the transition period.
+2. Some users may deploy multiple cluster networks for the sake of their security considerations, and may desire to specify the default network for individual pods.
+
+Follow these steps to specify the default network on a pod-by-pod basis:
+
+1. First, you need to define all your cluster networks as network-attachment-definition objects.
+
+2. Next, you can specify the network you want in pods with the `v1.multus-cni.io/default-network` annotation. Pods which do not specify this annotation will keep using the CNI as defined in the Multus config file.
+
+```yaml
+apiVersion: v1
+kind: Pod
+metadata:
+name: pod-example
+annotations:
+ v1.multus-cni.io/default-network: calico-conf
+...
+```

doc/development.md

@@ -0,0 +1,52 @@
+## Development Information
+
+## How to utilize multus-cni code as library?
+
+Multus now uses [gopkg.in](http://gopkg.in/) to expose its code as library.
+You can use following command to import our code into your go code.
+
+```
+go get gopkg.in/intel/multus-cni.v3
+```
+
+
+## How do I submit an issue?
+
+Use GitHub as normally, you'll be presented with an option to submit a issue or enhancement request.
+
+Issues are considered stale after 90 days. After which, the maintainers reserve the right to close an issue.
+
+Typically, we'll tag the submitter and ask for more information if necessary before closing.
+
+If an issue is closed that you don't feel is sufficiently resolved, please feel free to re-open the issue and provide any necessary information.
+
+## How do I build multus-cni?
+
+You can use the built in `./build` script!
+
+```
+git clone https://github.com/intel/multus-cni.git
+cd multus-cni
+./build
+```
+
+## How do I run CI tests?
+
+Multus has go unit tests (based on ginkgo framework).The following commands drive CI tests manually in your environment:
+
+```
+sudo ./test.sh
+```
+
+## What are the best practices for logging?
+
+The following are the best practices for multus logging:
+
+* Add `logging.Debugf()` at the begining of functions
+* In case of error handling, use `logging.Errorf()` with given error info
+* `logging.Panicf()` only be used for critical errors (it should NOT normally be used)
+
+
+## CI Introduction
+
+TBD

doc/how-to-use.md

@@ -0,0 +1,625 @@
+## Multus CNI usage guide
+
+### Prerequisites
+
+* Kubelet configured to use CNI
+* Kubernetes version with CRD support (generally )
+
+Your Kubelet(s) must be configured to run with the CNI network plugin. Please see [Kubernetes document for CNI](https://kubernetes.io/docs/concepts/extend-kubernetes/compute-storage-net/network-plugins/#cni) for more details.
+
+### Install Multus
+
+Generally we recommend two options: Manually place a Multus binary in your `/opt/cni/bin`, or use our [quick-start method](quickstart.md) -- which creates a daemonset that has an opinionated way of how to install & configure Multus CNI (recommended).
+
+*Copy Multus Binary into place*
+
+You may acquire the Multus binary via compilation (see the [developer guide](development.md)) or download the a binary from the [GitHub releases](https://github.com/intel/multus-cni/releases) page. Copy multus binary into CNI binary directory, usually `/opt/cni/bin`. Perform this on all nodes in your cluster (master and nodes).
+
+    $ cp multus /opt/cni/bin
+
+*Via Daemonset method*
+
+As a [quickstart](quickstart.md), you may apply these YAML files (included in the clone of this repository). Run this command (typically you would run this on the master, or wherever you have access to the `kubectl` command to manage your cluster).
+
+    $ cat ./images/{multus-daemonset.yml,flannel-daemonset.yml} | kubectl apply -f -
+
+If you need more comprehensive detail, continue along with this guide, otherwise, you may wish to either [follow the quickstart guide]() or skip to the ['Create network attachment definition'](#create-network-attachment-definition) section.
+
+### Set up conf file in /etc/cni/net.d/ (Installed automatically by Daemonset)
+
+**If you use daemonset to install multus, skip this section and go to "Create network attachment"**
+
+You put CNI config file in `/etc/cni/net.d`. Kubernetes CNI runtime uses the alphabetically first file in the directory. (`"NOTE1"`, `"NOTE2"` are just comments, you can remove them at your configuration)
+
+Execute following commands at all Kubernetes nodes (i.e. master and minions)
+
+```
+$ mkdir -p /etc/cni/net.d
+$ cat >/etc/cni/net.d/30-multus.conf <<EOF
+{
+  "name": "multus-cni-network",
+  "type": "multus",
+  "readinessindicatorfile": "/var/run/flannel/subnet.env",
+  "delegates": [
+    {
+      "NOTE1": "This is example, wrote your CNI config in delegates",
+      "NOTE2": "If you use flannel, you also need to run flannel daemonset before!",
+      "type": "flannel",
+      "name": "flannel.1",
+      "delegate": {
+        "isDefaultGateway": true
+      }
+    }
+  ],
+  "kubeconfig": "/etc/cni/net.d/multus.d/multus.kubeconfig"
+}
+EOF
+```
+
+For the detail, please take a look into [Configuration Reference](configuration.md)
+
+**NOTE: You can use "clusterNetwork"/"defaultNetworks" instead of "delegates", see []() for the detail**
+
+As above config, you need to set `"kubeconfig"` in the config file for NetworkAttachmentDefinition(CRD).
+
+##### Which network will be used for "Pod IP"?
+
+In case of "delegates", the first delegates network will be used for "Pod IP". Otherwise, "clusterNetwork" will be used for "Pod IP".
+
+#### Create ServiceAccount, ClusterRole and its binding
+
+Create resources for multus to access CRD objects as following command:
+
+```
+# Execute following commands at Kubernetes master
+$ cat <<EOF | kubectl create -f -
+apiVersion: v1
+kind: ServiceAccount
+metadata:
+  name: multus
+  namespace: kube-system
+---
+kind: ClusterRole
+apiVersion: rbac.authorization.k8s.io/v1beta1
+metadata:
+  name: multus
+rules:
+  - apiGroups: ["k8s.cni.cncf.io"]
+    resources:
+      - '*'
+    verbs:
+      - '*'
+  - apiGroups:
+      - ""
+    resources:
+      - pods
+      - pods/status
+    verbs:
+      - get
+      - update
+---
+kind: ClusterRoleBinding
+apiVersion: rbac.authorization.k8s.io/v1beta1
+metadata:
+  name: multus
+roleRef:
+  apiGroup: rbac.authorization.k8s.io
+  kind: ClusterRole
+  name: multus
+subjects:
+- kind: ServiceAccount
+  name: multus
+  namespace: kube-system
+EOF
+```
+
+#### Set up kubeconfig file
+
+Create kubeconfig at master node as following commands:
+
+```
+# Execute following command at Kubernetes master
+$ mkdir -p /etc/cni/net.d/multus.d
+$ SERVICEACCOUNT_CA=$(kubectl get secrets -n=kube-system -o json | jq -r '.items[]|select(.metadata.annotations."kubernetes.io/service-account.name"=="multus")| .data."ca.crt"')
+$ SERVICEACCOUNT_TOKEN=$(kubectl get secrets -n=kube-system -o json | jq -r '.items[]|select(.metadata.annotations."kubernetes.io/service-account.name"=="multus")| .data.token' | base64 -d )
+$ KUBERNETES_SERVICE_PROTO=$(kubectl get all -o json | jq -r .items[0].spec.ports[0].name)
+$ KUBERNETES_SERVICE_HOST=$(kubectl get all -o json | jq -r .items[0].spec.clusterIP)
+$ KUBERNETES_SERVICE_PORT=$(kubectl get all -o json | jq -r .items[0].spec.ports[0].port)
+$ cat > /etc/cni/net.d/multus.d/multus.kubeconfig <<EOF
+# Kubeconfig file for Multus CNI plugin.
+apiVersion: v1
+kind: Config
+clusters:
+- name: local
+  cluster:
+    server: ${KUBERNETES_SERVICE_PROTOCOL:-https}://${KUBERNETES_SERVICE_HOST}:${KUBERNETES_SERVICE_PORT}
+    certificate-authority-data: ${SERVICEACCOUNT_CA}
+users:
+- name: multus
+  user:
+    token: "${SERVICEACCOUNT_TOKEN}"
+contexts:
+- name: multus-context
+  context:
+    cluster: local
+    user: multus
+current-context: multus-context
+EOF
+```
+
+Copy `/etc/cni/net.d/multus.d/multus.kubeconfig` into other Kubernetes nodes
+**NOTE: Recommend to exec 'chmod 600 /etc/cni/net.d/multus.d/multus.kubeconfig' to keep secure**
+
+```
+$ scp /etc/cni/net.d/multus.d/multus.kubeconfig ...
+```
+
+### Setup CRDs (daemonset automatically does)
+
+**If you use daemonset to install multus, skip this section and go to "Create network attachment"**
+
+Create CRD definition in Kubernetes as following command at master node:
+
+```
+# Execute following command at Kubernetes master
+$ cat <<EOF | kubectl create -f -
+apiVersion: apiextensions.k8s.io/v1beta1
+kind: CustomResourceDefinition
+metadata:
+  name: network-attachment-definitions.k8s.cni.cncf.io
+spec:
+  group: k8s.cni.cncf.io
+  version: v1
+  scope: Namespaced
+  names:
+    plural: network-attachment-definitions
+    singular: network-attachment-definition
+    kind: NetworkAttachmentDefinition
+    shortNames:
+    - net-attach-def
+  validation:
+    openAPIV3Schema:
+      properties:
+        spec:
+          properties:
+            config:
+                 type: string
+EOF
+```
+
+### Create network attachment definition
+
+The 'NetworkAttachmentDefinition' is used to setup the network attachment, i.e. secondary interface for the pod, There are two ways to configure the 'NetworkAttachmentDefinition' as following:
+
+- NetworkAttachmentDefinition with json CNI config
+- NetworkAttachmentDefinition with CNI config file
+
+#### NetworkAttachmentDefinition with json CNI config:
+
+Following command creates NetworkAttachmentDefinition. CNI config is in `config:` field.
+
+```
+# Execute following command at Kubernetes master
+$ cat <<EOF | kubectl create -f -
+apiVersion: "k8s.cni.cncf.io/v1"
+kind: NetworkAttachmentDefinition
+metadata:
+  name: macvlan-conf-1
+spec:
+  config: '{
+            "cniVersion": "0.3.0",
+            "type": "macvlan",
+            "master": "eth1",
+            "mode": "bridge",
+            "ipam": {
+                "type": "host-local",
+                "ranges": [
+                    [ {
+                         "subnet": "10.10.0.0/16",
+                         "rangeStart": "10.10.1.20",
+                         "rangeEnd": "10.10.3.50",
+                         "gateway": "10.10.0.254"
+                    } ]
+                ]
+            }
+        }'
+EOF
+```
+
+#### NetworkAttachmentDefinition with CNI config file:
+
+If NetworkAttachmentDefinition has no spec, multus find a file in defaultConfDir ('/etc/cni/multus/net.d', with same name in the 'name' field of CNI config.
+
+```
+# Execute following command at Kubernetes master
+$ cat <<EOF | kubectl create -f -
+apiVersion: "k8s.cni.cncf.io/v1"
+kind: NetworkAttachmentDefinition
+metadata:
+  name: macvlan-conf-2
+EOF
+```
+
+```
+# Execute following commands at all Kubernetes nodes (i.e. master and minions)
+$ cat <<EOF > /etc/cni/multus/net.d/macvlan2.conf
+{
+  "cniVersion": "0.3.0",
+  "type": "macvlan",
+  "name": "macvlan-conf-2",
+  "master": "eth1",
+  "mode": "bridge",
+  "ipam": {
+      "type": "host-local",
+      "ranges": [
+          [ {
+               "subnet": "11.10.0.0/16",
+               "rangeStart": "11.10.1.20",
+               "rangeEnd": "11.10.3.50"
+          } ]
+      ]
+  }
+}
+```
+
+### Run pod with network annotation
+
+#### Lauch pod with text annotation
+
+```
+# Execute following command at Kubernetes master
+$ cat <<EOF | kubectl create -f -
+apiVersion: v1
+kind: Pod
+metadata:
+  name: pod-case-01
+  annotations:
+    k8s.v1.cni.cncf.io/networks: macvlan-conf-1, macvlan-conf-2
+spec:
+  containers:
+  - name: pod-case-01
+    image: docker.io/centos/tools:latest
+    command:
+    - /sbin/init
+EOF
+```
+
+#### Lauch pod with text annotation for NetworkAttachmentDefinition in different namespace
+
+You can also specify NetworkAttachmentDefinition with its namespace as adding `<namespace>/`
+
+```
+# Execute following command at Kubernetes master
+$ cat <<EOF | kubectl create -f -
+apiVersion: "k8s.cni.cncf.io/v1"
+kind: NetworkAttachmentDefinition
+metadata:
+  name: macvlan-conf-3
+  namespace: testns1
+spec:
+  config: '{
+            "cniVersion": "0.3.0",
+            "type": "macvlan",
+            "master": "eth1",
+            "mode": "bridge",
+            "ipam": {
+                "type": "host-local",
+                "ranges": [
+                    [ {
+                         "subnet": "12.10.0.0/16",
+                         "rangeStart": "12.10.1.20",
+                         "rangeEnd": "12.10.3.50"
+                    } ]
+                ]
+            }
+        }'
+EOF
+$ cat <<EOF | kubectl create -f -
+apiVersion: v1
+kind: Pod
+metadata:
+  name: pod-case-02
+  annotations:
+    k8s.v1.cni.cncf.io/networks: testns1/macvlan-conf-3
+spec:
+  containers:
+  - name: pod-case-02
+    image: docker.io/centos/tools:latest
+    command:
+    - /sbin/init
+EOF
+```
+
+#### Lauch pod with text annotation with interface name
+
+You can also specify interface name as adding `@<ifname>`.
+
+```
+# Execute following command at Kubernetes master
+$ cat <<EOF | kubectl create -f -
+apiVersion: v1
+kind: Pod
+metadata:
+  name: pod-case-03
+  annotations:
+    k8s.v1.cni.cncf.io/networks: macvlan-conf-1@macvlan1
+spec:
+  containers:
+  - name: pod-case-03
+    image: docker.io/centos/tools:latest
+    command:
+    - /sbin/init
+EOF
+```
+
+#### Lauch pod with json annotation
+
+```
+# Execute following command at Kubernetes master
+$ cat <<EOF | kubectl create -f -
+apiVersion: v1
+kind: Pod
+metadata:
+  name: pod-case-04
+  annotations:
+    k8s.v1.cni.cncf.io/networks: '[
+            { "name" : "macvlan-conf-1" },
+            { "name" : "macvlan-conf-2" }
+    ]'
+spec:
+  containers:
+  - name: pod-case-04
+    image: docker.io/centos/tools:latest
+    command:
+    - /sbin/init
+EOF
+```
+
+#### Lauch pod with json annotation for NetworkAttachmentDefinition in different namespace
+
+You can also specify NetworkAttachmentDefinition with its namespace as adding `"namespace": "<namespace>"`.
+
+```
+# Execute following command at Kubernetes master
+$ cat <<EOF | kubectl create -f -
+apiVersion: v1
+kind: Pod
+metadata:
+  name: pod-case-05
+  annotations:
+    k8s.v1.cni.cncf.io/networks: '[
+            { "name" : "macvlan-conf-1",
+              "namespace": "testns1" }
+    ]'
+spec:
+  containers:
+  - name: pod-case-05
+    image: docker.io/centos/tools:latest
+    command:
+    - /sbin/init
+EOF
+```
+
+#### Lauch pod with json annotation with interface
+
+You can also specify interface name as adding `"interface": "<ifname>"`.
+
+```
+# Execute following command at Kubernetes master
+$ cat <<EOF | kubectl create -f -
+apiVersion: v1
+kind: Pod
+metadata:
+  name: pod-case-06
+  annotations:
+    k8s.v1.cni.cncf.io/networks: '[
+            { "name" : "macvlan-conf-1",
+              "interface": "macvlan1" },
+            { "name" : "macvlan-conf-2" }
+    ]'
+spec:
+  containers:
+  - name: pod-case-06
+    image: docker.io/centos/tools:latest
+    command:
+    - /sbin/init
+EOF
+```
+
+### Verifying pod network
+
+Following the example of `ip -d address` output of above pod, "pod-case-06":
+
+```
+# Execute following command at Kubernetes master
+$ kubectl exec -it pod-case-06 -- ip -d address
+1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1
+    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 promiscuity 0 numtxqueues 1 numrxqueues 1 gso_max_size 65536 gso_max_segs 65535
+    inet 127.0.0.1/8 scope host lo
+       valid_lft forever preferred_lft forever
+    inet6 ::1/128 scope host
+       valid_lft forever preferred_lft forever
+3: eth0@if11: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1450 qdisc noqueue state UP group default
+    link/ether 0a:58:0a:f4:02:06 brd ff:ff:ff:ff:ff:ff link-netnsid 0 promiscuity 0
+    veth numtxqueues 1 numrxqueues 1 gso_max_size 65536 gso_max_segs 65535
+    inet 10.244.2.6/24 scope global eth0
+       valid_lft forever preferred_lft forever
+    inet6 fe80::ac66:45ff:fe7c:3a19/64 scope link
+       valid_lft forever preferred_lft forever
+4: macvlan1@if3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN group default
+    link/ether 4e:6d:7a:4e:14:87 brd ff:ff:ff:ff:ff:ff link-netnsid 0 promiscuity 0
+    macvlan mode bridge numtxqueues 1 numrxqueues 1 gso_max_size 65536 gso_max_segs 65535
+    inet 10.10.1.22/16 scope global macvlan1
+       valid_lft forever preferred_lft forever
+    inet6 fe80::4c6d:7aff:fe4e:1487/64 scope link
+       valid_lft forever preferred_lft forever
+5: net2@if3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN group default
+    link/ether 6e:e3:71:7f:86:f7 brd ff:ff:ff:ff:ff:ff link-netnsid 0 promiscuity 0
+    macvlan mode bridge numtxqueues 1 numrxqueues 1 gso_max_size 65536 gso_max_segs 65535
+    inet 11.10.1.22/16 scope global net2
+       valid_lft forever preferred_lft forever
+    inet6 fe80::6ce3:71ff:fe7f:86f7/64 scope link
+       valid_lft forever preferred_lft forever
+```
+
+| Interface name | Description |
+| --- | --- |
+| lo | loopback |
+| eth0 | Default network interface (flannel) |
+| macvlan1 | macvlan interface (macvlan-conf-1) |
+| net2 | macvlan interface (macvlan-conf-2) |
+
+## Specifying a default route for a specific attachment
+
+Typically, the default route for a pod will route traffic over the `eth0` and therefore over the cluster-wide default network. You may wish to specify that a different network attachment will have the default route.
+
+You can achieve this by using the JSON formatted annotation and specifying a `default-route` key.
+
+*NOTE*: It's important that you consider that this may impact some functionality of getting traffic to route over the cluster-wide default network.
+
+For example, we have a this configuration for macvlan:
+
+```
+cat <<EOF | kubectl create -f -
+apiVersion: "k8s.cni.cncf.io/v1"
+kind: NetworkAttachmentDefinition
+metadata:
+  name: macvlan-conf
+spec:
+  config: '{
+      "cniVersion": "0.3.0",
+      "type": "macvlan",
+      "master": "eth0",
+      "mode": "bridge",
+      "ipam": {
+        "type": "host-local",
+        "subnet": "192.168.2.0/24",
+        "rangeStart": "192.168.2.200",
+        "rangeEnd": "192.168.2.216",
+        "routes": [
+          { "dst": "0.0.0.0/0" }
+        ],
+        "gateway": "192.168.2.1"
+      }
+    }'
+EOF
+```
+
+We can then create a pod which uses the `default-route` key in the JSON formatted `k8s.v1.cni.cncf.io/networks` annotation. 
+
+```
+cat <<EOF | kubectl create -f -
+apiVersion: v1
+kind: Pod
+metadata:
+  name: samplepod
+  annotations:
+    k8s.v1.cni.cncf.io/networks: '[{
+      "name": "macvlan-conf",
+      "default-route": ["192.168.2.1"]
+    }]'
+spec:
+  containers:
+  - name: samplepod
+    command: ["/bin/bash", "-c", "trap : TERM INT; sleep infinity & wait"]
+    image: dougbtv/centos-network
+EOF
+```
+
+This will set `192.168.2.1` as the default route over the `net1` interface, such as:
+
+```
+$ kubectl exec -it samplepod -- ip route
+default via 192.168.2.1 dev net1 
+10.244.0.0/24 dev eth0  proto kernel  scope link  src 10.244.0.169 
+10.244.0.0/16 via 10.244.0.1 dev eth0 
+```
+
+## Entrypoint Parameters
+
+Multus CNI, when installed using the daemonset-style installation uses an entrypoint script which copies the Multus binary into place, places CNI configurations. This entrypoint takes a variety of parameters for customization.
+
+Typically, you'd modified the daemonset YAML itself to specify these parameters.
+
+For example, the `command` and `args` parameters in the `containers` section of the DaemonSet may look something like:
+
+```
+  command: ["/entrypoint.sh"]
+  args:
+  - "--multus-conf-file=auto"
+  - "--namespace-isolation=true"
+  - "--multus-log-level=verbose"
+```
+
+Note that some of the defaults have directories inside the root directory named `/host/`, this is because it is deployed as a container and we have host file system locations mapped into this directory inside the container. If you use other directories, you may have to change the mounted volumes.
+
+### Entrypoint script parameters
+
+Each parameter is shown with the default as the value.
+
+    --cni-conf-dir=/host/etc/cni/net.d
+
+This is the configuration directory where Multus will write its configuration file.
+
+    --cni-bin-dir=/host/opt/cni/bin
+
+This the directory in which the Multus binary will be installed.
+
+    --namespace-isolation=false
+
+Setting this option to true enables the Namespace isolation feature, which insists that custom resources must be created in the same namespace as the pods, otherwise it will refuse to attach those definitions as additional interfaces.
+
+    --multus-bin-file=/usr/src/multus-cni/bin/multus
+
+This option lets you set which binary executable to copy from the container onto the host (into the directory specified by `--cni-bin-dir`), allowing one to copy an alternate version or build of Multus CNI.
+
+    --multus-conf-file=/usr/src/multus-cni/images/70-multus.conf
+
+The `--multus-conf-file` is one of two options; it can be set to a source file to be copied into the location specified by `--cni-conf-dir`. Or, to a value of `auto`, that is: `--multus-conf-file=auto`.
+
+The automatic configuration option is used to automatically generate Multus configurations given existing on-disk CNI configurations for your default network.
+
+In the case that `--multus-conf-file=auto` -- The entrypoint script will look at the `--multus-autoconfig-dir` (by default, the same as the `--cni-conf-dir`). Multus will wait (600 seconds) until there's a CNI configuration file there, and it will take the alphabetically first configuration there, and it will wrap that configuration into a Multus configuration.
+
+    --multus-autoconfig-dir=/host/etc/cni/net.d
+
+Used only with `--multus-conf-file=auto`. This option allows one to set which directory will be used to generate configuration files.
+
+This can be used if you have your CNI configuration stored in an alternate location, or, you have constraints on race conditions where you'd like to generate your default network configuration first, and then only have Multus write its configuration when it finds that configuration -- allowing only Multus to write the CNI configuration in the `--cni-conf-dir`, therefore notifying the Kubelet that the node is in a ready state.
+
+    --multus-kubeconfig-file-host=/etc/cni/net.d/multus.d/multus.kubeconfig
+
+Used only with `--multus-conf-file=auto`. Allows you to specify an alternate path to the Kubeconfig.
+
+    --multus-log-level=
+    --multus-log-file=
+
+Used only with `--multus-conf-file=auto`. See the documentation for logging for which values are permitted.
+
+Used only with `--multus-conf-file=auto`. Allows you to specify CNI spec version. Please set if you need to speicfy CNI spec version.
+
+    --cni-version=
+
+In some cases, the original CNI configuration that the Multus configuration was generated from (using `--multus-conf-file=auto`) may be used as a sort of semaphor for network readiness -- as this model is used by the Kubelet itself. If you need to disable Multus' availablity, you may wish to clean out the generated configuration file when the source file for autogeneration of the config file is no longer present. You can use this functionality by setting:
+
+    --cleanup-config-on-exit=true
+
+When using CRIO, you may need to restart CRIO to get the Multus configuration file to take -- this is rarely necessary.
+
+    --restart-crio=false
+
+Additionally when using CRIO, you may wish to have the CNI config file that's used as the source for `--multus-conf-file=auto` renamed. This boolean option when set to true automatically renames the file with a `.old` suffix to the original filename.
+
+    --rename-conf-file=true
+
+When using `--multus-conf-file=auto` you may also care to specify a `binDir` in the configuration, this can be accomplished using the `--additional-bin-dir` option.
+
+    --additional-bin-dir=/opt/multus/bin
+
+Sometimes, you may wish to not have the entrypoint copy the binary file onto the host. Potentially, you have another way to copy in a specific version of Multus, for example. By default, it's always copied, but you may disable the copy with:
+
+    --skip-multus-binary-copy=true
+
+If you wish to have auto configuration use the `readinessindicatorfile` in the configuration, you can use the `--readiness-indicator-file` to express which file should be used as the readiness indicator.
+
+    --readiness-indicator-file=/path/to/file

doc/node-kubeconfig.yaml

@@ -0,0 +1,18 @@
+apiVersion: v1
+kind: Config
+clusters:
+- name: local
+  cluster:
+    insecure-skip-tls-verify: true
+    server: https://<master-IP-Address>:6443
+contexts:
+- context:
+    cluster: local
+    user: admin
+  name: kubelet-context
+current-context: kubelet-context
+users:
+- name: admin
+  user:
+    password: admin
+    username: admin

doc/quickstart.md

@@ -0,0 +1,230 @@
+# Quickstart Guide
+
+This guide is intended as a way to get you off the ground, using Multus CNI to create Kubernetes pods with multiple interfaces. If you're already using Multus and need more detail, see the [comprehensive usage guide](how-to-use.md). This document is a quickstart and a getting started guide in one, intended for your first run-through of Multus CNI.
+
+We'll first install Multus CNI, and then we'll setup some configurations so that you can see how multiple interfaces are created for pods.
+
+## Key Concepts
+
+Two things we'll refer to a number of times through this document are:
+
+* "Default network" -- This is your pod-to-pod network. This is how pods communicate among one another in your cluster, how they have connectivity. Generally speaking, this is presented as the interface named `eth0`. This interface is always attached to your pods, so that they can have connectivity among themselves. We'll add interfaces in addition to this.
+* "CRDs" -- Custom Resource Definitions. Custom Resources are a way that the Kubernetes API is extended. We use these here to store some information that Multus can read. Primarily, we use these to store the configurations for each of the additional interfaces that are attached to your pods.
+
+## Prerequisites
+
+Our installation method requires that you first have installed Kubernetes and have configured a default network -- that is, a CNI plugin that's used for your pod-to-pod connectivity. 
+
+We recommend Kubernetes 1.16 or later.
+
+To install Kubernetes, you may decide to use [kubeadm](https://kubernetes.io/docs/setup/independent/create-cluster-kubeadm/), or potentially [kubespray](https://github.com/kubernetes-sigs/kubespray).
+
+After installing Kubernetes, you must install a default network CNI plugin. If you're using kubeadm, refer to the "[Installing a pod network add-on](https://kubernetes.io/docs/setup/independent/create-cluster-kubeadm/#pod-network)" section in the kubeadm documentation. If it's your first time, we generally recommend using Flannel for the sake of simplicity.
+
+Alternatively, for advanced use cases, for installing Multus and a default network plugin at the same time, you may refer to the [Kubernetes Network Plumbing Group's Reference Deployments](https://github.com/k8snetworkplumbingwg/reference-deployment).
+
+To verify that you default network is ready, you may list your Kubernetes nodes with:
+
+```
+kubectl get nodes
+```
+
+In the case that your default network is ready
+
+## Installation
+
+Our recommended quickstart method to deploy Multus is to deploy using a Daemonset (a method of running pods on each nodes in your cluster), this spins up pods which install a Multus binary and configure Multus for usage.
+
+Firstly, clone this GitHub repository. 
+
+```
+git clone https://github.com/intel/multus-cni.git && cd multus-cni
+```
+
+We'll apply a YAML file with `kubectl` from this repo.
+
+```
+$ cat ./images/multus-daemonset.yml | kubectl apply -f -
+```
+
+### What the Multus daemonset does
+
+* Starts a Multus daemonset, this runs a pod on each node which places a Multus binary on each node in `/opt/cni/bin`
+* Reads the lexicographically (alphabetically) first configuration file in `/etc/cni/net.d`, and creates a new configuration file for Multus as `/etc/cni/net.d/00-multus.conf`, this configuration is auto-generated and is based on the default network configuration (which is assumed to be the alphabetically first configuration)
+* Creates a `/etc/cni/net.d/multus.d` directory on each node with authentication information for Multus to access the Kubernetes API.
+
+
+### Validating your installation
+
+Generally, the first step in validating your installation is to ensure that the Multus pods have run without error, you may see an overview of those by looking at:
+
+```
+$ kubectl get pods --all-namespaces | grep -i multus
+```
+
+You may further validate that it has ran by looking at the `/etc/cni/net.d/` directory and ensure that the auto-generated `/etc/cni/net.d/00-multus.conf` exists corresponding to the alphabetically first configuration file.
+
+## Creating additional interfaces
+
+The first thing we'll do is create configurations for each of the additional interfaces that we attach to pods. We'll do this by creating Custom Resources. Part of the quickstart installation creates a "CRD" -- a custom resource definition that is the home where we keep these custom resources -- we'll store our configurations for each interface in these.
+
+### CNI Configurations
+
+Each configuration we'll add is a CNI configuration. If you're not familiar with them, let's break them down quickly. Here's an example CNI configuration:
+
+```
+{
+  "cniVersion": "0.3.0",
+  "type": "loopback",
+  "additional": "information"
+}
+```
+
+CNI configurations are JSON, and we have a structure here that has a few things we're interested in:
+
+1. `cniVersion`: Tells each CNI plugin which version is being used and can give the plugin information if it's using a too late (or too early) version.
+2. `type`: This tells CNI which binary to call on disk. Each CNI plugin is a binary that's called. Typically, these binaries are stored in `/opt/cni/bin` on each node, and CNI executes this binary. In this case we've specified the `loopback` binary (which create a loopback-type network interface). If this is your first time installing Multus, you might want to verify that the plugins that are in the "type" field are actually on disk in the `/opt/cni/bin` directory.
+3. `additional`: This field is put here as an example, each CNI plugin can specify whatever configuration parameters they'd like in JSON. These are specific to the binary you're calling in the `type` field.
+
+For an even further example -- take a look at the [bridge CNI plugin README](https://github.com/containernetworking/plugins/tree/master/plugins/main/bridge) which shows additional details.
+
+If you'd like more information about CNI configuration, you can read [the entire CNI specification](https://github.com/containernetworking/cni/blob/master/SPEC.md). It might also be useful to look at the [CNI reference plugins](https://github.com/containernetworking/plugins) and see how they're configured.
+
+You do not need to reload or refresh the Kubelets when CNI configurations change. These are read on each creation & deletion of pods. So if you change a configuration, it'll apply the next time a pod is created. Existing pods may need to be restarted if they need the new configuration.
+
+### Storing a configuration as a Custom Resource
+
+So, we want to create an additional interface. Let's create a macvlan interface for pods to use. We'll create a custom resource that defines the CNI configuration for interfaces.
+
+Note in the following command that there's a `kind: NetworkAttachmentDefinition`. This is our fancy name for our configuration -- it's a custom extension of Kubernetes that defines how we attach networks to our pods. 
+
+Secondarily, note the `config` field. You'll see that this is a CNI configuration just like we explained earlier.
+
+Lastly but *very* importantly, note under `metadata` the `name` field -- here's where we give this configuration a name, and it's how we tell pods to use this configuration. The name here is `macvlan-conf` -- as we're creating a configuration for macvlan.
+
+Here's the command to create this example configuration:
+
+```
+cat <<EOF | kubectl create -f -
+apiVersion: "k8s.cni.cncf.io/v1"
+kind: NetworkAttachmentDefinition
+metadata:
+  name: macvlan-conf
+spec:
+  config: '{
+      "cniVersion": "0.3.0",
+      "type": "macvlan",
+      "master": "eth0",
+      "mode": "bridge",
+      "ipam": {
+        "type": "host-local",
+        "subnet": "192.168.1.0/24",
+        "rangeStart": "192.168.1.200",
+        "rangeEnd": "192.168.1.216",
+        "routes": [
+          { "dst": "0.0.0.0/0" }
+        ],
+        "gateway": "192.168.1.1"
+      }
+    }'
+EOF
+```
+
+*NOTE*: This example uses `eth0` as the `master` parameter, this master parameter should match the interface name on the hosts in your cluster.
+
+You can see which configurations you've created using `kubectl` here's how you can do that:
+
+```
+kubectl get network-attachment-definitions
+```
+
+You can get more detail by describing them:
+
+```
+kubectl describe network-attachment-definitions macvlan-conf
+```
+
+### Creating a pod that attaches an additional interface
+
+We're going to create a pod. This will look familiar as any pod you might have created before, but, we'll have a special `annotations` field -- in this case we'll have an annotation called `k8s.v1.cni.cncf.io/networks`. This field takes a comma delimited list of the names of your `NetworkAttachmentDefinition`s as we created above. Note in the command below that we have the annotation of `k8s.v1.cni.cncf.io/networks: macvlan-conf` where `macvlan-conf` is the name we used above when we created our configuration.
+
+Let's go ahead and create a pod (that just sleeps for a really long time) with this command:
+
+```
+cat <<EOF | kubectl create -f -
+apiVersion: v1
+kind: Pod
+metadata:
+  name: samplepod
+  annotations:
+    k8s.v1.cni.cncf.io/networks: macvlan-conf
+spec:
+  containers:
+  - name: samplepod
+    command: ["/bin/ash", "-c", "trap : TERM INT; sleep infinity & wait"]
+    image: alpine
+EOF
+```
+
+You may now inspect the pod and see what interfaces are attached, like so:
+
+```
+$ kubectl exec -it samplepod -- ip a
+```
+
+You should note that there are 3 interfaces:
+
+* `lo` a loopback interface
+* `eth0` our default network
+* `net1` the new interface we created with the macvlan configuration.
+
+### Network Status Annotations
+
+For additional confirmation, use `kubectl describe pod samplepod` and there will be an annotations section, similar to the following:
+
+```
+Annotations:        k8s.v1.cni.cncf.io/networks: macvlan-conf
+                    k8s.v1.cni.cncf.io/networks-status:
+                      [{
+                          "name": "cbr0",
+                          "ips": [
+                              "10.244.1.73"
+                          ],
+                          "default": true,
+                          "dns": {}
+                      },{
+                          "name": "macvlan-conf",
+                          "interface": "net1",
+                          "ips": [
+                              "192.168.1.205"
+                          ],
+                          "mac": "86:1d:96:ff:55:0d",
+                          "dns": {}
+                      }]
+```
+
+This metadata tells us that we have two CNI plugins running successfully.
+
+### What if I want more interfaces?
+
+You can add more interfaces to a pod by creating more custom resources and then referring to them in pod's annotation. You can also reuse configurations, so for example, to attach two macvlan interfaces to a pod, you could create a pod like so:
+
+```
+cat <<EOF | kubectl create -f -
+apiVersion: v1
+kind: Pod
+metadata:
+  name: samplepod
+  annotations:
+    k8s.v1.cni.cncf.io/networks: macvlan-conf,macvlan-conf
+spec:
+  containers:
+  - name: samplepod
+    command: ["/bin/ash", "-c", "trap : TERM INT; sleep infinity & wait"]
+    image: alpine
+EOF
+```
+
+Note that the annotation now reads `k8s.v1.cni.cncf.io/networks: macvlan-conf,macvlan-conf`. Where we have the same configuration used twice, separated by a comma. 
+
+If you were to create another custom resource with the name `foo` you could use that such as: `k8s.v1.cni.cncf.io/networks: foo,macvlan-conf`, and use any number of attachments.

e2e/README.md

@@ -0,0 +1,12 @@
+## Multus e2e test with kind
+
+### How to test e2e
+
+
+```
+$ git clone https://github.com/intel/multus-cni.git
+$ cd multus-cni/e2e
+$ ./get_tools.sh
+$ ./setup_cluster.sh
+$ ./test-simple-macvlan1.sh
+```

e2e/cni-install.yml

@@ -0,0 +1,64 @@
+---
+kind: ConfigMap
+apiVersion: v1
+metadata:
+  name: cni-install-sh
+  namespace: kube-system
+data:
+  install_cni.sh: |
+    cd /tmp
+    wget https://github.com/containernetworking/plugins/releases/download/v0.8.5/cni-plugins-linux-amd64-v0.8.5.tgz
+    cd /host/opt/cni/bin
+    tar xvfzp /tmp/cni-plugins-linux-amd64-v0.8.5.tgz
+    sleep infinite
+---
+apiVersion: apps/v1
+kind: DaemonSet
+metadata:
+  name: install-cni-plugins
+  namespace: kube-system
+  labels:
+    name: cni-plugins
+spec:
+  selector:
+    matchLabels:
+      name: cni-plugins
+  template:
+    metadata:
+      labels:
+        name: cni-plugins
+    spec:
+      hostNetwork: true
+      nodeSelector:
+        kubernetes.io/arch: amd64
+      tolerations:
+      - operator: Exists
+        effect: NoSchedule
+      containers:
+      - name: install-cni-plugins
+        image: alpine
+        command: ["/bin/sh", "/scripts/install_cni.sh"]
+        resources:
+          requests:
+            cpu: "100m"
+            memory: "50Mi"
+          limits:
+            cpu: "100m"
+            memory: "50Mi"
+        securityContext:
+          privileged: true
+        volumeMounts:
+        - name: cni-bin
+          mountPath: /host/opt/cni/bin
+        - name: scripts
+          mountPath: /scripts
+      volumes:
+        - name: cni-bin
+          hostPath:
+            path: /opt/cni/bin
+        - name: scripts
+          configMap:
+            name: cni-install-sh
+            items:
+            - key: install_cni.sh
+              path: install_cni.sh

e2e/get_tools.sh

@@ -0,0 +1,15 @@
+#!/bin/sh
+set -o errexit
+
+if [ ! -d bin ]; then
+	mkdir bin
+fi
+
+curl -Lo ./bin/kind "https://github.com/kubernetes-sigs/kind/releases/download/v0.7.0/kind-$(uname)-amd64"
+chmod +x ./bin/kind
+curl -Lo ./bin/kubectl https://storage.googleapis.com/kubernetes-release/release/`curl -s https://storage.googleapis.com/kubernetes-release/release/stable.txt`/bin/linux/amd64/kubectl
+chmod +x ./bin/kubectl
+curl -Lo ./bin/koko https://github.com/redhat-nfvpe/koko/releases/download/v0.82/koko_0.82_linux_amd64
+chmod +x ./bin/koko
+curl -Lo ./bin/jq https://github.com/stedolan/jq/releases/download/jq-1.6/jq-linux64
+chmod +x ./bin/jq

e2e/macvlan1.yml

@@ -0,0 +1,63 @@
+---
+apiVersion: "k8s.cni.cncf.io/v1"
+kind: NetworkAttachmentDefinition
+metadata:
+  name: macvlan1-config
+spec: 
+  config: '{
+            "cniVersion": "0.3.1",
+            "plugins": [
+                {
+                    "type": "macvlan",
+                    "capabilities": { "ips": true },
+                    "master": "eth1",
+                    "mode": "bridge",
+                    "ipam": {
+                        "type": "static"
+                    }
+                }, {
+                    "type": "tuning"
+                } ]
+        }'
+---
+apiVersion: v1
+kind: Pod
+metadata:
+  name: macvlan1-worker1
+  annotations:
+    k8s.v1.cni.cncf.io/networks: '[
+            { "name": "macvlan1-config",
+              "ips": [ "10.1.1.11/24" ] }
+    ]'
+  labels:
+    app: macvlan
+spec:
+  containers:
+  - name: macvlan-worker1
+    image: centos:8
+    command: ["/bin/sleep", "10000"]
+    securityContext:
+      privileged: true
+  nodeSelector:
+    kubernetes.io/hostname: kind-worker
+---
+apiVersion: v1
+kind: Pod
+metadata:
+  name: macvlan1-worker2
+  annotations:
+    k8s.v1.cni.cncf.io/networks: '[
+            { "name": "macvlan1-config",
+              "ips": [ "10.1.1.12/24" ] }
+    ]'
+  labels:
+    app: macvlan
+spec:
+  containers:
+  - name: macvlan-worker2
+    image: centos:8
+    command: ["/bin/sleep", "10000"]
+    securityContext:
+      privileged: true
+  nodeSelector:
+    kubernetes.io/hostname: kind-worker2

e2e/multus-daemonset.yml

@@ -0,0 +1,247 @@
+---
+apiVersion: apiextensions.k8s.io/v1
+kind: CustomResourceDefinition
+metadata:
+  name: network-attachment-definitions.k8s.cni.cncf.io
+spec:
+  group: k8s.cni.cncf.io
+  scope: Namespaced
+  names:
+    plural: network-attachment-definitions
+    singular: network-attachment-definition
+    kind: NetworkAttachmentDefinition
+    shortNames:
+    - net-attach-def
+  versions:
+    - name: v1
+      served: true
+      storage: true
+      schema:
+        openAPIV3Schema:
+          type: object
+          properties:
+            spec:
+              type: object
+              properties:
+                config:
+                  type: string
+---
+kind: ClusterRole
+apiVersion: rbac.authorization.k8s.io/v1
+metadata:
+  name: multus
+rules:
+  - apiGroups: ["k8s.cni.cncf.io"]
+    resources:
+      - '*'
+    verbs:
+      - '*'
+  - apiGroups:
+      - ""
+    resources:
+      - pods
+      - pods/status
+    verbs:
+      - get
+      - update
+---
+kind: ClusterRoleBinding
+apiVersion: rbac.authorization.k8s.io/v1
+metadata:
+  name: multus
+roleRef:
+  apiGroup: rbac.authorization.k8s.io
+  kind: ClusterRole
+  name: multus
+subjects:
+- kind: ServiceAccount
+  name: multus
+  namespace: kube-system
+---
+apiVersion: v1
+kind: ServiceAccount
+metadata:
+  name: multus
+  namespace: kube-system
+---
+kind: ConfigMap
+apiVersion: v1
+metadata:
+  name: multus-cni-config
+  namespace: kube-system
+  labels:
+    tier: node
+    app: multus
+data:
+  # NOTE: If you'd prefer to manually apply a configuration file, you may create one here.
+  # In the case you'd like to customize the Multus installation, you should change the arguments to the Multus pod
+  # change the "args" line below from
+  # - "--multus-conf-file=auto"
+  # to:
+  # "--multus-conf-file=/tmp/multus-conf/70-multus.conf"
+  # Additionally -- you should ensure that the name "70-multus.conf" is the alphabetically first name in the
+  # /etc/cni/net.d/ directory on each node, otherwise, it will not be used by the Kubelet.
+  cni-conf.json: |
+    {
+      "name": "multus-cni-network",
+      "type": "multus",
+      "capabilities": {
+        "portMappings": true
+      },
+      "delegates": [
+        {
+          "cniVersion": "0.3.1",
+          "name": "default-cni-network",
+          "plugins": [
+            {
+              "type": "flannel",
+              "name": "flannel.1",
+                "delegate": {
+                  "isDefaultGateway": true,
+                  "hairpinMode": true
+                }
+              },
+              {
+                "type": "portmap",
+                "capabilities": {
+                  "portMappings": true
+                }
+              }
+          ]
+        }
+      ],
+      "kubeconfig": "/etc/cni/net.d/multus.d/multus.kubeconfig"
+    }
+---
+apiVersion: apps/v1
+kind: DaemonSet
+metadata:
+  name: kube-multus-ds-amd64
+  namespace: kube-system
+  labels:
+    tier: node
+    app: multus
+    name: multus
+spec:
+  selector:
+    matchLabels:
+      name: multus
+  updateStrategy:
+    type: RollingUpdate
+  template:
+    metadata:
+      labels:
+        tier: node
+        app: multus
+        name: multus
+    spec:
+      hostNetwork: true
+      nodeSelector:
+        kubernetes.io/arch: amd64
+      tolerations:
+      - operator: Exists
+        effect: NoSchedule
+      serviceAccountName: multus
+      containers:
+      - name: kube-multus
+        image: localhost:5000/multus:e2e
+        command: ["/entrypoint.sh"]
+        args:
+        - "--multus-conf-file=auto"
+        - "--cni-version=0.3.1"
+        resources:
+          requests:
+            cpu: "100m"
+            memory: "50Mi"
+          limits:
+            cpu: "100m"
+            memory: "50Mi"
+        securityContext:
+          privileged: true
+        volumeMounts:
+        - name: cni
+          mountPath: /host/etc/cni/net.d
+        - name: cnibin
+          mountPath: /host/opt/cni/bin
+        - name: multus-cfg
+          mountPath: /tmp/multus-conf
+      volumes:
+        - name: cni
+          hostPath:
+            path: /etc/cni/net.d
+        - name: cnibin
+          hostPath:
+            path: /opt/cni/bin
+        - name: multus-cfg
+          configMap:
+            name: multus-cni-config
+            items:
+            - key: cni-conf.json
+              path: 70-multus.conf
+---
+apiVersion: apps/v1
+kind: DaemonSet
+metadata:
+  name: kube-multus-ds-ppc64le
+  namespace: kube-system
+  labels:
+    tier: node
+    app: multus
+    name: multus
+spec:
+  selector:
+    matchLabels:
+      name: multus
+  updateStrategy:
+    type: RollingUpdate
+  template:
+    metadata:
+      labels:
+        tier: node
+        app: multus
+        name: multus
+    spec:
+      hostNetwork: true
+      nodeSelector:
+        kubernetes.io/arch: ppc64le
+      tolerations:
+      - operator: Exists
+        effect: NoSchedule
+      serviceAccountName: multus
+      containers:
+      - name: kube-multus
+        # ppc64le support requires multus:latest for now. support 3.3 or later.
+        image: nfvpe/multus:latest-ppc64le
+        command: ["/entrypoint.sh"]
+        args:
+        - "--multus-conf-file=auto"
+        - "--cni-version=0.3.1"
+        resources:
+          requests:
+            cpu: "100m"
+            memory: "90Mi"
+          limits:
+            cpu: "100m"
+            memory: "90Mi"
+        securityContext:
+          privileged: true
+        volumeMounts:
+        - name: cni
+          mountPath: /host/etc/cni/net.d
+        - name: cnibin
+          mountPath: /host/opt/cni/bin
+        - name: multus-cfg
+          mountPath: /tmp/multus-conf
+      volumes:
+        - name: cni
+          hostPath:
+            path: /etc/cni/net.d
+        - name: cnibin
+          hostPath:
+            path: /opt/cni/bin
+        - name: multus-cfg
+          configMap:
+            name: multus-cni-config
+            items:
+            - key: cni-conf.json
+              path: 70-multus.conf

e2e/setup_cluster.sh

@@ -0,0 +1,36 @@
+#!/bin/sh
+set -o errexit
+
+export PATH=${PATH}:./bin
+
+reg_name='kind-registry'
+reg_port='5000'
+running="$(docker inspect -f '{{.State.Running}}' "${reg_name}" 2>/dev/null || true)"
+if [ "${running}" != 'true' ]; then
+  docker run -d --restart=always -p "${reg_port}:5000" --name "${reg_name}" registry:2
+fi
+reg_ip="$(docker inspect -f '{{.NetworkSettings.IPAddress}}' "${reg_name}")"
+
+cat <<EOF | kind create cluster --config=-
+kind: Cluster
+apiVersion: kind.x-k8s.io/v1alpha4
+containerdConfigPatches:
+- |-
+  [plugins."io.containerd.grpc.v1.cri".registry.mirrors."localhost:${reg_port}"]
+    endpoint = ["http://${reg_ip}:${reg_port}"]
+nodes:
+  - role: control-plane
+  - role: worker
+  - role: worker
+EOF
+
+kind export kubeconfig
+sudo env PATH=${PATH} koko -d kind-worker,eth1 -d kind-worker2,eth1
+sleep 1
+kubectl -n kube-system wait --for=condition=available deploy/coredns --timeout=300s
+kubectl create -f https://raw.githubusercontent.com/intel/multus-cni/master/images/multus-daemonset.yml
+sleep 1
+kubectl -n kube-system wait --for=condition=ready -l name=multus pod --timeout=300s
+kubectl create -f cni-install.yml
+sleep 1
+kubectl -n kube-system wait --for=condition=ready -l name=cni-plugins pod --timeout=300s

e2e/setup_multus.sh

@@ -0,0 +1,15 @@
+#!/bin/sh
+set -o errexit
+
+export PATH=${PATH}:./bin
+
+kind export kubeconfig
+sudo koko -d kind-worker,eth1 -d kind-worker2,eth1
+sleep 1
+kubectl -n kube-system wait --for=condition=available deploy/coredns --timeout=300s
+kubectl create -f https://raw.githubusercontent.com/intel/multus-cni/master/images/multus-daemonset.yml
+sleep 1
+kubectl -n kube-system wait --for=condition=ready -l name=multus pod --timeout=300s
+kubectl create -f cni-install.yml
+sleep 1
+kubectl -n kube-system wait --for=condition=ready -l name=cni-plugins pod --timeout=300s

e2e/test-simple-macvlan1.sh

@@ -0,0 +1,27 @@
+#!/bin/sh
+set -o errexit
+
+export PATH=${PATH}:./bin
+
+kubectl create -f macvlan1.yml
+kubectl wait --for=condition=ready -l app=macvlan --timeout=300s pod
+
+echo "check macvlan1-worker1 interface: net1"
+kubectl exec macvlan1-worker1 ip a show dev net1
+
+echo "check macvlan1-worker1 interface address: net1"
+ipaddr=$(kubectl exec macvlan1-worker1 -- ip -j a show  | jq -r \
+	'.[]|select(.ifname =="net1")|.addr_info[]|select(.family=="inet").local')
+if [ $ipaddr != "10.1.1.11" ]; then
+	echo "macvlan1-worker1 IP address is different: ${ipaddr}"
+fi
+
+echo "check macvlan1-worker2 interface: net1"
+kubectl exec macvlan1-worker2 ip a show dev net1
+
+echo "check macvlan1-worker2 interface address: net1"
+ipaddr=$(kubectl exec macvlan1-worker2 -- ip -j a show  | jq -r \
+	'.[]|select(.ifname =="net1")|.addr_info[]|select(.family=="inet").local')
+if [ $ipaddr != "10.1.1.12" ]; then
+	echo "macvlan1-worker2 IP address is different: ${ipaddr}"
+fi

examples/README.md

@@ -0,0 +1,94 @@
+# Example Configurations & Pod Specs
+
+In the `./examples` folder some example configurations are provided for using Multus, especially with CRDs, and doubly so in reference to their usage with the [defacto standard for CRDs](https://docs.google.com/document/d/1Ny03h6IDVy_e_vmElOqR7UdTPAG_RNydhVE1Kx54kFQ/edit) as proposed by the Network Plumbing Working Group.
+
+## Examples overview
+
+Generally, the examples here show a setup using Multus with CRD support. The examples here demonstrate a setup with Multus as the meta-plugin used by Kubernetes, and delegating to either Flannel (which will be the default pod network), or to macvlan. The CRDs are intended to be alignment with the defacto standard.
+
+It is expected that aspects of your own setup will vary, at least in part, from some of what's demonstrated here. Namely, the IP address spaces, and likely the host ethernet interface names used in the macvlan part of the configuration.
+
+More specifically, these examples show:
+
+* Multus configured, using CNI a `.conf` file, with CRD support, specifying that we will use a "default network".
+* A resource definition with a daemonset that places the `.conf` on each node in the cluster.
+* A CRD defining the "networks" @ `network-attachment-definitions.k8s.cni.cncf.io` 
+* CRD objects containing the configuration for both Flannel & macvlan.
+
+## Quick-start instructions
+
+* Compile Multus and place binaries into (typically) `/opt/cni/bin/`
+    - Refer to the primary README.md for more details on compilation.
+* Allow `system:node` access to enable Multus to pull CRD objects.
+    - See "RBAC configuration section below for details."
+* Create the Flannel + Multus setup with the daemonset provided
+    - As in: `kubectl create -f multus-with-flannel.yml`
+    - Optionally, verify that the `/etc/cni/net.d/*.conf` exists on each node.
+* Create the CRDs
+    - Create the CRD itself, `kubectl create -f crd.yml`
+    - Create the network attachment configurations (i.e. CNI configurations packed into CRD objects)
+        + `kubectl create -f flannel-conf.yml`
+        + `kubectl create -f macvlan-conf.yml`
+        + Verify the CRD objects are created with: `kubectl get networks`
+* Spin up an sample pod
+    - `kubectl create -f sample-pod.yml`
+    - Verify that it has multiple interfaces with:
+        + `kubectl exec -it samplepod -- ip a`
+
+## RBAC configuration
+
+You'll need to enable the `system:node` users access to the API endpoints that will deliver the CRD objects to Multus. 
+
+Using these examples, you'll first create a cluster role with the provided sample:
+
+```
+kubectl create -f clusterrole.yml
+```
+
+You will then create a `clusterrolebinding` for each hostname in the Kubernetes cluster. Replace `HOSTNAME` below with the host name of a node, and then repeat for all hostnames in the cluster.
+
+```
+kubectl create clusterrolebinding multus-node-HOSTNAME \
+    --clusterrole=multus-crd-overpowered \
+    --user=system:node:HOSTNAME
+```
+
+## CNI Configuration
+
+A sample `cni-configuration.conf` is provided, typically this file is placed in `/etc/cni/net.d/`. It must be the first file alphabetically in this folder in order for the Kubelet to honor its use. However, if you opt to use the provided Flannel + Multus YAML file, this will deploy a configuration (packed inside a daemonset therein) on each node in your Kubernetes cluster.
+
+## Other considerations
+
+Primarily in this setup one thing that one should consider are the aspects of the `macvlan-conf.yml`, which is likely specific to the configuration of the node on which this resides.
+
+## Passing down device information
+Some CNI plugins require specific device information which maybe pre-allocated by K8s device plugin. This could be indicated by providing `k8s.v1.cni.cncf.io/resourceName` annotaton in its network attachment definition CRD. The file [`examples/sriov-net.yaml`](./sriov-net.yaml) shows an example on how to define a Network attachment definition with specific device allocation information. Multus will get allocated device information and make them available for CNI plugin to work on.
+
+In this exmaple (shown below), it is expected that an [SRIOV Device Plugin](https://github.com/intel/sriov-network-device-plugin/) making a pool of SRIOV VFs available to the K8s with `intel.com/sriov` as their resourceName. Any device allocated from this resource pool will be passed down by Multus to the [sriov-cni](https://github.com/intel/sriov-cni/tree/dev/k8s-deviceid-model) plugin in `deviceID` field. This is up to the sriov-cni plugin to capture this information and work with this specific device information.
+
+```yaml
+apiVersion: "k8s.cni.cncf.io/v1"
+kind: NetworkAttachmentDefinition
+metadata:
+  name: sriov-net-a
+  annotations:
+    k8s.v1.cni.cncf.io/resourceName: intel.com/sriov
+spec:
+  config: '{
+  "type": "sriov",
+  "vlan": 1000,
+  "ipam": {
+    "type": "host-local",
+    "subnet": "10.56.217.0/24",
+    "rangeStart": "10.56.217.171",
+    "rangeEnd": "10.56.217.181",
+    "routes": [{
+      "dst": "0.0.0.0/0"
+    }],
+    "gateway": "10.56.217.1"
+  }
+}'
+```
+The [net-resource-sample-pod.yaml](./net-resource-sample-pod.yaml) is an exmaple Pod manifest file that requesting a SRIOV device from a host which is then configured using the above network attachement definition.
+
+>For further information on how to configure SRIOV Device Plugin and SRIOV-CNI please refer to the links given above.

examples/macvlan-pod.yml

@@ -0,0 +1,56 @@
+---
+# This net-attach-def defines macvlan-conf with 
+#   + ips capabilities to specify ip in pod annotation and 
+#   + mac capabilities to specify mac address in pod annotation
+# default gateway is defined as well
+apiVersion: "k8s.cni.cncf.io/v1"
+kind: NetworkAttachmentDefinition
+metadata:
+  name: macvlan-conf
+spec: 
+  config: '{
+      "cniVersion": "0.3.1",
+      "plugins": [
+        {
+          "type": "macvlan",
+          "capabilities": { "ips": true },
+          "master": "eth0",
+          "mode": "bridge",
+          "ipam": {
+            "type": "static",
+            "routes": [
+              {
+                "dst": "0.0.0.0/0",
+                "gw": "10.1.1.1"
+              }
+            ] 
+          }
+        }, {
+          "capabilities": { "mac": true },
+          "type": "tuning"
+        }
+      ]
+    }'
+---
+# Define a pod with macvlan-conf, defined above, with ip address and mac, and 
+# "gateway" overrides default gateway to use macvlan-conf's one. 
+# without "gateway" in k8s.v1.cni.cncf.io/networks, default route will be cluster
+# network interface, eth0, even tough macvlan-conf has default gateway config.
+apiVersion: v1
+kind: Pod
+metadata:
+  name: samplepod
+  annotations:
+    k8s.v1.cni.cncf.io/networks: '[
+            { "name": "macvlan-conf",
+              "ips": [ "10.1.1.101/24" ],
+              "mac": "c2:b0:57:49:47:f1",
+              "gateway": [ "10.1.1.1" ]
+            }]'
+spec:
+  containers:
+  - name: samplepod
+    command: ["/bin/bash", "-c", "trap : TERM INT; sleep infinity & wait"]
+    image: dougbtv/centos-network
+    ports:
+    - containerPort: 80

examples/sriov-pod.yml

@@ -0,0 +1,47 @@
+# This net-attach-def defines SR-IOV CNI config
+# Please see https://github.com/intel/sriov-cni and https://github.com/intel/sriov-network-device-plugin
+# for its detail.
+---
+apiVersion: "k8s.cni.cncf.io/v1"
+kind: NetworkAttachmentDefinition
+metadata:
+  name: sriov-net-a
+  annotations:
+    k8s.v1.cni.cncf.io/resourceName: intel.com/sriov
+spec:
+  config: '{
+  "type": "sriov",
+  "vlan": 1000,
+  "ipam": {
+    "type": "host-local",
+    "subnet": "10.56.217.0/24",
+    "rangeStart": "10.56.217.171",
+    "rangeEnd": "10.56.217.181",
+    "routes": [{
+      "dst": "0.0.0.0/0"
+    }],
+    "gateway": "10.56.217.1"
+  }
+}'
+---
+apiVersion: v1
+kind: Pod
+metadata:
+  name: testpod1
+  labels:
+    env: test
+  annotations:
+    k8s.v1.cni.cncf.io/networks: sriov-net-a
+spec:
+  containers:
+  - name: appcntr1
+    image: centos/tools
+    imagePullPolicy: IfNotPresent
+    command: [ "/bin/bash", "-c", "--" ]
+    args: [ "while true; do sleep 300000; done;" ]
+    resources:
+      requests:
+        intel.com/sriov: '1'
+      limits:
+        intel.com/sriov: '1'
+  restartPolicy: "Never"

go.mod

@@ -0,0 +1,29 @@
+module gopkg.in/intel/multus-cni.v3
+
+go 1.12
+
+require (
+	github.com/Microsoft/go-winio v0.4.14 // indirect
+	github.com/containernetworking/cni v0.7.1
+	github.com/containernetworking/plugins v0.8.2
+	github.com/golang/protobuf v1.3.2 // indirect
+	github.com/json-iterator/go v1.1.9 // indirect
+	github.com/k8snetworkplumbingwg/network-attachment-definition-client v0.0.0-20200127152046-0ee521d56061
+	github.com/onsi/ginkgo v1.10.1
+	github.com/onsi/gomega v1.7.0
+	github.com/pkg/errors v0.8.1
+	github.com/vishvananda/netlink v0.0.0-20181108222139-023a6dafdcdf
+	github.com/vishvananda/netns v0.0.0-20190625233234-7109fa855b0f // indirect
+	golang.org/x/crypto v0.0.0-20191011191535-87dc89f01550 // indirect
+	golang.org/x/net v0.0.0-20200226121028-0de0cce0169b
+	golang.org/x/text v0.3.3 // indirect
+	google.golang.org/grpc v1.23.0
+	gopkg.in/natefinch/lumberjack.v2 v2.0.0
+	gopkg.in/yaml.v2 v2.2.8 // indirect
+	k8s.io/api v0.0.0-20181115043458-b799cb063522
+	k8s.io/apimachinery v0.0.0-20181110190943-2a7c93004028
+	k8s.io/client-go v0.0.0-20181115111358-9bea17718df8
+	k8s.io/klog v1.0.0
+	k8s.io/kube-openapi v0.0.0-20200121204235-bf4fb3bd569c // indirect
+	k8s.io/kubernetes v1.13.0
+)

go.sum

@@ -0,0 +1,291 @@
+cloud.google.com/go v0.26.0/go.mod h1:aQUYkXzVsufM+DwF1aE+0xfcU+56JwCaLick0ClmMTw=
+github.com/BurntSushi/toml v0.3.1 h1:WXkYYl6Yr3qBf1K79EBnL4mak0OimBfB0XUf9Vl28OQ=
+github.com/BurntSushi/toml v0.3.1/go.mod h1:xHWCNGjB5oqiDr8zfno3MHue2Ht5sIBksp03qcyfWMU=
+github.com/BurntSushi/xgb v0.0.0-20160522181843-27f122750802/go.mod h1:IVnqGOEym/WlBOVXweHU+Q+/VP0lqqI8lqeDx9IjBqo=
+github.com/Microsoft/go-winio v0.4.11/go.mod h1:VhR8bwka0BXejwEJY73c50VrPtXAaKcyvVC4A4RozmA=
+github.com/Microsoft/go-winio v0.4.14 h1:+hMXMk01us9KgxGb7ftKQt2Xpf5hH/yky+TDA+qxleU=
+github.com/Microsoft/go-winio v0.4.14/go.mod h1:qXqCSQ3Xa7+6tgxaGTIe4Kpcdsi+P8jBhyzoq1bpyYA=
+github.com/Microsoft/hcsshim v0.8.6/go.mod h1:Op3hHsoHPAvb6lceZHDtd9OkTew38wNoXnJs8iY7rUg=
+github.com/NYTimes/gziphandler v0.0.0-20170623195520-56545f4a5d46/go.mod h1:3wb06e3pkSAbeQ52E9H9iFoQsEEwGN64994WTCIhntQ=
+github.com/PuerkitoBio/purell v1.0.0/go.mod h1:c11w/QuzBsJSee3cPx9rAFu61PvFxuPbtSwDGJws/X0=
+github.com/PuerkitoBio/purell v1.1.1 h1:WEQqlqaGbrPkxLJWfBwQmfEAE1Z7ONdDLqrN38tNFfI=
+github.com/PuerkitoBio/purell v1.1.1/go.mod h1:c11w/QuzBsJSee3cPx9rAFu61PvFxuPbtSwDGJws/X0=
+github.com/PuerkitoBio/urlesc v0.0.0-20160726150825-5bd2802263f2/go.mod h1:uGdkoq3SwY9Y+13GIhn11/XLaGBb4BfwItxLd5jeuXE=
+github.com/PuerkitoBio/urlesc v0.0.0-20170810143723-de5bf2ad4578 h1:d+Bc7a5rLufV/sSk/8dngufqelfh6jnri85riMAaF/M=
+github.com/PuerkitoBio/urlesc v0.0.0-20170810143723-de5bf2ad4578/go.mod h1:uGdkoq3SwY9Y+13GIhn11/XLaGBb4BfwItxLd5jeuXE=
+github.com/alexflint/go-filemutex v0.0.0-20171022225611-72bdc8eae2ae/go.mod h1:CgnQgUtFrFz9mxFNtED3jI5tLDjKlOM+oUF/sTk6ps0=
+github.com/buger/jsonparser v0.0.0-20180808090653-f4dd9f5a6b44/go.mod h1:bbYlZJ7hK1yFx9hf58LP0zeX7UjIGs20ufpu3evjr+s=
+github.com/client9/misspell v0.3.4/go.mod h1:qj6jICC3Q7zFZvVWo7KLAzC3yx5G7kyvSDkc90ppPyw=
+github.com/containernetworking/cni v0.7.0 h1:1Qy7EwdC08mx5wUB0DpjCuBrk6e/uXg9yI9TvAvgox8=
+github.com/containernetworking/cni v0.7.0/go.mod h1:LGwApLUm2FpoOfxTDEeq8T9ipbpZ61X79hmU3w8FmsY=
+github.com/containernetworking/cni v0.7.1 h1:fE3r16wpSEyaqY4Z4oFrLMmIGfBYIKpPrHK31EJ9FzE=
+github.com/containernetworking/cni v0.7.1/go.mod h1:LGwApLUm2FpoOfxTDEeq8T9ipbpZ61X79hmU3w8FmsY=
+github.com/containernetworking/plugins v0.8.2 h1:5lnwfsAYO+V7yXhysJKy3E1A2Gy9oVut031zfdOzI9w=
+github.com/containernetworking/plugins v0.8.2/go.mod h1:TxALKWZpWL79BC3GOYKJzzXr7U8R23PdhwaLp6F3adc=
+github.com/coreos/go-iptables v0.4.2/go.mod h1:/mVI274lEDI2ns62jHCDnCyBF9Iwsmekav8Dbxlm1MU=
+github.com/coreos/go-systemd v0.0.0-20180511133405-39ca1b05acc7/go.mod h1:F5haX7vjVVG0kc13fIWeqUViNPyEJxv/OmvnBo0Yme4=
+github.com/d2g/dhcp4 v0.0.0-20170904100407-a1d1b6c41b1c/go.mod h1:Ct2BUK8SB0YC1SMSibvLzxjeJLnrYEVLULFNiHY9YfQ=
+github.com/d2g/dhcp4client v1.0.0/go.mod h1:j0hNfjhrt2SxUOw55nL0ATM/z4Yt3t2Kd1mW34z5W5s=
+github.com/d2g/dhcp4server v0.0.0-20181031114812-7d4a0a7f59a5/go.mod h1:Eo87+Kg/IX2hfWJfwxMzLyuSZyxSoAug2nGa1G2QAi8=
+github.com/d2g/hardwareaddr v0.0.0-20190221164911-e7d9fbe030e4/go.mod h1:bMl4RjIciD2oAxI7DmWRx6gbeqrkoLqv3MV0vzNad+I=
+github.com/davecgh/go-spew v1.1.0/go.mod h1:J7Y8YcW2NihsgmVo/mv3lAwl/skON4iLHjSsI+c5H38=
+github.com/davecgh/go-spew v1.1.1 h1:vj9j/u1bqnvCEfJOwUhtlOARqs3+rkHYY13jYWTU97c=
+github.com/davecgh/go-spew v1.1.1/go.mod h1:J7Y8YcW2NihsgmVo/mv3lAwl/skON4iLHjSsI+c5H38=
+github.com/emicklei/go-restful v0.0.0-20170410110728-ff4f55a20633/go.mod h1:otzb+WCGbkyDHkqmQmT5YD2WR4BBwUdeQoFo8l/7tVs=
+github.com/emicklei/go-restful v2.10.0+incompatible h1:l6Soi8WCOOVAeCo4W98iBFC6Og7/X8bpRt51oNLZ2C8=
+github.com/emicklei/go-restful v2.10.0+incompatible/go.mod h1:otzb+WCGbkyDHkqmQmT5YD2WR4BBwUdeQoFo8l/7tVs=
+github.com/evanphx/json-patch v4.1.0+incompatible h1:K1MDoo4AZ4wU0GIU/fPmtZg7VpzLjCxu+UwBD1FvwOc=
+github.com/evanphx/json-patch v4.1.0+incompatible/go.mod h1:50XU6AFN0ol/bzJsmQLiYLvXMP4fmwYFNcr97nuDLSk=
+github.com/fsnotify/fsnotify v1.4.7 h1:IXs+QLmnXW2CcXuY+8Mzv/fWEsPGWxqefPtCP5CnV9I=
+github.com/fsnotify/fsnotify v1.4.7/go.mod h1:jwhsz4b93w/PPRr/qN1Yymfu8t87LnFCMoQvtojpjFo=
+github.com/ghodss/yaml v0.0.0-20150909031657-73d445a93680/go.mod h1:4dBDuWmgqj2HViK6kFavaiC9ZROes6MMH2rRYeMEF04=
+github.com/go-logr/logr v0.1.0/go.mod h1:ixOQHD9gLJUVQQ2ZOR7zLEifBX6tGkNJF4QyIY7sIas=
+github.com/go-openapi/jsonpointer v0.0.0-20160704185906-46af16f9f7b1/go.mod h1:+35s3my2LFTysnkMfxsJBAMHj/DoqoB9knIWoYG/Vk0=
+github.com/go-openapi/jsonpointer v0.19.2/go.mod h1:3akKfEdA7DF1sugOqz1dVQHBcuDBPKZGEoHC/NkiQRg=
+github.com/go-openapi/jsonpointer v0.19.3 h1:gihV7YNZK1iK6Tgwwsxo2rJbD1GTbdm72325Bq8FI3w=
+github.com/go-openapi/jsonpointer v0.19.3/go.mod h1:Pl9vOtqEWErmShwVjC8pYs9cog34VGT37dQOVbmoatg=
+github.com/go-openapi/jsonreference v0.0.0-20160704190145-13c6e3589ad9/go.mod h1:W3Z9FmVs9qj+KR4zFKmDPGiLdk1D9Rlm7cyMvf57TTg=
+github.com/go-openapi/jsonreference v0.19.2 h1:o20suLFB4Ri0tuzpWtyHlh7E7HnkqTNLq6aR6WVNS1w=
+github.com/go-openapi/jsonreference v0.19.2/go.mod h1:jMjeRr2HHw6nAVajTXJ4eiUwohSTlpa0o73RUL1owJc=
+github.com/go-openapi/spec v0.0.0-20160808142527-6aced65f8501/go.mod h1:J8+jY1nAiCcj+friV/PDoE1/3eeccG9LYBs0tYvLOWc=
+github.com/go-openapi/spec v0.19.3 h1:0XRyw8kguri6Yw4SxhsQA/atC88yqrk0+G4YhI2wabc=
+github.com/go-openapi/spec v0.19.3/go.mod h1:FpwSN1ksY1eteniUU7X0N/BgJ7a4WvBFVA8Lj9mJglo=
+github.com/go-openapi/swag v0.0.0-20160704191624-1d0bd113de87/go.mod h1:DXUve3Dpr1UfpPtxFw+EFuQ41HhCWZfha5jSVRG7C7I=
+github.com/go-openapi/swag v0.19.2/go.mod h1:POnQmlKehdgb5mhVOsnJFsivZCEZ/vjK9gh66Z9tfKk=
+github.com/go-openapi/swag v0.19.5 h1:lTz6Ys4CmqqCQmZPBlbQENR1/GucA2bzYTE12Pw4tFY=
+github.com/go-openapi/swag v0.19.5/go.mod h1:POnQmlKehdgb5mhVOsnJFsivZCEZ/vjK9gh66Z9tfKk=
+github.com/godbus/dbus v0.0.0-20180201030542-885f9cc04c9c/go.mod h1:/YcGZj5zSblfDWMMoOzV4fas9FZnQYTkDnsGvmh2Grw=
+github.com/gogo/protobuf v1.1.1/go.mod h1:r8qH/GZQm5c6nD/R0oafs1akxWv10x8SbQlK7atdtwQ=
+github.com/gogo/protobuf v1.3.0 h1:G8O7TerXerS4F6sx9OV7/nRfJdnXgHZu/S/7F2SN+UE=
+github.com/gogo/protobuf v1.3.0/go.mod h1:SlYgWuQ5SjCEi6WLHjHCa1yvBfUnHcTbrrZtXPKa29o=
+github.com/golang/glog v0.0.0-20160126235308-23def4e6c14b h1:VKtxabqXZkF25pY9ekfRL6a582T4P37/31XEstQ5p58=
+github.com/golang/glog v0.0.0-20160126235308-23def4e6c14b/go.mod h1:SBH7ygxi8pfUlaOkMMuAQtPIUF8ecWP5IEl/CR7VP2Q=
+github.com/golang/groupcache v0.0.0-20190702054246-869f871628b6 h1:ZgQEtGgCBiWRM39fZuwSd1LwSqqSW0hOdXCYYDX0R3I=
+github.com/golang/groupcache v0.0.0-20190702054246-869f871628b6/go.mod h1:cIg4eruTrX1D+g88fzRXU5OdNfaM+9IcxsU14FzY7Hc=
+github.com/golang/mock v1.1.1/go.mod h1:oTYuIxOrZwtPieC+H1uAHpcLFnEyAGVDL/k47Jfbm0A=
+github.com/golang/protobuf v0.0.0-20161109072736-4bd1920723d7/go.mod h1:6lQm79b+lXiMfvg/cZm0SGofjICqVBUtrP5yJMmIC1U=
+github.com/golang/protobuf v1.2.0/go.mod h1:6lQm79b+lXiMfvg/cZm0SGofjICqVBUtrP5yJMmIC1U=
+github.com/golang/protobuf v1.3.1/go.mod h1:6lQm79b+lXiMfvg/cZm0SGofjICqVBUtrP5yJMmIC1U=
+github.com/golang/protobuf v1.3.2 h1:6nsPYzhq5kReh6QImI3k5qWzO4PEbvbIW2cwSfR/6xs=
+github.com/golang/protobuf v1.3.2/go.mod h1:6lQm79b+lXiMfvg/cZm0SGofjICqVBUtrP5yJMmIC1U=
+github.com/google/btree v1.0.0 h1:0udJVsspx3VBr5FwtLhQQtuAsVc79tTq0ocGIPAU6qo=
+github.com/google/btree v1.0.0/go.mod h1:lNA+9X1NB3Zf8V7Ke586lFgjr2dZNuvo3lPJSGZ5JPQ=
+github.com/google/go-cmp v0.2.0/go.mod h1:oXzfMopK8JAjlY9xF4vHSVASa0yLyX7SntLO5aqRK0M=
+github.com/google/gofuzz v0.0.0-20170612174753-24818f796faf h1:+RRA9JqSOZFfKrOeqr2z77+8R2RKyh8PG66dcu1V0ck=
+github.com/google/gofuzz v0.0.0-20170612174753-24818f796faf/go.mod h1:HP5RmnzzSNb993RKQDq4+1A4ia9nllfqcQFTQJedwGI=
+github.com/google/gofuzz v1.0.0 h1:A8PeW59pxE9IoFRqBp37U+mSNaQoZ46F1f0f863XSXw=
+github.com/google/gofuzz v1.0.0/go.mod h1:dBl0BpW6vV/+mYPU4Po3pmUjxk6FQPldtuIdl/M65Eg=
+github.com/googleapis/gnostic v0.0.0-20170729233727-0c5108395e2d/go.mod h1:sJBsCZ4ayReDTBIg8b9dl28c5xFWyhBTVRp3pOg5EKY=
+github.com/googleapis/gnostic v0.2.0 h1:l6N3VoaVzTncYYW+9yOz2LJJammFZGBO13sqgEhpy9g=
+github.com/googleapis/gnostic v0.2.0/go.mod h1:sJBsCZ4ayReDTBIg8b9dl28c5xFWyhBTVRp3pOg5EKY=
+github.com/gregjones/httpcache v0.0.0-20181110185634-c63ab54fda8f h1:ShTPMJQes6tubcjzGMODIVG5hlrCeImaBnZzKF2N8SM=
+github.com/gregjones/httpcache v0.0.0-20181110185634-c63ab54fda8f/go.mod h1:FecbI9+v66THATjSRHfNgh1IVFe/9kFxbXtjV0ctIMA=
+github.com/hashicorp/golang-lru v0.5.0/go.mod h1:/m3WP610KZHVQ1SGc6re/UDhFvYD7pJ4Ao+sR/qLZy8=
+github.com/hpcloud/tail v1.0.0 h1:nfCOvKYfkgYP8hkirhJocXT2+zOD8yUNjXaWfTlyFKI=
+github.com/hpcloud/tail v1.0.0/go.mod h1:ab1qPbhIpdTxEkNHXyeSf5vhxWSCs/tWer42PpOxQnU=
+github.com/imdario/mergo v0.0.0-20171009183408-7fe0c75c13ab h1:k/Biv+LJL35wkk0Hveko1nj7as4tSHkHdZaNlzn/gcQ=
+github.com/imdario/mergo v0.0.0-20171009183408-7fe0c75c13ab/go.mod h1:2EnlNZ0deacrJVfApfmtdGgDfMuh/nq6Ok1EcJh5FfA=
+github.com/j-keck/arping v0.0.0-20160618110441-2cf9dc699c56/go.mod h1:ymszkNOg6tORTn+6F6j+Jc8TOr5osrynvN6ivFWZ2GA=
+github.com/json-iterator/go v0.0.0-20180612202835-f2b4162afba3 h1:/UewZcckqhvnnS0C6r3Sher2hSEbVmM6Ogpcjen08+Y=
+github.com/json-iterator/go v0.0.0-20180612202835-f2b4162afba3/go.mod h1:+SdeFBvtyEkXs7REEP0seUULqWtbJapLOCVDaaPEHmU=
+github.com/json-iterator/go v1.1.6/go.mod h1:+SdeFBvtyEkXs7REEP0seUULqWtbJapLOCVDaaPEHmU=
+github.com/json-iterator/go v1.1.9 h1:9yzud/Ht36ygwatGx56VwCZtlI/2AD15T1X2sjSuGns=
+github.com/json-iterator/go v1.1.9/go.mod h1:KdQUCv79m/52Kvf8AW2vK1V8akMuk1QjK/uOdHXbAo4=
+github.com/juju/errors v0.0.0-20180806074554-22422dad46e1/go.mod h1:W54LbzXuIE0boCoNJfwqpmkKJ1O4TCTZMetAt6jGk7Q=
+github.com/juju/loggo v0.0.0-20190526231331-6e530bcce5d8/go.mod h1:vgyd7OREkbtVEN/8IXZe5Ooef3LQePvuBm9UWj6ZL8U=
+github.com/juju/testing v0.0.0-20190613124551-e81189438503/go.mod h1:63prj8cnj0tU0S9OHjGJn+b1h0ZghCndfnbQolrYTwA=
+github.com/k8snetworkplumbingwg/network-attachment-definition-client v0.0.0-20200127152046-0ee521d56061 h1:zz0mSqgjSJP6gqP2b7GdCiysj5OgD2DMJRNFJegLcs4=
+github.com/k8snetworkplumbingwg/network-attachment-definition-client v0.0.0-20200127152046-0ee521d56061/go.mod h1:MP2HbArq3QT+oVp8pmtHNZnSnkhdkHtDnc7h6nJXmBU=
+github.com/kisielk/errcheck v1.2.0/go.mod h1:/BMXB+zMLi60iA8Vv6Ksmxu/1UDYcXs4uQLJ+jE2L00=
+github.com/kisielk/gotool v1.0.0/go.mod h1:XhKaO+MFFWcvkIS/tQcRk01m1F5IRFswLeQ+oQHNcck=
+github.com/konsorten/go-windows-terminal-sequences v1.0.1/go.mod h1:T0+1ngSBFLxvqU3pZ+m/2kptfBszLMUkC4ZK/EgS/cQ=
+github.com/kr/pretty v0.1.0 h1:L/CwN0zerZDmRFUapSPitk6f+Q3+0za1rQkzVuMiMFI=
+github.com/kr/pretty v0.1.0/go.mod h1:dAy3ld7l9f0ibDNOQOHHMYYIIbhfbHSm3C4ZsoJORNo=
+github.com/kr/pty v1.1.1/go.mod h1:pFQYn66WHrOpPYNljwOMqo10TkYh1fy3cYio2l3bCsQ=
+github.com/kr/pty v1.1.5/go.mod h1:9r2w37qlBe7rQ6e1fg1S/9xpWHSnaqNdHD3WcMdbPDA=
+github.com/kr/text v0.1.0 h1:45sCR5RtlFHMR4UwH9sdQ5TC8v0qDQCHnXt+kaKSTVE=
+github.com/kr/text v0.1.0/go.mod h1:4Jbv+DJW3UT/LiOwJeYQe1efqtUx/iVham/4vfdArNI=
+github.com/mailru/easyjson v0.0.0-20160728113105-d5b7844b561a/go.mod h1:C1wdFJiN94OJF2b5HbByQZoLdCWB1Yqtg26g4irojpc=
+github.com/mailru/easyjson v0.0.0-20190614124828-94de47d64c63/go.mod h1:C1wdFJiN94OJF2b5HbByQZoLdCWB1Yqtg26g4irojpc=
+github.com/mailru/easyjson v0.0.0-20190626092158-b2ccc519800e h1:hB2xlXdHp/pmPZq0y3QnmWAArdw9PqbmotexnWx/FU8=
+github.com/mailru/easyjson v0.0.0-20190626092158-b2ccc519800e/go.mod h1:C1wdFJiN94OJF2b5HbByQZoLdCWB1Yqtg26g4irojpc=
+github.com/mattn/go-shellwords v1.0.3/go.mod h1:3xCvwCdWdlDJUrvuMn7Wuy9eWs4pE8vqg+NOMyg4B2o=
+github.com/modern-go/concurrent v0.0.0-20180228061459-e0a39a4cb421/go.mod h1:6dJC0mAP4ikYIbvyc7fijjWJddQyLn8Ig3JB5CqoB9Q=
+github.com/modern-go/concurrent v0.0.0-20180306012644-bacd9c7ef1dd h1:TRLaZ9cD/w8PVh93nsPXa1VrQ6jlwL5oN8l14QlcNfg=
+github.com/modern-go/concurrent v0.0.0-20180306012644-bacd9c7ef1dd/go.mod h1:6dJC0mAP4ikYIbvyc7fijjWJddQyLn8Ig3JB5CqoB9Q=
+github.com/modern-go/reflect2 v0.0.0-20180701023420-4b7aa43c6742 h1:Esafd1046DLDQ0W1YjYsBW+p8U2u7vzgW2SQVmlNazg=
+github.com/modern-go/reflect2 v0.0.0-20180701023420-4b7aa43c6742/go.mod h1:bx2lNnkwVCuqBIxFjflWJWanXIb3RllmbCylyMrvgv0=
+github.com/modern-go/reflect2 v1.0.1 h1:9f412s+6RmYXLWZSEzVVgPGK7C2PphHj5RJrvfx9AWI=
+github.com/modern-go/reflect2 v1.0.1/go.mod h1:bx2lNnkwVCuqBIxFjflWJWanXIb3RllmbCylyMrvgv0=
+github.com/munnerz/goautoneg v0.0.0-20120707110453-a547fc61f48d/go.mod h1:+n7T8mK8HuQTcFwEeznm/DIxMOiR9yIdICNftLE1DvQ=
+github.com/onsi/ginkgo v0.0.0-20151202141238-7f8ab55aaf3b h1:Ey6yH0acn50T/v6CB75bGP4EMJqnv9WvnjN7oZaj+xE=
+github.com/onsi/ginkgo v0.0.0-20151202141238-7f8ab55aaf3b/go.mod h1:lLunBs/Ym6LB5Z9jYTR76FiuTmxDTDusOGeTQH+WWjE=
+github.com/onsi/ginkgo v0.0.0-20170829012221-11459a886d9c/go.mod h1:lLunBs/Ym6LB5Z9jYTR76FiuTmxDTDusOGeTQH+WWjE=
+github.com/onsi/ginkgo v1.6.0/go.mod h1:lLunBs/Ym6LB5Z9jYTR76FiuTmxDTDusOGeTQH+WWjE=
+github.com/onsi/ginkgo v1.10.1 h1:q/mM8GF/n0shIN8SaAZ0V+jnLPzen6WIVZdiwrRlMlo=
+github.com/onsi/ginkgo v1.10.1/go.mod h1:lLunBs/Ym6LB5Z9jYTR76FiuTmxDTDusOGeTQH+WWjE=
+github.com/onsi/gomega v0.0.0-20151007035656-2152b45fa28a h1:KfNOeFvoAssuZLT7IntKZElKwi/5LRuxY71k+t6rfaM=
+github.com/onsi/gomega v0.0.0-20151007035656-2152b45fa28a/go.mod h1:C1qb7wdrVGGVU+Z6iS04AVkA3Q65CEZX59MT0QO5uiA=
+github.com/onsi/gomega v0.0.0-20170829124025-dcabb60a477c/go.mod h1:C1qb7wdrVGGVU+Z6iS04AVkA3Q65CEZX59MT0QO5uiA=
+github.com/onsi/gomega v1.7.0 h1:XPnZz8VVBHjVsy1vzJmRwIcSwiUO+JFfrv/xGiigmME=
+github.com/onsi/gomega v1.7.0/go.mod h1:ex+gbHU/CVuBBDIJjb2X0qEXbFg53c61hWP/1CpauHY=
+github.com/peterbourgon/diskv v2.0.1+incompatible h1:UBdAOUP5p4RWqPBg048CAvpKN+vxiaj6gdUUzhl4XmI=
+github.com/peterbourgon/diskv v2.0.1+incompatible/go.mod h1:uqqh8zWWbv1HBMNONnaR/tNboyR3/BZd58JJSHlUSCU=
+github.com/pkg/errors v0.8.1 h1:iURUrRGxPUNPdy5/HRSm+Yj6okJ6UtLINN0Q9M4+h3I=
+github.com/pkg/errors v0.8.1/go.mod h1:bwawxfHBFNV+L2hUp1rHADufV3IMtnDRdf1r5NINEl0=
+github.com/pmezard/go-difflib v1.0.0 h1:4DBwDE0NGyQoBHbLQYPwSUPoCMWR5BEzIk/f1lZbAQM=
+github.com/pmezard/go-difflib v1.0.0/go.mod h1:iKH77koFhYxTK1pcRnkKkqfTogsbg7gZNVY4sRDYZ/4=
+github.com/remyoudompheng/bigfft v0.0.0-20170806203942-52369c62f446/go.mod h1:uYEyJGbgTkfkS4+E/PavXkNJcbFIpEtjt2B0KDQ5+9M=
+github.com/safchain/ethtool v0.0.0-20190326074333-42ed695e3de8/go.mod h1:Z0q5wiBQGYcxhMZ6gUqHn6pYNLypFAvaL3UvgZLR0U4=
+github.com/sirupsen/logrus v1.0.6/go.mod h1:pMByvHTf9Beacp5x1UXfOR9xyW/9antXMhjMPG0dEzc=
+github.com/sirupsen/logrus v1.4.1/go.mod h1:ni0Sbl8bgC9z8RoU9G6nDWqqs/fq4eDPysMBDgk/93Q=
+github.com/spf13/pflag v0.0.0-20170130214245-9ff6c6923cff/go.mod h1:DYY7MBk1bdzusC3SYhjObp+wFpr4gzcvqqNjLnInEg4=
+github.com/spf13/pflag v1.0.1 h1:aCvUg6QPl3ibpQUxyLkrEkCHtPqYJL4x9AuhqVqFis4=
+github.com/spf13/pflag v1.0.1/go.mod h1:DYY7MBk1bdzusC3SYhjObp+wFpr4gzcvqqNjLnInEg4=
+github.com/spf13/pflag v1.0.5 h1:iy+VFUOCP1a+8yFto/drg2CJ5u0yRoB7fZw3DKv/JXA=
+github.com/spf13/pflag v1.0.5/go.mod h1:McXfInJRrz4CZXVZOBLb0bTZqETkiAhM9Iw0y3An2Bg=
+github.com/stretchr/objx v0.1.0 h1:4G4v2dO3VZwixGIRoQ5Lfboy6nUhCyYzaqnIAPPhYs4=
+github.com/stretchr/objx v0.1.0/go.mod h1:HFkY916IF+rwdDfMAkV7OtwuqBVzrE8GR6GFx+wExME=
+github.com/stretchr/objx v0.1.1/go.mod h1:HFkY916IF+rwdDfMAkV7OtwuqBVzrE8GR6GFx+wExME=
+github.com/stretchr/objx v0.2.0 h1:Hbg2NidpLE8veEBkEZTL3CvlkUIVzuU9jDplZO54c48=
+github.com/stretchr/objx v0.2.0/go.mod h1:qt09Ya8vawLte6SNmTgCsAVtYtaKzEcn8ATUoHMkEqE=
+github.com/stretchr/testify v1.2.2/go.mod h1:a8OnRcib4nhh0OaRAV+Yts87kKdq0PP7pXfy6kDkUVs=
+github.com/stretchr/testify v1.3.0 h1:TivCn/peBQ7UY8ooIcPgZFpTNSz0Q2U6UrFlUfqbe0Q=
+github.com/stretchr/testify v1.3.0/go.mod h1:M5WIy9Dh21IEIfnGCwXGc5bZfKNJtfHm1UVUgZn+9EI=
+github.com/stretchr/testify v1.4.0 h1:2E4SXV/wtOkTonXsotYi4li6zVWxYlZuYNCXe9XRJyk=
+github.com/stretchr/testify v1.4.0/go.mod h1:j7eGeouHqKxXV5pUuKE4zz7dFj8WfuZ+81PSLYec5m4=
+github.com/vishvananda/netlink v0.0.0-20181108222139-023a6dafdcdf h1:3J37+NPjNyGW/dbfXtj3yWuF9OEepIdGOXRaJGbORV8=
+github.com/vishvananda/netlink v0.0.0-20181108222139-023a6dafdcdf/go.mod h1:+SR5DhBJrl6ZM7CoCKvpw5BKroDKQ+PJqOg65H/2ktk=
+github.com/vishvananda/netns v0.0.0-20180720170159-13995c7128cc/go.mod h1:ZjcWmFBXmLKZu9Nxj3WKYEafiSqer2rnvPr0en9UNpI=
+github.com/vishvananda/netns v0.0.0-20190625233234-7109fa855b0f h1:nBX3nTcmxEtHSERBJaIo1Qa26VwRaopnZmfDQUXsF4I=
+github.com/vishvananda/netns v0.0.0-20190625233234-7109fa855b0f/go.mod h1:ZjcWmFBXmLKZu9Nxj3WKYEafiSqer2rnvPr0en9UNpI=
+golang.org/x/crypto v0.0.0-20181009213950-7c1a557ab941/go.mod h1:6SG95UA2DQfeDnfUPMdvaQW0Q7yPrPDi9nlGo2tz2b4=
+golang.org/x/crypto v0.0.0-20190308221718-c2843e01d9a2 h1:VklqNMn3ovrHsnt90PveolxSbWFaJdECFbxSq0Mqo2M=
+golang.org/x/crypto v0.0.0-20190308221718-c2843e01d9a2/go.mod h1:djNgcEr1/C05ACkg1iLfiJU5Ep61QUkGW8qpdssI0+w=
+golang.org/x/crypto v0.0.0-20190611184440-5c40567a22f8/go.mod h1:yigFU9vqHzYiE8UmvKecakEJjdnWj3jj499lnFckfCI=
+golang.org/x/crypto v0.0.0-20190927123631-a832865fa7ad h1:5E5raQxcv+6CZ11RrBYQe5WRbUIWpScjh0kvHZkZIrQ=
+golang.org/x/crypto v0.0.0-20190927123631-a832865fa7ad/go.mod h1:yigFU9vqHzYiE8UmvKecakEJjdnWj3jj499lnFckfCI=
+golang.org/x/crypto v0.0.0-20191011191535-87dc89f01550 h1:ObdrDkeb4kJdCP557AjRjq69pTHfNouLtWZG7j9rPN8=
+golang.org/x/crypto v0.0.0-20191011191535-87dc89f01550/go.mod h1:yigFU9vqHzYiE8UmvKecakEJjdnWj3jj499lnFckfCI=
+golang.org/x/exp v0.0.0-20190125153040-c74c464bbbf2/go.mod h1:CJ0aWSM057203Lf6IL+f9T1iT9GByDxfZKAQTCR3kQA=
+golang.org/x/exp v0.0.0-20190312203227-4b39c73a6495/go.mod h1:ZjyILWgesfNpC6sMxTJOJm9Kp84zZh5NQWvqDGG3Qr8=
+golang.org/x/image v0.0.0-20190227222117-0694c2d4d067/go.mod h1:kZ7UVZpmo3dzQBMxlp+ypCbDeSB+sBbTgSJuh5dn5js=
+golang.org/x/lint v0.0.0-20190313153728-d0100b6bd8b3 h1:XQyxROzUlZH+WIQwySDgnISgOivlhjIEwaQaJEJrrN0=
+golang.org/x/lint v0.0.0-20190313153728-d0100b6bd8b3/go.mod h1:6SW0HCj/g11FgYtHlgUYUwCkIfeOF89ocIRzGO/8vkc=
+golang.org/x/mobile v0.0.0-20190312151609-d3739f865fa6/go.mod h1:z+o9i4GpDbdi3rU15maQ/Ox0txvL9dWGYEHz965HBQE=
+golang.org/x/net v0.0.0-20170114055629-f2499483f923/go.mod h1:mL1N/T3taQHkDXs73rZJwtUhF3w3ftmwwsq0BUmARs4=
+golang.org/x/net v0.0.0-20180724234803-3673e40ba225/go.mod h1:mL1N/T3taQHkDXs73rZJwtUhF3w3ftmwwsq0BUmARs4=
+golang.org/x/net v0.0.0-20180906233101-161cd47e91fd/go.mod h1:mL1N/T3taQHkDXs73rZJwtUhF3w3ftmwwsq0BUmARs4=
+golang.org/x/net v0.0.0-20181011144130-49bb7cea24b1/go.mod h1:mL1N/T3taQHkDXs73rZJwtUhF3w3ftmwwsq0BUmARs4=
+golang.org/x/net v0.0.0-20190311183353-d8887717615a/go.mod h1:t9HGtf8HONx5eT2rtn7q6eTqICYqUVnKs3thJo3Qplg=
+golang.org/x/net v0.0.0-20190404232315-eb5bcb51f2a3/go.mod h1:t9HGtf8HONx5eT2rtn7q6eTqICYqUVnKs3thJo3Qplg=
+golang.org/x/net v0.0.0-20190613194153-d28f0bde5980/go.mod h1:z5CRVTTTmAJ677TzLLGU+0bjPO0LkuOLi4/5GtJWs/s=
+golang.org/x/net v0.0.0-20190620200207-3b0461eec859/go.mod h1:z5CRVTTTmAJ677TzLLGU+0bjPO0LkuOLi4/5GtJWs/s=
+golang.org/x/net v0.0.0-20190827160401-ba9fcec4b297/go.mod h1:z5CRVTTTmAJ677TzLLGU+0bjPO0LkuOLi4/5GtJWs/s=
+golang.org/x/net v0.0.0-20190930134127-c5a3c61f89f3 h1:6KET3Sqa7fkVfD63QnAM81ZeYg5n4HwApOJkufONnHA=
+golang.org/x/net v0.0.0-20190930134127-c5a3c61f89f3/go.mod h1:z5CRVTTTmAJ677TzLLGU+0bjPO0LkuOLi4/5GtJWs/s=
+golang.org/x/net v0.0.0-20200226121028-0de0cce0169b h1:0mm1VjtFUOIlE1SbDlwjYaDxZVDP2S5ou6y0gSgXHu8=
+golang.org/x/net v0.0.0-20200226121028-0de0cce0169b/go.mod h1:z5CRVTTTmAJ677TzLLGU+0bjPO0LkuOLi4/5GtJWs/s=
+golang.org/x/oauth2 v0.0.0-20180821212333-d2e6202438be/go.mod h1:N/0e6XlmueqKjAGxoOufVs8QHGRruUQn6yWY3a++T0U=
+golang.org/x/oauth2 v0.0.0-20181106182150-f42d05182288 h1:JIqe8uIcRBHXDQVvZtHwp80ai3Lw3IJAeJEs55Dc1W0=
+golang.org/x/oauth2 v0.0.0-20181106182150-f42d05182288/go.mod h1:N/0e6XlmueqKjAGxoOufVs8QHGRruUQn6yWY3a++T0U=
+golang.org/x/sync v0.0.0-20180314180146-1d60e4601c6f/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM=
+golang.org/x/sync v0.0.0-20190423024810-112230192c58/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM=
+golang.org/x/sync v0.0.0-20190911185100-cd5d95a43a6e/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM=
+golang.org/x/sys v0.0.0-20170830134202-bb24a47a89ea/go.mod h1:STP8DvDyc/dI5b8T5hshtkjS+E42TnysNCUPdjciGhY=
+golang.org/x/sys v0.0.0-20180905080454-ebe1bf3edb33/go.mod h1:STP8DvDyc/dI5b8T5hshtkjS+E42TnysNCUPdjciGhY=
+golang.org/x/sys v0.0.0-20180909124046-d0be0721c37e/go.mod h1:STP8DvDyc/dI5b8T5hshtkjS+E42TnysNCUPdjciGhY=
+golang.org/x/sys v0.0.0-20190215142949-d0b11bdaac8a h1:1BGLXjeY4akVXGgbC9HugT3Jv3hCI0z56oJR5vAMgBU=
+golang.org/x/sys v0.0.0-20190215142949-d0b11bdaac8a/go.mod h1:STP8DvDyc/dI5b8T5hshtkjS+E42TnysNCUPdjciGhY=
+golang.org/x/sys v0.0.0-20190312061237-fead79001313/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
+golang.org/x/sys v0.0.0-20190412213103-97732733099d/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
+golang.org/x/sys v0.0.0-20190507160741-ecd444e8653b/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
+golang.org/x/sys v0.0.0-20190616124812-15dcb6c0061f/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
+golang.org/x/sys v0.0.0-20190927073244-c990c680b611 h1:q9u40nxWT5zRClI/uU9dHCiYGottAg6Nzz4YUQyHxdA=
+golang.org/x/sys v0.0.0-20190927073244-c990c680b611/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
+golang.org/x/text v0.0.0-20160726164857-2910a502d2bf/go.mod h1:NqM8EUOU14njkJ3fqMW+pc6Ldnwhi/IjpwHt7yyuwOQ=
+golang.org/x/text v0.3.0 h1:g61tztE5qeGQ89tm6NTjjM9VPIm088od1l6aSorWRWg=
+golang.org/x/text v0.3.0/go.mod h1:NqM8EUOU14njkJ3fqMW+pc6Ldnwhi/IjpwHt7yyuwOQ=
+golang.org/x/text v0.3.2 h1:tW2bmiBqwgJj/UpqtC8EpXEZVYOwU0yG4iWbprSVAcs=
+golang.org/x/text v0.3.2/go.mod h1:bEr9sfX3Q8Zfm5fL9x+3itogRgK3+ptLWKqgva+5dAk=
+golang.org/x/text v0.3.3 h1:cokOdA+Jmi5PJGXLlLllQSgYigAEfHXJAERHVMaCc2k=
+golang.org/x/text v0.3.3/go.mod h1:5Zoc/QRtKVWzQhOtBMvqHzDpF6irO9z98xDceosuGiQ=
+golang.org/x/time v0.0.0-20181108054448-85acf8d2951c h1:fqgJT0MGcGpPgpWU7VRdRjuArfcOvC4AoJmILihzhDg=
+golang.org/x/time v0.0.0-20181108054448-85acf8d2951c/go.mod h1:tRJNPiyCQ0inRvYxbN9jk5I+vvW/OXSQhTDSoE431IQ=
+golang.org/x/tools v0.0.0-20180917221912-90fa682c2a6e/go.mod h1:n7NCudcB/nEzxVGmLbDWY5pfWTLqBcC2KZ6jyYvM4mQ=
+golang.org/x/tools v0.0.0-20181011042414-1f849cf54d09/go.mod h1:n7NCudcB/nEzxVGmLbDWY5pfWTLqBcC2KZ6jyYvM4mQ=
+golang.org/x/tools v0.0.0-20181030221726-6c7e314b6563/go.mod h1:n7NCudcB/nEzxVGmLbDWY5pfWTLqBcC2KZ6jyYvM4mQ=
+golang.org/x/tools v0.0.0-20190206041539-40960b6deb8e/go.mod h1:n7NCudcB/nEzxVGmLbDWY5pfWTLqBcC2KZ6jyYvM4mQ=
+golang.org/x/tools v0.0.0-20190311212946-11955173bddd/go.mod h1:LCzVGOaR6xXOjkQ3onu1FJEFr0SW1gC7cKk1uF8kGRs=
+golang.org/x/tools v0.0.0-20190312151545-0bb0c0a6e846/go.mod h1:LCzVGOaR6xXOjkQ3onu1FJEFr0SW1gC7cKk1uF8kGRs=
+golang.org/x/tools v0.0.0-20190328211700-ab21143f2384/go.mod h1:LCzVGOaR6xXOjkQ3onu1FJEFr0SW1gC7cKk1uF8kGRs=
+golang.org/x/tools v0.0.0-20190524140312-2c0ae7006135/go.mod h1:RgjU9mgBXZiqYHBnxXauZ1Gv1EHHAz9KjViQ78xBX0Q=
+golang.org/x/tools v0.0.0-20190614205625-5aca471b1d59/go.mod h1:/rFqwRUd4F7ZHNgwSSTFct+R/Kf4OFW1sUzUTQQTgfc=
+golang.org/x/tools v0.0.0-20190930201159-7c411dea38b0 h1:7+F62GGWUowoiJOUDivedlBECd/fTeUDJnCu0JetQO0=
+golang.org/x/tools v0.0.0-20190930201159-7c411dea38b0/go.mod h1:b+2E5dAYhXwXZwtnZ6UAqBI28+e2cm9otk0dWdXHAEo=
+golang.org/x/xerrors v0.0.0-20190717185122-a985d3407aa7/go.mod h1:I/5z698sn9Ka8TeJc9MKroUUfqBBauWjQqLJ2OPfmY0=
+gonum.org/v1/gonum v0.0.0-20190331200053-3d26580ed485 h1:OB/uP/Puiu5vS5QMRPrXCDWUPb+kt8f1KW8oQzFejQw=
+gonum.org/v1/gonum v0.0.0-20190331200053-3d26580ed485/go.mod h1:2ltnJ7xHfj0zHS40VVPYEAAMTa3ZGguvHGBSJeRWqE0=
+gonum.org/v1/netlib v0.0.0-20190313105609-8cb42192e0e0/go.mod h1:wa6Ws7BG/ESfp6dHfk7C6KdzKA7wR7u/rKwOGE66zvw=
+gonum.org/v1/netlib v0.0.0-20190331212654-76723241ea4e/go.mod h1:kS+toOQn6AQKjmKJ7gzohV1XkqsFehRA2FbsbkopSuQ=
+google.golang.org/appengine v1.1.0/go.mod h1:EbEs0AVv82hx2wNQdGPgUI5lhzA/G0D9YwlJXL52JkM=
+google.golang.org/appengine v1.3.0 h1:FBSsiFRMz3LBeXIomRnVzrQwSDj4ibvcRexLG0LZGQk=
+google.golang.org/appengine v1.3.0/go.mod h1:xpcJRLb0r/rnEns0DIKYYv+WjYCduHsrkT7/EB5XEv4=
+google.golang.org/genproto v0.0.0-20180817151627-c66870c02cf8 h1:Nw54tB0rB7hY/N0NQvRW8DG4Yk3Q6T9cu9RcFQDu1tc=
+google.golang.org/genproto v0.0.0-20180817151627-c66870c02cf8/go.mod h1:JiN7NxoALGmiZfu7CAH4rXhgtRTLTxftemlI0sWmxmc=
+google.golang.org/grpc v1.23.0 h1:AzbTB6ux+okLTzP8Ru1Xs41C303zdcfEht7MQnYJt5A=
+google.golang.org/grpc v1.23.0/go.mod h1:Y5yQAOtifL1yxbo5wqy6BxZv8vAUGQwXBOALyacEbxg=
+gopkg.in/airbrake/gobrake.v2 v2.0.9/go.mod h1:/h5ZAUhDkGaJfjzjKLSjv6zCL6O0LLBxU4K+aSYdM/U=
+gopkg.in/check.v1 v0.0.0-20161208181325-20d25e280405/go.mod h1:Co6ibVJAznAaIkqp8huTwlJQCZ016jof/cbN4VW5Yz0=
+gopkg.in/check.v1 v1.0.0-20180628173108-788fd7840127 h1:qIbj1fsPNlZgppZ+VLlY7N33q108Sa+fhmuc+sWQYwY=
+gopkg.in/check.v1 v1.0.0-20180628173108-788fd7840127/go.mod h1:Co6ibVJAznAaIkqp8huTwlJQCZ016jof/cbN4VW5Yz0=
+gopkg.in/check.v1 v1.0.0-20190902080502-41f04d3bba15 h1:YR8cESwS4TdDjEe65xsg0ogRM/Nc3DYOhEAlW+xobZo=
+gopkg.in/check.v1 v1.0.0-20190902080502-41f04d3bba15/go.mod h1:Co6ibVJAznAaIkqp8huTwlJQCZ016jof/cbN4VW5Yz0=
+gopkg.in/fsnotify.v1 v1.4.7 h1:xOHLXZwVvI9hhs+cLKq5+I5onOuwQLhQwiu63xxlHs4=
+gopkg.in/fsnotify.v1 v1.4.7/go.mod h1:Tz8NjZHkW78fSQdbUxIjBTcgA1z1m8ZHf0WmKUhAMys=
+gopkg.in/gemnasium/logrus-airbrake-hook.v2 v2.1.2/go.mod h1:Xk6kEKp8OKb+X14hQBKWaSkCsqBpgog8nAV2xsGOxlo=
+gopkg.in/inf.v0 v0.9.0 h1:3zYtXIO92bvsdS3ggAdA8Gb4Azj0YU+TVY1uGYNFA8o=
+gopkg.in/inf.v0 v0.9.0/go.mod h1:cWUDdTG/fYaXco+Dcufb5Vnc6Gp2YChqWtbxRZE0mXw=
+gopkg.in/mgo.v2 v2.0.0-20180705113604-9856a29383ce/go.mod h1:yeKp02qBN3iKW1OzL3MGk2IdtZzaj7SFntXj72NppTA=
+gopkg.in/natefinch/lumberjack.v2 v2.0.0 h1:1Lc07Kr7qY4U2YPouBjpCLxpiyxIVoxqXgkXLknAOE8=
+gopkg.in/natefinch/lumberjack.v2 v2.0.0/go.mod h1:l0ndWWf7gzL7RNwBG7wST/UCcT4T24xpD6X8LsfU/+k=
+gopkg.in/tomb.v1 v1.0.0-20141024135613-dd632973f1e7 h1:uRGJdciOHaEIrze2W8Q3AKkepLTh2hOroT7a+7czfdQ=
+gopkg.in/tomb.v1 v1.0.0-20141024135613-dd632973f1e7/go.mod h1:dt/ZhP58zS4L8KSrWDmTeBkI65Dw0HsyUHuEVlX15mw=
+gopkg.in/yaml.v2 v2.2.1/go.mod h1:hI93XBmqTisBFMUTm0b8Fm+jr3Dg1NNxqwp+5A1VGuI=
+gopkg.in/yaml.v2 v2.2.2 h1:ZCJp+EgiOT7lHqUV2J862kp8Qj64Jo6az82+3Td9dZw=
+gopkg.in/yaml.v2 v2.2.2/go.mod h1:hI93XBmqTisBFMUTm0b8Fm+jr3Dg1NNxqwp+5A1VGuI=
+gopkg.in/yaml.v2 v2.2.3 h1:fvjTMHxHEw/mxHbtzPi3JCcKXQRAnQTBRo6YCJSVHKI=
+gopkg.in/yaml.v2 v2.2.3/go.mod h1:hI93XBmqTisBFMUTm0b8Fm+jr3Dg1NNxqwp+5A1VGuI=
+gopkg.in/yaml.v2 v2.2.8 h1:obN1ZagJSUGI0Ek/LBmuj4SNLPfIny3KsKFopxRdj10=
+gopkg.in/yaml.v2 v2.2.8/go.mod h1:hI93XBmqTisBFMUTm0b8Fm+jr3Dg1NNxqwp+5A1VGuI=
+honnef.co/go/tools v0.0.0-20190523083050-ea95bdfd59fc/go.mod h1:rf3lG4BRIbNafJWhAfAdb/ePZxsR/4RtNHQocxwk9r4=
+k8s.io/api v0.0.0-20181115043458-b799cb063522 h1:JGWVL/WnKVncCXESoe8wCBZJbXbUvYi+AuSHJ/IuFN8=
+k8s.io/api v0.0.0-20181115043458-b799cb063522/go.mod h1:iuAfoD4hCxJ8Onx9kaTIt30j7jUFS00AXQi6QMi99vA=
+k8s.io/apimachinery v0.0.0-20181110190943-2a7c93004028 h1:5i4gnQYlGFIWZp34j0qnt3eeCEFs2ymFzRhzFOPG5/U=
+k8s.io/apimachinery v0.0.0-20181110190943-2a7c93004028/go.mod h1:ccL7Eh7zubPUSh9A3USN90/OzHNSVN6zxzde07TDCL0=
+k8s.io/client-go v0.0.0-20181115111358-9bea17718df8 h1:Q1pEgTyAmkwHY+lScBj4VZaGP3ykzGUoP9qC0/JxbEk=
+k8s.io/client-go v0.0.0-20181115111358-9bea17718df8/go.mod h1:7vJpHMYJwNQCWgzmNV+VYUl1zCObLyodBc8nIyt8L5s=
+k8s.io/code-generator v0.0.0-20181114232248-ae218e241252 h1:bURaCtXd+lhw5buMNdeUig98d3Eq97Muux1hUFvkqb0=
+k8s.io/code-generator v0.0.0-20181114232248-ae218e241252/go.mod h1:IPqxl/YHk05nodzupwjke6ctMjyNRdV2zZ5/j3/F204=
+k8s.io/gengo v0.0.0-20181106084056-51747d6e00da/go.mod h1:ezvh/TsK7cY6rbqRK0oQQ8IAqLxYwwyPxAX1Pzy0ii0=
+k8s.io/gengo v0.0.0-20190128074634-0689ccc1d7d6/go.mod h1:ezvh/TsK7cY6rbqRK0oQQ8IAqLxYwwyPxAX1Pzy0ii0=
+k8s.io/gengo v0.0.0-20190907103519-ebc107f98eab h1:j4L8spMe0tFfBvvW6lrc0c+Ql8+nnkcV3RYfi3eSwGY=
+k8s.io/gengo v0.0.0-20190907103519-ebc107f98eab/go.mod h1:ezvh/TsK7cY6rbqRK0oQQ8IAqLxYwwyPxAX1Pzy0ii0=
+k8s.io/klog v0.0.0-20181102134211-b9b56d5dfc92/go.mod h1:Gq+BEi5rUBO/HRz0bTSXDUcqjScdoY3a9IHpCEIOOfk=
+k8s.io/klog v0.0.0-20190306015804-8e90cee79f82/go.mod h1:Gq+BEi5rUBO/HRz0bTSXDUcqjScdoY3a9IHpCEIOOfk=
+k8s.io/klog v1.0.0 h1:Pt+yjF5aB1xDSVbau4VsWe+dQNzA0qv1LlXdC2dF6Q8=
+k8s.io/klog v1.0.0/go.mod h1:4Bi6QPql/J/LkTDqv7R/cd3hPo4k2DG6Ptcz060Ez5I=
+k8s.io/kube-openapi v0.0.0-20181114233023-0317810137be h1:aWEq4nbj7HRJ0mtKYjNSk/7X28Tl6TI6FeG8gKF+r7Q=
+k8s.io/kube-openapi v0.0.0-20181114233023-0317810137be/go.mod h1:BXM9ceUBTj2QnfH2MK1odQs778ajze1RxcmP6S8RVVc=
+k8s.io/kube-openapi v0.0.0-20200121204235-bf4fb3bd569c h1:/KUFqjjqAcY4Us6luF5RDNZ16KJtb49HfR3ZHB9qYXM=
+k8s.io/kube-openapi v0.0.0-20200121204235-bf4fb3bd569c/go.mod h1:GRQhZsXIAJ1xR0C9bd8UpWHZ5plfAS9fzPjJuQ6JL3E=
+k8s.io/kubernetes v1.13.0 h1:qTfB+u5M92k2fCCCVP2iuhgwwSOv1EkAkvQY1tQODD8=
+k8s.io/kubernetes v1.13.0/go.mod h1:ocZa8+6APFNC2tX1DZASIbocyYT5jHzqFVsY5aoB7Jk=
+modernc.org/cc v1.0.0/go.mod h1:1Sk4//wdnYJiUIxnW8ddKpaOJCF37yAdqYnkxUpaYxw=
+modernc.org/golex v1.0.0/go.mod h1:b/QX9oBD/LhixY6NDh+IdGv17hgB+51fET1i2kPSmvk=
+modernc.org/mathutil v1.0.0/go.mod h1:wU0vUrJsVWBZ4P6e7xtFJEhFSNsfRLJ8H458uRjg03k=
+modernc.org/strutil v1.0.0/go.mod h1:lstksw84oURvj9y3tn8lGvRxyRC1S2+g5uuIzNfIOBs=
+modernc.org/xc v1.0.0/go.mod h1:mRNCo0bvLjGhHO9WsyuKVU4q0ceiDDDoEeWDJHrNx8I=
+sigs.k8s.io/structured-merge-diff/v3 v3.0.0-20200116222232-67a7b8c61874/go.mod h1:PlARxl6Hbt/+BC80dRLi1qAmnMqwqDg62YvvVkZjemw=
+sigs.k8s.io/yaml v1.1.0 h1:4A07+ZFc2wgJwo8YNlQpr1rVlgUDlxXHhPJciaPY5gs=
+sigs.k8s.io/yaml v1.1.0/go.mod h1:UJmg0vDUVViEyp3mgSv9WPwZCDxu4rQW1olrI1uml+o=

images/README.md

@@ -0,0 +1,68 @@
+## Dockerfile build
+
+This is used for distribution of Multus in a Docker image.
+
+Typically you'd build this from the root of your Multus clone, as such:
+
+```
+$ docker build -t dougbtv/multus .
+```
+
+---
+
+## Daemonset deployment
+
+You may wish to deploy Multus as a daemonset, you can do so by starting with the example Daemonset shown here:
+
+```
+$ kubectl create -f ./images/multus-daemonset.yml
+```
+
+Note: The likely best practice here is to build your own image given the Dockerfile, and then push it to your preferred registry, and change the `image` fields in the Daemonset YAML to reference that image.
+
+---
+
+### `entrypoint.sh` parameters
+
+The entrypoint takes named parameters for the configuration
+
+You can get get help with the `--help` flag.
+
+```
+$ ./entrypoint.sh --help
+
+This is an entrypoint script for Multus CNI to overlay its binary and
+configuration into locations in a filesystem. The configuration & binary file
+will be copied to the corresponding configuration directory. When
+`--multus-conf-file=auto` is used, 00-multus.conf will be automatically
+generated from the CNI configuration file of the master plugin (the first file
+in lexicographical order in cni-conf-dir).
+
+./entrypoint.sh
+    -h --help
+    --cni-conf-dir=/host/etc/cni/net.d
+    --cni-bin-dir=/host/opt/cni/bin
+    --multus-conf-file=/usr/src/multus-cni/images/70-multus.conf
+    --multus-bin-file=/usr/src/multus-cni/bin/multus
+    --multus-kubeconfig-file-host=/etc/cni/net.d/multus.d/multus.kubeconfig
+```
+
+You must use an `=` to delimit the parameter name and the value. For example you may set a custom `cni-conf-dir` like so:
+
+```
+./entrypoint.sh --cni-conf-dir=/special/path/to/cni/configs/
+```
+
+Note: You'll noticed that there's a `/host/...` directory from the root for the default for both the `cni-conf-dir` and `cni-bin-dir` as it's intended for the host volumes to be mounted specially under this directory to help in the semantics of which paths belong to the host or container.
+
+---
+
+### Development notes
+
+Example docker run command:
+
+```
+$ docker run -it -v /opt/cni/bin/:/host/opt/cni/bin/ -v /etc/cni/net.d/:/host/etc/cni/net.d/ --entrypoint=/bin/bash dougbtv/multus
+```
+
+Originally inspired by and is a portmanteau of the [Flannel daemonset](https://github.com/coreos/flannel/blob/master/Documentation/kube-flannel.yml), the [Calico Daemonset](https://github.com/projectcalico/calico/blob/master/v2.0/getting-started/kubernetes/installation/hosted/k8s-backend-addon-manager/calico-daemonset.yaml), and the [Calico CNI install bash script](https://github.com/projectcalico/cni-plugin/blob/be4df4db2e47aa7378b1bdf6933724bac1f348d0/k8s-install/scripts/install-cni.sh#L104-L153).

images/deprecated/multus-daemonset-crio-pre1.16.yml

@@ -0,0 +1,263 @@
+---
+apiVersion: apiextensions.k8s.io/v1beta1
+kind: CustomResourceDefinition
+metadata:
+  name: network-attachment-definitions.k8s.cni.cncf.io
+spec:
+  group: k8s.cni.cncf.io
+  scope: Namespaced
+  names:
+    plural: network-attachment-definitions
+    singular: network-attachment-definition
+    kind: NetworkAttachmentDefinition
+    shortNames:
+    - net-attach-def
+  versions:
+    - name: v1
+      served: true
+      storage: true
+      schema:
+        openAPIV3Schema:
+          description: 'NetworkAttachmentDefinition is a CRD schema specified by the Network Plumbing
+            Working Group to express the intent for attaching pods to one or more logical or physical
+            networks. More information available at: https://github.com/k8snetworkplumbingwg/multi-net-spec'
+          type: object
+          properties:
+            spec:
+              description: 'NetworkAttachmentDefinition spec defines the desired state of a network attachment'
+              type: object
+              properties:
+                config:
+                  description: 'NetworkAttachmentDefinition config is a JSON-formatted CNI configuration'
+                  type: string
+---
+kind: ClusterRole
+apiVersion: rbac.authorization.k8s.io/v1beta1
+metadata:
+  name: multus
+rules:
+  - apiGroups: ["k8s.cni.cncf.io"]
+    resources:
+      - '*'
+    verbs:
+      - '*'
+  - apiGroups:
+      - ""
+    resources:
+      - pods
+      - pods/status
+    verbs:
+      - get
+      - update
+  - apiGroups:
+      - ""
+      - events.k8s.io
+    resources:
+      - events
+    verbs:
+      - create
+      - patch
+      - update
+---
+kind: ClusterRoleBinding
+apiVersion: rbac.authorization.k8s.io/v1beta1
+metadata:
+  name: multus
+roleRef:
+  apiGroup: rbac.authorization.k8s.io
+  kind: ClusterRole
+  name: multus
+subjects:
+- kind: ServiceAccount
+  name: multus
+  namespace: kube-system
+---
+apiVersion: v1
+kind: ServiceAccount
+metadata:
+  name: multus
+  namespace: kube-system
+---
+kind: ConfigMap
+apiVersion: v1
+metadata:
+  name: multus-cni-config
+  namespace: kube-system
+  labels:
+    tier: node
+    app: multus
+data:
+  # NOTE: If you'd prefer to manually apply a configuration file, you may create one here.
+  # In the case you'd like to customize the Multus installation, you should change the arguments to the Multus pod
+  # change the "args" line below from
+  # - "--multus-conf-file=auto"
+  # to:
+  # "--multus-conf-file=/tmp/multus-conf/70-multus.conf"
+  # Additionally -- you should ensure that the name "70-multus.conf" is the alphabetically first name in the
+  # /etc/cni/net.d/ directory on each node, otherwise, it will not be used by the Kubelet.
+  cni-conf.json: |
+    {
+      "name": "multus-cni-network",
+      "type": "multus",
+      "capabilities": {
+        "portMappings": true
+      },
+      "delegates": [
+        {
+          "cniVersion": "0.3.1",
+          "name": "default-cni-network",
+          "plugins": [
+            {
+              "type": "flannel",
+              "name": "flannel.1",
+                "delegate": {
+                  "isDefaultGateway": true,
+                  "hairpinMode": true
+                }
+              },
+              {
+                "type": "portmap",
+                "capabilities": {
+                  "portMappings": true
+                }
+              }
+          ]
+        }
+      ],
+      "kubeconfig": "/etc/cni/net.d/multus.d/multus.kubeconfig"
+    }
+---
+apiVersion: extensions/v1beta1
+kind: DaemonSet
+metadata:
+  name: kube-multus-ds-amd64
+  namespace: kube-system
+  labels:
+    tier: node
+    app: multus
+spec:
+  updateStrategy:
+    type: RollingUpdate
+  template:
+    metadata:
+      labels:
+        tier: node
+        app: multus
+    spec:
+      hostNetwork: true
+      nodeSelector:
+        beta.kubernetes.io/arch: amd64
+      tolerations:
+      - operator: Exists
+        effect: NoSchedule
+      serviceAccountName: multus
+      containers:
+      - name: kube-multus
+        # crio support requires multus:latest for now. support 3.3 or later.
+        image: nfvpe/multus:latest
+        command: ["/entrypoint.sh"]
+        args:
+        - "--cni-bin-dir=/host/usr/libexec/cni"
+        - "--multus-conf-file=auto"
+        - "--override-network-name=true"
+        - "--restart-crio=true"
+        resources:
+          requests:
+            cpu: "100m"
+            memory: "50Mi"
+          limits:
+            cpu: "100m"
+            memory: "50Mi"
+        securityContext:
+          privileged: true
+          capabilities:
+            add: ["SYS_ADMIN"]
+        volumeMounts:
+        - name: run
+          mountPath: /run
+        - name: cni
+          mountPath: /host/etc/cni/net.d
+        - name: cnibin
+          mountPath: /host/usr/libexec/cni
+        - name: multus-cfg
+          mountPath: /tmp/multus-conf
+      volumes:
+        - name: run
+          hostPath:
+            path: /run
+        - name: cni
+          hostPath:
+            path: /etc/cni/net.d
+        - name: cnibin
+          hostPath:
+            path: /usr/libexec/cni
+        - name: multus-cfg
+          configMap:
+            name: multus-cni-config
+            items:
+            - key: cni-conf.json
+              path: 70-multus.conf
+---
+apiVersion: extensions/v1beta1
+kind: DaemonSet
+metadata:
+  name: kube-multus-ds-ppc64le
+  namespace: kube-system
+  labels:
+    tier: node
+    app: multus
+spec:
+  updateStrategy:
+    type: RollingUpdate
+  template:
+    metadata:
+      labels:
+        tier: node
+        app: multus
+    spec:
+      hostNetwork: true
+      nodeSelector:
+        beta.kubernetes.io/arch: ppc64le
+      tolerations:
+      - operator: Exists
+        effect: NoSchedule
+      serviceAccountName: multus
+      containers:
+      - name: kube-multus
+        # crio support requires multus:latest for now. support 3.3 or later.
+        image: nfvpe/multus:latest-ppc64le
+        command: ["/entrypoint.sh"]
+        args:
+        - "--cni-bin-dir=/host/usr/libexec/cni"
+        - "--multus-conf-file=auto"
+        - "--override-network-name=true"
+        - "--restart-crio=true"
+        resources:
+          requests:
+            cpu: "100m"
+            memory: "90Mi"
+          limits:
+            cpu: "100m"
+            memory: "90Mi"
+        securityContext:
+          privileged: true
+        volumeMounts:
+        - name: cni
+          mountPath: /host/etc/cni/net.d
+        - name: cnibin
+          mountPath: /host/usr/libexec/cni
+        - name: multus-cfg
+          mountPath: /tmp/multus-conf
+      volumes:
+        - name: cni
+          hostPath:
+            path: /etc/cni/net.d
+        - name: cnibin
+          hostPath:
+            path: /usr/libexec/cni
+        - name: multus-cfg
+          configMap:
+            name: multus-cni-config
+            items:
+            - key: cni-conf.json
+              path: 70-multus.conf

images/deprecated/multus-daemonset-gke-pre-1.16.yml

@@ -0,0 +1,232 @@
+---
+apiVersion: apiextensions.k8s.io/v1beta1
+kind: CustomResourceDefinition
+metadata:
+  name: network-attachment-definitions.k8s.cni.cncf.io
+spec:
+  group: k8s.cni.cncf.io
+  version: v1
+  scope: Namespaced
+  names:
+    plural: network-attachment-definitions
+    singular: network-attachment-definition
+    kind: NetworkAttachmentDefinition
+    shortNames:
+      - net-attach-def
+  validation:
+    openAPIV3Schema:
+      properties:
+        spec:
+          properties:
+            config:
+              type: string
+---
+kind: ClusterRole
+apiVersion: rbac.authorization.k8s.io/v1beta1
+metadata:
+  name: multus
+rules:
+  - apiGroups: ["k8s.cni.cncf.io"]
+    resources:
+      - '*'
+    verbs:
+      - '*'
+  - apiGroups:
+      - ""
+    resources:
+      - pods
+      - pods/status
+    verbs:
+      - get
+      - update
+---
+kind: ClusterRoleBinding
+apiVersion: rbac.authorization.k8s.io/v1beta1
+metadata:
+  name: multus
+roleRef:
+  apiGroup: rbac.authorization.k8s.io
+  kind: ClusterRole
+  name: multus
+subjects:
+  - kind: ServiceAccount
+    name: multus
+    namespace: kube-system
+---
+apiVersion: v1
+kind: ServiceAccount
+metadata:
+  name: multus
+  namespace: kube-system
+---
+kind: ConfigMap
+apiVersion: v1
+metadata:
+  name: multus-cni-config
+  namespace: kube-system
+  labels:
+    tier: node
+    app: multus
+data:
+  # NOTE: If you'd prefer to manually apply a configuration file, you may create one here.
+  # In the case you'd like to customize the Multus installation, you should change the arguments to the Multus pod
+  # change the "args" line below from
+  # - "--multus-conf-file=auto"
+  # to:
+  # "--multus-conf-file=/tmp/multus-conf/70-multus.conf"
+  # Additionally -- you should ensure that the name "70-multus.conf" is the alphabetically first name in the
+  # /etc/cni/net.d/ directory on each node, otherwise, it will not be used by the Kubelet.
+  cni-conf.json: |
+    {
+      "name": "multus-cni-network",
+      "type": "multus",
+      "capabilities": {
+        "portMappings": true
+      },
+      "delegates": [
+        {
+          "cniVersion": "0.3.1",
+          "name": "default-cni-network",
+          "plugins": [
+            {
+              "type": "flannel",
+              "name": "flannel.1",
+                "delegate": {
+                  "isDefaultGateway": true,
+                  "hairpinMode": true
+                }
+              },
+              {
+                "type": "portmap",
+                "capabilities": {
+                  "portMappings": true
+                }
+              }
+          ]
+        }
+      ],
+      "kubeconfig": "/etc/cni/net.d/multus.d/multus.kubeconfig"
+    }
+---
+apiVersion: extensions/v1beta1
+kind: DaemonSet
+metadata:
+  name: kube-multus-ds-amd64
+  namespace: kube-system
+  labels:
+    tier: node
+    app: multus
+spec:
+  updateStrategy:
+    type: RollingUpdate
+  template:
+    metadata:
+      labels:
+        tier: node
+        app: multus
+    spec:
+      hostNetwork: true
+      nodeSelector:
+        beta.kubernetes.io/arch: amd64
+      tolerations:
+        - operator: Exists
+          effect: NoSchedule
+      serviceAccountName: multus
+      containers:
+        - name: kube-multus
+          image: nfvpe/multus:v3.4
+          command: ["/entrypoint.sh"]
+          args:
+            - "--multus-conf-file=auto"
+            - "--cni-bin-dir=/host/home/kubernetes/bin"
+          resources:
+            requests:
+              cpu: "100m"
+              memory: "50Mi"
+            limits:
+              cpu: "100m"
+              memory: "50Mi"
+          securityContext:
+            privileged: true
+          volumeMounts:
+            - name: cni
+              mountPath: /host/etc/cni/net.d
+            - name: cnibin
+              mountPath: /host/home/kubernetes/bin
+            - name: multus-cfg
+              mountPath: /tmp/multus-conf
+      volumes:
+        - name: cni
+          hostPath:
+            path: /etc/cni/net.d
+        - name: cnibin
+          hostPath:
+            path: /home/kubernetes/bin
+        - name: multus-cfg
+          configMap:
+            name: multus-cni-config
+            items:
+              - key: cni-conf.json
+                path: 70-multus.conf
+---
+apiVersion: extensions/v1beta1
+kind: DaemonSet
+metadata:
+  name: kube-multus-ds-ppc64le
+  namespace: kube-system
+  labels:
+    tier: node
+    app: multus
+spec:
+  updateStrategy:
+    type: RollingUpdate
+  template:
+    metadata:
+      labels:
+        tier: node
+        app: multus
+    spec:
+      hostNetwork: true
+      nodeSelector:
+        beta.kubernetes.io/arch: ppc64le
+      tolerations:
+        - operator: Exists
+          effect: NoSchedule
+      serviceAccountName: multus
+      containers:
+        - name: kube-multus
+          # ppc64le support requires multus:latest for now. support 3.3 or later.
+          image: nfvpe/multus:latest-ppc64le
+          command: ["/entrypoint.sh"]
+          args:
+            - "--multus-conf-file=auto"
+            - "--cni-bin-dir=/host/home/kubernetes/bin"
+          resources:
+            requests:
+              cpu: "100m"
+              memory: "90Mi"
+            limits:
+              cpu: "100m"
+              memory: "90Mi"
+          securityContext:
+            privileged: true
+          volumeMounts:
+            - name: cni
+              mountPath: /host/etc/cni/net.d
+            - name: cnibin
+              mountPath: /host/home/kubernetes/bin
+            - name: multus-cfg
+              mountPath: /tmp/multus-conf
+      volumes:
+        - name: cni
+          hostPath:
+            path: /etc/cni/net.d
+        - name: cnibin
+          hostPath:
+            path: /home/kubernetes/bin
+        - name: multus-cfg
+          configMap:
+            name: multus-cni-config
+            items:
+              - key: cni-conf.json
+                path: 70-multus.conf

images/deprecated/multus-daemonset-pre-1.16.yml

@@ -0,0 +1,249 @@
+---
+apiVersion: apiextensions.k8s.io/v1beta1
+kind: CustomResourceDefinition
+metadata:
+  name: network-attachment-definitions.k8s.cni.cncf.io
+spec:
+  group: k8s.cni.cncf.io
+  scope: Namespaced
+  names:
+    plural: network-attachment-definitions
+    singular: network-attachment-definition
+    kind: NetworkAttachmentDefinition
+    shortNames:
+    - net-attach-def
+  versions:
+    - name: v1
+      served: true
+      storage: true
+      schema:
+        openAPIV3Schema:
+          description: 'NetworkAttachmentDefinition is a CRD schema specified by the Network Plumbing
+            Working Group to express the intent for attaching pods to one or more logical or physical
+            networks. More information available at: https://github.com/k8snetworkplumbingwg/multi-net-spec'
+          type: object
+          properties:
+            spec:
+              description: 'NetworkAttachmentDefinition spec defines the desired state of a network attachment'
+              type: object
+              properties:
+                config:
+                  description: 'NetworkAttachmentDefinition config is a JSON-formatted CNI configuration'
+                  type: string
+---
+kind: ClusterRole
+apiVersion: rbac.authorization.k8s.io/v1beta1
+metadata:
+  name: multus
+rules:
+  - apiGroups: ["k8s.cni.cncf.io"]
+    resources:
+      - '*'
+    verbs:
+      - '*'
+  - apiGroups:
+      - ""
+    resources:
+      - pods
+      - pods/status
+    verbs:
+      - get
+      - update
+  - apiGroups:
+      - ""
+      - events.k8s.io
+    resources:
+      - events
+    verbs:
+      - create
+      - patch
+      - update
+---
+kind: ClusterRoleBinding
+apiVersion: rbac.authorization.k8s.io/v1beta1
+metadata:
+  name: multus
+roleRef:
+  apiGroup: rbac.authorization.k8s.io
+  kind: ClusterRole
+  name: multus
+subjects:
+- kind: ServiceAccount
+  name: multus
+  namespace: kube-system
+---
+apiVersion: v1
+kind: ServiceAccount
+metadata:
+  name: multus
+  namespace: kube-system
+---
+kind: ConfigMap
+apiVersion: v1
+metadata:
+  name: multus-cni-config
+  namespace: kube-system
+  labels:
+    tier: node
+    app: multus
+data:
+  # NOTE: If you'd prefer to manually apply a configuration file, you may create one here.
+  # In the case you'd like to customize the Multus installation, you should change the arguments to the Multus pod
+  # change the "args" line below from
+  # - "--multus-conf-file=auto"
+  # to:
+  # "--multus-conf-file=/tmp/multus-conf/70-multus.conf"
+  # Additionally -- you should ensure that the name "70-multus.conf" is the alphabetically first name in the
+  # /etc/cni/net.d/ directory on each node, otherwise, it will not be used by the Kubelet.
+  cni-conf.json: |
+    {
+      "name": "multus-cni-network",
+      "type": "multus",
+      "capabilities": {
+        "portMappings": true
+      },
+      "delegates": [
+        {
+          "cniVersion": "0.3.1",
+          "name": "default-cni-network",
+          "plugins": [
+            {
+              "type": "flannel",
+              "name": "flannel.1",
+                "delegate": {
+                  "isDefaultGateway": true,
+                  "hairpinMode": true
+                }
+              },
+              {
+                "type": "portmap",
+                "capabilities": {
+                  "portMappings": true
+                }
+              }
+          ]
+        }
+      ],
+      "kubeconfig": "/etc/cni/net.d/multus.d/multus.kubeconfig"
+    }
+---
+apiVersion: extensions/v1beta1
+kind: DaemonSet
+metadata:
+  name: kube-multus-ds-amd64
+  namespace: kube-system
+  labels:
+    tier: node
+    app: multus
+spec:
+  updateStrategy:
+    type: RollingUpdate
+  template:
+    metadata:
+      labels:
+        tier: node
+        app: multus
+    spec:
+      hostNetwork: true
+      nodeSelector:
+        beta.kubernetes.io/arch: amd64
+      tolerations:
+      - operator: Exists
+        effect: NoSchedule
+      serviceAccountName: multus
+      containers:
+      - name: kube-multus
+        image: nfvpe/multus:v3.4.1
+        command: ["/entrypoint.sh"]
+        args:
+        - "--multus-conf-file=auto"
+        resources:
+          requests:
+            cpu: "100m"
+            memory: "50Mi"
+          limits:
+            cpu: "100m"
+            memory: "50Mi"
+        securityContext:
+          privileged: true
+        volumeMounts:
+        - name: cni
+          mountPath: /host/etc/cni/net.d
+        - name: cnibin
+          mountPath: /host/opt/cni/bin
+        - name: multus-cfg
+          mountPath: /tmp/multus-conf
+      volumes:
+        - name: cni
+          hostPath:
+            path: /etc/cni/net.d
+        - name: cnibin
+          hostPath:
+            path: /opt/cni/bin
+        - name: multus-cfg
+          configMap:
+            name: multus-cni-config
+            items:
+            - key: cni-conf.json
+              path: 70-multus.conf
+---
+apiVersion: extensions/v1beta1
+kind: DaemonSet
+metadata:
+  name: kube-multus-ds-ppc64le
+  namespace: kube-system
+  labels:
+    tier: node
+    app: multus
+spec:
+  updateStrategy:
+    type: RollingUpdate
+  template:
+    metadata:
+      labels:
+        tier: node
+        app: multus
+    spec:
+      hostNetwork: true
+      nodeSelector:
+        beta.kubernetes.io/arch: ppc64le
+      tolerations:
+      - operator: Exists
+        effect: NoSchedule
+      serviceAccountName: multus
+      containers:
+      - name: kube-multus
+        # ppc64le support requires multus:latest for now. support 3.3 or later.
+        image: nfvpe/multus:latest-ppc64le
+        command: ["/entrypoint.sh"]
+        args:
+        - "--multus-conf-file=auto"
+        resources:
+          requests:
+            cpu: "100m"
+            memory: "90Mi"
+          limits:
+            cpu: "100m"
+            memory: "90Mi"
+        securityContext:
+          privileged: true
+        volumeMounts:
+        - name: cni
+          mountPath: /host/etc/cni/net.d
+        - name: cnibin
+          mountPath: /host/opt/cni/bin
+        - name: multus-cfg
+          mountPath: /tmp/multus-conf
+      volumes:
+        - name: cni
+          hostPath:
+            path: /etc/cni/net.d
+        - name: cnibin
+          hostPath:
+            path: /opt/cni/bin
+        - name: multus-cfg
+          configMap:
+            name: multus-cni-config
+            items:
+            - key: cni-conf.json
+              path: 70-multus.conf

images/entrypoint.sh

@@ -0,0 +1,378 @@
+#!/bin/bash
+
+# Always exit on errors.
+set -e
+
+# Set our known directories.
+CNI_CONF_DIR="/host/etc/cni/net.d"
+CNI_BIN_DIR="/host/opt/cni/bin"
+ADDITIONAL_BIN_DIR=""
+MULTUS_CONF_FILE="/usr/src/multus-cni/images/70-multus.conf"
+MULTUS_AUTOCONF_DIR="/host/etc/cni/net.d"
+MULTUS_BIN_FILE="/usr/src/multus-cni/bin/multus"
+MULTUS_KUBECONFIG_FILE_HOST="/etc/cni/net.d/multus.d/multus.kubeconfig"
+MULTUS_NAMESPACE_ISOLATION=false
+MULTUS_LOG_LEVEL=""
+MULTUS_LOG_FILE=""
+MULTUS_READINESS_INDICATOR_FILE=""
+OVERRIDE_NETWORK_NAME=false
+MULTUS_CLEANUP_CONFIG_ON_EXIT=false
+RESTART_CRIO=false
+CRIO_RESTARTED_ONCE=false
+RENAME_SOURCE_CONFIG_FILE=false
+SKIP_BINARY_COPY=false
+
+# Give help text for parameters.
+function usage()
+{
+    echo -e "This is an entrypoint script for Multus CNI to overlay its binary and "
+    echo -e "configuration into locations in a filesystem. The configuration & binary file "
+    echo -e "will be copied to the corresponding configuration directory. When "
+    echo -e "'--multus-conf-file=auto' is used, 00-multus.conf will be automatically "
+    echo -e "generated from the CNI configuration file of the master plugin (the first file "
+    echo -e "in lexicographical order in cni-conf-dir)."
+    echo -e ""
+    echo -e "./entrypoint.sh"
+    echo -e "\t-h --help"
+    echo -e "\t--cni-conf-dir=$CNI_CONF_DIR"
+    echo -e "\t--cni-bin-dir=$CNI_BIN_DIR"
+    echo -e "\t--cni-version=<cniVersion (e.g. 0.3.1)>"
+    echo -e "\t--multus-conf-file=$MULTUS_CONF_FILE"
+    echo -e "\t--multus-bin-file=$MULTUS_BIN_FILE"
+    echo -e "\t--skip-multus-binary-copy=$SKIP_BINARY_COPY"
+    echo -e "\t--multus-kubeconfig-file-host=$MULTUS_KUBECONFIG_FILE_HOST"
+    echo -e "\t--namespace-isolation=$MULTUS_NAMESPACE_ISOLATION"
+    echo -e "\t--multus-autoconfig-dir=$MULTUS_AUTOCONF_DIR (used only with --multus-conf-file=auto)"
+    echo -e "\t--multus-log-level=$MULTUS_LOG_LEVEL (empty by default, used only with --multus-conf-file=auto)"
+    echo -e "\t--multus-log-file=$MULTUS_LOG_FILE (empty by default, used only with --multus-conf-file=auto)"
+    echo -e "\t--override-network-name=false (used only with --multus-conf-file=auto)"
+    echo -e "\t--cleanup-config-on-exit=false (used only with --multus-conf-file=auto)"
+    echo -e "\t--rename-conf-file=false (used only with --multus-conf-file=auto)"
+    echo -e "\t--readiness-indicator-file=$MULTUS_READINESS_INDICATOR_FILE (used only with --multus-conf-file=auto)"
+    echo -e "\t--additional-bin-dir=$ADDITIONAL_BIN_DIR (adds binDir option to configuration, used only with --multus-conf-file=auto)"
+    echo -e "\t--restart-crio=false (restarts CRIO after config file is generated)"
+}
+
+function log()
+{
+    echo "$(date --iso-8601=seconds) ${1}"
+}
+
+function error()
+{
+    log "ERR:  {$1}"
+}
+
+function warn()
+{
+    log "WARN: {$1}"
+}
+
+# Parse parameters given as arguments to this script.
+while [ "$1" != "" ]; do
+    PARAM=`echo $1 | awk -F= '{print $1}'`
+    VALUE=`echo $1 | awk -F= '{print $2}'`
+    case $PARAM in
+        -h | --help)
+            usage
+            exit
+            ;;
+        --cni-version)
+            CNI_VERSION=$VALUE
+            ;;
+        --cni-conf-dir)
+            CNI_CONF_DIR=$VALUE
+            ;;
+        --cni-bin-dir)
+            CNI_BIN_DIR=$VALUE
+            ;;
+        --multus-conf-file)
+            MULTUS_CONF_FILE=$VALUE
+            ;;
+        --multus-bin-file)
+            MULTUS_BIN_FILE=$VALUE
+            ;;
+        --multus-kubeconfig-file-host)
+            MULTUS_KUBECONFIG_FILE_HOST=$VALUE
+            ;;
+        --namespace-isolation)
+            MULTUS_NAMESPACE_ISOLATION=$VALUE
+            ;;
+        --multus-log-level)
+            MULTUS_LOG_LEVEL=$VALUE
+            ;;
+        --multus-log-file)
+            MULTUS_LOG_FILE=$VALUE
+            ;;
+        --multus-autoconfig-dir)
+            MULTUS_AUTOCONF_DIR=$VALUE
+            ;;
+        --override-network-name)
+            OVERRIDE_NETWORK_NAME=$VALUE
+            ;;
+        --cleanup-config-on-exit)
+            MULTUS_CLEANUP_CONFIG_ON_EXIT=$VALUE
+            ;;
+        --restart-crio)
+            RESTART_CRIO=$VALUE
+            ;;
+        --rename-conf-file)
+            RENAME_SOURCE_CONFIG_FILE=$VALUE
+            ;;
+        --additional-bin-dir)
+            ADDITIONAL_BIN_DIR=$VALUE
+            ;;
+        --skip-multus-binary-copy)
+            SKIP_BINARY_COPY=$VALUE
+            ;;
+        --readiness-indicator-file)
+            MULTUS_READINESS_INDICATOR_FILE=$VALUE
+            ;;
+        *)
+            warn "unknown parameter \"$PARAM\""
+            ;;
+    esac
+    shift
+done
+
+
+# Create array of known locations
+declare -a arr=($CNI_CONF_DIR $CNI_BIN_DIR $MULTUS_BIN_FILE)
+if [ "$MULTUS_CONF_FILE" != "auto" ]; then
+  arr+=($MULTUS_CONF_FILE)
+fi
+
+
+# Loop through and verify each location each.
+for i in "${arr[@]}"
+do
+  if [ ! -e "$i" ]; then
+    warn "Location $i does not exist"
+    exit 1;
+  fi
+done
+
+# Copy files into place and atomically move into final binary name
+if [ "$SKIP_BINARY_COPY" = false ]; then
+  cp -f $MULTUS_BIN_FILE $CNI_BIN_DIR/_multus
+  mv -f $CNI_BIN_DIR/_multus $CNI_BIN_DIR/multus
+else
+  log "Entrypoint skipped copying Multus binary."
+fi
+
+if [ "$MULTUS_CONF_FILE" != "auto" ]; then
+  cp -f $MULTUS_CONF_FILE $CNI_CONF_DIR
+fi
+
+# Make a multus.d directory (for our kubeconfig)
+
+mkdir -p $CNI_CONF_DIR/multus.d
+MULTUS_KUBECONFIG=$CNI_CONF_DIR/multus.d/multus.kubeconfig
+
+# ------------------------------- Generate a "kube-config"
+# Inspired by: https://tinyurl.com/y7r2knme
+SERVICE_ACCOUNT_PATH=/var/run/secrets/kubernetes.io/serviceaccount
+KUBE_CA_FILE=${KUBE_CA_FILE:-$SERVICE_ACCOUNT_PATH/ca.crt}
+SERVICEACCOUNT_TOKEN=$(cat $SERVICE_ACCOUNT_PATH/token)
+SKIP_TLS_VERIFY=${SKIP_TLS_VERIFY:-false}
+
+
+# Check if we're running as a k8s pod.
+if [ -f "$SERVICE_ACCOUNT_PATH/token" ]; then
+  # We're running as a k8d pod - expect some variables.
+  if [ -z ${KUBERNETES_SERVICE_HOST} ]; then
+    error "KUBERNETES_SERVICE_HOST not set"; exit 1;
+  fi
+  if [ -z ${KUBERNETES_SERVICE_PORT} ]; then
+    error "KUBERNETES_SERVICE_PORT not set"; exit 1;
+  fi
+
+  if [ "$SKIP_TLS_VERIFY" == "true" ]; then
+    TLS_CFG="insecure-skip-tls-verify: true"
+  elif [ -f "$KUBE_CA_FILE" ]; then
+    TLS_CFG="certificate-authority-data: $(cat $KUBE_CA_FILE | base64 | tr -d '\n')"
+  fi
+
+  # Write a kubeconfig file for the CNI plugin.  Do this
+  # to skip TLS verification for now.  We should eventually support
+  # writing more complete kubeconfig files. This is only used
+  # if the provided CNI network config references it.
+  touch $MULTUS_KUBECONFIG
+  chmod ${KUBECONFIG_MODE:-600} $MULTUS_KUBECONFIG
+  cat > $MULTUS_KUBECONFIG <<EOF
+# Kubeconfig file for Multus CNI plugin.
+apiVersion: v1
+kind: Config
+clusters:
+- name: local
+  cluster:
+    server: ${KUBERNETES_SERVICE_PROTOCOL:-https}://[${KUBERNETES_SERVICE_HOST}]:${KUBERNETES_SERVICE_PORT}
+    $TLS_CFG
+users:
+- name: multus
+  user:
+    token: "${SERVICEACCOUNT_TOKEN}"
+contexts:
+- name: multus-context
+  context:
+    cluster: local
+    user: multus
+current-context: multus-context
+EOF
+
+else
+  warn "Doesn't look like we're running in a kubernetes environment (no serviceaccount token)"
+fi
+
+# ---------------------- end Generate a "kube-config".
+
+# ------------------------------- Generate "00-multus.conf"
+
+function generateMultusConf {
+if [ "$MULTUS_CONF_FILE" == "auto" ]; then
+  log "Generating Multus configuration file using files in $MULTUS_AUTOCONF_DIR..."
+  found_master=false
+  tries=0
+  while [ $found_master == false ]; do
+    MASTER_PLUGIN="$(ls $MULTUS_AUTOCONF_DIR | grep -E '\.conf(list)?$' | grep -Ev '00-multus\.conf' | head -1)"
+    if [ "$MASTER_PLUGIN" == "" ]; then
+      if [ $tries -lt 600 ]; then
+        if ! (($tries % 5)); then
+          log "Attemping to find master plugin configuration, attempt $tries"
+        fi
+        let "tries+=1"
+        sleep 1;
+      else
+        error "Multus could not be configured: no master plugin was found."
+        exit 1;
+      fi
+    else
+
+      found_master=true
+
+      ISOLATION_STRING=""
+      if [ "$MULTUS_NAMESPACE_ISOLATION" == true ]; then
+        ISOLATION_STRING="\"namespaceIsolation\": true,"
+      fi
+
+      LOG_LEVEL_STRING=""
+      if [ ! -z "${MULTUS_LOG_LEVEL// }" ]; then
+        case "$MULTUS_LOG_LEVEL" in
+          debug)
+              ;;
+          error)
+              ;;
+          panic)
+              ;;
+          verbose)
+              ;;
+          *)
+              error "Log levels should be one of: debug/verbose/error/panic, did not understand $MULTUS_LOG_LEVEL"
+              usage
+              exit 1     
+        esac
+        LOG_LEVEL_STRING="\"logLevel\": \"$MULTUS_LOG_LEVEL\","
+      fi
+
+      LOG_FILE_STRING=""
+      if [ ! -z "${MULTUS_LOG_FILE// }" ]; then
+        LOG_FILE_STRING="\"logFile\": \"$MULTUS_LOG_FILE\","
+      fi
+
+      CNI_VERSION_STRING=""
+      if [ ! -z "${CNI_VERSION// }" ]; then
+        CNI_VERSION_STRING="\"cniVersion\": \"$CNI_VERSION\","
+      fi
+
+      ADDITIONAL_BIN_DIR_STRING=""
+      if [ ! -z "${ADDITIONAL_BIN_DIR// }" ]; then
+        ADDITIONAL_BIN_DIR_STRING="\"binDir\": \"$ADDITIONAL_BIN_DIR\","
+      fi
+
+
+      READINESS_INDICATOR_FILE_STRING=""
+      if [ ! -z "${MULTUS_READINESS_INDICATOR_FILE// }" ]; then
+        READINESS_INDICATOR_FILE_STRING="\"readinessindicatorfile\": \"$MULTUS_READINESS_INDICATOR_FILE\","
+      fi
+
+      if [ "$OVERRIDE_NETWORK_NAME" == "true" ]; then
+        MASTER_PLUGIN_NET_NAME="$(cat $MULTUS_AUTOCONF_DIR/$MASTER_PLUGIN | \
+            python -c 'import json,sys;print json.load(sys.stdin)["name"]')"
+      else
+        MASTER_PLUGIN_NET_NAME="multus-cni-network"
+      fi
+
+      MASTER_PLUGIN_LOCATION=$MULTUS_AUTOCONF_DIR/$MASTER_PLUGIN
+      MASTER_PLUGIN_JSON="$(cat $MASTER_PLUGIN_LOCATION)"
+      log "Using $MASTER_PLUGIN_LOCATION as a source to generate the Multus configuration"
+      CONF=$(cat <<-EOF
+        {
+          $CNI_VERSION_STRING
+          "name": "$MASTER_PLUGIN_NET_NAME",
+          "type": "multus",
+          $ISOLATION_STRING
+          $LOG_LEVEL_STRING
+          $LOG_FILE_STRING
+          $ADDITIONAL_BIN_DIR_STRING
+          $READINESS_INDICATOR_FILE_STRING
+          "kubeconfig": "$MULTUS_KUBECONFIG_FILE_HOST",
+          "delegates": [
+            $MASTER_PLUGIN_JSON
+          ]
+        }
+EOF
+      )
+      tmpfile=$(mktemp)
+      echo $CONF > $tmpfile
+      mv $tmpfile $CNI_CONF_DIR/00-multus.conf
+      log "Config file created @ $CNI_CONF_DIR/00-multus.conf"
+      echo $CONF
+      
+      # If we're not performing the cleanup on exit, we can safely rename the config file.
+      if [ "$RENAME_SOURCE_CONFIG_FILE" == true ]; then
+        mv ${MULTUS_AUTOCONF_DIR}/${MASTER_PLUGIN} ${MULTUS_AUTOCONF_DIR}/${MASTER_PLUGIN}.old
+        log "Original master file moved to ${MULTUS_AUTOCONF_DIR}/${MASTER_PLUGIN}.old"
+      fi
+
+      if [ "$RESTART_CRIO" == true ]; then
+        # Restart CRIO only once.
+        if [ "$CRIO_RESTARTED_ONCE" == false ]; then
+          log "Restarting crio"
+          systemctl restart crio
+          CRIO_RESTARTED_ONCE=true
+        fi
+      fi
+    fi
+  done
+fi
+}
+generateMultusConf
+
+# ---------------------- end Generate "00-multus.conf".
+
+# Enter either sleep loop, or watch loop...
+if [ "$MULTUS_CLEANUP_CONFIG_ON_EXIT" == true ]; then
+  log "Entering watch loop..."
+  while true; do
+    # Check and see if the original master plugin configuration exists...
+    if [ ! -f "$MASTER_PLUGIN_LOCATION" ]; then
+      log "Master plugin @ $MASTER_PLUGIN_LOCATION has been deleted. Allowing 45 seconds for its restoration..."
+      sleep 10
+      for i in {1..35}
+      do
+        if [ -f "$MASTER_PLUGIN_LOCATION" ]; then
+          log "Master plugin @ $MASTER_PLUGIN_LOCATION was restored. Regenerating given configuration."
+          break
+        fi
+        sleep 1
+      done
+
+      generateMultusConf
+      log "Continuing watch loop after configuration regeneration..."
+    fi
+    sleep 1
+  done
+else
+  log "Entering sleep (success)..."
+  sleep infinity
+fi

images/multus-daemonset-crio.yml

@@ -0,0 +1,273 @@
+---
+apiVersion: apiextensions.k8s.io/v1
+kind: CustomResourceDefinition
+metadata:
+  name: network-attachment-definitions.k8s.cni.cncf.io
+spec:
+  group: k8s.cni.cncf.io
+  scope: Namespaced
+  names:
+    plural: network-attachment-definitions
+    singular: network-attachment-definition
+    kind: NetworkAttachmentDefinition
+    shortNames:
+    - net-attach-def
+  versions:
+    - name: v1
+      served: true
+      storage: true
+      schema:
+        openAPIV3Schema:
+          description: 'NetworkAttachmentDefinition is a CRD schema specified by the Network Plumbing
+            Working Group to express the intent for attaching pods to one or more logical or physical
+            networks. More information available at: https://github.com/k8snetworkplumbingwg/multi-net-spec'
+          type: object
+          properties:
+            spec:
+              description: 'NetworkAttachmentDefinition spec defines the desired state of a network attachment'
+              type: object
+              properties:
+                config:
+                  description: 'NetworkAttachmentDefinition config is a JSON-formatted CNI configuration'
+                  type: string
+---
+kind: ClusterRole
+apiVersion: rbac.authorization.k8s.io/v1
+metadata:
+  name: multus
+rules:
+  - apiGroups: ["k8s.cni.cncf.io"]
+    resources:
+      - '*'
+    verbs:
+      - '*'
+  - apiGroups:
+      - ""
+    resources:
+      - pods
+      - pods/status
+    verbs:
+      - get
+      - update
+  - apiGroups:
+      - ""
+      - events.k8s.io
+    resources:
+      - events
+    verbs:
+      - create
+      - patch
+      - update
+---
+kind: ClusterRoleBinding
+apiVersion: rbac.authorization.k8s.io/v1
+metadata:
+  name: multus
+roleRef:
+  apiGroup: rbac.authorization.k8s.io
+  kind: ClusterRole
+  name: multus
+subjects:
+- kind: ServiceAccount
+  name: multus
+  namespace: kube-system
+---
+apiVersion: v1
+kind: ServiceAccount
+metadata:
+  name: multus
+  namespace: kube-system
+---
+kind: ConfigMap
+apiVersion: v1
+metadata:
+  name: multus-cni-config
+  namespace: kube-system
+  labels:
+    tier: node
+    app: multus
+data:
+  # NOTE: If you'd prefer to manually apply a configuration file, you may create one here.
+  # In the case you'd like to customize the Multus installation, you should change the arguments to the Multus pod
+  # change the "args" line below from
+  # - "--multus-conf-file=auto"
+  # to:
+  # "--multus-conf-file=/tmp/multus-conf/70-multus.conf"
+  # Additionally -- you should ensure that the name "70-multus.conf" is the alphabetically first name in the
+  # /etc/cni/net.d/ directory on each node, otherwise, it will not be used by the Kubelet.
+  cni-conf.json: |
+    {
+      "name": "multus-cni-network",
+      "type": "multus",
+      "capabilities": {
+        "portMappings": true
+      },
+      "delegates": [
+        {
+          "cniVersion": "0.3.1",
+          "name": "default-cni-network",
+          "plugins": [
+            {
+              "type": "flannel",
+              "name": "flannel.1",
+                "delegate": {
+                  "isDefaultGateway": true,
+                  "hairpinMode": true
+                }
+              },
+              {
+                "type": "portmap",
+                "capabilities": {
+                  "portMappings": true
+                }
+              }
+          ]
+        }
+      ],
+      "kubeconfig": "/etc/cni/net.d/multus.d/multus.kubeconfig"
+    }
+---
+apiVersion: apps/v1
+kind: DaemonSet
+metadata:
+  name: kube-multus-ds-amd64
+  namespace: kube-system
+  labels:
+    tier: node
+    app: multus
+    name: multus
+spec:
+  selector:
+    matchLabels:
+      name: multus
+  updateStrategy:
+    type: RollingUpdate
+  template:
+    metadata:
+      labels:
+        tier: node
+        app: multus
+        name: multus
+    spec:
+      hostNetwork: true
+      nodeSelector:
+        kubernetes.io/arch: amd64
+      tolerations:
+      - operator: Exists
+        effect: NoSchedule
+      serviceAccountName: multus
+      containers:
+      - name: kube-multus
+        # crio support requires multus:latest for now. support 3.3 or later.
+        image: nfvpe/multus:latest
+        command: ["/entrypoint.sh"]
+        args:
+        - "--cni-version=0.3.1"
+        - "--cni-bin-dir=/host/usr/libexec/cni"
+        - "--multus-conf-file=auto"
+        - "--restart-crio=true"
+        resources:
+          requests:
+            cpu: "100m"
+            memory: "50Mi"
+          limits:
+            cpu: "100m"
+            memory: "50Mi"
+        securityContext:
+          privileged: true
+          capabilities:
+            add: ["SYS_ADMIN"]
+        volumeMounts:
+        - name: run
+          mountPath: /run
+        - name: cni
+          mountPath: /host/etc/cni/net.d
+        - name: cnibin
+          mountPath: /host/usr/libexec/cni
+        - name: multus-cfg
+          mountPath: /tmp/multus-conf
+      volumes:
+        - name: run
+          hostPath:
+            path: /run
+        - name: cni
+          hostPath:
+            path: /etc/cni/net.d
+        - name: cnibin
+          hostPath:
+            path: /usr/libexec/cni
+        - name: multus-cfg
+          configMap:
+            name: multus-cni-config
+            items:
+            - key: cni-conf.json
+              path: 70-multus.conf
+---
+apiVersion: apps/v1
+kind: DaemonSet
+metadata:
+  name: kube-multus-ds-ppc64le
+  namespace: kube-system
+  labels:
+    tier: node
+    app: multus
+    name: multus
+spec:
+  selector:
+    matchLabels:
+      name: multus
+  updateStrategy:
+    type: RollingUpdate
+  template:
+    metadata:
+      labels:
+        tier: node
+        app: multus
+        name: multus
+    spec:
+      hostNetwork: true
+      nodeSelector:
+        kubernetes.io/arch: ppc64le
+      tolerations:
+      - operator: Exists
+        effect: NoSchedule
+      serviceAccountName: multus
+      containers:
+      - name: kube-multus
+        # crio support requires multus:latest for now. support 3.3 or later.
+        image: nfvpe/multus:latest-ppc64le
+        command: ["/entrypoint.sh"]
+        args:
+        - "--cni-version=0.3.1"
+        - "--cni-bin-dir=/host/usr/libexec/cni"
+        - "--multus-conf-file=auto"
+        - "--restart-crio=true"
+        resources:
+          requests:
+            cpu: "100m"
+            memory: "90Mi"
+          limits:
+            cpu: "100m"
+            memory: "90Mi"
+        securityContext:
+          privileged: true
+        volumeMounts:
+        - name: cni
+          mountPath: /host/etc/cni/net.d
+        - name: cnibin
+          mountPath: /host/usr/libexec/cni
+        - name: multus-cfg
+          mountPath: /tmp/multus-conf
+      volumes:
+        - name: cni
+          hostPath:
+            path: /etc/cni/net.d
+        - name: cnibin
+          hostPath:
+            path: /usr/libexec/cni
+        - name: multus-cfg
+          configMap:
+            name: multus-cni-config
+            items:
+            - key: cni-conf.json
+              path: 70-multus.conf

images/multus-daemonset-gke-1.16.yml

@@ -0,0 +1,249 @@
+---
+apiVersion: apiextensions.k8s.io/v1
+kind: CustomResourceDefinition
+metadata:
+  name: network-attachment-definitions.k8s.cni.cncf.io
+spec:
+  group: k8s.cni.cncf.io
+  scope: Namespaced
+  names:
+    plural: network-attachment-definitions
+    singular: network-attachment-definition
+    kind: NetworkAttachmentDefinition
+    shortNames:
+      - net-attach-def
+  versions:
+    - name: v1
+      served: true
+      storage: true
+      schema:
+        openAPIV3Schema:
+          type: object
+          properties:
+            spec:
+              type: object
+              properties:
+                config:
+                  type: string
+---
+kind: ClusterRole
+apiVersion: rbac.authorization.k8s.io/v1
+metadata:
+  name: multus
+rules:
+  - apiGroups: ["k8s.cni.cncf.io"]
+    resources:
+      - '*'
+    verbs:
+      - '*'
+  - apiGroups:
+      - ""
+    resources:
+      - pods
+      - pods/status
+    verbs:
+      - get
+      - update
+---
+kind: ClusterRoleBinding
+apiVersion: rbac.authorization.k8s.io/v1
+metadata:
+  name: multus
+roleRef:
+  apiGroup: rbac.authorization.k8s.io
+  kind: ClusterRole
+  name: multus
+subjects:
+  - kind: ServiceAccount
+    name: multus
+    namespace: kube-system
+---
+apiVersion: v1
+kind: ServiceAccount
+metadata:
+  name: multus
+  namespace: kube-system
+---
+kind: ConfigMap
+apiVersion: v1
+metadata:
+  name: multus-cni-config
+  namespace: kube-system
+  labels:
+    tier: node
+    app: multus
+data:
+  # NOTE: If you'd prefer to manually apply a configuration file, you may create one here.
+  # In the case you'd like to customize the Multus installation, you should change the arguments to the Multus pod
+  # change the "args" line below from
+  # - "--multus-conf-file=auto"
+  # to:
+  # "--multus-conf-file=/tmp/multus-conf/70-multus.conf"
+  # Additionally -- you should ensure that the name "70-multus.conf" is the alphabetically first name in the
+  # /etc/cni/net.d/ directory on each node, otherwise, it will not be used by the Kubelet.
+  cni-conf.json: |
+    {
+      "name": "multus-cni-network",
+      "type": "multus",
+      "capabilities": {
+        "portMappings": true
+      },
+      "delegates": [
+        {
+          "cniVersion": "0.3.1",
+          "name": "default-cni-network",
+          "plugins": [
+            {
+              "type": "flannel",
+              "name": "flannel.1",
+                "delegate": {
+                  "isDefaultGateway": true,
+                  "hairpinMode": true
+                }
+              },
+              {
+                "type": "portmap",
+                "capabilities": {
+                  "portMappings": true
+                }
+              }
+          ]
+        }
+      ],
+      "kubeconfig": "/etc/cni/net.d/multus.d/multus.kubeconfig"
+    }
+---
+apiVersion: apps/v1
+kind: DaemonSet
+metadata:
+  name: kube-multus-ds-amd64
+  namespace: kube-system
+  labels:
+    tier: node
+    app: multus
+    name: multus
+spec:
+  selector:
+    matchLabels:
+      name: multus
+  updateStrategy:
+    type: RollingUpdate
+  template:
+    metadata:
+      labels:
+        tier: node
+        app: multus
+        name: multus
+    spec:
+      hostNetwork: true
+      nodeSelector:
+        kubernetes.io/arch: amd64
+      tolerations:
+        - operator: Exists
+          effect: NoSchedule
+      serviceAccountName: multus
+      containers:
+        - name: kube-multus
+          image: nfvpe/multus:v3.4
+          command: ["/entrypoint.sh"]
+          args:
+            - "--multus-conf-file=auto"
+            - "--cni-version=0.3.1"
+            - "--cni-bin-dir=/host/home/kubernetes/bin"
+          resources:
+            requests:
+              cpu: "100m"
+              memory: "50Mi"
+            limits:
+              cpu: "100m"
+              memory: "50Mi"
+          securityContext:
+            privileged: true
+          volumeMounts:
+            - name: cni
+              mountPath: /host/etc/cni/net.d
+            - name: cnibin
+              mountPath: /host/home/kubernetes/bin
+            - name: multus-cfg
+              mountPath: /tmp/multus-conf
+      volumes:
+        - name: cni
+          hostPath:
+            path: /etc/cni/net.d
+        - name: cnibin
+          hostPath:
+            path: /home/kubernetes/bin
+        - name: multus-cfg
+          configMap:
+            name: multus-cni-config
+            items:
+              - key: cni-conf.json
+                path: 70-multus.conf
+---
+apiVersion: apps/v1
+kind: DaemonSet
+metadata:
+  name: kube-multus-ds-ppc64le
+  namespace: kube-system
+  labels:
+    tier: node
+    app: multus
+    name: multus
+spec:
+  selector:
+    matchLabels:
+      name: multus
+  updateStrategy:
+    type: RollingUpdate
+  template:
+    metadata:
+      labels:
+        tier: node
+        app: multus
+        name: multus
+    spec:
+      hostNetwork: true
+      nodeSelector:
+        kubernetes.io/arch: ppc64le
+      tolerations:
+        - operator: Exists
+          effect: NoSchedule
+      serviceAccountName: multus
+      containers:
+        - name: kube-multus
+          # ppc64le support requires multus:latest for now. support 3.3 or later.
+          image: nfvpe/multus:latest-ppc64le
+          command: ["/entrypoint.sh"]
+          args:
+            - "--multus-conf-file=auto"
+            - "--cni-version=0.3.1"
+            - "--cni-bin-dir=/host/home/kubernetes/bin"
+          resources:
+            requests:
+              cpu: "100m"
+              memory: "90Mi"
+            limits:
+              cpu: "100m"
+              memory: "90Mi"
+          securityContext:
+            privileged: true
+          volumeMounts:
+            - name: cni
+              mountPath: /host/etc/cni/net.d
+            - name: cnibin
+              mountPath: /host/home/kubernetes/bin
+            - name: multus-cfg
+              mountPath: /tmp/multus-conf
+      volumes:
+        - name: cni
+          hostPath:
+            path: /etc/cni/net.d
+        - name: cnibin
+          hostPath:
+            path: /home/kubernetes/bin
+        - name: multus-cfg
+          configMap:
+            name: multus-cni-config
+            items:
+              - key: cni-conf.json
+                path: 70-multus.conf

images/multus-daemonset.yml

@@ -0,0 +1,261 @@
+---
+apiVersion: apiextensions.k8s.io/v1
+kind: CustomResourceDefinition
+metadata:
+  name: network-attachment-definitions.k8s.cni.cncf.io
+spec:
+  group: k8s.cni.cncf.io
+  scope: Namespaced
+  names:
+    plural: network-attachment-definitions
+    singular: network-attachment-definition
+    kind: NetworkAttachmentDefinition
+    shortNames:
+    - net-attach-def
+  versions:
+    - name: v1
+      served: true
+      storage: true
+      schema:
+        openAPIV3Schema:
+          description: 'NetworkAttachmentDefinition is a CRD schema specified by the Network Plumbing
+            Working Group to express the intent for attaching pods to one or more logical or physical
+            networks. More information available at: https://github.com/k8snetworkplumbingwg/multi-net-spec'
+          type: object
+          properties:
+            spec:
+              description: 'NetworkAttachmentDefinition spec defines the desired state of a network attachment'
+              type: object
+              properties:
+                config:
+                  description: 'NetworkAttachmentDefinition config is a JSON-formatted CNI configuration'
+                  type: string
+---
+kind: ClusterRole
+apiVersion: rbac.authorization.k8s.io/v1
+metadata:
+  name: multus
+rules:
+  - apiGroups: ["k8s.cni.cncf.io"]
+    resources:
+      - '*'
+    verbs:
+      - '*'
+  - apiGroups:
+      - ""
+    resources:
+      - pods
+      - pods/status
+    verbs:
+      - get
+      - update
+  - apiGroups:
+      - ""
+      - events.k8s.io
+    resources:
+      - events
+    verbs:
+      - create
+      - patch
+      - update
+---
+kind: ClusterRoleBinding
+apiVersion: rbac.authorization.k8s.io/v1
+metadata:
+  name: multus
+roleRef:
+  apiGroup: rbac.authorization.k8s.io
+  kind: ClusterRole
+  name: multus
+subjects:
+- kind: ServiceAccount
+  name: multus
+  namespace: kube-system
+---
+apiVersion: v1
+kind: ServiceAccount
+metadata:
+  name: multus
+  namespace: kube-system
+---
+kind: ConfigMap
+apiVersion: v1
+metadata:
+  name: multus-cni-config
+  namespace: kube-system
+  labels:
+    tier: node
+    app: multus
+data:
+  # NOTE: If you'd prefer to manually apply a configuration file, you may create one here.
+  # In the case you'd like to customize the Multus installation, you should change the arguments to the Multus pod
+  # change the "args" line below from
+  # - "--multus-conf-file=auto"
+  # to:
+  # "--multus-conf-file=/tmp/multus-conf/70-multus.conf"
+  # Additionally -- you should ensure that the name "70-multus.conf" is the alphabetically first name in the
+  # /etc/cni/net.d/ directory on each node, otherwise, it will not be used by the Kubelet.
+  cni-conf.json: |
+    {
+      "name": "multus-cni-network",
+      "type": "multus",
+      "capabilities": {
+        "portMappings": true
+      },
+      "delegates": [
+        {
+          "cniVersion": "0.3.1",
+          "name": "default-cni-network",
+          "plugins": [
+            {
+              "type": "flannel",
+              "name": "flannel.1",
+                "delegate": {
+                  "isDefaultGateway": true,
+                  "hairpinMode": true
+                }
+              },
+              {
+                "type": "portmap",
+                "capabilities": {
+                  "portMappings": true
+                }
+              }
+          ]
+        }
+      ],
+      "kubeconfig": "/etc/cni/net.d/multus.d/multus.kubeconfig"
+    }
+---
+apiVersion: apps/v1
+kind: DaemonSet
+metadata:
+  name: kube-multus-ds-amd64
+  namespace: kube-system
+  labels:
+    tier: node
+    app: multus
+    name: multus
+spec:
+  selector:
+    matchLabels:
+      name: multus
+  updateStrategy:
+    type: RollingUpdate
+  template:
+    metadata:
+      labels:
+        tier: node
+        app: multus
+        name: multus
+    spec:
+      hostNetwork: true
+      nodeSelector:
+        kubernetes.io/arch: amd64
+      tolerations:
+      - operator: Exists
+        effect: NoSchedule
+      serviceAccountName: multus
+      containers:
+      - name: kube-multus
+        image: nfvpe/multus:v3.4.1
+        command: ["/entrypoint.sh"]
+        args:
+        - "--multus-conf-file=auto"
+        - "--cni-version=0.3.1"
+        resources:
+          requests:
+            cpu: "100m"
+            memory: "50Mi"
+          limits:
+            cpu: "100m"
+            memory: "50Mi"
+        securityContext:
+          privileged: true
+        volumeMounts:
+        - name: cni
+          mountPath: /host/etc/cni/net.d
+        - name: cnibin
+          mountPath: /host/opt/cni/bin
+        - name: multus-cfg
+          mountPath: /tmp/multus-conf
+      volumes:
+        - name: cni
+          hostPath:
+            path: /etc/cni/net.d
+        - name: cnibin
+          hostPath:
+            path: /opt/cni/bin
+        - name: multus-cfg
+          configMap:
+            name: multus-cni-config
+            items:
+            - key: cni-conf.json
+              path: 70-multus.conf
+---
+apiVersion: apps/v1
+kind: DaemonSet
+metadata:
+  name: kube-multus-ds-ppc64le
+  namespace: kube-system
+  labels:
+    tier: node
+    app: multus
+    name: multus
+spec:
+  selector:
+    matchLabels:
+      name: multus
+  updateStrategy:
+    type: RollingUpdate
+  template:
+    metadata:
+      labels:
+        tier: node
+        app: multus
+        name: multus
+    spec:
+      hostNetwork: true
+      nodeSelector:
+        kubernetes.io/arch: ppc64le
+      tolerations:
+      - operator: Exists
+        effect: NoSchedule
+      serviceAccountName: multus
+      containers:
+      - name: kube-multus
+        # ppc64le support requires multus:latest for now. support 3.3 or later.
+        image: nfvpe/multus:latest-ppc64le
+        command: ["/entrypoint.sh"]
+        args:
+        - "--multus-conf-file=auto"
+        - "--cni-version=0.3.1"
+        resources:
+          requests:
+            cpu: "100m"
+            memory: "90Mi"
+          limits:
+            cpu: "100m"
+            memory: "90Mi"
+        securityContext:
+          privileged: true
+        volumeMounts:
+        - name: cni
+          mountPath: /host/etc/cni/net.d
+        - name: cnibin
+          mountPath: /host/opt/cni/bin
+        - name: multus-cfg
+          mountPath: /tmp/multus-conf
+      volumes:
+        - name: cni
+          hostPath:
+            path: /etc/cni/net.d
+        - name: cnibin
+          hostPath:
+            path: /opt/cni/bin
+        - name: multus-cfg
+          configMap:
+            name: multus-cni-config
+            items:
+            - key: cni-conf.json
+              path: 70-multus.conf

k8sclient/k8sclient.go

@@ -0,0 +1,599 @@
+// Copyright (c) 2017 Intel Corporation
+//
+// Licensed under the Apache License, Version 2.0 (the "License");
+// you may not use this file except in compliance with the License.
+// You may obtain a copy of the License at
+//
+//     http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+package k8sclient
+
+import (
+	"encoding/json"
+	"fmt"
+	"net"
+	"os"
+	"regexp"
+	"strings"
+	"time"
+
+	v1 "k8s.io/api/core/v1"
+	metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
+	"k8s.io/apimachinery/pkg/runtime"
+	"k8s.io/client-go/kubernetes"
+	"k8s.io/client-go/kubernetes/scheme"
+	v1core "k8s.io/client-go/kubernetes/typed/core/v1"
+	"k8s.io/client-go/rest"
+	"k8s.io/client-go/tools/clientcmd"
+	"k8s.io/client-go/tools/record"
+	"k8s.io/klog"
+
+	"github.com/containernetworking/cni/libcni"
+	"github.com/containernetworking/cni/pkg/skel"
+	cnitypes "github.com/containernetworking/cni/pkg/types"
+	nettypes "github.com/k8snetworkplumbingwg/network-attachment-definition-client/pkg/apis/k8s.cni.cncf.io/v1"
+	netclient "github.com/k8snetworkplumbingwg/network-attachment-definition-client/pkg/client/clientset/versioned/typed/k8s.cni.cncf.io/v1"
+	netutils "github.com/k8snetworkplumbingwg/network-attachment-definition-client/pkg/utils"
+	"gopkg.in/intel/multus-cni.v3/kubeletclient"
+	"gopkg.in/intel/multus-cni.v3/logging"
+	"gopkg.in/intel/multus-cni.v3/types"
+)
+
+const (
+	resourceNameAnnot      = "k8s.v1.cni.cncf.io/resourceName"
+	defaultNetAnnot        = "v1.multus-cni.io/default-network"
+	networkAttachmentAnnot = "k8s.v1.cni.cncf.io/networks"
+)
+
+// NoK8sNetworkError indicates error, no network in kubernetes
+type NoK8sNetworkError struct {
+	message string
+}
+
+// ClientInfo contains information given from k8s client
+type ClientInfo struct {
+	Client           kubernetes.Interface
+	NetClient        netclient.K8sCniCncfIoV1Interface
+	EventBroadcaster record.EventBroadcaster
+	EventRecorder    record.EventRecorder
+}
+
+// AddPod adds pod into kubernetes
+func (c *ClientInfo) AddPod(pod *v1.Pod) (*v1.Pod, error) {
+	return c.Client.Core().Pods(pod.ObjectMeta.Namespace).Create(pod)
+}
+
+// GetPod gets pod from kubernetes
+func (c *ClientInfo) GetPod(namespace, name string) (*v1.Pod, error) {
+	return c.Client.Core().Pods(namespace).Get(name, metav1.GetOptions{})
+}
+
+// DeletePod deletes a pod from kubernetes
+func (c *ClientInfo) DeletePod(namespace, name string) error {
+	return c.Client.Core().Pods(namespace).Delete(name, &metav1.DeleteOptions{})
+}
+
+// AddNetAttachDef adds net-attach-def into kubernetes
+func (c *ClientInfo) AddNetAttachDef(netattach *nettypes.NetworkAttachmentDefinition) (*nettypes.NetworkAttachmentDefinition, error) {
+	return c.NetClient.NetworkAttachmentDefinitions(netattach.ObjectMeta.Namespace).Create(netattach)
+}
+
+// Eventf puts event into kubernetes events
+func (c *ClientInfo) Eventf(object runtime.Object, eventtype, reason, messageFmt string, args ...interface{}) {
+	if c != nil && c.EventRecorder != nil {
+		c.EventRecorder.Eventf(object, eventtype, reason, messageFmt, args...)
+	}
+}
+
+func (e *NoK8sNetworkError) Error() string { return string(e.message) }
+
+// SetNetworkStatus sets network status into Pod annotation
+func SetNetworkStatus(client *ClientInfo, k8sArgs *types.K8sArgs, netStatus []nettypes.NetworkStatus, conf *types.NetConf) error {
+	var err error
+	logging.Debugf("SetNetworkStatus: %v, %v, %v, %v", client, k8sArgs, netStatus, conf)
+
+	client, err = GetK8sClient(conf.Kubeconfig, client)
+	if err != nil {
+		return logging.Errorf("SetNetworkStatus: %v", err)
+	}
+	if client == nil || client.Client == nil {
+		if len(conf.Delegates) == 0 {
+			// No available kube client and no delegates, we can't do anything
+			return logging.Errorf("SetNetworkStatus: must have either Kubernetes config or delegates")
+		}
+		logging.Debugf("SetNetworkStatus: kube client info is not defined, skip network status setup")
+		return nil
+	}
+
+	podName := string(k8sArgs.K8S_POD_NAME)
+	podNamespace := string(k8sArgs.K8S_POD_NAMESPACE)
+	pod, err := client.GetPod(podNamespace, podName)
+	if err != nil {
+		return logging.Errorf("SetNetworkStatus: failed to query the pod %v in out of cluster comm: %v", podName, err)
+	}
+
+	if netStatus != nil {
+		err = netutils.SetNetworkStatus(client.Client, pod, netStatus)
+		if err != nil {
+			return logging.Errorf("SetNetworkStatus: failed to update the pod %v in out of cluster comm: %v", podName, err)
+		}
+	}
+
+	return nil
+}
+
+func parsePodNetworkObjectName(podnetwork string) (string, string, string, error) {
+	var netNsName string
+	var netIfName string
+	var networkName string
+
+	logging.Debugf("parsePodNetworkObjectName: %s", podnetwork)
+	slashItems := strings.Split(podnetwork, "/")
+	if len(slashItems) == 2 {
+		netNsName = strings.TrimSpace(slashItems[0])
+		networkName = slashItems[1]
+	} else if len(slashItems) == 1 {
+		networkName = slashItems[0]
+	} else {
+		return "", "", "", logging.Errorf("parsePodNetworkObjectName: Invalid network object (failed at '/')")
+	}
+
+	atItems := strings.Split(networkName, "@")
+	networkName = strings.TrimSpace(atItems[0])
+	if len(atItems) == 2 {
+		netIfName = strings.TrimSpace(atItems[1])
+	} else if len(atItems) != 1 {
+		return "", "", "", logging.Errorf("parsePodNetworkObjectName: Invalid network object (failed at '@')")
+	}
+
+	// Check and see if each item matches the specification for valid attachment name.
+	// "Valid attachment names must be comprised of units of the DNS-1123 label format"
+	// [a-z0-9]([-a-z0-9]*[a-z0-9])?
+	// And we allow at (@), and forward slash (/) (units separated by commas)
+	// It must start and end alphanumerically.
+	allItems := []string{netNsName, networkName, netIfName}
+	for i := range allItems {
+		matched, _ := regexp.MatchString("^[a-z0-9]([-a-z0-9]*[a-z0-9])?$", allItems[i])
+		if !matched && len([]rune(allItems[i])) > 0 {
+			return "", "", "", logging.Errorf(fmt.Sprintf("parsePodNetworkObjectName: Failed to parse: one or more items did not match comma-delimited format (must consist of lower case alphanumeric characters). Must start and end with an alphanumeric character), mismatch @ '%v'", allItems[i]))
+		}
+	}
+
+	logging.Debugf("parsePodNetworkObjectName: parsed: %s, %s, %s", netNsName, networkName, netIfName)
+	return netNsName, networkName, netIfName, nil
+}
+
+func parsePodNetworkAnnotation(podNetworks, defaultNamespace string) ([]*types.NetworkSelectionElement, error) {
+	var networks []*types.NetworkSelectionElement
+
+	logging.Debugf("parsePodNetworkAnnotation: %s, %s", podNetworks, defaultNamespace)
+	if podNetworks == "" {
+		return nil, logging.Errorf("parsePodNetworkAnnotation: pod annotation does not have \"network\" as key")
+	}
+
+	if strings.IndexAny(podNetworks, "[{\"") >= 0 {
+		if err := json.Unmarshal([]byte(podNetworks), &networks); err != nil {
+			return nil, logging.Errorf("parsePodNetworkAnnotation: failed to parse pod Network Attachment Selection Annotation JSON format: %v", err)
+		}
+	} else {
+		// Comma-delimited list of network attachment object names
+		for _, item := range strings.Split(podNetworks, ",") {
+			// Remove leading and trailing whitespace.
+			item = strings.TrimSpace(item)
+
+			// Parse network name (i.e. <namespace>/<network name>@<ifname>)
+			netNsName, networkName, netIfName, err := parsePodNetworkObjectName(item)
+			if err != nil {
+				return nil, logging.Errorf("parsePodNetworkAnnotation: %v", err)
+			}
+
+			networks = append(networks, &types.NetworkSelectionElement{
+				Name:             networkName,
+				Namespace:        netNsName,
+				InterfaceRequest: netIfName,
+			})
+		}
+	}
+
+	for _, n := range networks {
+		if n.Namespace == "" {
+			n.Namespace = defaultNamespace
+		}
+		if n.MacRequest != "" {
+			// validate MAC address
+			if _, err := net.ParseMAC(n.MacRequest); err != nil {
+				return nil, logging.Errorf("parsePodNetworkAnnotation: failed to mac: %v", err)
+			}
+		}
+		if n.InfinibandGUIDRequest != "" {
+			// validate GUID address
+			if _, err := net.ParseMAC(n.InfinibandGUIDRequest); err != nil {
+				return nil, logging.Errorf("parsePodNetworkAnnotation: failed to validate infiniband GUID: %v", err)
+			}
+		}
+		if n.IPRequest != nil {
+			for _, ip := range n.IPRequest {
+				// validate IP address
+				if strings.Contains(ip, "/") {
+					if _, _, err := net.ParseCIDR(ip); err != nil {
+						return nil, logging.Errorf("failed to parse CIDR %q: %v", ip, err)
+					}
+				} else if net.ParseIP(ip) == nil {
+					return nil, logging.Errorf("failed to parse IP address %q", ip)
+				}
+			}
+		}
+		// compatibility pre v3.2, will be removed in v4.0
+		if n.DeprecatedInterfaceRequest != "" && n.InterfaceRequest == "" {
+			n.InterfaceRequest = n.DeprecatedInterfaceRequest
+		}
+	}
+
+	return networks, nil
+}
+
+func getKubernetesDelegate(client *ClientInfo, net *types.NetworkSelectionElement, confdir string, pod *v1.Pod, resourceMap map[string]*types.ResourceInfo) (*types.DelegateNetConf, map[string]*types.ResourceInfo, error) {
+
+	logging.Debugf("getKubernetesDelegate: %v, %v, %s, %v, %v", client, net, confdir, pod, resourceMap)
+	customResource, err := client.NetClient.NetworkAttachmentDefinitions(net.Namespace).Get(net.Name, metav1.GetOptions{})
+	if err != nil {
+		errMsg := fmt.Sprintf("cannot find a network-attachment-definition (%s) in namespace (%s): %v", net.Name, net.Namespace, err)
+		if client != nil {
+			client.Eventf(pod, v1.EventTypeWarning, "NoNetworkFound", errMsg)
+		}
+		return nil, resourceMap, logging.Errorf("getKubernetesDelegate: " + errMsg)
+	}
+
+	// Get resourceName annotation from NetworkAttachmentDefinition
+	deviceID := ""
+	resourceName, ok := customResource.GetAnnotations()[resourceNameAnnot]
+	if ok && pod.Name != "" && pod.Namespace != "" {
+		// ResourceName annotation is found; try to get device info from resourceMap
+		logging.Debugf("getKubernetesDelegate: found resourceName annotation : %s", resourceName)
+
+		if resourceMap == nil {
+			ck, err := kubeletclient.GetResourceClient()
+			if err != nil {
+				return nil, resourceMap, logging.Errorf("getKubernetesDelegate: failed to get a ResourceClient instance: %v", err)
+			}
+			resourceMap, err = ck.GetPodResourceMap(pod)
+			if err != nil {
+				return nil, resourceMap, logging.Errorf("getKubernetesDelegate: failed to get resourceMap from ResourceClient: %v", err)
+			}
+			logging.Debugf("getKubernetesDelegate: resourceMap instance: %+v", resourceMap)
+		}
+
+		entry, ok := resourceMap[resourceName]
+		if ok {
+			if idCount := len(entry.DeviceIDs); idCount > 0 && idCount > entry.Index {
+				deviceID = entry.DeviceIDs[entry.Index]
+				logging.Debugf("getKubernetesDelegate: podName: %s deviceID: %s", pod.Name, deviceID)
+				entry.Index++ // increment Index for next delegate
+			}
+		}
+	}
+
+	configBytes, err := netutils.GetCNIConfig(customResource, confdir)
+	if err != nil {
+		return nil, resourceMap, err
+	}
+
+	delegate, err := types.LoadDelegateNetConf(configBytes, net, deviceID)
+	if err != nil {
+		return nil, resourceMap, err
+	}
+
+	return delegate, resourceMap, nil
+}
+
+// GetK8sArgs gets k8s related args from CNI args
+func GetK8sArgs(args *skel.CmdArgs) (*types.K8sArgs, error) {
+	k8sArgs := &types.K8sArgs{}
+
+	logging.Debugf("GetK8sArgs: %v", args)
+	err := cnitypes.LoadArgs(args.Args, k8sArgs)
+	if err != nil {
+		return nil, err
+	}
+
+	return k8sArgs, nil
+}
+
+// TryLoadPodDelegates attempts to load Kubernetes-defined delegates and add them to the Multus config.
+// Returns the number of Kubernetes-defined delegates added or an error.
+func TryLoadPodDelegates(pod *v1.Pod, conf *types.NetConf, clientInfo *ClientInfo, resourceMap map[string]*types.ResourceInfo) (int, *ClientInfo, error) {
+	var err error
+
+	logging.Debugf("TryLoadPodDelegates: %v, %v, %v", pod, conf, clientInfo)
+	clientInfo, err = GetK8sClient(conf.Kubeconfig, clientInfo)
+	if err != nil {
+		return 0, nil, err
+	}
+
+	if clientInfo == nil {
+		if len(conf.Delegates) == 0 {
+			// No available kube client and no delegates, we can't do anything
+			return 0, nil, logging.Errorf("TryLoadPodDelegates: must have either Kubernetes config or delegates")
+		}
+		return 0, nil, nil
+	}
+
+	delegate, err := tryLoadK8sPodDefaultNetwork(clientInfo, pod, conf)
+	if err != nil {
+		return 0, nil, logging.Errorf("TryLoadPodDelegates: error in loading K8s cluster default network from pod annotation: %v", err)
+	}
+	if delegate != nil {
+		logging.Debugf("TryLoadPodDelegates: Overwrite the cluster default network with %v from pod annotations", delegate)
+
+		conf.Delegates[0] = delegate
+	}
+
+	networks, err := GetPodNetwork(pod)
+	if networks != nil {
+		delegates, err := GetNetworkDelegates(clientInfo, pod, networks, conf.ConfDir, conf.NamespaceIsolation, resourceMap)
+
+		if err != nil {
+			if _, ok := err.(*NoK8sNetworkError); ok {
+				return 0, clientInfo, nil
+			}
+			return 0, nil, logging.Errorf("TryLoadPodDelegates: error in getting k8s network for pod: %v", err)
+		}
+
+		if err = conf.AddDelegates(delegates); err != nil {
+			return 0, nil, err
+		}
+
+		// Check gatewayRequest is configured in delegates
+		// and mark its config if gateway filter is required
+		isGatewayConfigured := false
+		for _, delegate := range conf.Delegates {
+			if delegate.GatewayRequest != nil {
+				isGatewayConfigured = true
+				break
+			}
+		}
+
+		if isGatewayConfigured == true {
+			types.CheckGatewayConfig(conf.Delegates)
+		}
+
+		return len(delegates), clientInfo, nil
+	}
+
+	return 0, clientInfo, nil
+}
+
+// GetK8sClient gets client info from kubeconfig
+func GetK8sClient(kubeconfig string, kubeClient *ClientInfo) (*ClientInfo, error) {
+	logging.Debugf("GetK8sClient: %s, %v", kubeconfig, kubeClient)
+	// If we get a valid kubeClient (eg from testcases) just return that
+	// one.
+	if kubeClient != nil {
+		return kubeClient, nil
+	}
+
+	var err error
+	var config *rest.Config
+
+	// Otherwise try to create a kubeClient from a given kubeConfig
+	if kubeconfig != "" {
+		// uses the current context in kubeconfig
+		config, err = clientcmd.BuildConfigFromFlags("", kubeconfig)
+		if err != nil {
+			return nil, logging.Errorf("GetK8sClient: failed to get context for the kubeconfig %v: %v", kubeconfig, err)
+		}
+	} else if os.Getenv("KUBERNETES_SERVICE_HOST") != "" && os.Getenv("KUBERNETES_SERVICE_PORT") != "" {
+		// Try in-cluster config where multus might be running in a kubernetes pod
+		config, err = rest.InClusterConfig()
+		if err != nil {
+			return nil, logging.Errorf("GetK8sClient: failed to get context for in-cluster kube config: %v", err)
+		}
+	} else {
+		// No kubernetes config; assume we shouldn't talk to Kube at all
+		return nil, nil
+	}
+
+	// Specify that we use gRPC
+	config.AcceptContentTypes = "application/vnd.kubernetes.protobuf,application/json"
+	config.ContentType = "application/vnd.kubernetes.protobuf"
+	// Set the config timeout to one minute.
+	config.Timeout = time.Minute
+
+	// creates the clientset
+	client, err := kubernetes.NewForConfig(config)
+	if err != nil {
+		return nil, err
+	}
+
+	netclient, err := netclient.NewForConfig(config)
+	if err != nil {
+		return nil, err
+	}
+
+	broadcaster := record.NewBroadcaster()
+	broadcaster.StartLogging(klog.Infof)
+	broadcaster.StartRecordingToSink(&v1core.EventSinkImpl{Interface: client.CoreV1().Events("")})
+	recorder := broadcaster.NewRecorder(scheme.Scheme, v1.EventSource{Component: "multus"})
+
+	return &ClientInfo{
+		Client:           client,
+		NetClient:        netclient,
+		EventBroadcaster: broadcaster,
+		EventRecorder:    recorder,
+	}, nil
+}
+
+// GetPodNetwork gets net-attach-def annotation from pod
+func GetPodNetwork(pod *v1.Pod) ([]*types.NetworkSelectionElement, error) {
+	logging.Debugf("GetPodNetwork: %v", pod)
+
+	netAnnot := pod.Annotations[networkAttachmentAnnot]
+	defaultNamespace := pod.ObjectMeta.Namespace
+
+	if len(netAnnot) == 0 {
+		return nil, &NoK8sNetworkError{"no kubernetes network found"}
+	}
+
+	networks, err := parsePodNetworkAnnotation(netAnnot, defaultNamespace)
+	if err != nil {
+		return nil, err
+	}
+	return networks, nil
+}
+
+// GetNetworkDelegates returns delegatenetconf from net-attach-def annotation in pod
+func GetNetworkDelegates(k8sclient *ClientInfo, pod *v1.Pod, networks []*types.NetworkSelectionElement, confdir string, confnamespaceIsolation bool, resourceMap map[string]*types.ResourceInfo) ([]*types.DelegateNetConf, error) {
+	logging.Debugf("GetNetworkDelegates: %v, %v, %v, %v, %v, %v", k8sclient, pod, networks, confdir, confnamespaceIsolation, resourceMap)
+	// Read all network objects referenced by 'networks'
+	var delegates []*types.DelegateNetConf
+	defaultNamespace := pod.ObjectMeta.Namespace
+
+	for _, net := range networks {
+
+		// The pods namespace (stored as defaultNamespace, does not equal the annotation's target namespace in net.Namespace)
+		// In the case that this is a mismatch when namespaceisolation is enabled, this should be an error.
+		if confnamespaceIsolation {
+			if defaultNamespace != net.Namespace {
+				// There is an exception however, we always allow a reference to the default namespace.
+				if net.Namespace != "default" {
+					return nil, logging.Errorf("GetNetworkDelegates: namespace isolation enabled, annotation violates permission, pod is in namespace %v but refers to target namespace %v", defaultNamespace, net.Namespace)
+				}
+			}
+		}
+
+		delegate, updatedResourceMap, err := getKubernetesDelegate(k8sclient, net, confdir, pod, resourceMap)
+		if err != nil {
+			return nil, logging.Errorf("GetNetworkDelegates: failed getting the delegate: %v", err)
+		}
+		delegates = append(delegates, delegate)
+		resourceMap = updatedResourceMap
+	}
+
+	return delegates, nil
+}
+
+func getNetDelegate(client *ClientInfo, pod *v1.Pod, netname, confdir, namespace string, resourceMap map[string]*types.ResourceInfo) (*types.DelegateNetConf, map[string]*types.ResourceInfo, error) {
+	logging.Debugf("getNetDelegate: %v, %v, %v, %s", client, netname, confdir, namespace)
+	// option1) search CRD object for the network
+	net := &types.NetworkSelectionElement{
+		Name:      netname,
+		Namespace: namespace,
+	}
+	delegate, resourceMap, err := getKubernetesDelegate(client, net, confdir, pod, resourceMap)
+	if err == nil {
+		return delegate, resourceMap, nil
+	}
+
+	// option2) search CNI json config file
+	var configBytes []byte
+	configBytes, err = netutils.GetCNIConfigFromFile(netname, confdir)
+	if err == nil {
+		delegate, err := types.LoadDelegateNetConf(configBytes, nil, "")
+		if err != nil {
+			return nil, resourceMap, err
+		}
+		return delegate, resourceMap, nil
+	}
+
+	// option3) search directry
+	fInfo, err := os.Stat(netname)
+	if err == nil {
+		if fInfo.IsDir() {
+			files, err := libcni.ConfFiles(netname, []string{".conf", ".conflist"})
+			if err != nil {
+				return nil, resourceMap, err
+			}
+			if len(files) > 0 {
+				var configBytes []byte
+				configBytes, err = netutils.GetCNIConfigFromFile("", netname)
+				if err == nil {
+					delegate, err := types.LoadDelegateNetConf(configBytes, nil, "")
+					if err != nil {
+						return nil, resourceMap, err
+					}
+					return delegate, resourceMap, nil
+				}
+				return nil, resourceMap, err
+			}
+		}
+	}
+	return nil, resourceMap, logging.Errorf("getNetDelegate: cannot find network: %v", netname)
+}
+
+// GetDefaultNetworks parses 'defaultNetwork' config, gets network json and put it into netconf.Delegates.
+func GetDefaultNetworks(pod *v1.Pod, conf *types.NetConf, kubeClient *ClientInfo, resourceMap map[string]*types.ResourceInfo) (map[string]*types.ResourceInfo, error) {
+	logging.Debugf("GetDefaultNetworks: %v, %v, %v, %v", pod, conf, kubeClient, resourceMap)
+	var delegates []*types.DelegateNetConf
+
+	kubeClient, err := GetK8sClient(conf.Kubeconfig, kubeClient)
+	if err != nil {
+		return resourceMap, err
+	}
+	if kubeClient == nil {
+		if len(conf.Delegates) == 0 {
+			// No available kube client and no delegates, we can't do anything
+			return resourceMap, logging.Errorf("GetDefaultNetworks: must have either Kubernetes config or delegates")
+		}
+		return resourceMap, nil
+	}
+
+	delegate, resourceMap, err := getNetDelegate(kubeClient, pod, conf.ClusterNetwork, conf.ConfDir, conf.MultusNamespace, resourceMap)
+
+	if err != nil {
+		return resourceMap, logging.Errorf("GetDefaultNetworks: failed to get clusterNetwork %s in namespace %s", conf.ClusterNetwork, conf.MultusNamespace)
+	}
+	delegate.MasterPlugin = true
+	delegates = append(delegates, delegate)
+
+	// Pod in kube-system namespace does not have default network for now.
+	if !types.CheckSystemNamespaces(pod.ObjectMeta.Namespace, conf.SystemNamespaces) {
+		for _, netname := range conf.DefaultNetworks {
+			delegate, resourceMap, err := getNetDelegate(kubeClient, pod, netname, conf.ConfDir, conf.MultusNamespace, resourceMap)
+			if err != nil {
+				return resourceMap, err
+			}
+			delegates = append(delegates, delegate)
+		}
+	}
+
+	if err = conf.AddDelegates(delegates); err != nil {
+		return resourceMap, err
+	}
+
+	return resourceMap, nil
+}
+
+// tryLoadK8sPodDefaultNetwork get pod default network from annotations
+func tryLoadK8sPodDefaultNetwork(kubeClient *ClientInfo, pod *v1.Pod, conf *types.NetConf) (*types.DelegateNetConf, error) {
+	var netAnnot string
+	logging.Debugf("tryLoadK8sPodDefaultNetwork: %v, %v, %v", kubeClient, pod, conf)
+
+	netAnnot, ok := pod.Annotations[defaultNetAnnot]
+	if !ok {
+		logging.Debugf("tryLoadK8sPodDefaultNetwork: Pod default network annotation is not defined")
+		return nil, nil
+	}
+
+	// The CRD object of default network should only be defined in multusNamespace
+	networks, err := parsePodNetworkAnnotation(netAnnot, conf.MultusNamespace)
+	if err != nil {
+		return nil, logging.Errorf("tryLoadK8sPodDefaultNetwork: failed to parse CRD object: %v", err)
+	}
+	if len(networks) > 1 {
+		return nil, logging.Errorf("tryLoadK8sPodDefaultNetwork: more than one default network is specified: %s", netAnnot)
+	}
+
+	delegate, _, err := getKubernetesDelegate(kubeClient, networks[0], conf.ConfDir, pod, nil)
+	if err != nil {
+		return nil, logging.Errorf("tryLoadK8sPodDefaultNetwork: failed getting the delegate: %v", err)
+	}
+	delegate.MasterPlugin = true
+
+	return delegate, nil
+}

kubeletclient/kubeletclient.go

@@ -0,0 +1,113 @@
+package kubeletclient
+
+import (
+	"os"
+	"path/filepath"
+	"time"
+
+	"golang.org/x/net/context"
+	"gopkg.in/intel/multus-cni.v3/checkpoint"
+	"gopkg.in/intel/multus-cni.v3/logging"
+	"gopkg.in/intel/multus-cni.v3/types"
+	v1 "k8s.io/api/core/v1"
+	"k8s.io/kubernetes/pkg/kubelet/apis/podresources"
+	podresourcesapi "k8s.io/kubernetes/pkg/kubelet/apis/podresources/v1alpha1"
+	"k8s.io/kubernetes/pkg/kubelet/util"
+)
+
+const (
+	defaultKubeletSocketFile   = "kubelet.sock"
+	defaultPodResourcesMaxSize = 1024 * 1024 * 16 // 16 Mb
+)
+
+var (
+	kubeletSocket           string
+	defaultPodResourcesPath = "/var/lib/kubelet/pod-resources"
+)
+
+// GetResourceClient returns an instance of ResourceClient interface initialized with Pod resource information
+func GetResourceClient() (types.ResourceClient, error) {
+	// If Kubelet resource API endpoint exist use that by default
+	// Or else fallback with checkpoint file
+	if hasKubeletAPIEndpoint() {
+		logging.Debugf("GetResourceClient: using Kubelet resource API endpoint")
+		return getKubeletClient()
+	}
+
+	logging.Debugf("GetResourceClient: using Kubelet device plugin checkpoint")
+	return checkpoint.GetCheckpoint()
+}
+
+func getKubeletClient() (types.ResourceClient, error) {
+	newClient := &kubeletClient{}
+	if kubeletSocket == "" {
+		kubeletSocket = util.LocalEndpoint(defaultPodResourcesPath, podresources.Socket)
+	}
+
+	client, conn, err := podresources.GetClient(kubeletSocket, 10*time.Second, defaultPodResourcesMaxSize)
+	if err != nil {
+		return nil, logging.Errorf("getKubeletClient: error getting grpc client: %v\n", err)
+	}
+	defer conn.Close()
+
+	if err := newClient.getPodResources(client); err != nil {
+		return nil, logging.Errorf("getKubeletClient: error ge tting pod resources from client: %v\n", err)
+	}
+
+	return newClient, nil
+}
+
+type kubeletClient struct {
+	resources []*podresourcesapi.PodResources
+}
+
+func (rc *kubeletClient) getPodResources(client podresourcesapi.PodResourcesListerClient) error {
+
+	ctx, cancel := context.WithTimeout(context.Background(), 10*time.Second)
+	defer cancel()
+
+	resp, err := client.List(ctx, &podresourcesapi.ListPodResourcesRequest{})
+	if err != nil {
+		return logging.Errorf("getPodResources: failed to list pod resources, %v.Get(_) = _, %v", client, err)
+	}
+
+	rc.resources = resp.PodResources
+	return nil
+}
+
+// GetPodResourceMap returns an instance of a map of Pod ResourceInfo given a (Pod name, namespace) tuple
+func (rc *kubeletClient) GetPodResourceMap(pod *v1.Pod) (map[string]*types.ResourceInfo, error) {
+	resourceMap := make(map[string]*types.ResourceInfo)
+
+	name := pod.Name
+	ns := pod.Namespace
+
+	if name == "" || ns == "" {
+		return nil, logging.Errorf("GetPodResourcesMap: Pod name or namespace cannot be empty")
+	}
+
+	for _, pr := range rc.resources {
+		if pr.Name == name && pr.Namespace == ns {
+			for _, cnt := range pr.Containers {
+				for _, dev := range cnt.Devices {
+					if rInfo, ok := resourceMap[dev.ResourceName]; ok {
+						rInfo.DeviceIDs = append(rInfo.DeviceIDs, dev.DeviceIds...)
+					} else {
+						resourceMap[dev.ResourceName] = &types.ResourceInfo{DeviceIDs: dev.DeviceIds}
+					}
+				}
+			}
+		}
+	}
+	return resourceMap, nil
+}
+
+func hasKubeletAPIEndpoint() bool {
+	// Check for kubelet resource API socket file
+	kubeletAPISocket := filepath.Join(defaultPodResourcesPath, defaultKubeletSocketFile)
+	if _, err := os.Stat(kubeletAPISocket); err != nil {
+		logging.Debugf("hasKubeletAPIEndpoint: error looking up kubelet resource api socket file: %q", err)
+		return false
+	}
+	return true
+}

kubeletclient/kubeletclient_test.go

@@ -0,0 +1,250 @@
+package kubeletclient
+
+import (
+	"context"
+	"io/ioutil"
+	"os"
+	"path/filepath"
+	"testing"
+
+	. "github.com/onsi/ginkgo"
+	. "github.com/onsi/gomega"
+	"google.golang.org/grpc"
+	v1 "k8s.io/api/core/v1"
+	metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
+	k8sTypes "k8s.io/apimachinery/pkg/types"
+	"k8s.io/kubernetes/pkg/kubelet/util"
+
+	mtypes "gopkg.in/intel/multus-cni.v3/types"
+	podresourcesapi "k8s.io/kubernetes/pkg/kubelet/apis/podresources/v1alpha1"
+)
+
+var (
+	socketDir  string
+	socketName string
+	fakeServer *fakeResourceServer
+)
+
+type fakeResourceServer struct {
+	server *grpc.Server
+}
+
+func (m *fakeResourceServer) List(ctx context.Context, req *podresourcesapi.ListPodResourcesRequest) (*podresourcesapi.ListPodResourcesResponse, error) {
+	podName := "pod-name"
+	podNamespace := "pod-namespace"
+	containerName := "container-name"
+
+	devs := []*podresourcesapi.ContainerDevices{
+		{
+			ResourceName: "resource",
+			DeviceIds:    []string{"dev0", "dev1"},
+		},
+	}
+
+	resp := &podresourcesapi.ListPodResourcesResponse{
+		PodResources: []*podresourcesapi.PodResources{
+			{
+				Name:      podName,
+				Namespace: podNamespace,
+				Containers: []*podresourcesapi.ContainerResources{
+					{
+						Name:    containerName,
+						Devices: devs,
+					},
+				},
+			},
+		},
+	}
+	return resp, nil
+}
+
+func TestKubeletclient(t *testing.T) {
+	RegisterFailHandler(Fail)
+	RunSpecs(t, "Kubeletclient Suite")
+}
+
+func setUp() error {
+	tempSocketDir, err := ioutil.TempDir("", "kubelet-resource-client")
+	if err != nil {
+		return err
+	}
+	defaultPodResourcesPath = filepath.Join(tempSocketDir, defaultPodResourcesPath)
+
+	if err := os.MkdirAll(defaultPodResourcesPath, os.ModeDir); err != nil {
+		return err
+	}
+
+	socketDir = defaultPodResourcesPath
+	socketName = filepath.Join(socketDir, "kubelet.sock")
+
+	fakeServer = &fakeResourceServer{server: grpc.NewServer()}
+	podresourcesapi.RegisterPodResourcesListerServer(fakeServer.server, fakeServer)
+	lis, err := util.CreateListener(socketName)
+	if err != nil {
+		return nil
+	}
+	go fakeServer.server.Serve(lis)
+	return nil
+}
+
+func tearDown(path string) error {
+	if fakeServer != nil {
+		fakeServer.server.Stop()
+	}
+	if err := os.RemoveAll(path); err != nil {
+		return err
+	}
+	return nil
+}
+
+var _ = BeforeSuite(func() {
+	err := setUp()
+	Expect(err).NotTo(HaveOccurred())
+})
+
+var _ = AfterSuite(func() {
+	err := tearDown(socketDir)
+	Expect(err).NotTo(HaveOccurred())
+})
+
+var _ = Describe("Kubelet resource endpoint data read operations", func() {
+
+	Context("GetResourceClient()", func() {
+		It("should return no error", func() {
+			kubeletSocket = socketName
+			_, err := GetResourceClient()
+			Expect(err).NotTo(HaveOccurred())
+		})
+
+		It("should fail with missing file", func() {
+			kubeletSocket = "sampleSocketString"
+			_, err := GetResourceClient()
+			Expect(err).To(HaveOccurred())
+		})
+	})
+
+	Context("GetPodResourceMap() with valid pod name and namespace", func() {
+		It("should return no error", func() {
+			podUID := k8sTypes.UID("970a395d-bb3b-11e8-89df-408d5c537d23")
+			fakePod := &v1.Pod{
+				ObjectMeta: metav1.ObjectMeta{
+					Name:      "pod-name",
+					Namespace: "pod-namespace",
+					UID:       podUID,
+				},
+				Spec: v1.PodSpec{
+					Containers: []v1.Container{
+						{
+							Name: "container-name",
+						},
+					},
+				},
+			}
+			kubeletSocket = socketName
+			client, err := getKubeletClient()
+			Expect(err).NotTo(HaveOccurred())
+
+			outputRMap := map[string]*mtypes.ResourceInfo{
+				"resource": &mtypes.ResourceInfo{DeviceIDs: []string{"dev0", "dev1"}},
+			}
+			resourceMap, err := client.GetPodResourceMap(fakePod)
+			Expect(err).NotTo(HaveOccurred())
+			Expect(resourceMap).ShouldNot(BeNil())
+			Expect(resourceMap).To(Equal(outputRMap))
+		})
+
+		It("should return no error with empty socket", func() {
+			podUID := k8sTypes.UID("970a395d-bb3b-11e8-89df-408d5c537d23")
+			fakePod := &v1.Pod{
+				ObjectMeta: metav1.ObjectMeta{
+					Name:      "pod-name",
+					Namespace: "pod-namespace",
+					UID:       podUID,
+				},
+				Spec: v1.PodSpec{
+					Containers: []v1.Container{
+						{
+							Name: "container-name",
+						},
+					},
+				},
+			}
+			kubeletSocket = ""
+			client, err := getKubeletClient()
+			Expect(err).NotTo(HaveOccurred())
+
+			outputRMap := map[string]*mtypes.ResourceInfo{
+				"resource": &mtypes.ResourceInfo{DeviceIDs: []string{"dev0", "dev1"}},
+			}
+			resourceMap, err := client.GetPodResourceMap(fakePod)
+			Expect(err).NotTo(HaveOccurred())
+			Expect(resourceMap).ShouldNot(BeNil())
+			Expect(resourceMap).To(Equal(outputRMap))
+		})
+
+		It("should return an error with garbage socket value", func() {
+			kubeletSocket = "/badfilepath!?//"
+			_, err := getKubeletClient()
+			Expect(err).To(HaveOccurred())
+		})
+	})
+
+	Context("GetPodResourceMap() with empty podname", func() {
+		It("should return error", func() {
+			podUID := k8sTypes.UID("970a395d-bb3b-11e8-89df-408d5c537d23")
+			fakePod := &v1.Pod{
+				ObjectMeta: metav1.ObjectMeta{
+					Name:      "",
+					Namespace: "pod-namespace",
+					UID:       podUID,
+				},
+			}
+			kubeletSocket = socketName
+			client, err := getKubeletClient()
+			Expect(err).NotTo(HaveOccurred())
+			_, err = client.GetPodResourceMap(fakePod)
+			Expect(err).To(HaveOccurred())
+		})
+	})
+
+	Context("GetPodResourceMap() with empty namespace", func() {
+		It("should return error", func() {
+			podUID := k8sTypes.UID("970a395d-bb3b-11e8-89df-408d5c537d23")
+			fakePod := &v1.Pod{
+				ObjectMeta: metav1.ObjectMeta{
+					Name:      "pod-name",
+					Namespace: "",
+					UID:       podUID,
+				},
+			}
+			kubeletSocket = socketName
+			client, err := getKubeletClient()
+			Expect(err).NotTo(HaveOccurred())
+			_, err = client.GetPodResourceMap(fakePod)
+			Expect(err).To(HaveOccurred())
+		})
+	})
+
+	Context("GetPodResourceMap() with non-existent podname and namespace", func() {
+		It("should return no error", func() {
+			podUID := k8sTypes.UID("970a395d-bb3b-11e8-89df-408d5c537d23")
+			fakePod := &v1.Pod{
+				ObjectMeta: metav1.ObjectMeta{
+					Name:      "whateverpod",
+					Namespace: "whatevernamespace",
+					UID:       podUID,
+				},
+			}
+
+			kubeletSocket = socketName
+			client, err := getKubeletClient()
+			Expect(err).NotTo(HaveOccurred())
+
+			emptyRMap := map[string]*mtypes.ResourceInfo{}
+			resourceMap, err := client.GetPodResourceMap(fakePod)
+			Expect(err).NotTo(HaveOccurred())
+			Expect(resourceMap).ShouldNot(BeNil())
+			Expect(resourceMap).To(Equal(emptyRMap))
+		})
+	})
+})

logging/logging.go

@@ -0,0 +1,158 @@
+// Copyright (c) 2018 Intel Corporation
+//
+// Licensed under the Apache License, Version 2.0 (the "License");
+// you may not use this file except in compliance with the License.
+// You may obtain a copy of the License at
+//
+//     http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+package logging
+
+import (
+	"fmt"
+	"os"
+	"strings"
+	"time"
+	"io"
+
+	"github.com/pkg/errors"
+	lumberjack "gopkg.in/natefinch/lumberjack.v2"
+)
+
+// Level type
+type Level uint32
+
+// PanicLevel...MaxLevel indicates the logging level
+const (
+	PanicLevel Level = iota
+	ErrorLevel
+	VerboseLevel
+	DebugLevel
+	MaxLevel
+	UnknownLevel
+)
+
+var loggingStderr bool
+var loggingW io.Writer
+var loggingLevel Level
+
+const defaultTimestampFormat = time.RFC3339
+
+func (l Level) String() string {
+	switch l {
+	case PanicLevel:
+		return "panic"
+	case VerboseLevel:
+		return "verbose"
+	case ErrorLevel:
+		return "error"
+	case DebugLevel:
+		return "debug"
+	}
+	return "unknown"
+}
+
+func printf(level Level, format string, a ...interface{}) {
+	header := "%s [%s] "
+	t := time.Now()
+	if level > loggingLevel {
+		return
+	}
+
+	if loggingStderr {
+		fmt.Fprintf(os.Stderr, header, t.Format(defaultTimestampFormat), level)
+		fmt.Fprintf(os.Stderr, format, a...)
+		fmt.Fprintf(os.Stderr, "\n")
+	}
+
+	if loggingW != nil {
+		fmt.Fprintf(loggingW, header, t.Format(defaultTimestampFormat), level)
+		fmt.Fprintf(loggingW, format, a...)
+		fmt.Fprintf(loggingW, "\n")
+	}
+}
+
+// Debugf prints logging if logging level >= debug
+func Debugf(format string, a ...interface{}) {
+	printf(DebugLevel, format, a...)
+}
+
+// Verbosef prints logging if logging level >= verbose
+func Verbosef(format string, a ...interface{}) {
+	printf(VerboseLevel, format, a...)
+}
+
+// Errorf prints logging if logging level >= error
+func Errorf(format string, a ...interface{}) error {
+	printf(ErrorLevel, format, a...)
+	return fmt.Errorf(format, a...)
+}
+
+// Panicf prints logging plus stack trace. This should be used only for unrecoverble error
+func Panicf(format string, a ...interface{}) {
+	printf(PanicLevel, format, a...)
+	printf(PanicLevel, "========= Stack trace output ========")
+	printf(PanicLevel, "%+v", errors.New("Multus Panic"))
+	printf(PanicLevel, "========= Stack trace output end ========")
+}
+
+// GetLoggingLevel gets current logging level
+func GetLoggingLevel() Level {
+	return loggingLevel
+}
+
+func getLoggingLevel(levelStr string) Level {
+	switch strings.ToLower(levelStr) {
+	case "debug":
+		return DebugLevel
+	case "verbose":
+		return VerboseLevel
+	case "error":
+		return ErrorLevel
+	case "panic":
+		return PanicLevel
+	}
+	fmt.Fprintf(os.Stderr, "multus logging: cannot set logging level to %s\n", levelStr)
+	return UnknownLevel
+}
+
+// SetLogLevel sets logging level
+func SetLogLevel(levelStr string) {
+	level := getLoggingLevel(levelStr)
+	if level < MaxLevel {
+		loggingLevel = level
+	}
+}
+
+// SetLogStderr sets flag for logging stderr output
+func SetLogStderr(enable bool) {
+	loggingStderr = enable
+}
+
+// SetLogFile sets logging file
+func SetLogFile(filename string) {
+	if filename == "" {
+		return
+	}
+
+	loggingW = &lumberjack.Logger{
+                        Filename:   filename,
+                        MaxSize:    100, // megabytes
+                        MaxBackups: 5,
+                        MaxAge:     5, // days
+                        Compress:   true,
+                    }
+
+}
+
+func init() {
+	loggingStderr = true
+	loggingW = nil
+	loggingLevel = PanicLevel
+}

logging/logging_test.go

@@ -0,0 +1,82 @@
+// Copyright (c) 2018 Intel Corporation
+//
+// Licensed under the Apache License, Version 2.0 (the "License");
+// you may not use this file except in compliance with the License.
+// You may obtain a copy of the License at
+//
+//     http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+package logging
+
+import (
+	"testing"
+
+	. "github.com/onsi/ginkgo"
+	. "github.com/onsi/gomega"
+)
+
+func TestLogging(t *testing.T) {
+	RegisterFailHandler(Fail)
+	RunSpecs(t, "Logging")
+}
+
+var _ = Describe("logging operations", func() {
+
+	BeforeEach(func() {
+		loggingStderr = false
+		loggingW = nil
+		loggingLevel = PanicLevel
+	})
+
+	It("Check file setter with empty", func() {
+		SetLogFile("")
+		Expect(loggingW).To(BeNil())
+	})
+
+	It("Check file setter with empty", func() {
+		SetLogFile("/tmp/foobar.logging")
+		Expect(loggingW).NotTo(Equal(nil))
+		// check file existance
+	})
+
+	It("Check file setter with bad filepath", func() {
+		SetLogFile("/invalid/filepath")
+		Expect(loggingW).NotTo(Equal(nil))
+		// check file existance
+	})
+
+	It("Check loglevel setter", func() {
+		SetLogLevel("debug")
+		Expect(loggingLevel).To(Equal(DebugLevel))
+		SetLogLevel("Error")
+		Expect(loggingLevel).To(Equal(ErrorLevel))
+		SetLogLevel("VERbose")
+		Expect(loggingLevel).To(Equal(VerboseLevel))
+		SetLogLevel("PANIC")
+		Expect(loggingLevel).To(Equal(PanicLevel))
+	})
+
+	It("Check loglevel setter with invalid level", func() {
+		currentLevel := loggingLevel
+		SetLogLevel("XXXX")
+		Expect(loggingLevel).To(Equal(currentLevel))
+	})
+
+	It("Check log to stderr setter with invalid level", func() {
+		currentVal := loggingStderr
+		SetLogStderr(!currentVal)
+		Expect(loggingStderr).NotTo(Equal(currentVal))
+	})
+
+	// Tests public getter
+	It("Check getter for logging level with current level", func() {
+		currentLevel := loggingLevel
+		Expect(currentLevel).To(Equal(GetLoggingLevel()))
+	})
+})

netutils/netutils.go

@@ -0,0 +1,119 @@
+// Copyright (c) 2019 Multus Authors
+//
+// Licensed under the Apache License, Version 2.0 (the "License");
+// you may not use this file except in compliance with the License.
+// You may obtain a copy of the License at
+//
+//     http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+//
+
+package netutils
+
+import (
+	"github.com/containernetworking/cni/pkg/skel"
+	cnitypes "github.com/containernetworking/cni/pkg/types"
+	"github.com/containernetworking/cni/pkg/types/current"
+	"github.com/containernetworking/plugins/pkg/ns"
+	"github.com/vishvananda/netlink"
+	"gopkg.in/intel/multus-cni.v3/logging"
+	"net"
+	"strings"
+)
+
+// DeleteDefaultGW removes the default gateway from marked interfaces.
+func DeleteDefaultGW(args *skel.CmdArgs, ifName string, res *cnitypes.Result) (*current.Result, error) {
+	result, err := current.NewResultFromResult(*res)
+	if err != nil {
+		return nil, logging.Errorf("DeleteDefaultGW: Error creating new from current CNI result: %v", err)
+	}
+
+	netns, err := ns.GetNS(args.Netns)
+	if err != nil {
+		return nil, logging.Errorf("DeleteDefaultGW: Error getting namespace %v", err)
+	}
+	defer netns.Close()
+
+	err = netns.Do(func(_ ns.NetNS) error {
+		var err error
+		link, _ := netlink.LinkByName(ifName)
+		routes, _ := netlink.RouteList(link, netlink.FAMILY_ALL)
+		for _, nlroute := range routes {
+			if nlroute.Dst == nil {
+				err = netlink.RouteDel(&nlroute)
+			}
+		}
+		return err
+	})
+	var newRoutes []*cnitypes.Route
+	for _, route := range result.Routes {
+		if mask, _ := route.Dst.Mask.Size(); mask != 0 {
+			newRoutes = append(newRoutes, route)
+		}
+	}
+	result.Routes = newRoutes
+	return result, err
+}
+
+// SetDefaultGW adds a default gateway on a specific interface
+func SetDefaultGW(args *skel.CmdArgs, ifName string, gateways []net.IP, res *cnitypes.Result) (*current.Result, error) {
+
+	// Use the current CNI result...
+	result, err := current.NewResultFromResult(*res)
+	if err != nil {
+		return nil, logging.Errorf("SetDefaultGW: Error creating new CNI result from current: %v", err)
+	}
+
+	// This ensures we're acting within the net namespace for the pod.
+	netns, err := ns.GetNS(args.Netns)
+	if err != nil {
+		return nil, logging.Errorf("SetDefaultGW: Error getting namespace %v", err)
+	}
+	defer netns.Close()
+
+	var newResultDefaultRoutes []*cnitypes.Route
+
+	// Do this within the net namespace.
+	err = netns.Do(func(_ ns.NetNS) error {
+		var err error
+
+		// Pick up the link info as we need the index.
+		link, _ := netlink.LinkByName(ifName)
+
+		// Cycle through all the desired gateways.
+		for _, gw := range gateways {
+
+			// Create a new route (note: dst is nil by default)
+			logging.Debugf("SetDefaultGW: Adding default route on %v (index: %v) to %v", ifName, link.Attrs().Index, gw)
+			newDefaultRoute := netlink.Route{
+				LinkIndex: link.Attrs().Index,
+				Gw:        gw,
+			}
+
+			// Build a new element for the results route
+
+			// Set a correct CIDR depending on IP type
+			_, dstipnet, _ := net.ParseCIDR("::0/0")
+			if strings.Count(gw.String(), ":") < 2 {
+				_, dstipnet, _ = net.ParseCIDR("0.0.0.0/0")
+			}
+			newResultDefaultRoutes = append(newResultDefaultRoutes, &cnitypes.Route{Dst: *dstipnet, GW: gw})
+
+			// Perform the creation of the default route....
+			err = netlink.RouteAdd(&newDefaultRoute)
+			if err != nil {
+				logging.Errorf("SetDefaultGW: Error adding route: %v", err)
+			}
+		}
+		return err
+	})
+
+	result.Routes = newResultDefaultRoutes
+	return result, err
+
+}

test.sh

@@ -0,0 +1,23 @@
+#!/usr/bin/env bash
+set -e
+
+# this if... will be removed when gomodules goes default
+if [ "$GO111MODULE" == "off" ]; then
+	echo "Warning: this will be deprecated in near future so please use go modules!"
+
+	ORG_PATH="github.com/intel"
+	REPO_PATH="${ORG_PATH}/multus-cni"
+
+	if [ ! -h gopath/src/${REPO_PATH} ]; then
+		mkdir -p gopath/src/${ORG_PATH}
+		ln -s ../../../.. gopath/src/${REPO_PATH} || exit 255
+	fi
+
+	export GO15VENDOREXPERIMENT=1
+	export GOBIN=${PWD}/bin
+	export GOPATH=${PWD}/gopath
+	bash -c "umask 0; cd ${GOPATH}/src/${REPO_PATH}; PATH=${GOROOT}/bin:$(pwd)/bin:${PATH} go test -v -covermode=count -coverprofile=coverage.out ./..."
+else
+	# test with go modules
+	bash -c "umask 0; go test -v -covermode=count -coverprofile=coverage.out ./..."
+fi

testing/testing.go

@@ -0,0 +1,168 @@
+// Copyright (c) 2017 Intel Corporation
+//
+// Licensed under the Apache License, Version 2.0 (the "License");
+// you may not use this file except in compliance with the License.
+// You may obtain a copy of the License at
+//
+//     http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+//
+
+package testing
+
+import (
+	"encoding/json"
+	"fmt"
+	"io"
+	"io/ioutil"
+	"net"
+	"os"
+	"strings"
+
+	netv1 "github.com/k8snetworkplumbingwg/network-attachment-definition-client/pkg/apis/k8s.cni.cncf.io/v1"
+	v1 "k8s.io/api/core/v1"
+	metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
+
+	"github.com/containernetworking/cni/pkg/types"
+	types020 "github.com/containernetworking/cni/pkg/types/020"
+
+	"github.com/onsi/gomega"
+)
+
+// NewFakeNetAttachDef returns net-attach-def for testing
+func NewFakeNetAttachDef(namespace, name, config string) *netv1.NetworkAttachmentDefinition {
+	return &netv1.NetworkAttachmentDefinition{
+		ObjectMeta: metav1.ObjectMeta{
+			Name:      name,
+			Namespace: namespace,
+		},
+		Spec: netv1.NetworkAttachmentDefinitionSpec{
+			Config: config,
+		},
+	}
+}
+
+// NewFakeNetAttachDefFile returns net-attach-def for testing with conf file
+func NewFakeNetAttachDefFile(namespace, name, filePath, fileData string) *netv1.NetworkAttachmentDefinition {
+	netAttach := &netv1.NetworkAttachmentDefinition{
+		ObjectMeta: metav1.ObjectMeta{
+			Name:      name,
+			Namespace: namespace,
+		},
+	}
+	err := ioutil.WriteFile(filePath, []byte(fileData), 0600)
+	gomega.Expect(err).NotTo(gomega.HaveOccurred())
+	return netAttach
+}
+
+// NewFakeNetAttachDefAnnotation returns net-attach-def with resource annotation
+func NewFakeNetAttachDefAnnotation(namespace, name, config string) *netv1.NetworkAttachmentDefinition {
+	return &netv1.NetworkAttachmentDefinition{
+		ObjectMeta: metav1.ObjectMeta{
+			Name:      name,
+			Namespace: namespace,
+			Annotations: map[string]string{
+				"k8s.v1.cni.cncf.io/resourceName": "intel.com/sriov",
+			},
+		},
+		Spec: netv1.NetworkAttachmentDefinitionSpec{
+			Config: config,
+		},
+	}
+}
+
+// NewFakePod creates fake Pod object
+func NewFakePod(name string, netAnnotation string, defaultNetAnnotation string) *v1.Pod {
+	pod := &v1.Pod{
+		ObjectMeta: metav1.ObjectMeta{
+			Name:      name,
+			Namespace: "test",
+			UID:       "testUID",
+		},
+		Spec: v1.PodSpec{
+			Containers: []v1.Container{
+				{Name: "ctr1", Image: "image"},
+			},
+		},
+	}
+	annotations := make(map[string]string)
+
+	if netAnnotation != "" {
+		netAnnotation = strings.Replace(netAnnotation, "\n", "", -1)
+		netAnnotation = strings.Replace(netAnnotation, "\t", "", -1)
+		annotations["k8s.v1.cni.cncf.io/networks"] = netAnnotation
+	}
+
+	if defaultNetAnnotation != "" {
+		annotations["v1.multus-cni.io/default-network"] = defaultNetAnnotation
+	}
+
+	pod.ObjectMeta.Annotations = annotations
+	return pod
+}
+
+// EnsureCIDR parses/verify CIDR ip string and convert to net.IPNet
+func EnsureCIDR(cidr string) *net.IPNet {
+	ip, net, err := net.ParseCIDR(cidr)
+	gomega.Expect(err).NotTo(gomega.HaveOccurred())
+	net.IP = ip
+	return net
+}
+
+// Result is stub Result for testing
+type Result struct {
+	CNIVersion string             `json:"cniVersion,omitempty"`
+	IP4        *types020.IPConfig `json:"ip4,omitempty"`
+	IP6        *types020.IPConfig `json:"ip6,omitempty"`
+	DNS        types.DNS          `json:"dns,omitempty"`
+}
+
+// Version returns current CNIVersion of the given Result
+func (r *Result) Version() string {
+	return r.CNIVersion
+}
+
+// GetAsVersion returns a Result object given a version
+func (r *Result) GetAsVersion(version string) (types.Result, error) {
+	for _, supportedVersion := range types020.SupportedVersions {
+		if version == supportedVersion {
+			r.CNIVersion = version
+			return r, nil
+		}
+	}
+	return nil, fmt.Errorf("cannot convert version %q to %s", types020.SupportedVersions, version)
+}
+
+// Print prints a Result's information to std out
+func (r *Result) Print() error {
+	return r.PrintTo(os.Stdout)
+}
+
+// PrintTo prints a Result's information to the provided writer
+func (r *Result) PrintTo(writer io.Writer) error {
+	data, err := json.MarshalIndent(r, "", "    ")
+	if err != nil {
+		return err
+	}
+	_, err = writer.Write(data)
+	return err
+}
+
+// String returns a formatted string in the form of "[IP4: $1,][ IP6: $2,] DNS: $3" where
+// $1 represents the receiver's IPv4, $2 represents the receiver's IPv6 and $3 the
+// receiver's DNS. If $1 or $2 are nil, they won't be present in the returned string.
+func (r *Result) String() string {
+	var str string
+	if r.IP4 != nil {
+		str = fmt.Sprintf("IP4:%+v, ", *r.IP4)
+	}
+	if r.IP6 != nil {
+		str += fmt.Sprintf("IP6:%+v, ", *r.IP6)
+	}
+	return fmt.Sprintf("%sDNS:%+v", str, r.DNS)
+}

types/conf.go

@@ -0,0 +1,466 @@
+// Copyright (c) 2017 Intel Corporation
+//
+// Licensed under the Apache License, Version 2.0 (the "License");
+// you may not use this file except in compliance with the License.
+// You may obtain a copy of the License at
+//
+//     http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+//
+
+package types
+
+import (
+	"encoding/json"
+	"fmt"
+	"net"
+
+	"github.com/containernetworking/cni/libcni"
+	"github.com/containernetworking/cni/pkg/skel"
+	"github.com/containernetworking/cni/pkg/types/current"
+	"github.com/containernetworking/cni/pkg/version"
+	"gopkg.in/intel/multus-cni.v3/logging"
+)
+
+const (
+	defaultCNIDir                 = "/var/lib/cni/multus"
+	defaultConfDir                = "/etc/cni/multus/net.d"
+	defaultBinDir                 = "/opt/cni/bin"
+	defaultReadinessIndicatorFile = ""
+	defaultMultusNamespace        = "kube-system"
+)
+
+// LoadDelegateNetConfList reads DelegateNetConf from bytes
+func LoadDelegateNetConfList(bytes []byte, delegateConf *DelegateNetConf) error {
+	logging.Debugf("LoadDelegateNetConfList: %s, %v", string(bytes), delegateConf)
+
+	if err := json.Unmarshal(bytes, &delegateConf.ConfList); err != nil {
+		return logging.Errorf("LoadDelegateNetConfList: error unmarshalling delegate conflist: %v", err)
+	}
+
+	if delegateConf.ConfList.Plugins == nil {
+		return logging.Errorf("LoadDelegateNetConfList: delegate must have the 'type' or 'plugin' field")
+	}
+
+	if delegateConf.ConfList.Plugins[0].Type == "" {
+		return logging.Errorf("LoadDelegateNetConfList: a plugin delegate must have the 'type' field")
+	}
+	delegateConf.ConfListPlugin = true
+	return nil
+}
+
+// LoadDelegateNetConf converts raw CNI JSON into a DelegateNetConf structure
+func LoadDelegateNetConf(bytes []byte, net *NetworkSelectionElement, deviceID string) (*DelegateNetConf, error) {
+	var err error
+	logging.Debugf("LoadDelegateNetConf: %s, %v, %s", string(bytes), net, deviceID)
+
+	delegateConf := &DelegateNetConf{}
+	if err := json.Unmarshal(bytes, &delegateConf.Conf); err != nil {
+		return nil, logging.Errorf("LoadDelegateNetConf: error unmarshalling delegate config: %v", err)
+	}
+
+	// Do some minimal validation
+	if delegateConf.Conf.Type == "" {
+		if err := LoadDelegateNetConfList(bytes, delegateConf); err != nil {
+			return nil, logging.Errorf("LoadDelegateNetConf: failed with: %v", err)
+		}
+		if deviceID != "" {
+			bytes, err = addDeviceIDInConfList(bytes, deviceID)
+			if err != nil {
+				return nil, logging.Errorf("LoadDelegateNetConf: failed to add deviceID in NetConfList bytes: %v", err)
+			}
+		}
+		if net != nil && net.CNIArgs != nil {
+			bytes, err = addCNIArgsInConfList(bytes, net.CNIArgs)
+			if err != nil {
+				return nil, logging.Errorf("LoadDelegateNetConf(): failed to add cni-args in NetConfList bytes: %v", err)
+			}
+		}
+	} else {
+		if deviceID != "" {
+			bytes, err = delegateAddDeviceID(bytes, deviceID)
+			if err != nil {
+				return nil, logging.Errorf("LoadDelegateNetConf: failed to add deviceID in NetConf bytes: %v", err)
+			}
+		}
+		if net != nil && net.CNIArgs != nil {
+			bytes, err = addCNIArgsInConfig(bytes, net.CNIArgs)
+			if err != nil {
+				return nil, logging.Errorf("LoadDelegateNetConf(): failed to add cni-args in NetConfList bytes: %v", err)
+			}
+		}
+	}
+
+	if net != nil {
+		if net.Name != "" {
+			// Overwrite CNI config name with net-attach-def name
+			delegateConf.Name = fmt.Sprintf("%s/%s", net.Namespace, net.Name)
+		}
+		if net.InterfaceRequest != "" {
+			delegateConf.IfnameRequest = net.InterfaceRequest
+		}
+		if net.MacRequest != "" {
+			delegateConf.MacRequest = net.MacRequest
+		}
+		if net.IPRequest != nil {
+			delegateConf.IPRequest = net.IPRequest
+		}
+		if net.BandwidthRequest != nil {
+			delegateConf.BandwidthRequest = net.BandwidthRequest
+		}
+		if net.PortMappingsRequest != nil {
+			delegateConf.PortMappingsRequest = net.PortMappingsRequest
+		}
+		if net.GatewayRequest != nil {
+			delegateConf.GatewayRequest = append(delegateConf.GatewayRequest, net.GatewayRequest...)
+		}
+		if net.InfinibandGUIDRequest != "" {
+			delegateConf.InfinibandGUIDRequest = net.InfinibandGUIDRequest
+		}
+	}
+
+	delegateConf.Bytes = bytes
+
+	return delegateConf, nil
+}
+
+// MergeCNIRuntimeConfig creates CNI runtimeconfig from delegate
+func MergeCNIRuntimeConfig(runtimeConfig *RuntimeConfig, delegate *DelegateNetConf) *RuntimeConfig {
+	logging.Debugf("MergeCNIRuntimeConfig: %v %v", runtimeConfig, delegate)
+	if runtimeConfig == nil {
+		runtimeConfig = &RuntimeConfig{}
+	}
+
+	// multus inject RuntimeConfig only in case of non MasterPlugin.
+	if delegate.MasterPlugin != true {
+		logging.Debugf("MergeCNIRuntimeConfig: add runtimeConfig for net-attach-def: %v", runtimeConfig)
+		if delegate.PortMappingsRequest != nil {
+			runtimeConfig.PortMaps = delegate.PortMappingsRequest
+		}
+		if delegate.BandwidthRequest != nil {
+			runtimeConfig.Bandwidth = delegate.BandwidthRequest
+		}
+		if delegate.IPRequest != nil {
+			runtimeConfig.IPs = delegate.IPRequest
+		}
+		if delegate.MacRequest != "" {
+			runtimeConfig.Mac = delegate.MacRequest
+		}
+		if delegate.InfinibandGUIDRequest != "" {
+			runtimeConfig.InfinibandGUID = delegate.InfinibandGUIDRequest
+		}
+	}
+
+	return runtimeConfig
+}
+
+// CreateCNIRuntimeConf create CNI RuntimeConf
+func CreateCNIRuntimeConf(args *skel.CmdArgs, k8sArgs *K8sArgs, ifName string, rc *RuntimeConfig) *libcni.RuntimeConf {
+	logging.Debugf("LoadCNIRuntimeConf: %v, %v, %s, %v", args, k8sArgs, ifName, rc)
+
+	// In part, adapted from K8s pkg/kubelet/dockershim/network/cni/cni.go#buildCNIRuntimeConf
+	// Todo
+	// ingress, egress and bandwidth capability features as same as kubelet.
+	rt := &libcni.RuntimeConf{
+		ContainerID: args.ContainerID,
+		NetNS:       args.Netns,
+		IfName:      ifName,
+		// NOTE: Verbose logging depends on this order, so please keep Args order.
+		Args: [][2]string{
+			{"IgnoreUnknown", string("true")},
+			{"K8S_POD_NAMESPACE", string(k8sArgs.K8S_POD_NAMESPACE)},
+			{"K8S_POD_NAME", string(k8sArgs.K8S_POD_NAME)},
+			{"K8S_POD_INFRA_CONTAINER_ID", string(k8sArgs.K8S_POD_INFRA_CONTAINER_ID)},
+		},
+	}
+
+	if rc != nil {
+		capabilityArgs := map[string]interface{}{}
+		if len(rc.PortMaps) != 0 {
+			capabilityArgs["portMappings"] = rc.PortMaps
+		}
+		if rc.Bandwidth != nil {
+			capabilityArgs["bandwidth"] = rc.Bandwidth
+		}
+		if len(rc.IPs) != 0 {
+			capabilityArgs["ips"] = rc.IPs
+		}
+		if len(rc.Mac) != 0 {
+			capabilityArgs["mac"] = rc.Mac
+		}
+		if len(rc.InfinibandGUID) != 0 {
+			capabilityArgs["infinibandGUID"] = rc.InfinibandGUID
+		}
+		rt.CapabilityArgs = capabilityArgs
+	}
+	return rt
+}
+
+// GetGatewayFromResult retrieves gateway IP addresses from CNI result
+func GetGatewayFromResult(result *current.Result) []net.IP {
+	var gateways []net.IP
+
+	for _, route := range result.Routes {
+		if mask, _ := route.Dst.Mask.Size(); mask == 0 {
+			gateways = append(gateways, route.GW)
+		}
+	}
+	return gateways
+}
+
+// LoadNetConf converts inputs (i.e. stdin) to NetConf
+func LoadNetConf(bytes []byte) (*NetConf, error) {
+	netconf := &NetConf{}
+
+	logging.Debugf("LoadNetConf: %s", string(bytes))
+	if err := json.Unmarshal(bytes, netconf); err != nil {
+		return nil, logging.Errorf("LoadNetConf: failed to load netconf: %v", err)
+	}
+
+	// Logging
+	if netconf.LogFile != "" {
+		logging.SetLogFile(netconf.LogFile)
+	}
+	if netconf.LogLevel != "" {
+		logging.SetLogLevel(netconf.LogLevel)
+	}
+
+	// Parse previous result
+	if netconf.RawPrevResult != nil {
+		resultBytes, err := json.Marshal(netconf.RawPrevResult)
+		if err != nil {
+			return nil, logging.Errorf("LoadNetConf: could not serialize prevResult: %v", err)
+		}
+		res, err := version.NewResult(netconf.CNIVersion, resultBytes)
+		if err != nil {
+			return nil, logging.Errorf("LoadNetConf: could not parse prevResult: %v", err)
+		}
+		netconf.RawPrevResult = nil
+		netconf.PrevResult, err = current.NewResultFromResult(res)
+		if err != nil {
+			return nil, logging.Errorf("LoadNetConf: could not convert result to current version: %v", err)
+		}
+	}
+
+	// Delegates must always be set. If no kubeconfig is present, the
+	// delegates are executed in-order.  If a kubeconfig is present,
+	// at least one delegate must be present and the first delegate is
+	// the master plugin. Kubernetes CRD delegates are then appended to
+	// the existing delegate list and all delegates executed in-order.
+
+	if len(netconf.RawDelegates) == 0 && netconf.ClusterNetwork == "" {
+		return nil, logging.Errorf("LoadNetConf: at least one delegate/defaultNetwork must be specified")
+	}
+
+	if netconf.CNIDir == "" {
+		netconf.CNIDir = defaultCNIDir
+	}
+
+	if netconf.ConfDir == "" {
+		netconf.ConfDir = defaultConfDir
+	}
+
+	if netconf.BinDir == "" {
+		netconf.BinDir = defaultBinDir
+	}
+
+	if netconf.ReadinessIndicatorFile == "" {
+		netconf.ReadinessIndicatorFile = defaultReadinessIndicatorFile
+	}
+
+	if len(netconf.SystemNamespaces) == 0 {
+		netconf.SystemNamespaces = []string{"kube-system"}
+	}
+
+	if netconf.MultusNamespace == "" {
+		netconf.MultusNamespace = defaultMultusNamespace
+	}
+
+	// get RawDelegates and put delegates field
+	if netconf.ClusterNetwork == "" {
+		// for Delegates
+		if len(netconf.RawDelegates) == 0 {
+			return nil, logging.Errorf("LoadNetConf: at least one delegate must be specified")
+		}
+		for idx, rawConf := range netconf.RawDelegates {
+			bytes, err := json.Marshal(rawConf)
+			if err != nil {
+				return nil, logging.Errorf("LoadNetConf: error marshalling delegate %d config: %v", idx, err)
+			}
+			delegateConf, err := LoadDelegateNetConf(bytes, nil, "")
+			if err != nil {
+				return nil, logging.Errorf("LoadNetConf: failed to load delegate %d config: %v", idx, err)
+			}
+			netconf.Delegates = append(netconf.Delegates, delegateConf)
+		}
+		netconf.RawDelegates = nil
+
+		// First delegate is always the master plugin
+		netconf.Delegates[0].MasterPlugin = true
+	}
+
+	return netconf, nil
+}
+
+// AddDelegates appends the new delegates to the delegates list
+func (n *NetConf) AddDelegates(newDelegates []*DelegateNetConf) error {
+	logging.Debugf("AddDelegates: %v", newDelegates)
+	n.Delegates = append(n.Delegates, newDelegates...)
+	return nil
+}
+
+// delegateAddDeviceID injects deviceID information in delegate bytes
+func delegateAddDeviceID(inBytes []byte, deviceID string) ([]byte, error) {
+	var rawConfig map[string]interface{}
+	var err error
+
+	err = json.Unmarshal(inBytes, &rawConfig)
+	if err != nil {
+		return nil, logging.Errorf("delegateAddDeviceID: failed to unmarshal inBytes: %v", err)
+	}
+	// Inject deviceID
+	rawConfig["deviceID"] = deviceID
+	rawConfig["pciBusID"] = deviceID
+	configBytes, err := json.Marshal(rawConfig)
+	if err != nil {
+		return nil, logging.Errorf("delegateAddDeviceID: failed to re-marshal Spec.Config: %v", err)
+	}
+	logging.Debugf("delegateAddDeviceID updated configBytes %s", string(configBytes))
+	return configBytes, nil
+}
+
+// addDeviceIDInConfList injects deviceID information in delegate bytes
+func addDeviceIDInConfList(inBytes []byte, deviceID string) ([]byte, error) {
+	var rawConfig map[string]interface{}
+	var err error
+
+	err = json.Unmarshal(inBytes, &rawConfig)
+	if err != nil {
+		return nil, logging.Errorf("addDeviceIDInConfList: failed to unmarshal inBytes: %v", err)
+	}
+
+	pList, ok := rawConfig["plugins"]
+	if !ok {
+		return nil, logging.Errorf("addDeviceIDInConfList: unable to get plugin list")
+	}
+
+	pMap, ok := pList.([]interface{})
+	if !ok {
+		return nil, logging.Errorf("addDeviceIDInConfList: unable to typecast plugin list")
+	}
+
+	for idx, plugin := range pMap {
+		currentPlugin, ok := plugin.(map[string]interface{})
+		if !ok {
+			return nil, logging.Errorf("addDeviceIDInConfList: unable to typecast plugin #%d", idx)
+		}
+		// Inject deviceID
+		currentPlugin["deviceID"] = deviceID
+		currentPlugin["pciBusID"] = deviceID
+	}
+
+	configBytes, err := json.Marshal(rawConfig)
+	if err != nil {
+		return nil, logging.Errorf("addDeviceIDInConfList: failed to re-marshal: %v", err)
+	}
+	logging.Debugf("addDeviceIDInConfList: updated configBytes %s", string(configBytes))
+	return configBytes, nil
+}
+
+// injectCNIArgs injects given args to cniConfig
+func injectCNIArgs(cniConfig *map[string]interface{}, args *map[string]interface{}) error {
+	if argsval, ok := (*cniConfig)["args"]; ok {
+		argsvalmap := argsval.(map[string]interface{})
+		if cnival, ok := argsvalmap["cni"]; ok {
+			cnivalmap := cnival.(map[string]interface{})
+			// merge it if conf has args
+			for key, val := range *args {
+				cnivalmap[key] = val
+			}
+		} else {
+			argsvalmap["cni"] = *args
+		}
+	} else {
+		argsval := map[string]interface{}{}
+		argsval["cni"] = *args
+		(*cniConfig)["args"] = argsval
+	}
+	return nil
+}
+
+// addCNIArgsInConfig injects given cniArgs to CNI config in inBytes
+func addCNIArgsInConfig(inBytes []byte, cniArgs *map[string]interface{}) ([]byte, error) {
+	var rawConfig map[string]interface{}
+	var err error
+
+	err = json.Unmarshal(inBytes, &rawConfig)
+	if err != nil {
+		return nil, logging.Errorf("addCNIArgsInConfig(): failed to unmarshal inBytes: %v", err)
+	}
+
+	injectCNIArgs(&rawConfig, cniArgs)
+
+	configBytes, err := json.Marshal(rawConfig)
+	if err != nil {
+		return nil, logging.Errorf("addCNIArgsInConfig(): failed to re-marshal: %v", err)
+	}
+	return configBytes, nil
+}
+
+// addCNIArgsInConfList injects given cniArgs to CNI conflist in inBytes
+func addCNIArgsInConfList(inBytes []byte, cniArgs *map[string]interface{}) ([]byte, error) {
+	var rawConfig map[string]interface{}
+	var err error
+
+	err = json.Unmarshal(inBytes, &rawConfig)
+	if err != nil {
+		return nil, logging.Errorf("addCNIArgsInConfList(): failed to unmarshal inBytes: %v", err)
+	}
+
+	pList, ok := rawConfig["plugins"]
+	if !ok {
+		return nil, logging.Errorf("addCNIArgsInConfList(): unable to get plugin list")
+	}
+
+	pMap, ok := pList.([]interface{})
+	if !ok {
+		return nil, logging.Errorf("addCNIArgsInConfList(): unable to typecast plugin list")
+	}
+
+	for idx := range pMap {
+		valMap := pMap[idx].(map[string]interface{})
+		injectCNIArgs(&valMap, cniArgs)
+	}
+
+	configBytes, err := json.Marshal(rawConfig)
+	if err != nil {
+		return nil, logging.Errorf("addCNIArgsInConfList(): failed to re-marshal: %v", err)
+	}
+	return configBytes, nil
+}
+
+// CheckGatewayConfig check gatewayRequest and mark IsFilterGateway flag if
+// gw filtering is required
+func CheckGatewayConfig(delegates []*DelegateNetConf) {
+	// Check the Gateway
+	for i, delegate := range delegates {
+		if delegate.GatewayRequest == nil {
+			delegates[i].IsFilterGateway = true
+		}
+	}
+}
+
+// CheckSystemNamespaces checks whether given namespace is in systemNamespaces or not.
+func CheckSystemNamespaces(namespace string, systemNamespaces []string) bool {
+	for _, nsname := range systemNamespaces {
+		if namespace == nsname {
+			return true
+		}
+	}
+	return false
+}

types/conf_test.go

@@ -0,0 +1,765 @@
+// Copyright (c) 2017 Intel Corporation
+//
+// Licensed under the Apache License, Version 2.0 (the "License");
+// you may not use this file except in compliance with the License.
+// You may obtain a copy of the License at
+//
+//     http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+//
+
+package types
+
+import (
+	"encoding/json"
+	"fmt"
+	"io/ioutil"
+	"os"
+	"testing"
+
+	"github.com/containernetworking/cni/pkg/skel"
+	types020 "github.com/containernetworking/cni/pkg/types/020"
+	"github.com/containernetworking/plugins/pkg/ns"
+	"github.com/containernetworking/plugins/pkg/testutils"
+	netutils "github.com/k8snetworkplumbingwg/network-attachment-definition-client/pkg/utils"
+	testhelpers "gopkg.in/intel/multus-cni.v3/testing"
+
+	. "github.com/onsi/ginkgo"
+	. "github.com/onsi/gomega"
+)
+
+func TestConf(t *testing.T) {
+	RegisterFailHandler(Fail)
+	RunSpecs(t, "conf")
+}
+
+var _ = Describe("config operations", func() {
+	var testNS ns.NetNS
+	var tmpDir string
+
+	BeforeEach(func() {
+		// Create a new NetNS so we don't modify the host
+		var err error
+		testNS, err = testutils.NewNS()
+		Expect(err).NotTo(HaveOccurred())
+		os.Setenv("CNI_NETNS", testNS.Path())
+		os.Setenv("CNI_PATH", "/some/path")
+
+		tmpDir, err = ioutil.TempDir("", "multus_tmp")
+		Expect(err).NotTo(HaveOccurred())
+	})
+
+	AfterEach(func() {
+		Expect(testNS.Close()).To(Succeed())
+		os.Unsetenv("CNI_PATH")
+		os.Unsetenv("CNI_ARGS")
+		err := os.RemoveAll(tmpDir)
+		Expect(err).NotTo(HaveOccurred())
+	})
+
+	It("parses a valid multus configuration", func() {
+		conf := `{
+    "name": "node-cni-network",
+    "type": "multus",
+    "kubeconfig": "/etc/kubernetes/node-kubeconfig.yaml",
+    "delegates": [{
+        "type": "weave-net"
+    }],
+	"runtimeConfig": {
+      "portMappings": [
+        {"hostPort": 8080, "containerPort": 80, "protocol": "tcp"}
+      ]
+    }
+
+}`
+		netConf, err := LoadNetConf([]byte(conf))
+		Expect(err).NotTo(HaveOccurred())
+		Expect(len(netConf.Delegates)).To(Equal(1))
+		Expect(netConf.Delegates[0].Conf.Type).To(Equal("weave-net"))
+		Expect(netConf.Delegates[0].MasterPlugin).To(BeTrue())
+		Expect(len(netConf.RuntimeConfig.PortMaps)).To(Equal(1))
+	})
+
+	It("fails to load invalid multus configuration (bad json)", func() {
+		conf := `{
+	  "name": "node-cni-network",
+	  "type": "multus",
+	  "kubeconfig": "/etc/kubernetes/node-kubeconfig.yaml",
+	  "delegates": [{
+	      "type": "weave-net"
+	  }],
+	"runtimeConfig": {
+	    "portMappings": [
+	      {"hostPort": 8080, "containerPort": 80, "protocol": "tcp"}
+	    ]
+		}`
+		// Error in conf json: missing end bracket
+
+		_, err := LoadNetConf([]byte(conf))
+		Expect(err).To(HaveOccurred())
+		_, err = LoadDelegateNetConf([]byte(conf), nil, "")
+		Expect(err).To(HaveOccurred())
+		err = LoadDelegateNetConfList([]byte(conf), &DelegateNetConf{})
+		Expect(err).To(HaveOccurred())
+		_, err = addDeviceIDInConfList([]byte(conf), "")
+		Expect(err).To(HaveOccurred())
+		_, err = delegateAddDeviceID([]byte(conf), "")
+		Expect(err).To(HaveOccurred())
+	})
+
+	It("checks if logFile and logLevel are set correctly", func() {
+		conf := `{
+	    "name": "node-cni-network",
+			"type": "multus",
+			"logLevel": "debug",
+			"logFile": "/var/log/multus.log",
+	    "kubeconfig": "/etc/kubernetes/node-kubeconfig.yaml",
+	    "delegates": [{
+	        "type": "weave-net"
+	    }],
+		"runtimeConfig": {
+	      "portMappings": [
+	        {"hostPort": 8080, "containerPort": 80, "protocol": "tcp"}
+	      ]
+	    }
+
+	}`
+		netConf, err := LoadNetConf([]byte(conf))
+		Expect(err).NotTo(HaveOccurred())
+		Expect(netConf.LogLevel).To(Equal("debug"))
+		Expect(netConf.LogFile).To(Equal("/var/log/multus.log"))
+	})
+
+	It("prevResult with no errors", func() {
+		conf := `{
+	    "name": "node-cni-network",
+			"type": "multus",
+	    "kubeconfig": "/etc/kubernetes/node-kubeconfig.yaml",
+			"prevResult": {
+				"ips": [
+					{
+						"version": "4",
+						"address": "10.0.0.5/32",
+						"interface": 2
+					}
+			]},
+			"delegates": [{
+	        "type": "weave-net"
+			}],
+		"runtimeConfig": {
+	      "portMappings": [
+	        {"hostPort": 8080, "containerPort": 80, "protocol": "tcp"}
+	      ]
+	    }
+
+	}`
+		_, err := LoadNetConf([]byte(conf))
+		Expect(err).NotTo(HaveOccurred())
+	})
+
+	It("succeeds if only delegates are set", func() {
+		conf := `{
+    "name": "node-cni-network",
+    "type": "multus",
+    "delegates": [{
+        "type": "weave-net"
+    },{
+        "type": "foobar"
+    }]
+}`
+		netConf, err := LoadNetConf([]byte(conf))
+		Expect(err).NotTo(HaveOccurred())
+		Expect(len(netConf.Delegates)).To(Equal(2))
+		Expect(netConf.Delegates[0].Conf.Type).To(Equal("weave-net"))
+		Expect(netConf.Delegates[0].MasterPlugin).To(BeTrue())
+		Expect(netConf.Delegates[1].Conf.Type).To(Equal("foobar"))
+		Expect(netConf.Delegates[1].MasterPlugin).To(BeFalse())
+	})
+
+	It("fails if no kubeconfig or delegates are set", func() {
+		conf := `{
+    "name": "node-cni-network",
+    "type": "multus"
+}`
+		_, err := LoadNetConf([]byte(conf))
+		Expect(err).To(HaveOccurred())
+	})
+
+	It("fails if kubeconfig is present but no delegates are set", func() {
+		conf := `{
+    "name": "node-cni-network",
+    "type": "multus",
+    "kubeconfig": "/etc/kubernetes/node-kubeconfig.yaml"
+}`
+		_, err := LoadNetConf([]byte(conf))
+		Expect(err).To(HaveOccurred())
+	})
+
+	It("fails when delegate field exists but fields are named incorrectly", func() {
+		conf := `{
+	"name": "node-cni-network",
+		"type": "multus",
+    "kubeconfig": "/etc/kubernetes/node-kubeconfig.yaml",
+		"prevResult": {
+			"ips": [
+				{
+					"version": "4",
+					"address": "10.0.0.5/32",
+					"interface": 2
+				}
+			]
+		},
+		"delegates": [{
+			"_not_type": "weave-net"
+		}],
+	"runtimeConfig": {
+		"portMappings": [
+			{"hostPort": 8080, "containerPort": 80, "protocol": "tcp"}
+	    ]
+	}
+}`
+		_, err := LoadNetConf([]byte(conf))
+		Expect(err).To(HaveOccurred())
+	})
+
+	It("has defaults set for network readiness", func() {
+		conf := `{
+    "name": "defaultnetwork",
+    "type": "multus",
+    "kubeconfig": "/etc/kubernetes/kubelet.conf",
+    "delegates": [{
+      "cniVersion": "0.3.0",
+      "name": "defaultnetwork",
+      "type": "flannel",
+      "isDefaultGateway": true
+    }]
+}`
+		netConf, err := LoadNetConf([]byte(conf))
+		Expect(err).NotTo(HaveOccurred())
+		Expect(netConf.ReadinessIndicatorFile).To(Equal(""))
+	})
+
+	It("honors overrides for network readiness", func() {
+		conf := `{
+    "name": "defaultnetwork",
+    "type": "multus",
+    "readinessindicatorfile": "/etc/cni/net.d/foo",
+    "kubeconfig": "/etc/kubernetes/kubelet.conf",
+    "delegates": [{
+      "cniVersion": "0.3.0",
+      "name": "defaultnetwork",
+      "type": "flannel",
+      "isDefaultGateway": true
+    }]
+}`
+		netConf, err := LoadNetConf([]byte(conf))
+		Expect(err).NotTo(HaveOccurred())
+		Expect(netConf.ReadinessIndicatorFile).To(Equal("/etc/cni/net.d/foo"))
+	})
+
+	It("check CheckSystemNamespaces() works fine", func() {
+		b1 := CheckSystemNamespaces("foobar", []string{"barfoo", "bafoo", "foobar"})
+		Expect(b1).To(Equal(true))
+		b2 := CheckSystemNamespaces("foobar1", []string{"barfoo", "bafoo", "foobar"})
+		Expect(b2).To(Equal(false))
+	})
+
+	It("assigns deviceID in delegated conf", func() {
+		conf := `{
+    "name": "second-network",
+    "type": "sriov"
+}`
+		type sriovNetConf struct {
+			Name     string `json:"name"`
+			Type     string `json:"type"`
+			DeviceID string `json:"deviceID"`
+		}
+		sriovConf := &sriovNetConf{}
+		delegateNetConf, err := LoadDelegateNetConf([]byte(conf), nil, "0000:00:00.0")
+		Expect(err).NotTo(HaveOccurred())
+
+		err = json.Unmarshal(delegateNetConf.Bytes, &sriovConf)
+		Expect(err).NotTo(HaveOccurred())
+		Expect(sriovConf.DeviceID).To(Equal("0000:00:00.0"))
+	})
+
+	It("assigns deviceID in delegated conf list", func() {
+		conf := `{
+    "name": "second-network",
+    "plugins": [
+      {
+        "type": "sriov"
+      }
+    ]
+}`
+		type sriovNetConf struct {
+			Name     string `json:"name"`
+			Type     string `json:"type"`
+			DeviceID string `json:"deviceID"`
+		}
+		type sriovNetConfList struct {
+			Plugins []*sriovNetConf `json:"plugins"`
+		}
+		sriovConfList := &sriovNetConfList{}
+		delegateNetConf, err := LoadDelegateNetConf([]byte(conf), nil, "0000:00:00.1")
+		Expect(err).NotTo(HaveOccurred())
+
+		err = json.Unmarshal(delegateNetConf.Bytes, &sriovConfList)
+		Expect(err).NotTo(HaveOccurred())
+		Expect(sriovConfList.Plugins[0].DeviceID).To(Equal("0000:00:00.1"))
+	})
+
+	It("assigns deviceID in delegated conf list multiple plugins", func() {
+		conf := `{
+    "name": "second-network",
+    "plugins": [
+      {
+        "type": "sriov"
+      },
+      {
+        "type": "other-cni"
+      }
+    ]
+}`
+		type sriovNetConf struct {
+			Name     string `json:"name"`
+			Type     string `json:"type"`
+			DeviceID string `json:"deviceID"`
+		}
+		type sriovNetConfList struct {
+			Plugins []*sriovNetConf `json:"plugins"`
+		}
+		sriovConfList := &sriovNetConfList{}
+		delegateNetConf, err := LoadDelegateNetConf([]byte(conf), nil, "0000:00:00.1")
+		Expect(err).NotTo(HaveOccurred())
+
+		err = json.Unmarshal(delegateNetConf.Bytes, &sriovConfList)
+		Expect(err).NotTo(HaveOccurred())
+		for _, plugin := range sriovConfList.Plugins {
+			Expect(plugin.DeviceID).To(Equal("0000:00:00.1"))
+		}
+	})
+
+	It("assigns pciBusID in delegated conf", func() {
+		conf := `{
+    "name": "second-network",
+    "type": "host-device"
+}`
+		type hostDeviceNetConf struct {
+			Name     string `json:"name"`
+			Type     string `json:"type"`
+			PCIBusID string `json:"pciBusID"`
+		}
+		hostDeviceConf := &hostDeviceNetConf{}
+		delegateNetConf, err := LoadDelegateNetConf([]byte(conf), nil, "0000:00:00.2")
+		Expect(err).NotTo(HaveOccurred())
+
+		err = json.Unmarshal(delegateNetConf.Bytes, &hostDeviceConf)
+		Expect(err).NotTo(HaveOccurred())
+		Expect(hostDeviceConf.PCIBusID).To(Equal("0000:00:00.2"))
+	})
+
+	It("assigns pciBusID in delegated conf list", func() {
+		conf := `{
+    "name": "second-network",
+    "plugins": [
+      {
+        "type": "host-device"
+      }
+    ]
+}`
+		type hostDeviceNetConf struct {
+			Name     string `json:"name"`
+			Type     string `json:"type"`
+			PCIBusID string `json:"pciBusID"`
+		}
+		type hostDeviceNetConfList struct {
+			Plugins []*hostDeviceNetConf `json:"plugins"`
+		}
+		hostDeviceConfList := &hostDeviceNetConfList{}
+		delegateNetConf, err := LoadDelegateNetConf([]byte(conf), nil, "0000:00:00.3")
+		Expect(err).NotTo(HaveOccurred())
+
+		err = json.Unmarshal(delegateNetConf.Bytes, &hostDeviceConfList)
+		Expect(err).NotTo(HaveOccurred())
+		Expect(hostDeviceConfList.Plugins[0].PCIBusID).To(Equal("0000:00:00.3"))
+	})
+
+	It("assigns pciBusID in delegated conf list multiple plugins", func() {
+		conf := `{
+    "name": "second-network",
+    "plugins": [
+      {
+        "type": "host-device"
+      },
+      {
+        "type": "other-cni"
+      }
+    ]
+}`
+		type hostDeviceNetConf struct {
+			Name     string `json:"name"`
+			Type     string `json:"type"`
+			PCIBusID string `json:"pciBusID"`
+		}
+		type hostDeviceNetConfList struct {
+			Plugins []*hostDeviceNetConf `json:"plugins"`
+		}
+		hostDeviceConfList := &hostDeviceNetConfList{}
+		delegateNetConf, err := LoadDelegateNetConf([]byte(conf), nil, "0000:00:00.3")
+		Expect(err).NotTo(HaveOccurred())
+
+		err = json.Unmarshal(delegateNetConf.Bytes, &hostDeviceConfList)
+		Expect(err).NotTo(HaveOccurred())
+		for _, plugin := range hostDeviceConfList.Plugins {
+			Expect(plugin.PCIBusID).To(Equal("0000:00:00.3"))
+		}
+	})
+
+	It("add cni-args in config", func() {
+		var args map[string]interface{}
+		conf := `{
+    "name": "second-network",
+    "type": "bridge"
+}`
+		cniArgs := `{
+    "args1": "val1"
+}`
+		type bridgeNetConf struct {
+			Name string `json:"name"`
+			Type string `json:"type"`
+			Args struct {
+				CNI map[string]string `json:"cni"`
+			} `json:"args"`
+		}
+
+		err := json.Unmarshal([]byte(cniArgs), &args)
+		Expect(err).NotTo(HaveOccurred())
+		net := &NetworkSelectionElement{
+			Name:    "test-elem",
+			CNIArgs: &args,
+		}
+		delegateNetConf, err := LoadDelegateNetConf([]byte(conf), net, "")
+		Expect(err).NotTo(HaveOccurred())
+		bridgeConf := &bridgeNetConf{}
+		err = json.Unmarshal(delegateNetConf.Bytes, bridgeConf)
+		Expect(bridgeConf.Args.CNI["args1"]).To(Equal("val1"))
+	})
+
+	It("add cni-args in config which has cni args already (merge case)", func() {
+		var args map[string]interface{}
+		conf := `{
+    "name": "second-network",
+    "type": "bridge",
+    "args": {
+       "cni": {
+         "args0": "val0",
+         "args1": "val1"
+       }
+    }
+}`
+		cniArgs := `{
+    "args1": "val1a"
+}`
+		type bridgeNetConf struct {
+			Name string `json:"name"`
+			Type string `json:"type"`
+			Args struct {
+				CNI map[string]string `json:"cni"`
+			} `json:"args"`
+		}
+
+		err := json.Unmarshal([]byte(cniArgs), &args)
+		Expect(err).NotTo(HaveOccurred())
+		net := &NetworkSelectionElement{
+			Name:    "test-elem",
+			CNIArgs: &args,
+		}
+		delegateNetConf, err := LoadDelegateNetConf([]byte(conf), net, "")
+		Expect(err).NotTo(HaveOccurred())
+		bridgeConf := &bridgeNetConf{}
+		err = json.Unmarshal(delegateNetConf.Bytes, bridgeConf)
+		Expect(bridgeConf.Args.CNI["args0"]).To(Equal("val0"))
+		Expect(bridgeConf.Args.CNI["args1"]).To(Equal("val1a"))
+	})
+
+	It("add cni-args in conflist", func() {
+		var args map[string]interface{}
+		conf := `{
+    "name": "second-network",
+    "plugins": [
+      {
+        "type": "bridge"
+      }
+    ]
+}`
+		cniArgs := `{
+    "args1": "val1"
+}`
+		type bridgeNetConf struct {
+			Type string `json:"type"`
+			Args struct {
+				CNI map[string]string `json:"cni"`
+			} `json:"args"`
+		}
+		type bridgeNetConfList struct {
+			Name    string           `json:"name"`
+			Plugins []*bridgeNetConf `json:"plugins"`
+		}
+
+		err := json.Unmarshal([]byte(cniArgs), &args)
+		Expect(err).NotTo(HaveOccurred())
+		net := &NetworkSelectionElement{
+			Name:    "test-elem",
+			CNIArgs: &args,
+		}
+		delegateNetConf, err := LoadDelegateNetConf([]byte(conf), net, "")
+		Expect(err).NotTo(HaveOccurred())
+		bridgeConflist := &bridgeNetConfList{}
+		err = json.Unmarshal(delegateNetConf.Bytes, bridgeConflist)
+		Expect(bridgeConflist.Plugins[0].Args.CNI["args1"]).To(Equal("val1"))
+	})
+
+	It("creates a valid CNI runtime config", func() {
+		args := &skel.CmdArgs{
+			ContainerID: "123456789",
+			Netns:       testNS.Path(),
+			IfName:      "eth0",
+			StdinData: []byte(`{
+    "name": "node-cni-network",
+    "type": "multus",
+    "defaultnetworkfile": "/tmp/foo.multus.conf",
+    "defaultnetworkwaitseconds": 3,
+    "delegates": [{
+        "name": "weave1",
+        "cniVersion": "0.2.0",
+        "type": "weave-net"
+    },{
+        "name": "other1",
+        "cniVersion": "0.2.0",
+        "type": "other-plugin"
+    }]
+}`),
+		}
+
+		k8sArgs := &K8sArgs{K8S_POD_NAME: "dummy", K8S_POD_NAMESPACE: "namespacedummy", K8S_POD_INFRA_CONTAINER_ID: "123456789"}
+
+		rc := &RuntimeConfig{}
+		rc.PortMaps = make([]*PortMapEntry, 2)
+
+		rc.PortMaps[0] = &PortMapEntry{
+			HostPort:      0,
+			ContainerPort: 1,
+			Protocol:      "sampleProtocol",
+			HostIP:        "sampleHostIP",
+		}
+		rc.PortMaps[1] = &PortMapEntry{
+			HostPort:      1,
+			ContainerPort: 2,
+			Protocol:      "anotherSampleProtocol",
+			HostIP:        "anotherSampleHostIP",
+		}
+
+		rt := CreateCNIRuntimeConf(args, k8sArgs, "", rc)
+		fmt.Println("rt.ContainerID: ", rt.ContainerID)
+		Expect(rt.ContainerID).To(Equal("123456789"))
+		Expect(rt.NetNS).To(Equal(args.Netns))
+		Expect(rt.IfName).To(Equal(""))
+		Expect(rt.CapabilityArgs["portMappings"]).To(Equal(rc.PortMaps))
+	})
+
+	It("can loadnetworkstatus", func() {
+		result := &types020.Result{
+			CNIVersion: "0.2.0",
+			IP4: &types020.IPConfig{
+				IP: *testhelpers.EnsureCIDR("1.1.1.2/24"),
+			},
+		}
+
+		conf := `{
+    "name": "node-cni-network",
+    "type": "multus",
+    "kubeconfig": "/etc/kubernetes/node-kubeconfig.yaml",
+    "delegates": [{
+        "type": "weave-net"
+    }],
+  "runtimeConfig": {
+      "portMappings": [
+        {"hostPort": 8080, "containerPort": 80, "protocol": "tcp"}
+      ]
+    }
+}`
+
+		delegate, err := LoadDelegateNetConf([]byte(conf), nil, "0000:00:00.0")
+		Expect(err).NotTo(HaveOccurred())
+
+		delegateNetStatus, err := netutils.CreateNetworkStatus(result, delegate.Conf.Name, delegate.MasterPlugin)
+
+		GinkgoT().Logf("delegateNetStatus %+v\n", delegateNetStatus)
+
+		Expect(err).NotTo(HaveOccurred())
+	})
+
+	It("cannot loadnetworkstatus given incompatible CNIVersion", func() {
+
+		result := &testhelpers.Result{
+			CNIVersion: "1.2.3",
+			IP4: &types020.IPConfig{
+				IP: *testhelpers.EnsureCIDR("1.1.1.2/24"),
+			},
+		}
+
+		conf := `{
+    "name": "node-cni-network",
+    "type": "multus",
+    "kubeconfig": "/etc/kubernetes/node-kubeconfig.yaml",
+    "delegates": [{
+        "type": "weave-net"
+    }],
+  "runtimeConfig": {
+      "portMappings": [
+        {"hostPort": 8080, "containerPort": 80, "protocol": "tcp"}
+      ]
+    }
+}`
+
+		delegate, err := LoadDelegateNetConf([]byte(conf), nil, "0000:00:00.0")
+		Expect(err).NotTo(HaveOccurred())
+		fmt.Println("result.Version: ", result.Version())
+		delegateNetStatus, err := netutils.CreateNetworkStatus(result, delegate.Conf.Name, delegate.MasterPlugin)
+
+		GinkgoT().Logf("delegateNetStatus %+v\n", delegateNetStatus)
+
+		Expect(err).To(HaveOccurred())
+	})
+
+	It("verify the network selection elements goes into delegateconf", func() {
+		cniConfig := `{
+        "name": "weave1",
+        "cniVersion": "0.2.0",
+        "type": "weave-net"
+    }`
+		bandwidthEntry1 := &BandwidthEntry{
+			IngressRate:  100,
+			IngressBurst: 200,
+			EgressRate:   100,
+			EgressBurst:  200,
+		}
+
+		portMapEntry1 := &PortMapEntry{
+			HostPort:      8080,
+			ContainerPort: 80,
+			Protocol:      "tcp",
+			HostIP:        "10.0.0.1",
+		}
+
+		networkSelection := &NetworkSelectionElement{
+			Name:                  "testname",
+			InterfaceRequest:      "testIF1",
+			MacRequest:            "c2:11:22:33:44:66",
+			InfinibandGUIDRequest: "24:8a:07:03:00:8d:ae:2e",
+			IPRequest:             []string{"10.0.0.1/24"},
+			BandwidthRequest:      bandwidthEntry1,
+			PortMappingsRequest:   []*PortMapEntry{portMapEntry1},
+		}
+
+		delegateConf, err := LoadDelegateNetConf([]byte(cniConfig), networkSelection, "")
+		Expect(err).NotTo(HaveOccurred())
+		Expect(delegateConf.IfnameRequest).To(Equal(networkSelection.InterfaceRequest))
+		Expect(delegateConf.MacRequest).To(Equal(networkSelection.MacRequest))
+		Expect(delegateConf.InfinibandGUIDRequest).To(Equal(networkSelection.InfinibandGUIDRequest))
+		Expect(delegateConf.IPRequest).To(Equal(networkSelection.IPRequest))
+		Expect(delegateConf.BandwidthRequest).To(Equal(networkSelection.BandwidthRequest))
+		Expect(delegateConf.PortMappingsRequest).To(Equal(networkSelection.PortMappingsRequest))
+	})
+
+	It("test MergeCNIRuntimeConfig with masterPlugin", func() {
+		conf := `{
+			"name": "node-cni-network",
+			"type": "multus",
+			"kubeconfig": "/etc/kubernetes/node-kubeconfig.yaml",
+			"delegates": [{
+				"type": "weave-net"
+			}],
+		  "runtimeConfig": {
+			  "portMappings": [
+				{"hostPort": 8080, "containerPort": 80, "protocol": "tcp"}
+			  ]
+			}
+		}`
+		bandwidthEntry1 := &BandwidthEntry{
+			IngressRate:  100,
+			IngressBurst: 200,
+			EgressRate:   100,
+			EgressBurst:  200,
+		}
+		portMapEntry1 := &PortMapEntry{
+			HostPort:      8080,
+			ContainerPort: 80,
+			Protocol:      "tcp",
+			HostIP:        "10.0.0.1",
+		}
+
+		networkSelection := &NetworkSelectionElement{
+			Name:                  "testname",
+			InterfaceRequest:      "testIF1",
+			MacRequest:            "c2:11:22:33:44:66",
+			InfinibandGUIDRequest: "24:8a:07:03:00:8d:ae:2e",
+			IPRequest:             []string{"10.0.0.1/24"},
+			BandwidthRequest:      bandwidthEntry1,
+			PortMappingsRequest:   []*PortMapEntry{portMapEntry1},
+		}
+		delegate, err := LoadDelegateNetConf([]byte(conf), networkSelection, "")
+		delegate.MasterPlugin = true
+		Expect(err).NotTo(HaveOccurred())
+		runtimeConf := MergeCNIRuntimeConfig(&RuntimeConfig{}, delegate)
+		Expect(runtimeConf.PortMaps).To(BeNil())
+		Expect(runtimeConf.Bandwidth).To(BeNil())
+		Expect(runtimeConf.InfinibandGUID).To(Equal(""))
+	})
+
+	It("test MergeCNIRuntimeConfig with delegate plugin", func() {
+		conf := `{
+			"name": "weave1",
+			"cniVersion": "0.2.0",
+			"type": "weave-net"
+		}`
+		bandwidthEntry1 := &BandwidthEntry{
+			IngressRate:  100,
+			IngressBurst: 200,
+			EgressRate:   100,
+			EgressBurst:  200,
+		}
+		portMapEntry1 := &PortMapEntry{
+			HostPort:      8080,
+			ContainerPort: 80,
+			Protocol:      "tcp",
+			HostIP:        "10.0.0.1",
+		}
+
+		networkSelection := &NetworkSelectionElement{
+			Name:                  "testname",
+			InterfaceRequest:      "testIF1",
+			MacRequest:            "c2:11:22:33:44:66",
+			InfinibandGUIDRequest: "24:8a:07:03:00:8d:ae:2e",
+			IPRequest:             []string{"10.0.0.1/24"},
+			BandwidthRequest:      bandwidthEntry1,
+			PortMappingsRequest:   []*PortMapEntry{portMapEntry1},
+		}
+		delegate, err := LoadDelegateNetConf([]byte(conf), networkSelection, "")
+		Expect(err).NotTo(HaveOccurred())
+		runtimeConf := MergeCNIRuntimeConfig(&RuntimeConfig{}, delegate)
+		Expect(runtimeConf.PortMaps).NotTo(BeNil())
+		Expect(len(runtimeConf.PortMaps)).To(BeEquivalentTo(1))
+		Expect(runtimeConf.PortMaps[0]).To(Equal(portMapEntry1))
+		Expect(runtimeConf.Bandwidth).To(Equal(bandwidthEntry1))
+		Expect(len(runtimeConf.IPs)).To(BeEquivalentTo(1))
+		Expect(runtimeConf.Mac).To(Equal("c2:11:22:33:44:66"))
+		Expect(runtimeConf.InfinibandGUID).To(Equal("24:8a:07:03:00:8d:ae:2e"))
+	})
+})

types/types.go

@@ -0,0 +1,171 @@
+// Copyright (c) 2017 Intel Corporation
+//
+// Licensed under the Apache License, Version 2.0 (the "License");
+// you may not use this file except in compliance with the License.
+// You may obtain a copy of the License at
+//
+//     http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+//
+
+package types
+
+import (
+	"net"
+
+	"github.com/containernetworking/cni/pkg/types"
+	"github.com/containernetworking/cni/pkg/types/current"
+	v1 "k8s.io/api/core/v1"
+)
+
+// NetConf for cni config file written in json
+type NetConf struct {
+	types.NetConf
+
+	// support chaining for master interface and IP decisions
+	// occurring prior to running ipvlan plugin
+	RawPrevResult *map[string]interface{} `json:"prevResult"`
+	PrevResult    *current.Result         `json:"-"`
+
+	ConfDir string `json:"confDir"`
+	CNIDir  string `json:"cniDir"`
+	BinDir  string `json:"binDir"`
+	// RawDelegates is private to the NetConf class; use Delegates instead
+	RawDelegates    []map[string]interface{} `json:"delegates"`
+	Delegates       []*DelegateNetConf       `json:"-"`
+	NetStatus       []*NetworkStatus         `json:"-"`
+	Kubeconfig      string                   `json:"kubeconfig"`
+	ClusterNetwork  string                   `json:"clusterNetwork"`
+	DefaultNetworks []string                 `json:"defaultNetworks"`
+	LogFile         string                   `json:"logFile"`
+	LogLevel        string                   `json:"logLevel"`
+	RuntimeConfig   *RuntimeConfig           `json:"runtimeConfig,omitempty"`
+	// Default network readiness options
+	ReadinessIndicatorFile string `json:"readinessindicatorfile"`
+	// Option to isolate the usage of CR's to the namespace in which a pod resides.
+	NamespaceIsolation bool `json:"namespaceIsolation"`
+	// Option to set system namespaces (to avoid to add defaultNetworks)
+	SystemNamespaces []string `json:"systemNamespaces"`
+	// Option to set the namespace that multus-cni uses (clusterNetwork/defaultNetworks)
+	MultusNamespace string `json:"multusNamespace"`
+}
+
+// RuntimeConfig specifies CNI RuntimeConfig
+type RuntimeConfig struct {
+	PortMaps       []*PortMapEntry `json:"portMappings,omitempty"`
+	Bandwidth      *BandwidthEntry `json:"bandwidth,omitempty"`
+	IPs            []string        `json:"ips,omitempty"`
+	Mac            string          `json:"mac,omitempty"`
+	InfinibandGUID string          `json:"infinibandGUID,omitempty"`
+}
+
+// PortMapEntry for CNI PortMapEntry
+type PortMapEntry struct {
+	HostPort      int    `json:"hostPort"`
+	ContainerPort int    `json:"containerPort"`
+	Protocol      string `json:"protocol,omitempty"`
+	HostIP        string `json:"hostIP,omitempty"`
+}
+
+// BandwidthEntry for CNI BandwidthEntry
+type BandwidthEntry struct {
+	IngressRate  int `json:"ingressRate"`
+	IngressBurst int `json:"ingressBurst"`
+
+	EgressRate  int `json:"egressRate"`
+	EgressBurst int `json:"egressBurst"`
+}
+
+// NetworkStatus is for network status annotation for pod
+type NetworkStatus struct {
+	Name      string    `json:"name"`
+	Interface string    `json:"interface,omitempty"`
+	IPs       []string  `json:"ips,omitempty"`
+	Mac       string    `json:"mac,omitempty"`
+	DNS       types.DNS `json:"dns,omitempty"`
+	Gateway   []net.IP  `json:"default-route,omitempty"`
+}
+
+// DelegateNetConf for net-attach-def for pod
+type DelegateNetConf struct {
+	Conf                  types.NetConf
+	ConfList              types.NetConfList
+	Name                  string
+	IfnameRequest         string          `json:"ifnameRequest,omitempty"`
+	MacRequest            string          `json:"macRequest,omitempty"`
+	InfinibandGUIDRequest string          `json:"infinibandGUIDRequest,omitempty"`
+	IPRequest             []string        `json:"ipRequest,omitempty"`
+	PortMappingsRequest   []*PortMapEntry `json:"-"`
+	BandwidthRequest      *BandwidthEntry `json:"-"`
+	GatewayRequest        []net.IP        `json:"default-route,omitempty"`
+	IsFilterGateway       bool
+	// MasterPlugin is only used internal housekeeping
+	MasterPlugin bool `json:"-"`
+	// Conflist plugin is only used internal housekeeping
+	ConfListPlugin bool `json:"-"`
+
+	// Raw JSON
+	Bytes []byte
+}
+
+// NetworkSelectionElement represents one element of the JSON format
+// Network Attachment Selection Annotation as described in section 4.1.2
+// of the CRD specification.
+type NetworkSelectionElement struct {
+	// Name contains the name of the Network object this element selects
+	Name string `json:"name"`
+	// Namespace contains the optional namespace that the network referenced
+	// by Name exists in
+	Namespace string `json:"namespace,omitempty"`
+	// IPRequest contains an optional requested IP address for this network
+	// attachment
+	IPRequest []string `json:"ips,omitempty"`
+	// MacRequest contains an optional requested MAC address for this
+	// network attachment
+	MacRequest string `json:"mac,omitempty"`
+	// InfinibandGUID request contains an optional requested Infiniband GUID address
+	// for this network attachment
+	InfinibandGUIDRequest string `json:"infiniband-guid,omitempty"`
+	// InterfaceRequest contains an optional requested name for the
+	// network interface this attachment will create in the container
+	InterfaceRequest string `json:"interface,omitempty"`
+	// DeprecatedInterfaceRequest is obsolated parameter at pre 3.2.
+	// This will be removed in 4.0 release.
+	DeprecatedInterfaceRequest string `json:"interfaceRequest,omitempty"`
+	// PortMappingsRequest contains an optional requested port mapping
+	// for the network
+	PortMappingsRequest []*PortMapEntry `json:"portMappings,omitempty"`
+	// BandwidthRequest contains an optional requested bandwidth for
+	// the network
+	BandwidthRequest *BandwidthEntry `json:"bandwidth,omitempty"`
+	// CNIArgs contains additional CNI arguments for the network interface
+	CNIArgs *map[string]interface{} `json:"cni-args"`
+	// GatewayRequest contains default route IP address for the pod
+	GatewayRequest []net.IP `json:"default-route,omitempty"`
+}
+
+// K8sArgs is the valid CNI_ARGS used for Kubernetes
+type K8sArgs struct {
+	types.CommonArgs
+	IP                         net.IP
+	K8S_POD_NAME               types.UnmarshallableString
+	K8S_POD_NAMESPACE          types.UnmarshallableString
+	K8S_POD_INFRA_CONTAINER_ID types.UnmarshallableString
+}
+
+// ResourceInfo is struct to hold Pod device allocation information
+type ResourceInfo struct {
+	Index     int
+	DeviceIDs []string
+}
+
+// ResourceClient provides a kubelet Pod resource handle
+type ResourceClient interface {
+	// GetPodResourceMap returns an instance of a map of Pod ResourceInfo given a (Pod name, namespace) tuple
+	GetPodResourceMap(*v1.Pod) (map[string]*ResourceInfo, error)
+}

vendor/github.com/Microsoft/go-winio/.gitignore

@@ -0,0 +1 @@
+*.exe

vendor/github.com/Microsoft/go-winio/LICENSE

@@ -0,0 +1,22 @@
+The MIT License (MIT)
+
+Copyright (c) 2015 Microsoft
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.
+

vendor/github.com/Microsoft/go-winio/README.md

@@ -0,0 +1,22 @@
+# go-winio
+
+This repository contains utilities for efficiently performing Win32 IO operations in
+Go. Currently, this is focused on accessing named pipes and other file handles, and
+for using named pipes as a net transport.
+
+This code relies on IO completion ports to avoid blocking IO on system threads, allowing Go
+to reuse the thread to schedule another goroutine. This limits support to Windows Vista and
+newer operating systems. This is similar to the implementation of network sockets in Go's net
+package.
+
+Please see the LICENSE file for licensing information.
+
+This project has adopted the [Microsoft Open Source Code of
+Conduct](https://opensource.microsoft.com/codeofconduct/). For more information
+see the [Code of Conduct
+FAQ](https://opensource.microsoft.com/codeofconduct/faq/) or contact
+[opencode@microsoft.com](mailto:opencode@microsoft.com) with any additional
+questions or comments.
+
+Thanks to natefinch for the inspiration for this library. See https://github.com/natefinch/npipe
+for another named pipe implementation.

vendor/github.com/Microsoft/go-winio/backup.go

@@ -0,0 +1,280 @@
+// +build windows
+
+package winio
+
+import (
+	"encoding/binary"
+	"errors"
+	"fmt"
+	"io"
+	"io/ioutil"
+	"os"
+	"runtime"
+	"syscall"
+	"unicode/utf16"
+)
+
+//sys backupRead(h syscall.Handle, b []byte, bytesRead *uint32, abort bool, processSecurity bool, context *uintptr) (err error) = BackupRead
+//sys backupWrite(h syscall.Handle, b []byte, bytesWritten *uint32, abort bool, processSecurity bool, context *uintptr) (err error) = BackupWrite
+
+const (
+	BackupData = uint32(iota + 1)
+	BackupEaData
+	BackupSecurity
+	BackupAlternateData
+	BackupLink
+	BackupPropertyData
+	BackupObjectId
+	BackupReparseData
+	BackupSparseBlock
+	BackupTxfsData
+)
+
+const (
+	StreamSparseAttributes = uint32(8)
+)
+
+const (
+	WRITE_DAC              = 0x40000
+	WRITE_OWNER            = 0x80000
+	ACCESS_SYSTEM_SECURITY = 0x1000000
+)
+
+// BackupHeader represents a backup stream of a file.
+type BackupHeader struct {
+	Id         uint32 // The backup stream ID
+	Attributes uint32 // Stream attributes
+	Size       int64  // The size of the stream in bytes
+	Name       string // The name of the stream (for BackupAlternateData only).
+	Offset     int64  // The offset of the stream in the file (for BackupSparseBlock only).
+}
+
+type win32StreamId struct {
+	StreamId   uint32
+	Attributes uint32
+	Size       uint64
+	NameSize   uint32
+}
+
+// BackupStreamReader reads from a stream produced by the BackupRead Win32 API and produces a series
+// of BackupHeader values.
+type BackupStreamReader struct {
+	r         io.Reader
+	bytesLeft int64
+}
+
+// NewBackupStreamReader produces a BackupStreamReader from any io.Reader.
+func NewBackupStreamReader(r io.Reader) *BackupStreamReader {
+	return &BackupStreamReader{r, 0}
+}
+
+// Next returns the next backup stream and prepares for calls to Read(). It skips the remainder of the current stream if
+// it was not completely read.
+func (r *BackupStreamReader) Next() (*BackupHeader, error) {
+	if r.bytesLeft > 0 {
+		if s, ok := r.r.(io.Seeker); ok {
+			// Make sure Seek on io.SeekCurrent sometimes succeeds
+			// before trying the actual seek.
+			if _, err := s.Seek(0, io.SeekCurrent); err == nil {
+				if _, err = s.Seek(r.bytesLeft, io.SeekCurrent); err != nil {
+					return nil, err
+				}
+				r.bytesLeft = 0
+			}
+		}
+		if _, err := io.Copy(ioutil.Discard, r); err != nil {
+			return nil, err
+		}
+	}
+	var wsi win32StreamId
+	if err := binary.Read(r.r, binary.LittleEndian, &wsi); err != nil {
+		return nil, err
+	}
+	hdr := &BackupHeader{
+		Id:         wsi.StreamId,
+		Attributes: wsi.Attributes,
+		Size:       int64(wsi.Size),
+	}
+	if wsi.NameSize != 0 {
+		name := make([]uint16, int(wsi.NameSize/2))
+		if err := binary.Read(r.r, binary.LittleEndian, name); err != nil {
+			return nil, err
+		}
+		hdr.Name = syscall.UTF16ToString(name)
+	}
+	if wsi.StreamId == BackupSparseBlock {
+		if err := binary.Read(r.r, binary.LittleEndian, &hdr.Offset); err != nil {
+			return nil, err
+		}
+		hdr.Size -= 8
+	}
+	r.bytesLeft = hdr.Size
+	return hdr, nil
+}
+
+// Read reads from the current backup stream.
+func (r *BackupStreamReader) Read(b []byte) (int, error) {
+	if r.bytesLeft == 0 {
+		return 0, io.EOF
+	}
+	if int64(len(b)) > r.bytesLeft {
+		b = b[:r.bytesLeft]
+	}
+	n, err := r.r.Read(b)
+	r.bytesLeft -= int64(n)
+	if err == io.EOF {
+		err = io.ErrUnexpectedEOF
+	} else if r.bytesLeft == 0 && err == nil {
+		err = io.EOF
+	}
+	return n, err
+}
+
+// BackupStreamWriter writes a stream compatible with the BackupWrite Win32 API.
+type BackupStreamWriter struct {
+	w         io.Writer
+	bytesLeft int64
+}
+
+// NewBackupStreamWriter produces a BackupStreamWriter on top of an io.Writer.
+func NewBackupStreamWriter(w io.Writer) *BackupStreamWriter {
+	return &BackupStreamWriter{w, 0}
+}
+
+// WriteHeader writes the next backup stream header and prepares for calls to Write().
+func (w *BackupStreamWriter) WriteHeader(hdr *BackupHeader) error {
+	if w.bytesLeft != 0 {
+		return fmt.Errorf("missing %d bytes", w.bytesLeft)
+	}
+	name := utf16.Encode([]rune(hdr.Name))
+	wsi := win32StreamId{
+		StreamId:   hdr.Id,
+		Attributes: hdr.Attributes,
+		Size:       uint64(hdr.Size),
+		NameSize:   uint32(len(name) * 2),
+	}
+	if hdr.Id == BackupSparseBlock {
+		// Include space for the int64 block offset
+		wsi.Size += 8
+	}
+	if err := binary.Write(w.w, binary.LittleEndian, &wsi); err != nil {
+		return err
+	}
+	if len(name) != 0 {
+		if err := binary.Write(w.w, binary.LittleEndian, name); err != nil {
+			return err
+		}
+	}
+	if hdr.Id == BackupSparseBlock {
+		if err := binary.Write(w.w, binary.LittleEndian, hdr.Offset); err != nil {
+			return err
+		}
+	}
+	w.bytesLeft = hdr.Size
+	return nil
+}
+
+// Write writes to the current backup stream.
+func (w *BackupStreamWriter) Write(b []byte) (int, error) {
+	if w.bytesLeft < int64(len(b)) {
+		return 0, fmt.Errorf("too many bytes by %d", int64(len(b))-w.bytesLeft)
+	}
+	n, err := w.w.Write(b)
+	w.bytesLeft -= int64(n)
+	return n, err
+}
+
+// BackupFileReader provides an io.ReadCloser interface on top of the BackupRead Win32 API.
+type BackupFileReader struct {
+	f               *os.File
+	includeSecurity bool
+	ctx             uintptr
+}
+
+// NewBackupFileReader returns a new BackupFileReader from a file handle. If includeSecurity is true,
+// Read will attempt to read the security descriptor of the file.
+func NewBackupFileReader(f *os.File, includeSecurity bool) *BackupFileReader {
+	r := &BackupFileReader{f, includeSecurity, 0}
+	return r
+}
+
+// Read reads a backup stream from the file by calling the Win32 API BackupRead().
+func (r *BackupFileReader) Read(b []byte) (int, error) {
+	var bytesRead uint32
+	err := backupRead(syscall.Handle(r.f.Fd()), b, &bytesRead, false, r.includeSecurity, &r.ctx)
+	if err != nil {
+		return 0, &os.PathError{"BackupRead", r.f.Name(), err}
+	}
+	runtime.KeepAlive(r.f)
+	if bytesRead == 0 {
+		return 0, io.EOF
+	}
+	return int(bytesRead), nil
+}
+
+// Close frees Win32 resources associated with the BackupFileReader. It does not close
+// the underlying file.
+func (r *BackupFileReader) Close() error {
+	if r.ctx != 0 {
+		backupRead(syscall.Handle(r.f.Fd()), nil, nil, true, false, &r.ctx)
+		runtime.KeepAlive(r.f)
+		r.ctx = 0
+	}
+	return nil
+}
+
+// BackupFileWriter provides an io.WriteCloser interface on top of the BackupWrite Win32 API.
+type BackupFileWriter struct {
+	f               *os.File
+	includeSecurity bool
+	ctx             uintptr
+}
+
+// NewBackupFileWriter returns a new BackupFileWriter from a file handle. If includeSecurity is true,
+// Write() will attempt to restore the security descriptor from the stream.
+func NewBackupFileWriter(f *os.File, includeSecurity bool) *BackupFileWriter {
+	w := &BackupFileWriter{f, includeSecurity, 0}
+	return w
+}
+
+// Write restores a portion of the file using the provided backup stream.
+func (w *BackupFileWriter) Write(b []byte) (int, error) {
+	var bytesWritten uint32
+	err := backupWrite(syscall.Handle(w.f.Fd()), b, &bytesWritten, false, w.includeSecurity, &w.ctx)
+	if err != nil {
+		return 0, &os.PathError{"BackupWrite", w.f.Name(), err}
+	}
+	runtime.KeepAlive(w.f)
+	if int(bytesWritten) != len(b) {
+		return int(bytesWritten), errors.New("not all bytes could be written")
+	}
+	return len(b), nil
+}
+
+// Close frees Win32 resources associated with the BackupFileWriter. It does not
+// close the underlying file.
+func (w *BackupFileWriter) Close() error {
+	if w.ctx != 0 {
+		backupWrite(syscall.Handle(w.f.Fd()), nil, nil, true, false, &w.ctx)
+		runtime.KeepAlive(w.f)
+		w.ctx = 0
+	}
+	return nil
+}
+
+// OpenForBackup opens a file or directory, potentially skipping access checks if the backup
+// or restore privileges have been acquired.
+//
+// If the file opened was a directory, it cannot be used with Readdir().
+func OpenForBackup(path string, access uint32, share uint32, createmode uint32) (*os.File, error) {
+	winPath, err := syscall.UTF16FromString(path)
+	if err != nil {
+		return nil, err
+	}
+	h, err := syscall.CreateFile(&winPath[0], access, share, nil, createmode, syscall.FILE_FLAG_BACKUP_SEMANTICS|syscall.FILE_FLAG_OPEN_REPARSE_POINT, 0)
+	if err != nil {
+		err = &os.PathError{Op: "open", Path: path, Err: err}
+		return nil, err
+	}
+	return os.NewFile(uintptr(h), path), nil
+}

vendor/github.com/Microsoft/go-winio/ea.go

@@ -0,0 +1,137 @@
+package winio
+
+import (
+	"bytes"
+	"encoding/binary"
+	"errors"
+)
+
+type fileFullEaInformation struct {
+	NextEntryOffset uint32
+	Flags           uint8
+	NameLength      uint8
+	ValueLength     uint16
+}
+
+var (
+	fileFullEaInformationSize = binary.Size(&fileFullEaInformation{})
+
+	errInvalidEaBuffer = errors.New("invalid extended attribute buffer")
+	errEaNameTooLarge  = errors.New("extended attribute name too large")
+	errEaValueTooLarge = errors.New("extended attribute value too large")
+)
+
+// ExtendedAttribute represents a single Windows EA.
+type ExtendedAttribute struct {
+	Name  string
+	Value []byte
+	Flags uint8
+}
+
+func parseEa(b []byte) (ea ExtendedAttribute, nb []byte, err error) {
+	var info fileFullEaInformation
+	err = binary.Read(bytes.NewReader(b), binary.LittleEndian, &info)
+	if err != nil {
+		err = errInvalidEaBuffer
+		return
+	}
+
+	nameOffset := fileFullEaInformationSize
+	nameLen := int(info.NameLength)
+	valueOffset := nameOffset + int(info.NameLength) + 1
+	valueLen := int(info.ValueLength)
+	nextOffset := int(info.NextEntryOffset)
+	if valueLen+valueOffset > len(b) || nextOffset < 0 || nextOffset > len(b) {
+		err = errInvalidEaBuffer
+		return
+	}
+
+	ea.Name = string(b[nameOffset : nameOffset+nameLen])
+	ea.Value = b[valueOffset : valueOffset+valueLen]
+	ea.Flags = info.Flags
+	if info.NextEntryOffset != 0 {
+		nb = b[info.NextEntryOffset:]
+	}
+	return
+}
+
+// DecodeExtendedAttributes decodes a list of EAs from a FILE_FULL_EA_INFORMATION
+// buffer retrieved from BackupRead, ZwQueryEaFile, etc.
+func DecodeExtendedAttributes(b []byte) (eas []ExtendedAttribute, err error) {
+	for len(b) != 0 {
+		ea, nb, err := parseEa(b)
+		if err != nil {
+			return nil, err
+		}
+
+		eas = append(eas, ea)
+		b = nb
+	}
+	return
+}
+
+func writeEa(buf *bytes.Buffer, ea *ExtendedAttribute, last bool) error {
+	if int(uint8(len(ea.Name))) != len(ea.Name) {
+		return errEaNameTooLarge
+	}
+	if int(uint16(len(ea.Value))) != len(ea.Value) {
+		return errEaValueTooLarge
+	}
+	entrySize := uint32(fileFullEaInformationSize + len(ea.Name) + 1 + len(ea.Value))
+	withPadding := (entrySize + 3) &^ 3
+	nextOffset := uint32(0)
+	if !last {
+		nextOffset = withPadding
+	}
+	info := fileFullEaInformation{
+		NextEntryOffset: nextOffset,
+		Flags:           ea.Flags,
+		NameLength:      uint8(len(ea.Name)),
+		ValueLength:     uint16(len(ea.Value)),
+	}
+
+	err := binary.Write(buf, binary.LittleEndian, &info)
+	if err != nil {
+		return err
+	}
+
+	_, err = buf.Write([]byte(ea.Name))
+	if err != nil {
+		return err
+	}
+
+	err = buf.WriteByte(0)
+	if err != nil {
+		return err
+	}
+
+	_, err = buf.Write(ea.Value)
+	if err != nil {
+		return err
+	}
+
+	_, err = buf.Write([]byte{0, 0, 0}[0 : withPadding-entrySize])
+	if err != nil {
+		return err
+	}
+
+	return nil
+}
+
+// EncodeExtendedAttributes encodes a list of EAs into a FILE_FULL_EA_INFORMATION
+// buffer for use with BackupWrite, ZwSetEaFile, etc.
+func EncodeExtendedAttributes(eas []ExtendedAttribute) ([]byte, error) {
+	var buf bytes.Buffer
+	for i := range eas {
+		last := false
+		if i == len(eas)-1 {
+			last = true
+		}
+
+		err := writeEa(&buf, &eas[i], last)
+		if err != nil {
+			return nil, err
+		}
+	}
+	return buf.Bytes(), nil
+}

vendor/github.com/Microsoft/go-winio/file.go

@@ -0,0 +1,323 @@
+// +build windows
+
+package winio
+
+import (
+	"errors"
+	"io"
+	"runtime"
+	"sync"
+	"sync/atomic"
+	"syscall"
+	"time"
+)
+
+//sys cancelIoEx(file syscall.Handle, o *syscall.Overlapped) (err error) = CancelIoEx
+//sys createIoCompletionPort(file syscall.Handle, port syscall.Handle, key uintptr, threadCount uint32) (newport syscall.Handle, err error) = CreateIoCompletionPort
+//sys getQueuedCompletionStatus(port syscall.Handle, bytes *uint32, key *uintptr, o **ioOperation, timeout uint32) (err error) = GetQueuedCompletionStatus
+//sys setFileCompletionNotificationModes(h syscall.Handle, flags uint8) (err error) = SetFileCompletionNotificationModes
+//sys wsaGetOverlappedResult(h syscall.Handle, o *syscall.Overlapped, bytes *uint32, wait bool, flags *uint32) (err error) = ws2_32.WSAGetOverlappedResult
+
+type atomicBool int32
+
+func (b *atomicBool) isSet() bool { return atomic.LoadInt32((*int32)(b)) != 0 }
+func (b *atomicBool) setFalse()   { atomic.StoreInt32((*int32)(b), 0) }
+func (b *atomicBool) setTrue()    { atomic.StoreInt32((*int32)(b), 1) }
+func (b *atomicBool) swap(new bool) bool {
+	var newInt int32
+	if new {
+		newInt = 1
+	}
+	return atomic.SwapInt32((*int32)(b), newInt) == 1
+}
+
+const (
+	cFILE_SKIP_COMPLETION_PORT_ON_SUCCESS = 1
+	cFILE_SKIP_SET_EVENT_ON_HANDLE        = 2
+)
+
+var (
+	ErrFileClosed = errors.New("file has already been closed")
+	ErrTimeout    = &timeoutError{}
+)
+
+type timeoutError struct{}
+
+func (e *timeoutError) Error() string   { return "i/o timeout" }
+func (e *timeoutError) Timeout() bool   { return true }
+func (e *timeoutError) Temporary() bool { return true }
+
+type timeoutChan chan struct{}
+
+var ioInitOnce sync.Once
+var ioCompletionPort syscall.Handle
+
+// ioResult contains the result of an asynchronous IO operation
+type ioResult struct {
+	bytes uint32
+	err   error
+}
+
+// ioOperation represents an outstanding asynchronous Win32 IO
+type ioOperation struct {
+	o  syscall.Overlapped
+	ch chan ioResult
+}
+
+func initIo() {
+	h, err := createIoCompletionPort(syscall.InvalidHandle, 0, 0, 0xffffffff)
+	if err != nil {
+		panic(err)
+	}
+	ioCompletionPort = h
+	go ioCompletionProcessor(h)
+}
+
+// win32File implements Reader, Writer, and Closer on a Win32 handle without blocking in a syscall.
+// It takes ownership of this handle and will close it if it is garbage collected.
+type win32File struct {
+	handle        syscall.Handle
+	wg            sync.WaitGroup
+	wgLock        sync.RWMutex
+	closing       atomicBool
+	socket        bool
+	readDeadline  deadlineHandler
+	writeDeadline deadlineHandler
+}
+
+type deadlineHandler struct {
+	setLock     sync.Mutex
+	channel     timeoutChan
+	channelLock sync.RWMutex
+	timer       *time.Timer
+	timedout    atomicBool
+}
+
+// makeWin32File makes a new win32File from an existing file handle
+func makeWin32File(h syscall.Handle) (*win32File, error) {
+	f := &win32File{handle: h}
+	ioInitOnce.Do(initIo)
+	_, err := createIoCompletionPort(h, ioCompletionPort, 0, 0xffffffff)
+	if err != nil {
+		return nil, err
+	}
+	err = setFileCompletionNotificationModes(h, cFILE_SKIP_COMPLETION_PORT_ON_SUCCESS|cFILE_SKIP_SET_EVENT_ON_HANDLE)
+	if err != nil {
+		return nil, err
+	}
+	f.readDeadline.channel = make(timeoutChan)
+	f.writeDeadline.channel = make(timeoutChan)
+	return f, nil
+}
+
+func MakeOpenFile(h syscall.Handle) (io.ReadWriteCloser, error) {
+	// If we return the result of makeWin32File directly, it can result in an
+	// interface-wrapped nil, rather than a nil interface value.
+	f, err := makeWin32File(h)
+	if err != nil {
+		return nil, err
+	}
+	return f, nil
+}
+
+// closeHandle closes the resources associated with a Win32 handle
+func (f *win32File) closeHandle() {
+	f.wgLock.Lock()
+	// Atomically set that we are closing, releasing the resources only once.
+	if !f.closing.swap(true) {
+		f.wgLock.Unlock()
+		// cancel all IO and wait for it to complete
+		cancelIoEx(f.handle, nil)
+		f.wg.Wait()
+		// at this point, no new IO can start
+		syscall.Close(f.handle)
+		f.handle = 0
+	} else {
+		f.wgLock.Unlock()
+	}
+}
+
+// Close closes a win32File.
+func (f *win32File) Close() error {
+	f.closeHandle()
+	return nil
+}
+
+// prepareIo prepares for a new IO operation.
+// The caller must call f.wg.Done() when the IO is finished, prior to Close() returning.
+func (f *win32File) prepareIo() (*ioOperation, error) {
+	f.wgLock.RLock()
+	if f.closing.isSet() {
+		f.wgLock.RUnlock()
+		return nil, ErrFileClosed
+	}
+	f.wg.Add(1)
+	f.wgLock.RUnlock()
+	c := &ioOperation{}
+	c.ch = make(chan ioResult)
+	return c, nil
+}
+
+// ioCompletionProcessor processes completed async IOs forever
+func ioCompletionProcessor(h syscall.Handle) {
+	for {
+		var bytes uint32
+		var key uintptr
+		var op *ioOperation
+		err := getQueuedCompletionStatus(h, &bytes, &key, &op, syscall.INFINITE)
+		if op == nil {
+			panic(err)
+		}
+		op.ch <- ioResult{bytes, err}
+	}
+}
+
+// asyncIo processes the return value from ReadFile or WriteFile, blocking until
+// the operation has actually completed.
+func (f *win32File) asyncIo(c *ioOperation, d *deadlineHandler, bytes uint32, err error) (int, error) {
+	if err != syscall.ERROR_IO_PENDING {
+		return int(bytes), err
+	}
+
+	if f.closing.isSet() {
+		cancelIoEx(f.handle, &c.o)
+	}
+
+	var timeout timeoutChan
+	if d != nil {
+		d.channelLock.Lock()
+		timeout = d.channel
+		d.channelLock.Unlock()
+	}
+
+	var r ioResult
+	select {
+	case r = <-c.ch:
+		err = r.err
+		if err == syscall.ERROR_OPERATION_ABORTED {
+			if f.closing.isSet() {
+				err = ErrFileClosed
+			}
+		} else if err != nil && f.socket {
+			// err is from Win32. Query the overlapped structure to get the winsock error.
+			var bytes, flags uint32
+			err = wsaGetOverlappedResult(f.handle, &c.o, &bytes, false, &flags)
+		}
+	case <-timeout:
+		cancelIoEx(f.handle, &c.o)
+		r = <-c.ch
+		err = r.err
+		if err == syscall.ERROR_OPERATION_ABORTED {
+			err = ErrTimeout
+		}
+	}
+
+	// runtime.KeepAlive is needed, as c is passed via native
+	// code to ioCompletionProcessor, c must remain alive
+	// until the channel read is complete.
+	runtime.KeepAlive(c)
+	return int(r.bytes), err
+}
+
+// Read reads from a file handle.
+func (f *win32File) Read(b []byte) (int, error) {
+	c, err := f.prepareIo()
+	if err != nil {
+		return 0, err
+	}
+	defer f.wg.Done()
+
+	if f.readDeadline.timedout.isSet() {
+		return 0, ErrTimeout
+	}
+
+	var bytes uint32
+	err = syscall.ReadFile(f.handle, b, &bytes, &c.o)
+	n, err := f.asyncIo(c, &f.readDeadline, bytes, err)
+	runtime.KeepAlive(b)
+
+	// Handle EOF conditions.
+	if err == nil && n == 0 && len(b) != 0 {
+		return 0, io.EOF
+	} else if err == syscall.ERROR_BROKEN_PIPE {
+		return 0, io.EOF
+	} else {
+		return n, err
+	}
+}
+
+// Write writes to a file handle.
+func (f *win32File) Write(b []byte) (int, error) {
+	c, err := f.prepareIo()
+	if err != nil {
+		return 0, err
+	}
+	defer f.wg.Done()
+
+	if f.writeDeadline.timedout.isSet() {
+		return 0, ErrTimeout
+	}
+
+	var bytes uint32
+	err = syscall.WriteFile(f.handle, b, &bytes, &c.o)
+	n, err := f.asyncIo(c, &f.writeDeadline, bytes, err)
+	runtime.KeepAlive(b)
+	return n, err
+}
+
+func (f *win32File) SetReadDeadline(deadline time.Time) error {
+	return f.readDeadline.set(deadline)
+}
+
+func (f *win32File) SetWriteDeadline(deadline time.Time) error {
+	return f.writeDeadline.set(deadline)
+}
+
+func (f *win32File) Flush() error {
+	return syscall.FlushFileBuffers(f.handle)
+}
+
+func (f *win32File) Fd() uintptr {
+	return uintptr(f.handle)
+}
+
+func (d *deadlineHandler) set(deadline time.Time) error {
+	d.setLock.Lock()
+	defer d.setLock.Unlock()
+
+	if d.timer != nil {
+		if !d.timer.Stop() {
+			<-d.channel
+		}
+		d.timer = nil
+	}
+	d.timedout.setFalse()
+
+	select {
+	case <-d.channel:
+		d.channelLock.Lock()
+		d.channel = make(chan struct{})
+		d.channelLock.Unlock()
+	default:
+	}
+
+	if deadline.IsZero() {
+		return nil
+	}
+
+	timeoutIO := func() {
+		d.timedout.setTrue()
+		close(d.channel)
+	}
+
+	now := time.Now()
+	duration := deadline.Sub(now)
+	if deadline.After(now) {
+		// Deadline is in the future, set a timer to wait
+		d.timer = time.AfterFunc(duration, timeoutIO)
+	} else {
+		// Deadline is in the past. Cancel all pending IO now.
+		timeoutIO()
+	}
+	return nil
+}

vendor/github.com/Microsoft/go-winio/fileinfo.go

@@ -0,0 +1,61 @@
+// +build windows
+
+package winio
+
+import (
+	"os"
+	"runtime"
+	"syscall"
+	"unsafe"
+)
+
+//sys getFileInformationByHandleEx(h syscall.Handle, class uint32, buffer *byte, size uint32) (err error) = GetFileInformationByHandleEx
+//sys setFileInformationByHandle(h syscall.Handle, class uint32, buffer *byte, size uint32) (err error) = SetFileInformationByHandle
+
+const (
+	fileBasicInfo = 0
+	fileIDInfo    = 0x12
+)
+
+// FileBasicInfo contains file access time and file attributes information.
+type FileBasicInfo struct {
+	CreationTime, LastAccessTime, LastWriteTime, ChangeTime syscall.Filetime
+	FileAttributes                                          uint32
+	pad                                                     uint32 // padding
+}
+
+// GetFileBasicInfo retrieves times and attributes for a file.
+func GetFileBasicInfo(f *os.File) (*FileBasicInfo, error) {
+	bi := &FileBasicInfo{}
+	if err := getFileInformationByHandleEx(syscall.Handle(f.Fd()), fileBasicInfo, (*byte)(unsafe.Pointer(bi)), uint32(unsafe.Sizeof(*bi))); err != nil {
+		return nil, &os.PathError{Op: "GetFileInformationByHandleEx", Path: f.Name(), Err: err}
+	}
+	runtime.KeepAlive(f)
+	return bi, nil
+}
+
+// SetFileBasicInfo sets times and attributes for a file.
+func SetFileBasicInfo(f *os.File, bi *FileBasicInfo) error {
+	if err := setFileInformationByHandle(syscall.Handle(f.Fd()), fileBasicInfo, (*byte)(unsafe.Pointer(bi)), uint32(unsafe.Sizeof(*bi))); err != nil {
+		return &os.PathError{Op: "SetFileInformationByHandle", Path: f.Name(), Err: err}
+	}
+	runtime.KeepAlive(f)
+	return nil
+}
+
+// FileIDInfo contains the volume serial number and file ID for a file. This pair should be
+// unique on a system.
+type FileIDInfo struct {
+	VolumeSerialNumber uint64
+	FileID             [16]byte
+}
+
+// GetFileID retrieves the unique (volume, file ID) pair for a file.
+func GetFileID(f *os.File) (*FileIDInfo, error) {
+	fileID := &FileIDInfo{}
+	if err := getFileInformationByHandleEx(syscall.Handle(f.Fd()), fileIDInfo, (*byte)(unsafe.Pointer(fileID)), uint32(unsafe.Sizeof(*fileID))); err != nil {
+		return nil, &os.PathError{Op: "GetFileInformationByHandleEx", Path: f.Name(), Err: err}
+	}
+	runtime.KeepAlive(f)
+	return fileID, nil
+}

vendor/github.com/Microsoft/go-winio/go.mod

@@ -0,0 +1,9 @@
+module github.com/Microsoft/go-winio
+
+go 1.12
+
+require (
+	github.com/pkg/errors v0.8.1
+	github.com/sirupsen/logrus v1.4.1
+	golang.org/x/sys v0.0.0-20190507160741-ecd444e8653b
+)

vendor/github.com/Microsoft/go-winio/go.sum

@@ -0,0 +1,16 @@
+github.com/davecgh/go-spew v1.1.1 h1:vj9j/u1bqnvCEfJOwUhtlOARqs3+rkHYY13jYWTU97c=
+github.com/davecgh/go-spew v1.1.1/go.mod h1:J7Y8YcW2NihsgmVo/mv3lAwl/skON4iLHjSsI+c5H38=
+github.com/konsorten/go-windows-terminal-sequences v1.0.1 h1:mweAR1A6xJ3oS2pRaGiHgQ4OO8tzTaLawm8vnODuwDk=
+github.com/konsorten/go-windows-terminal-sequences v1.0.1/go.mod h1:T0+1ngSBFLxvqU3pZ+m/2kptfBszLMUkC4ZK/EgS/cQ=
+github.com/pkg/errors v0.8.1 h1:iURUrRGxPUNPdy5/HRSm+Yj6okJ6UtLINN0Q9M4+h3I=
+github.com/pkg/errors v0.8.1/go.mod h1:bwawxfHBFNV+L2hUp1rHADufV3IMtnDRdf1r5NINEl0=
+github.com/pmezard/go-difflib v1.0.0 h1:4DBwDE0NGyQoBHbLQYPwSUPoCMWR5BEzIk/f1lZbAQM=
+github.com/pmezard/go-difflib v1.0.0/go.mod h1:iKH77koFhYxTK1pcRnkKkqfTogsbg7gZNVY4sRDYZ/4=
+github.com/sirupsen/logrus v1.4.1 h1:GL2rEmy6nsikmW0r8opw9JIRScdMF5hA8cOYLH7In1k=
+github.com/sirupsen/logrus v1.4.1/go.mod h1:ni0Sbl8bgC9z8RoU9G6nDWqqs/fq4eDPysMBDgk/93Q=
+github.com/stretchr/objx v0.1.1/go.mod h1:HFkY916IF+rwdDfMAkV7OtwuqBVzrE8GR6GFx+wExME=
+github.com/stretchr/testify v1.2.2 h1:bSDNvY7ZPG5RlJ8otE/7V6gMiyenm9RtJ7IUVIAoJ1w=
+github.com/stretchr/testify v1.2.2/go.mod h1:a8OnRcib4nhh0OaRAV+Yts87kKdq0PP7pXfy6kDkUVs=
+golang.org/x/sys v0.0.0-20180905080454-ebe1bf3edb33/go.mod h1:STP8DvDyc/dI5b8T5hshtkjS+E42TnysNCUPdjciGhY=
+golang.org/x/sys v0.0.0-20190507160741-ecd444e8653b h1:ag/x1USPSsqHud38I9BAC88qdNLDHHtQ4mlgQIZPPNA=
+golang.org/x/sys v0.0.0-20190507160741-ecd444e8653b/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=

vendor/github.com/Microsoft/go-winio/hvsock.go

@@ -0,0 +1,305 @@
+package winio
+
+import (
+	"fmt"
+	"io"
+	"net"
+	"os"
+	"syscall"
+	"time"
+	"unsafe"
+
+	"github.com/Microsoft/go-winio/pkg/guid"
+)
+
+//sys bind(s syscall.Handle, name unsafe.Pointer, namelen int32) (err error) [failretval==socketError] = ws2_32.bind
+
+const (
+	afHvSock = 34 // AF_HYPERV
+
+	socketError = ^uintptr(0)
+)
+
+// An HvsockAddr is an address for a AF_HYPERV socket.
+type HvsockAddr struct {
+	VMID      guid.GUID
+	ServiceID guid.GUID
+}
+
+type rawHvsockAddr struct {
+	Family    uint16
+	_         uint16
+	VMID      guid.GUID
+	ServiceID guid.GUID
+}
+
+// Network returns the address's network name, "hvsock".
+func (addr *HvsockAddr) Network() string {
+	return "hvsock"
+}
+
+func (addr *HvsockAddr) String() string {
+	return fmt.Sprintf("%s:%s", &addr.VMID, &addr.ServiceID)
+}
+
+// VsockServiceID returns an hvsock service ID corresponding to the specified AF_VSOCK port.
+func VsockServiceID(port uint32) guid.GUID {
+	g, _ := guid.FromString("00000000-facb-11e6-bd58-64006a7986d3")
+	g.Data1 = port
+	return g
+}
+
+func (addr *HvsockAddr) raw() rawHvsockAddr {
+	return rawHvsockAddr{
+		Family:    afHvSock,
+		VMID:      addr.VMID,
+		ServiceID: addr.ServiceID,
+	}
+}
+
+func (addr *HvsockAddr) fromRaw(raw *rawHvsockAddr) {
+	addr.VMID = raw.VMID
+	addr.ServiceID = raw.ServiceID
+}
+
+// HvsockListener is a socket listener for the AF_HYPERV address family.
+type HvsockListener struct {
+	sock *win32File
+	addr HvsockAddr
+}
+
+// HvsockConn is a connected socket of the AF_HYPERV address family.
+type HvsockConn struct {
+	sock          *win32File
+	local, remote HvsockAddr
+}
+
+func newHvSocket() (*win32File, error) {
+	fd, err := syscall.Socket(afHvSock, syscall.SOCK_STREAM, 1)
+	if err != nil {
+		return nil, os.NewSyscallError("socket", err)
+	}
+	f, err := makeWin32File(fd)
+	if err != nil {
+		syscall.Close(fd)
+		return nil, err
+	}
+	f.socket = true
+	return f, nil
+}
+
+// ListenHvsock listens for connections on the specified hvsock address.
+func ListenHvsock(addr *HvsockAddr) (_ *HvsockListener, err error) {
+	l := &HvsockListener{addr: *addr}
+	sock, err := newHvSocket()
+	if err != nil {
+		return nil, l.opErr("listen", err)
+	}
+	sa := addr.raw()
+	err = bind(sock.handle, unsafe.Pointer(&sa), int32(unsafe.Sizeof(sa)))
+	if err != nil {
+		return nil, l.opErr("listen", os.NewSyscallError("socket", err))
+	}
+	err = syscall.Listen(sock.handle, 16)
+	if err != nil {
+		return nil, l.opErr("listen", os.NewSyscallError("listen", err))
+	}
+	return &HvsockListener{sock: sock, addr: *addr}, nil
+}
+
+func (l *HvsockListener) opErr(op string, err error) error {
+	return &net.OpError{Op: op, Net: "hvsock", Addr: &l.addr, Err: err}
+}
+
+// Addr returns the listener's network address.
+func (l *HvsockListener) Addr() net.Addr {
+	return &l.addr
+}
+
+// Accept waits for the next connection and returns it.
+func (l *HvsockListener) Accept() (_ net.Conn, err error) {
+	sock, err := newHvSocket()
+	if err != nil {
+		return nil, l.opErr("accept", err)
+	}
+	defer func() {
+		if sock != nil {
+			sock.Close()
+		}
+	}()
+	c, err := l.sock.prepareIo()
+	if err != nil {
+		return nil, l.opErr("accept", err)
+	}
+	defer l.sock.wg.Done()
+
+	// AcceptEx, per documentation, requires an extra 16 bytes per address.
+	const addrlen = uint32(16 + unsafe.Sizeof(rawHvsockAddr{}))
+	var addrbuf [addrlen * 2]byte
+
+	var bytes uint32
+	err = syscall.AcceptEx(l.sock.handle, sock.handle, &addrbuf[0], 0, addrlen, addrlen, &bytes, &c.o)
+	_, err = l.sock.asyncIo(c, nil, bytes, err)
+	if err != nil {
+		return nil, l.opErr("accept", os.NewSyscallError("acceptex", err))
+	}
+	conn := &HvsockConn{
+		sock: sock,
+	}
+	conn.local.fromRaw((*rawHvsockAddr)(unsafe.Pointer(&addrbuf[0])))
+	conn.remote.fromRaw((*rawHvsockAddr)(unsafe.Pointer(&addrbuf[addrlen])))
+	sock = nil
+	return conn, nil
+}
+
+// Close closes the listener, causing any pending Accept calls to fail.
+func (l *HvsockListener) Close() error {
+	return l.sock.Close()
+}
+
+/* Need to finish ConnectEx handling
+func DialHvsock(ctx context.Context, addr *HvsockAddr) (*HvsockConn, error) {
+	sock, err := newHvSocket()
+	if err != nil {
+		return nil, err
+	}
+	defer func() {
+		if sock != nil {
+			sock.Close()
+		}
+	}()
+	c, err := sock.prepareIo()
+	if err != nil {
+		return nil, err
+	}
+	defer sock.wg.Done()
+	var bytes uint32
+	err = windows.ConnectEx(windows.Handle(sock.handle), sa, nil, 0, &bytes, &c.o)
+	_, err = sock.asyncIo(ctx, c, nil, bytes, err)
+	if err != nil {
+		return nil, err
+	}
+	conn := &HvsockConn{
+		sock:   sock,
+		remote: *addr,
+	}
+	sock = nil
+	return conn, nil
+}
+*/
+
+func (conn *HvsockConn) opErr(op string, err error) error {
+	return &net.OpError{Op: op, Net: "hvsock", Source: &conn.local, Addr: &conn.remote, Err: err}
+}
+
+func (conn *HvsockConn) Read(b []byte) (int, error) {
+	c, err := conn.sock.prepareIo()
+	if err != nil {
+		return 0, conn.opErr("read", err)
+	}
+	defer conn.sock.wg.Done()
+	buf := syscall.WSABuf{Buf: &b[0], Len: uint32(len(b))}
+	var flags, bytes uint32
+	err = syscall.WSARecv(conn.sock.handle, &buf, 1, &bytes, &flags, &c.o, nil)
+	n, err := conn.sock.asyncIo(c, &conn.sock.readDeadline, bytes, err)
+	if err != nil {
+		if _, ok := err.(syscall.Errno); ok {
+			err = os.NewSyscallError("wsarecv", err)
+		}
+		return 0, conn.opErr("read", err)
+	} else if n == 0 {
+		err = io.EOF
+	}
+	return n, err
+}
+
+func (conn *HvsockConn) Write(b []byte) (int, error) {
+	t := 0
+	for len(b) != 0 {
+		n, err := conn.write(b)
+		if err != nil {
+			return t + n, err
+		}
+		t += n
+		b = b[n:]
+	}
+	return t, nil
+}
+
+func (conn *HvsockConn) write(b []byte) (int, error) {
+	c, err := conn.sock.prepareIo()
+	if err != nil {
+		return 0, conn.opErr("write", err)
+	}
+	defer conn.sock.wg.Done()
+	buf := syscall.WSABuf{Buf: &b[0], Len: uint32(len(b))}
+	var bytes uint32
+	err = syscall.WSASend(conn.sock.handle, &buf, 1, &bytes, 0, &c.o, nil)
+	n, err := conn.sock.asyncIo(c, &conn.sock.writeDeadline, bytes, err)
+	if err != nil {
+		if _, ok := err.(syscall.Errno); ok {
+			err = os.NewSyscallError("wsasend", err)
+		}
+		return 0, conn.opErr("write", err)
+	}
+	return n, err
+}
+
+// Close closes the socket connection, failing any pending read or write calls.
+func (conn *HvsockConn) Close() error {
+	return conn.sock.Close()
+}
+
+func (conn *HvsockConn) shutdown(how int) error {
+	err := syscall.Shutdown(conn.sock.handle, syscall.SHUT_RD)
+	if err != nil {
+		return os.NewSyscallError("shutdown", err)
+	}
+	return nil
+}
+
+// CloseRead shuts down the read end of the socket.
+func (conn *HvsockConn) CloseRead() error {
+	err := conn.shutdown(syscall.SHUT_RD)
+	if err != nil {
+		return conn.opErr("close", err)
+	}
+	return nil
+}
+
+// CloseWrite shuts down the write end of the socket, notifying the other endpoint that
+// no more data will be written.
+func (conn *HvsockConn) CloseWrite() error {
+	err := conn.shutdown(syscall.SHUT_WR)
+	if err != nil {
+		return conn.opErr("close", err)
+	}
+	return nil
+}
+
+// LocalAddr returns the local address of the connection.
+func (conn *HvsockConn) LocalAddr() net.Addr {
+	return &conn.local
+}
+
+// RemoteAddr returns the remote address of the connection.
+func (conn *HvsockConn) RemoteAddr() net.Addr {
+	return &conn.remote
+}
+
+// SetDeadline implements the net.Conn SetDeadline method.
+func (conn *HvsockConn) SetDeadline(t time.Time) error {
+	conn.SetReadDeadline(t)
+	conn.SetWriteDeadline(t)
+	return nil
+}
+
+// SetReadDeadline implements the net.Conn SetReadDeadline method.
+func (conn *HvsockConn) SetReadDeadline(t time.Time) error {
+	return conn.sock.SetReadDeadline(t)
+}
+
+// SetWriteDeadline implements the net.Conn SetWriteDeadline method.
+func (conn *HvsockConn) SetWriteDeadline(t time.Time) error {
+	return conn.sock.SetWriteDeadline(t)
+}

vendor/github.com/Microsoft/go-winio/pipe.go

@@ -0,0 +1,510 @@
+// +build windows
+
+package winio
+
+import (
+	"context"
+	"errors"
+	"fmt"
+	"io"
+	"net"
+	"os"
+	"runtime"
+	"syscall"
+	"time"
+	"unsafe"
+)
+
+//sys connectNamedPipe(pipe syscall.Handle, o *syscall.Overlapped) (err error) = ConnectNamedPipe
+//sys createNamedPipe(name string, flags uint32, pipeMode uint32, maxInstances uint32, outSize uint32, inSize uint32, defaultTimeout uint32, sa *syscall.SecurityAttributes) (handle syscall.Handle, err error)  [failretval==syscall.InvalidHandle] = CreateNamedPipeW
+//sys createFile(name string, access uint32, mode uint32, sa *syscall.SecurityAttributes, createmode uint32, attrs uint32, templatefile syscall.Handle) (handle syscall.Handle, err error) [failretval==syscall.InvalidHandle] = CreateFileW
+//sys getNamedPipeInfo(pipe syscall.Handle, flags *uint32, outSize *uint32, inSize *uint32, maxInstances *uint32) (err error) = GetNamedPipeInfo
+//sys getNamedPipeHandleState(pipe syscall.Handle, state *uint32, curInstances *uint32, maxCollectionCount *uint32, collectDataTimeout *uint32, userName *uint16, maxUserNameSize uint32) (err error) = GetNamedPipeHandleStateW
+//sys localAlloc(uFlags uint32, length uint32) (ptr uintptr) = LocalAlloc
+//sys ntCreateNamedPipeFile(pipe *syscall.Handle, access uint32, oa *objectAttributes, iosb *ioStatusBlock, share uint32, disposition uint32, options uint32, typ uint32, readMode uint32, completionMode uint32, maxInstances uint32, inboundQuota uint32, outputQuota uint32, timeout *int64) (status ntstatus) = ntdll.NtCreateNamedPipeFile
+//sys rtlNtStatusToDosError(status ntstatus) (winerr error) = ntdll.RtlNtStatusToDosErrorNoTeb
+//sys rtlDosPathNameToNtPathName(name *uint16, ntName *unicodeString, filePart uintptr, reserved uintptr) (status ntstatus) = ntdll.RtlDosPathNameToNtPathName_U
+//sys rtlDefaultNpAcl(dacl *uintptr) (status ntstatus) = ntdll.RtlDefaultNpAcl
+
+type ioStatusBlock struct {
+	Status, Information uintptr
+}
+
+type objectAttributes struct {
+	Length             uintptr
+	RootDirectory      uintptr
+	ObjectName         *unicodeString
+	Attributes         uintptr
+	SecurityDescriptor *securityDescriptor
+	SecurityQoS        uintptr
+}
+
+type unicodeString struct {
+	Length        uint16
+	MaximumLength uint16
+	Buffer        uintptr
+}
+
+type securityDescriptor struct {
+	Revision byte
+	Sbz1     byte
+	Control  uint16
+	Owner    uintptr
+	Group    uintptr
+	Sacl     uintptr
+	Dacl     uintptr
+}
+
+type ntstatus int32
+
+func (status ntstatus) Err() error {
+	if status >= 0 {
+		return nil
+	}
+	return rtlNtStatusToDosError(status)
+}
+
+const (
+	cERROR_PIPE_BUSY      = syscall.Errno(231)
+	cERROR_NO_DATA        = syscall.Errno(232)
+	cERROR_PIPE_CONNECTED = syscall.Errno(535)
+	cERROR_SEM_TIMEOUT    = syscall.Errno(121)
+
+	cSECURITY_SQOS_PRESENT = 0x100000
+	cSECURITY_ANONYMOUS    = 0
+
+	cPIPE_TYPE_MESSAGE = 4
+
+	cPIPE_READMODE_MESSAGE = 2
+
+	cFILE_OPEN   = 1
+	cFILE_CREATE = 2
+
+	cFILE_PIPE_MESSAGE_TYPE          = 1
+	cFILE_PIPE_REJECT_REMOTE_CLIENTS = 2
+
+	cSE_DACL_PRESENT = 4
+)
+
+var (
+	// ErrPipeListenerClosed is returned for pipe operations on listeners that have been closed.
+	// This error should match net.errClosing since docker takes a dependency on its text.
+	ErrPipeListenerClosed = errors.New("use of closed network connection")
+
+	errPipeWriteClosed = errors.New("pipe has been closed for write")
+)
+
+type win32Pipe struct {
+	*win32File
+	path string
+}
+
+type win32MessageBytePipe struct {
+	win32Pipe
+	writeClosed bool
+	readEOF     bool
+}
+
+type pipeAddress string
+
+func (f *win32Pipe) LocalAddr() net.Addr {
+	return pipeAddress(f.path)
+}
+
+func (f *win32Pipe) RemoteAddr() net.Addr {
+	return pipeAddress(f.path)
+}
+
+func (f *win32Pipe) SetDeadline(t time.Time) error {
+	f.SetReadDeadline(t)
+	f.SetWriteDeadline(t)
+	return nil
+}
+
+// CloseWrite closes the write side of a message pipe in byte mode.
+func (f *win32MessageBytePipe) CloseWrite() error {
+	if f.writeClosed {
+		return errPipeWriteClosed
+	}
+	err := f.win32File.Flush()
+	if err != nil {
+		return err
+	}
+	_, err = f.win32File.Write(nil)
+	if err != nil {
+		return err
+	}
+	f.writeClosed = true
+	return nil
+}
+
+// Write writes bytes to a message pipe in byte mode. Zero-byte writes are ignored, since
+// they are used to implement CloseWrite().
+func (f *win32MessageBytePipe) Write(b []byte) (int, error) {
+	if f.writeClosed {
+		return 0, errPipeWriteClosed
+	}
+	if len(b) == 0 {
+		return 0, nil
+	}
+	return f.win32File.Write(b)
+}
+
+// Read reads bytes from a message pipe in byte mode. A read of a zero-byte message on a message
+// mode pipe will return io.EOF, as will all subsequent reads.
+func (f *win32MessageBytePipe) Read(b []byte) (int, error) {
+	if f.readEOF {
+		return 0, io.EOF
+	}
+	n, err := f.win32File.Read(b)
+	if err == io.EOF {
+		// If this was the result of a zero-byte read, then
+		// it is possible that the read was due to a zero-size
+		// message. Since we are simulating CloseWrite with a
+		// zero-byte message, ensure that all future Read() calls
+		// also return EOF.
+		f.readEOF = true
+	} else if err == syscall.ERROR_MORE_DATA {
+		// ERROR_MORE_DATA indicates that the pipe's read mode is message mode
+		// and the message still has more bytes. Treat this as a success, since
+		// this package presents all named pipes as byte streams.
+		err = nil
+	}
+	return n, err
+}
+
+func (s pipeAddress) Network() string {
+	return "pipe"
+}
+
+func (s pipeAddress) String() string {
+	return string(s)
+}
+
+// tryDialPipe attempts to dial the pipe at `path` until `ctx` cancellation or timeout.
+func tryDialPipe(ctx context.Context, path *string) (syscall.Handle, error) {
+	for {
+		select {
+		case <-ctx.Done():
+			return syscall.Handle(0), ctx.Err()
+		default:
+			h, err := createFile(*path, syscall.GENERIC_READ|syscall.GENERIC_WRITE, 0, nil, syscall.OPEN_EXISTING, syscall.FILE_FLAG_OVERLAPPED|cSECURITY_SQOS_PRESENT|cSECURITY_ANONYMOUS, 0)
+			if err == nil {
+				return h, nil
+			}
+			if err != cERROR_PIPE_BUSY {
+				return h, &os.PathError{Err: err, Op: "open", Path: *path}
+			}
+			// Wait 10 msec and try again. This is a rather simplistic
+			// view, as we always try each 10 milliseconds.
+			time.Sleep(time.Millisecond * 10)
+		}
+	}
+}
+
+// DialPipe connects to a named pipe by path, timing out if the connection
+// takes longer than the specified duration. If timeout is nil, then we use
+// a default timeout of 2 seconds.  (We do not use WaitNamedPipe.)
+func DialPipe(path string, timeout *time.Duration) (net.Conn, error) {
+	var absTimeout time.Time
+	if timeout != nil {
+		absTimeout = time.Now().Add(*timeout)
+	} else {
+		absTimeout = time.Now().Add(time.Second * 2)
+	}
+	ctx, _ := context.WithDeadline(context.Background(), absTimeout)
+	conn, err := DialPipeContext(ctx, path)
+	if err == context.DeadlineExceeded {
+		return nil, ErrTimeout
+	}
+	return conn, err
+}
+
+// DialPipeContext attempts to connect to a named pipe by `path` until `ctx`
+// cancellation or timeout.
+func DialPipeContext(ctx context.Context, path string) (net.Conn, error) {
+	var err error
+	var h syscall.Handle
+	h, err = tryDialPipe(ctx, &path)
+	if err != nil {
+		return nil, err
+	}
+
+	var flags uint32
+	err = getNamedPipeInfo(h, &flags, nil, nil, nil)
+	if err != nil {
+		return nil, err
+	}
+
+	f, err := makeWin32File(h)
+	if err != nil {
+		syscall.Close(h)
+		return nil, err
+	}
+
+	// If the pipe is in message mode, return a message byte pipe, which
+	// supports CloseWrite().
+	if flags&cPIPE_TYPE_MESSAGE != 0 {
+		return &win32MessageBytePipe{
+			win32Pipe: win32Pipe{win32File: f, path: path},
+		}, nil
+	}
+	return &win32Pipe{win32File: f, path: path}, nil
+}
+
+type acceptResponse struct {
+	f   *win32File
+	err error
+}
+
+type win32PipeListener struct {
+	firstHandle syscall.Handle
+	path        string
+	config      PipeConfig
+	acceptCh    chan (chan acceptResponse)
+	closeCh     chan int
+	doneCh      chan int
+}
+
+func makeServerPipeHandle(path string, sd []byte, c *PipeConfig, first bool) (syscall.Handle, error) {
+	path16, err := syscall.UTF16FromString(path)
+	if err != nil {
+		return 0, &os.PathError{Op: "open", Path: path, Err: err}
+	}
+
+	var oa objectAttributes
+	oa.Length = unsafe.Sizeof(oa)
+
+	var ntPath unicodeString
+	if err := rtlDosPathNameToNtPathName(&path16[0], &ntPath, 0, 0).Err(); err != nil {
+		return 0, &os.PathError{Op: "open", Path: path, Err: err}
+	}
+	defer localFree(ntPath.Buffer)
+	oa.ObjectName = &ntPath
+
+	// The security descriptor is only needed for the first pipe.
+	if first {
+		if sd != nil {
+			len := uint32(len(sd))
+			sdb := localAlloc(0, len)
+			defer localFree(sdb)
+			copy((*[0xffff]byte)(unsafe.Pointer(sdb))[:], sd)
+			oa.SecurityDescriptor = (*securityDescriptor)(unsafe.Pointer(sdb))
+		} else {
+			// Construct the default named pipe security descriptor.
+			var dacl uintptr
+			if err := rtlDefaultNpAcl(&dacl).Err(); err != nil {
+				return 0, fmt.Errorf("getting default named pipe ACL: %s", err)
+			}
+			defer localFree(dacl)
+
+			sdb := &securityDescriptor{
+				Revision: 1,
+				Control:  cSE_DACL_PRESENT,
+				Dacl:     dacl,
+			}
+			oa.SecurityDescriptor = sdb
+		}
+	}
+
+	typ := uint32(cFILE_PIPE_REJECT_REMOTE_CLIENTS)
+	if c.MessageMode {
+		typ |= cFILE_PIPE_MESSAGE_TYPE
+	}
+
+	disposition := uint32(cFILE_OPEN)
+	access := uint32(syscall.GENERIC_READ | syscall.GENERIC_WRITE | syscall.SYNCHRONIZE)
+	if first {
+		disposition = cFILE_CREATE
+		// By not asking for read or write access, the named pipe file system
+		// will put this pipe into an initially disconnected state, blocking
+		// client connections until the next call with first == false.
+		access = syscall.SYNCHRONIZE
+	}
+
+	timeout := int64(-50 * 10000) // 50ms
+
+	var (
+		h    syscall.Handle
+		iosb ioStatusBlock
+	)
+	err = ntCreateNamedPipeFile(&h, access, &oa, &iosb, syscall.FILE_SHARE_READ|syscall.FILE_SHARE_WRITE, disposition, 0, typ, 0, 0, 0xffffffff, uint32(c.InputBufferSize), uint32(c.OutputBufferSize), &timeout).Err()
+	if err != nil {
+		return 0, &os.PathError{Op: "open", Path: path, Err: err}
+	}
+
+	runtime.KeepAlive(ntPath)
+	return h, nil
+}
+
+func (l *win32PipeListener) makeServerPipe() (*win32File, error) {
+	h, err := makeServerPipeHandle(l.path, nil, &l.config, false)
+	if err != nil {
+		return nil, err
+	}
+	f, err := makeWin32File(h)
+	if err != nil {
+		syscall.Close(h)
+		return nil, err
+	}
+	return f, nil
+}
+
+func (l *win32PipeListener) makeConnectedServerPipe() (*win32File, error) {
+	p, err := l.makeServerPipe()
+	if err != nil {
+		return nil, err
+	}
+
+	// Wait for the client to connect.
+	ch := make(chan error)
+	go func(p *win32File) {
+		ch <- connectPipe(p)
+	}(p)
+
+	select {
+	case err = <-ch:
+		if err != nil {
+			p.Close()
+			p = nil
+		}
+	case <-l.closeCh:
+		// Abort the connect request by closing the handle.
+		p.Close()
+		p = nil
+		err = <-ch
+		if err == nil || err == ErrFileClosed {
+			err = ErrPipeListenerClosed
+		}
+	}
+	return p, err
+}
+
+func (l *win32PipeListener) listenerRoutine() {
+	closed := false
+	for !closed {
+		select {
+		case <-l.closeCh:
+			closed = true
+		case responseCh := <-l.acceptCh:
+			var (
+				p   *win32File
+				err error
+			)
+			for {
+				p, err = l.makeConnectedServerPipe()
+				// If the connection was immediately closed by the client, try
+				// again.
+				if err != cERROR_NO_DATA {
+					break
+				}
+			}
+			responseCh <- acceptResponse{p, err}
+			closed = err == ErrPipeListenerClosed
+		}
+	}
+	syscall.Close(l.firstHandle)
+	l.firstHandle = 0
+	// Notify Close() and Accept() callers that the handle has been closed.
+	close(l.doneCh)
+}
+
+// PipeConfig contain configuration for the pipe listener.
+type PipeConfig struct {
+	// SecurityDescriptor contains a Windows security descriptor in SDDL format.
+	SecurityDescriptor string
+
+	// MessageMode determines whether the pipe is in byte or message mode. In either
+	// case the pipe is read in byte mode by default. The only practical difference in
+	// this implementation is that CloseWrite() is only supported for message mode pipes;
+	// CloseWrite() is implemented as a zero-byte write, but zero-byte writes are only
+	// transferred to the reader (and returned as io.EOF in this implementation)
+	// when the pipe is in message mode.
+	MessageMode bool
+
+	// InputBufferSize specifies the size the input buffer, in bytes.
+	InputBufferSize int32
+
+	// OutputBufferSize specifies the size the input buffer, in bytes.
+	OutputBufferSize int32
+}
+
+// ListenPipe creates a listener on a Windows named pipe path, e.g. \\.\pipe\mypipe.
+// The pipe must not already exist.
+func ListenPipe(path string, c *PipeConfig) (net.Listener, error) {
+	var (
+		sd  []byte
+		err error
+	)
+	if c == nil {
+		c = &PipeConfig{}
+	}
+	if c.SecurityDescriptor != "" {
+		sd, err = SddlToSecurityDescriptor(c.SecurityDescriptor)
+		if err != nil {
+			return nil, err
+		}
+	}
+	h, err := makeServerPipeHandle(path, sd, c, true)
+	if err != nil {
+		return nil, err
+	}
+	l := &win32PipeListener{
+		firstHandle: h,
+		path:        path,
+		config:      *c,
+		acceptCh:    make(chan (chan acceptResponse)),
+		closeCh:     make(chan int),
+		doneCh:      make(chan int),
+	}
+	go l.listenerRoutine()
+	return l, nil
+}
+
+func connectPipe(p *win32File) error {
+	c, err := p.prepareIo()
+	if err != nil {
+		return err
+	}
+	defer p.wg.Done()
+
+	err = connectNamedPipe(p.handle, &c.o)
+	_, err = p.asyncIo(c, nil, 0, err)
+	if err != nil && err != cERROR_PIPE_CONNECTED {
+		return err
+	}
+	return nil
+}
+
+func (l *win32PipeListener) Accept() (net.Conn, error) {
+	ch := make(chan acceptResponse)
+	select {
+	case l.acceptCh <- ch:
+		response := <-ch
+		err := response.err
+		if err != nil {
+			return nil, err
+		}
+		if l.config.MessageMode {
+			return &win32MessageBytePipe{
+				win32Pipe: win32Pipe{win32File: response.f, path: l.path},
+			}, nil
+		}
+		return &win32Pipe{win32File: response.f, path: l.path}, nil
+	case <-l.doneCh:
+		return nil, ErrPipeListenerClosed
+	}
+}
+
+func (l *win32PipeListener) Close() error {
+	select {
+	case l.closeCh <- 1:
+		<-l.doneCh
+	case <-l.doneCh:
+	}
+	return nil
+}
+
+func (l *win32PipeListener) Addr() net.Addr {
+	return pipeAddress(l.path)
+}

vendor/github.com/Microsoft/go-winio/pkg/guid/guid.go

@@ -0,0 +1,235 @@
+// Package guid provides a GUID type. The backing structure for a GUID is
+// identical to that used by the golang.org/x/sys/windows GUID type.
+// There are two main binary encodings used for a GUID, the big-endian encoding,
+// and the Windows (mixed-endian) encoding. See here for details:
+// https://en.wikipedia.org/wiki/Universally_unique_identifier#Encoding
+package guid
+
+import (
+	"crypto/rand"
+	"crypto/sha1"
+	"encoding"
+	"encoding/binary"
+	"fmt"
+	"strconv"
+
+	"golang.org/x/sys/windows"
+)
+
+// Variant specifies which GUID variant (or "type") of the GUID. It determines
+// how the entirety of the rest of the GUID is interpreted.
+type Variant uint8
+
+// The variants specified by RFC 4122.
+const (
+	// VariantUnknown specifies a GUID variant which does not conform to one of
+	// the variant encodings specified in RFC 4122.
+	VariantUnknown Variant = iota
+	VariantNCS
+	VariantRFC4122
+	VariantMicrosoft
+	VariantFuture
+)
+
+// Version specifies how the bits in the GUID were generated. For instance, a
+// version 4 GUID is randomly generated, and a version 5 is generated from the
+// hash of an input string.
+type Version uint8
+
+var _ = (encoding.TextMarshaler)(GUID{})
+var _ = (encoding.TextUnmarshaler)(&GUID{})
+
+// GUID represents a GUID/UUID. It has the same structure as
+// golang.org/x/sys/windows.GUID so that it can be used with functions expecting
+// that type. It is defined as its own type so that stringification and
+// marshaling can be supported. The representation matches that used by native
+// Windows code.
+type GUID windows.GUID
+
+// NewV4 returns a new version 4 (pseudorandom) GUID, as defined by RFC 4122.
+func NewV4() (GUID, error) {
+	var b [16]byte
+	if _, err := rand.Read(b[:]); err != nil {
+		return GUID{}, err
+	}
+
+	g := FromArray(b)
+	g.setVersion(4) // Version 4 means randomly generated.
+	g.setVariant(VariantRFC4122)
+
+	return g, nil
+}
+
+// NewV5 returns a new version 5 (generated from a string via SHA-1 hashing)
+// GUID, as defined by RFC 4122. The RFC is unclear on the encoding of the name,
+// and the sample code treats it as a series of bytes, so we do the same here.
+//
+// Some implementations, such as those found on Windows, treat the name as a
+// big-endian UTF16 stream of bytes. If that is desired, the string can be
+// encoded as such before being passed to this function.
+func NewV5(namespace GUID, name []byte) (GUID, error) {
+	b := sha1.New()
+	namespaceBytes := namespace.ToArray()
+	b.Write(namespaceBytes[:])
+	b.Write(name)
+
+	a := [16]byte{}
+	copy(a[:], b.Sum(nil))
+
+	g := FromArray(a)
+	g.setVersion(5) // Version 5 means generated from a string.
+	g.setVariant(VariantRFC4122)
+
+	return g, nil
+}
+
+func fromArray(b [16]byte, order binary.ByteOrder) GUID {
+	var g GUID
+	g.Data1 = order.Uint32(b[0:4])
+	g.Data2 = order.Uint16(b[4:6])
+	g.Data3 = order.Uint16(b[6:8])
+	copy(g.Data4[:], b[8:16])
+	return g
+}
+
+func (g GUID) toArray(order binary.ByteOrder) [16]byte {
+	b := [16]byte{}
+	order.PutUint32(b[0:4], g.Data1)
+	order.PutUint16(b[4:6], g.Data2)
+	order.PutUint16(b[6:8], g.Data3)
+	copy(b[8:16], g.Data4[:])
+	return b
+}
+
+// FromArray constructs a GUID from a big-endian encoding array of 16 bytes.
+func FromArray(b [16]byte) GUID {
+	return fromArray(b, binary.BigEndian)
+}
+
+// ToArray returns an array of 16 bytes representing the GUID in big-endian
+// encoding.
+func (g GUID) ToArray() [16]byte {
+	return g.toArray(binary.BigEndian)
+}
+
+// FromWindowsArray constructs a GUID from a Windows encoding array of bytes.
+func FromWindowsArray(b [16]byte) GUID {
+	return fromArray(b, binary.LittleEndian)
+}
+
+// ToWindowsArray returns an array of 16 bytes representing the GUID in Windows
+// encoding.
+func (g GUID) ToWindowsArray() [16]byte {
+	return g.toArray(binary.LittleEndian)
+}
+
+func (g GUID) String() string {
+	return fmt.Sprintf(
+		"%08x-%04x-%04x-%04x-%012x",
+		g.Data1,
+		g.Data2,
+		g.Data3,
+		g.Data4[:2],
+		g.Data4[2:])
+}
+
+// FromString parses a string containing a GUID and returns the GUID. The only
+// format currently supported is the `xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx`
+// format.
+func FromString(s string) (GUID, error) {
+	if len(s) != 36 {
+		return GUID{}, fmt.Errorf("invalid GUID %q", s)
+	}
+	if s[8] != '-' || s[13] != '-' || s[18] != '-' || s[23] != '-' {
+		return GUID{}, fmt.Errorf("invalid GUID %q", s)
+	}
+
+	var g GUID
+
+	data1, err := strconv.ParseUint(s[0:8], 16, 32)
+	if err != nil {
+		return GUID{}, fmt.Errorf("invalid GUID %q", s)
+	}
+	g.Data1 = uint32(data1)
+
+	data2, err := strconv.ParseUint(s[9:13], 16, 16)
+	if err != nil {
+		return GUID{}, fmt.Errorf("invalid GUID %q", s)
+	}
+	g.Data2 = uint16(data2)
+
+	data3, err := strconv.ParseUint(s[14:18], 16, 16)
+	if err != nil {
+		return GUID{}, fmt.Errorf("invalid GUID %q", s)
+	}
+	g.Data3 = uint16(data3)
+
+	for i, x := range []int{19, 21, 24, 26, 28, 30, 32, 34} {
+		v, err := strconv.ParseUint(s[x:x+2], 16, 8)
+		if err != nil {
+			return GUID{}, fmt.Errorf("invalid GUID %q", s)
+		}
+		g.Data4[i] = uint8(v)
+	}
+
+	return g, nil
+}
+
+func (g *GUID) setVariant(v Variant) {
+	d := g.Data4[0]
+	switch v {
+	case VariantNCS:
+		d = (d & 0x7f)
+	case VariantRFC4122:
+		d = (d & 0x3f) | 0x80
+	case VariantMicrosoft:
+		d = (d & 0x1f) | 0xc0
+	case VariantFuture:
+		d = (d & 0x0f) | 0xe0
+	case VariantUnknown:
+		fallthrough
+	default:
+		panic(fmt.Sprintf("invalid variant: %d", v))
+	}
+	g.Data4[0] = d
+}
+
+// Variant returns the GUID variant, as defined in RFC 4122.
+func (g GUID) Variant() Variant {
+	b := g.Data4[0]
+	if b&0x80 == 0 {
+		return VariantNCS
+	} else if b&0xc0 == 0x80 {
+		return VariantRFC4122
+	} else if b&0xe0 == 0xc0 {
+		return VariantMicrosoft
+	} else if b&0xe0 == 0xe0 {
+		return VariantFuture
+	}
+	return VariantUnknown
+}
+
+func (g *GUID) setVersion(v Version) {
+	g.Data3 = (g.Data3 & 0x0fff) | (uint16(v) << 12)
+}
+
+// Version returns the GUID version, as defined in RFC 4122.
+func (g GUID) Version() Version {
+	return Version((g.Data3 & 0xF000) >> 12)
+}
+
+// MarshalText returns the textual representation of the GUID.
+func (g GUID) MarshalText() ([]byte, error) {
+	return []byte(g.String()), nil
+}
+
+// UnmarshalText takes the textual representation of a GUID, and unmarhals it
+// into this GUID.
+func (g *GUID) UnmarshalText(text []byte) error {
+	g2, err := FromString(string(text))
+	if err != nil {
+		return err
+	}
+	*g = g2
+	return nil
+}

vendor/github.com/Microsoft/go-winio/privilege.go

@@ -0,0 +1,202 @@
+// +build windows
+
+package winio
+
+import (
+	"bytes"
+	"encoding/binary"
+	"fmt"
+	"runtime"
+	"sync"
+	"syscall"
+	"unicode/utf16"
+
+	"golang.org/x/sys/windows"
+)
+
+//sys adjustTokenPrivileges(token windows.Token, releaseAll bool, input *byte, outputSize uint32, output *byte, requiredSize *uint32) (success bool, err error) [true] = advapi32.AdjustTokenPrivileges
+//sys impersonateSelf(level uint32) (err error) = advapi32.ImpersonateSelf
+//sys revertToSelf() (err error) = advapi32.RevertToSelf
+//sys openThreadToken(thread syscall.Handle, accessMask uint32, openAsSelf bool, token *windows.Token) (err error) = advapi32.OpenThreadToken
+//sys getCurrentThread() (h syscall.Handle) = GetCurrentThread
+//sys lookupPrivilegeValue(systemName string, name string, luid *uint64) (err error) = advapi32.LookupPrivilegeValueW
+//sys lookupPrivilegeName(systemName string, luid *uint64, buffer *uint16, size *uint32) (err error) = advapi32.LookupPrivilegeNameW
+//sys lookupPrivilegeDisplayName(systemName string, name *uint16, buffer *uint16, size *uint32, languageId *uint32) (err error) = advapi32.LookupPrivilegeDisplayNameW
+
+const (
+	SE_PRIVILEGE_ENABLED = 2
+
+	ERROR_NOT_ALL_ASSIGNED syscall.Errno = 1300
+
+	SeBackupPrivilege  = "SeBackupPrivilege"
+	SeRestorePrivilege = "SeRestorePrivilege"
+)
+
+const (
+	securityAnonymous = iota
+	securityIdentification
+	securityImpersonation
+	securityDelegation
+)
+
+var (
+	privNames     = make(map[string]uint64)
+	privNameMutex sync.Mutex
+)
+
+// PrivilegeError represents an error enabling privileges.
+type PrivilegeError struct {
+	privileges []uint64
+}
+
+func (e *PrivilegeError) Error() string {
+	s := ""
+	if len(e.privileges) > 1 {
+		s = "Could not enable privileges "
+	} else {
+		s = "Could not enable privilege "
+	}
+	for i, p := range e.privileges {
+		if i != 0 {
+			s += ", "
+		}
+		s += `"`
+		s += getPrivilegeName(p)
+		s += `"`
+	}
+	return s
+}
+
+// RunWithPrivilege enables a single privilege for a function call.
+func RunWithPrivilege(name string, fn func() error) error {
+	return RunWithPrivileges([]string{name}, fn)
+}
+
+// RunWithPrivileges enables privileges for a function call.
+func RunWithPrivileges(names []string, fn func() error) error {
+	privileges, err := mapPrivileges(names)
+	if err != nil {
+		return err
+	}
+	runtime.LockOSThread()
+	defer runtime.UnlockOSThread()
+	token, err := newThreadToken()
+	if err != nil {
+		return err
+	}
+	defer releaseThreadToken(token)
+	err = adjustPrivileges(token, privileges, SE_PRIVILEGE_ENABLED)
+	if err != nil {
+		return err
+	}
+	return fn()
+}
+
+func mapPrivileges(names []string) ([]uint64, error) {
+	var privileges []uint64
+	privNameMutex.Lock()
+	defer privNameMutex.Unlock()
+	for _, name := range names {
+		p, ok := privNames[name]
+		if !ok {
+			err := lookupPrivilegeValue("", name, &p)
+			if err != nil {
+				return nil, err
+			}
+			privNames[name] = p
+		}
+		privileges = append(privileges, p)
+	}
+	return privileges, nil
+}
+
+// EnableProcessPrivileges enables privileges globally for the process.
+func EnableProcessPrivileges(names []string) error {
+	return enableDisableProcessPrivilege(names, SE_PRIVILEGE_ENABLED)
+}
+
+// DisableProcessPrivileges disables privileges globally for the process.
+func DisableProcessPrivileges(names []string) error {
+	return enableDisableProcessPrivilege(names, 0)
+}
+
+func enableDisableProcessPrivilege(names []string, action uint32) error {
+	privileges, err := mapPrivileges(names)
+	if err != nil {
+		return err
+	}
+
+	p, _ := windows.GetCurrentProcess()
+	var token windows.Token
+	err = windows.OpenProcessToken(p, windows.TOKEN_ADJUST_PRIVILEGES|windows.TOKEN_QUERY, &token)
+	if err != nil {
+		return err
+	}
+
+	defer token.Close()
+	return adjustPrivileges(token, privileges, action)
+}
+
+func adjustPrivileges(token windows.Token, privileges []uint64, action uint32) error {
+	var b bytes.Buffer
+	binary.Write(&b, binary.LittleEndian, uint32(len(privileges)))
+	for _, p := range privileges {
+		binary.Write(&b, binary.LittleEndian, p)
+		binary.Write(&b, binary.LittleEndian, action)
+	}
+	prevState := make([]byte, b.Len())
+	reqSize := uint32(0)
+	success, err := adjustTokenPrivileges(token, false, &b.Bytes()[0], uint32(len(prevState)), &prevState[0], &reqSize)
+	if !success {
+		return err
+	}
+	if err == ERROR_NOT_ALL_ASSIGNED {
+		return &PrivilegeError{privileges}
+	}
+	return nil
+}
+
+func getPrivilegeName(luid uint64) string {
+	var nameBuffer [256]uint16
+	bufSize := uint32(len(nameBuffer))
+	err := lookupPrivilegeName("", &luid, &nameBuffer[0], &bufSize)
+	if err != nil {
+		return fmt.Sprintf("<unknown privilege %d>", luid)
+	}
+
+	var displayNameBuffer [256]uint16
+	displayBufSize := uint32(len(displayNameBuffer))
+	var langID uint32
+	err = lookupPrivilegeDisplayName("", &nameBuffer[0], &displayNameBuffer[0], &displayBufSize, &langID)
+	if err != nil {
+		return fmt.Sprintf("<unknown privilege %s>", string(utf16.Decode(nameBuffer[:bufSize])))
+	}
+
+	return string(utf16.Decode(displayNameBuffer[:displayBufSize]))
+}
+
+func newThreadToken() (windows.Token, error) {
+	err := impersonateSelf(securityImpersonation)
+	if err != nil {
+		return 0, err
+	}
+
+	var token windows.Token
+	err = openThreadToken(getCurrentThread(), syscall.TOKEN_ADJUST_PRIVILEGES|syscall.TOKEN_QUERY, false, &token)
+	if err != nil {
+		rerr := revertToSelf()
+		if rerr != nil {
+			panic(rerr)
+		}
+		return 0, err
+	}
+	return token, nil
+}
+
+func releaseThreadToken(h windows.Token) {
+	err := revertToSelf()
+	if err != nil {
+		panic(err)
+	}
+	h.Close()
+}

vendor/github.com/Microsoft/go-winio/reparse.go

@@ -0,0 +1,128 @@
+package winio
+
+import (
+	"bytes"
+	"encoding/binary"
+	"fmt"
+	"strings"
+	"unicode/utf16"
+	"unsafe"
+)
+
+const (
+	reparseTagMountPoint = 0xA0000003
+	reparseTagSymlink    = 0xA000000C
+)
+
+type reparseDataBuffer struct {
+	ReparseTag           uint32
+	ReparseDataLength    uint16
+	Reserved             uint16
+	SubstituteNameOffset uint16
+	SubstituteNameLength uint16
+	PrintNameOffset      uint16
+	PrintNameLength      uint16
+}
+
+// ReparsePoint describes a Win32 symlink or mount point.
+type ReparsePoint struct {
+	Target       string
+	IsMountPoint bool
+}
+
+// UnsupportedReparsePointError is returned when trying to decode a non-symlink or
+// mount point reparse point.
+type UnsupportedReparsePointError struct {
+	Tag uint32
+}
+
+func (e *UnsupportedReparsePointError) Error() string {
+	return fmt.Sprintf("unsupported reparse point %x", e.Tag)
+}
+
+// DecodeReparsePoint decodes a Win32 REPARSE_DATA_BUFFER structure containing either a symlink
+// or a mount point.
+func DecodeReparsePoint(b []byte) (*ReparsePoint, error) {
+	tag := binary.LittleEndian.Uint32(b[0:4])
+	return DecodeReparsePointData(tag, b[8:])
+}
+
+func DecodeReparsePointData(tag uint32, b []byte) (*ReparsePoint, error) {
+	isMountPoint := false
+	switch tag {
+	case reparseTagMountPoint:
+		isMountPoint = true
+	case reparseTagSymlink:
+	default:
+		return nil, &UnsupportedReparsePointError{tag}
+	}
+	nameOffset := 8 + binary.LittleEndian.Uint16(b[4:6])
+	if !isMountPoint {
+		nameOffset += 4
+	}
+	nameLength := binary.LittleEndian.Uint16(b[6:8])
+	name := make([]uint16, nameLength/2)
+	err := binary.Read(bytes.NewReader(b[nameOffset:nameOffset+nameLength]), binary.LittleEndian, &name)
+	if err != nil {
+		return nil, err
+	}
+	return &ReparsePoint{string(utf16.Decode(name)), isMountPoint}, nil
+}
+
+func isDriveLetter(c byte) bool {
+	return (c >= 'a' && c <= 'z') || (c >= 'A' && c <= 'Z')
+}
+
+// EncodeReparsePoint encodes a Win32 REPARSE_DATA_BUFFER structure describing a symlink or
+// mount point.
+func EncodeReparsePoint(rp *ReparsePoint) []byte {
+	// Generate an NT path and determine if this is a relative path.
+	var ntTarget string
+	relative := false
+	if strings.HasPrefix(rp.Target, `\\?\`) {
+		ntTarget = `\??\` + rp.Target[4:]
+	} else if strings.HasPrefix(rp.Target, `\\`) {
+		ntTarget = `\??\UNC\` + rp.Target[2:]
+	} else if len(rp.Target) >= 2 && isDriveLetter(rp.Target[0]) && rp.Target[1] == ':' {
+		ntTarget = `\??\` + rp.Target
+	} else {
+		ntTarget = rp.Target
+		relative = true
+	}
+
+	// The paths must be NUL-terminated even though they are counted strings.
+	target16 := utf16.Encode([]rune(rp.Target + "\x00"))
+	ntTarget16 := utf16.Encode([]rune(ntTarget + "\x00"))
+
+	size := int(unsafe.Sizeof(reparseDataBuffer{})) - 8
+	size += len(ntTarget16)*2 + len(target16)*2
+
+	tag := uint32(reparseTagMountPoint)
+	if !rp.IsMountPoint {
+		tag = reparseTagSymlink
+		size += 4 // Add room for symlink flags
+	}
+
+	data := reparseDataBuffer{
+		ReparseTag:           tag,
+		ReparseDataLength:    uint16(size),
+		SubstituteNameOffset: 0,
+		SubstituteNameLength: uint16((len(ntTarget16) - 1) * 2),
+		PrintNameOffset:      uint16(len(ntTarget16) * 2),
+		PrintNameLength:      uint16((len(target16) - 1) * 2),
+	}
+
+	var b bytes.Buffer
+	binary.Write(&b, binary.LittleEndian, &data)
+	if !rp.IsMountPoint {
+		flags := uint32(0)
+		if relative {
+			flags |= 1
+		}
+		binary.Write(&b, binary.LittleEndian, flags)
+	}
+
+	binary.Write(&b, binary.LittleEndian, ntTarget16)
+	binary.Write(&b, binary.LittleEndian, target16)
+	return b.Bytes()
+}

vendor/github.com/Microsoft/go-winio/sd.go

@@ -0,0 +1,98 @@
+// +build windows
+
+package winio
+
+import (
+	"syscall"
+	"unsafe"
+)
+
+//sys lookupAccountName(systemName *uint16, accountName string, sid *byte, sidSize *uint32, refDomain *uint16, refDomainSize *uint32, sidNameUse *uint32) (err error) = advapi32.LookupAccountNameW
+//sys convertSidToStringSid(sid *byte, str **uint16) (err error) = advapi32.ConvertSidToStringSidW
+//sys convertStringSecurityDescriptorToSecurityDescriptor(str string, revision uint32, sd *uintptr, size *uint32) (err error) = advapi32.ConvertStringSecurityDescriptorToSecurityDescriptorW
+//sys convertSecurityDescriptorToStringSecurityDescriptor(sd *byte, revision uint32, secInfo uint32, sddl **uint16, sddlSize *uint32) (err error) = advapi32.ConvertSecurityDescriptorToStringSecurityDescriptorW
+//sys localFree(mem uintptr) = LocalFree
+//sys getSecurityDescriptorLength(sd uintptr) (len uint32) = advapi32.GetSecurityDescriptorLength
+
+const (
+	cERROR_NONE_MAPPED = syscall.Errno(1332)
+)
+
+type AccountLookupError struct {
+	Name string
+	Err  error
+}
+
+func (e *AccountLookupError) Error() string {
+	if e.Name == "" {
+		return "lookup account: empty account name specified"
+	}
+	var s string
+	switch e.Err {
+	case cERROR_NONE_MAPPED:
+		s = "not found"
+	default:
+		s = e.Err.Error()
+	}
+	return "lookup account " + e.Name + ": " + s
+}
+
+type SddlConversionError struct {
+	Sddl string
+	Err  error
+}
+
+func (e *SddlConversionError) Error() string {
+	return "convert " + e.Sddl + ": " + e.Err.Error()
+}
+
+// LookupSidByName looks up the SID of an account by name
+func LookupSidByName(name string) (sid string, err error) {
+	if name == "" {
+		return "", &AccountLookupError{name, cERROR_NONE_MAPPED}
+	}
+
+	var sidSize, sidNameUse, refDomainSize uint32
+	err = lookupAccountName(nil, name, nil, &sidSize, nil, &refDomainSize, &sidNameUse)
+	if err != nil && err != syscall.ERROR_INSUFFICIENT_BUFFER {
+		return "", &AccountLookupError{name, err}
+	}
+	sidBuffer := make([]byte, sidSize)
+	refDomainBuffer := make([]uint16, refDomainSize)
+	err = lookupAccountName(nil, name, &sidBuffer[0], &sidSize, &refDomainBuffer[0], &refDomainSize, &sidNameUse)
+	if err != nil {
+		return "", &AccountLookupError{name, err}
+	}
+	var strBuffer *uint16
+	err = convertSidToStringSid(&sidBuffer[0], &strBuffer)
+	if err != nil {
+		return "", &AccountLookupError{name, err}
+	}
+	sid = syscall.UTF16ToString((*[0xffff]uint16)(unsafe.Pointer(strBuffer))[:])
+	localFree(uintptr(unsafe.Pointer(strBuffer)))
+	return sid, nil
+}
+
+func SddlToSecurityDescriptor(sddl string) ([]byte, error) {
+	var sdBuffer uintptr
+	err := convertStringSecurityDescriptorToSecurityDescriptor(sddl, 1, &sdBuffer, nil)
+	if err != nil {
+		return nil, &SddlConversionError{sddl, err}
+	}
+	defer localFree(sdBuffer)
+	sd := make([]byte, getSecurityDescriptorLength(sdBuffer))
+	copy(sd, (*[0xffff]byte)(unsafe.Pointer(sdBuffer))[:len(sd)])
+	return sd, nil
+}
+
+func SecurityDescriptorToSddl(sd []byte) (string, error) {
+	var sddl *uint16
+	// The returned string length seems to including an aribtrary number of terminating NULs.
+	// Don't use it.
+	err := convertSecurityDescriptorToStringSecurityDescriptor(&sd[0], 1, 0xff, &sddl, nil)
+	if err != nil {
+		return "", err
+	}
+	defer localFree(uintptr(unsafe.Pointer(sddl)))
+	return syscall.UTF16ToString((*[0xffff]uint16)(unsafe.Pointer(sddl))[:]), nil
+}

vendor/github.com/Microsoft/go-winio/syscall.go

@@ -0,0 +1,3 @@
+package winio
+
+//go:generate go run $GOROOT/src/syscall/mksyscall_windows.go -output zsyscall_windows.go file.go pipe.go sd.go fileinfo.go privilege.go backup.go hvsock.go

vendor/github.com/Microsoft/go-winio/zsyscall_windows.go

@@ -0,0 +1,562 @@
+// Code generated by 'go generate'; DO NOT EDIT.
+
+package winio
+
+import (
+	"syscall"
+	"unsafe"
+
+	"golang.org/x/sys/windows"
+)
+
+var _ unsafe.Pointer
+
+// Do the interface allocations only once for common
+// Errno values.
+const (
+	errnoERROR_IO_PENDING = 997
+)
+
+var (
+	errERROR_IO_PENDING error = syscall.Errno(errnoERROR_IO_PENDING)
+)
+
+// errnoErr returns common boxed Errno values, to prevent
+// allocations at runtime.
+func errnoErr(e syscall.Errno) error {
+	switch e {
+	case 0:
+		return nil
+	case errnoERROR_IO_PENDING:
+		return errERROR_IO_PENDING
+	}
+	// TODO: add more here, after collecting data on the common
+	// error values see on Windows. (perhaps when running
+	// all.bat?)
+	return e
+}
+
+var (
+	modkernel32 = windows.NewLazySystemDLL("kernel32.dll")
+	modws2_32   = windows.NewLazySystemDLL("ws2_32.dll")
+	modntdll    = windows.NewLazySystemDLL("ntdll.dll")
+	modadvapi32 = windows.NewLazySystemDLL("advapi32.dll")
+
+	procCancelIoEx                                           = modkernel32.NewProc("CancelIoEx")
+	procCreateIoCompletionPort                               = modkernel32.NewProc("CreateIoCompletionPort")
+	procGetQueuedCompletionStatus                            = modkernel32.NewProc("GetQueuedCompletionStatus")
+	procSetFileCompletionNotificationModes                   = modkernel32.NewProc("SetFileCompletionNotificationModes")
+	procWSAGetOverlappedResult                               = modws2_32.NewProc("WSAGetOverlappedResult")
+	procConnectNamedPipe                                     = modkernel32.NewProc("ConnectNamedPipe")
+	procCreateNamedPipeW                                     = modkernel32.NewProc("CreateNamedPipeW")
+	procCreateFileW                                          = modkernel32.NewProc("CreateFileW")
+	procGetNamedPipeInfo                                     = modkernel32.NewProc("GetNamedPipeInfo")
+	procGetNamedPipeHandleStateW                             = modkernel32.NewProc("GetNamedPipeHandleStateW")
+	procLocalAlloc                                           = modkernel32.NewProc("LocalAlloc")
+	procNtCreateNamedPipeFile                                = modntdll.NewProc("NtCreateNamedPipeFile")
+	procRtlNtStatusToDosErrorNoTeb                           = modntdll.NewProc("RtlNtStatusToDosErrorNoTeb")
+	procRtlDosPathNameToNtPathName_U                         = modntdll.NewProc("RtlDosPathNameToNtPathName_U")
+	procRtlDefaultNpAcl                                      = modntdll.NewProc("RtlDefaultNpAcl")
+	procLookupAccountNameW                                   = modadvapi32.NewProc("LookupAccountNameW")
+	procConvertSidToStringSidW                               = modadvapi32.NewProc("ConvertSidToStringSidW")
+	procConvertStringSecurityDescriptorToSecurityDescriptorW = modadvapi32.NewProc("ConvertStringSecurityDescriptorToSecurityDescriptorW")
+	procConvertSecurityDescriptorToStringSecurityDescriptorW = modadvapi32.NewProc("ConvertSecurityDescriptorToStringSecurityDescriptorW")
+	procLocalFree                                            = modkernel32.NewProc("LocalFree")
+	procGetSecurityDescriptorLength                          = modadvapi32.NewProc("GetSecurityDescriptorLength")
+	procGetFileInformationByHandleEx                         = modkernel32.NewProc("GetFileInformationByHandleEx")
+	procSetFileInformationByHandle                           = modkernel32.NewProc("SetFileInformationByHandle")
+	procAdjustTokenPrivileges                                = modadvapi32.NewProc("AdjustTokenPrivileges")
+	procImpersonateSelf                                      = modadvapi32.NewProc("ImpersonateSelf")
+	procRevertToSelf                                         = modadvapi32.NewProc("RevertToSelf")
+	procOpenThreadToken                                      = modadvapi32.NewProc("OpenThreadToken")
+	procGetCurrentThread                                     = modkernel32.NewProc("GetCurrentThread")
+	procLookupPrivilegeValueW                                = modadvapi32.NewProc("LookupPrivilegeValueW")
+	procLookupPrivilegeNameW                                 = modadvapi32.NewProc("LookupPrivilegeNameW")
+	procLookupPrivilegeDisplayNameW                          = modadvapi32.NewProc("LookupPrivilegeDisplayNameW")
+	procBackupRead                                           = modkernel32.NewProc("BackupRead")
+	procBackupWrite                                          = modkernel32.NewProc("BackupWrite")
+	procbind                                                 = modws2_32.NewProc("bind")
+)
+
+func cancelIoEx(file syscall.Handle, o *syscall.Overlapped) (err error) {
+	r1, _, e1 := syscall.Syscall(procCancelIoEx.Addr(), 2, uintptr(file), uintptr(unsafe.Pointer(o)), 0)
+	if r1 == 0 {
+		if e1 != 0 {
+			err = errnoErr(e1)
+		} else {
+			err = syscall.EINVAL
+		}
+	}
+	return
+}
+
+func createIoCompletionPort(file syscall.Handle, port syscall.Handle, key uintptr, threadCount uint32) (newport syscall.Handle, err error) {
+	r0, _, e1 := syscall.Syscall6(procCreateIoCompletionPort.Addr(), 4, uintptr(file), uintptr(port), uintptr(key), uintptr(threadCount), 0, 0)
+	newport = syscall.Handle(r0)
+	if newport == 0 {
+		if e1 != 0 {
+			err = errnoErr(e1)
+		} else {
+			err = syscall.EINVAL
+		}
+	}
+	return
+}
+
+func getQueuedCompletionStatus(port syscall.Handle, bytes *uint32, key *uintptr, o **ioOperation, timeout uint32) (err error) {
+	r1, _, e1 := syscall.Syscall6(procGetQueuedCompletionStatus.Addr(), 5, uintptr(port), uintptr(unsafe.Pointer(bytes)), uintptr(unsafe.Pointer(key)), uintptr(unsafe.Pointer(o)), uintptr(timeout), 0)
+	if r1 == 0 {
+		if e1 != 0 {
+			err = errnoErr(e1)
+		} else {
+			err = syscall.EINVAL
+		}
+	}
+	return
+}
+
+func setFileCompletionNotificationModes(h syscall.Handle, flags uint8) (err error) {
+	r1, _, e1 := syscall.Syscall(procSetFileCompletionNotificationModes.Addr(), 2, uintptr(h), uintptr(flags), 0)
+	if r1 == 0 {
+		if e1 != 0 {
+			err = errnoErr(e1)
+		} else {
+			err = syscall.EINVAL
+		}
+	}
+	return
+}
+
+func wsaGetOverlappedResult(h syscall.Handle, o *syscall.Overlapped, bytes *uint32, wait bool, flags *uint32) (err error) {
+	var _p0 uint32
+	if wait {
+		_p0 = 1
+	} else {
+		_p0 = 0
+	}
+	r1, _, e1 := syscall.Syscall6(procWSAGetOverlappedResult.Addr(), 5, uintptr(h), uintptr(unsafe.Pointer(o)), uintptr(unsafe.Pointer(bytes)), uintptr(_p0), uintptr(unsafe.Pointer(flags)), 0)
+	if r1 == 0 {
+		if e1 != 0 {
+			err = errnoErr(e1)
+		} else {
+			err = syscall.EINVAL
+		}
+	}
+	return
+}
+
+func connectNamedPipe(pipe syscall.Handle, o *syscall.Overlapped) (err error) {
+	r1, _, e1 := syscall.Syscall(procConnectNamedPipe.Addr(), 2, uintptr(pipe), uintptr(unsafe.Pointer(o)), 0)
+	if r1 == 0 {
+		if e1 != 0 {
+			err = errnoErr(e1)
+		} else {
+			err = syscall.EINVAL
+		}
+	}
+	return
+}
+
+func createNamedPipe(name string, flags uint32, pipeMode uint32, maxInstances uint32, outSize uint32, inSize uint32, defaultTimeout uint32, sa *syscall.SecurityAttributes) (handle syscall.Handle, err error) {
+	var _p0 *uint16
+	_p0, err = syscall.UTF16PtrFromString(name)
+	if err != nil {
+		return
+	}
+	return _createNamedPipe(_p0, flags, pipeMode, maxInstances, outSize, inSize, defaultTimeout, sa)
+}
+
+func _createNamedPipe(name *uint16, flags uint32, pipeMode uint32, maxInstances uint32, outSize uint32, inSize uint32, defaultTimeout uint32, sa *syscall.SecurityAttributes) (handle syscall.Handle, err error) {
+	r0, _, e1 := syscall.Syscall9(procCreateNamedPipeW.Addr(), 8, uintptr(unsafe.Pointer(name)), uintptr(flags), uintptr(pipeMode), uintptr(maxInstances), uintptr(outSize), uintptr(inSize), uintptr(defaultTimeout), uintptr(unsafe.Pointer(sa)), 0)
+	handle = syscall.Handle(r0)
+	if handle == syscall.InvalidHandle {
+		if e1 != 0 {
+			err = errnoErr(e1)
+		} else {
+			err = syscall.EINVAL
+		}
+	}
+	return
+}
+
+func createFile(name string, access uint32, mode uint32, sa *syscall.SecurityAttributes, createmode uint32, attrs uint32, templatefile syscall.Handle) (handle syscall.Handle, err error) {
+	var _p0 *uint16
+	_p0, err = syscall.UTF16PtrFromString(name)
+	if err != nil {
+		return
+	}
+	return _createFile(_p0, access, mode, sa, createmode, attrs, templatefile)
+}
+
+func _createFile(name *uint16, access uint32, mode uint32, sa *syscall.SecurityAttributes, createmode uint32, attrs uint32, templatefile syscall.Handle) (handle syscall.Handle, err error) {
+	r0, _, e1 := syscall.Syscall9(procCreateFileW.Addr(), 7, uintptr(unsafe.Pointer(name)), uintptr(access), uintptr(mode), uintptr(unsafe.Pointer(sa)), uintptr(createmode), uintptr(attrs), uintptr(templatefile), 0, 0)
+	handle = syscall.Handle(r0)
+	if handle == syscall.InvalidHandle {
+		if e1 != 0 {
+			err = errnoErr(e1)
+		} else {
+			err = syscall.EINVAL
+		}
+	}
+	return
+}
+
+func getNamedPipeInfo(pipe syscall.Handle, flags *uint32, outSize *uint32, inSize *uint32, maxInstances *uint32) (err error) {
+	r1, _, e1 := syscall.Syscall6(procGetNamedPipeInfo.Addr(), 5, uintptr(pipe), uintptr(unsafe.Pointer(flags)), uintptr(unsafe.Pointer(outSize)), uintptr(unsafe.Pointer(inSize)), uintptr(unsafe.Pointer(maxInstances)), 0)
+	if r1 == 0 {
+		if e1 != 0 {
+			err = errnoErr(e1)
+		} else {
+			err = syscall.EINVAL
+		}
+	}
+	return
+}
+
+func getNamedPipeHandleState(pipe syscall.Handle, state *uint32, curInstances *uint32, maxCollectionCount *uint32, collectDataTimeout *uint32, userName *uint16, maxUserNameSize uint32) (err error) {
+	r1, _, e1 := syscall.Syscall9(procGetNamedPipeHandleStateW.Addr(), 7, uintptr(pipe), uintptr(unsafe.Pointer(state)), uintptr(unsafe.Pointer(curInstances)), uintptr(unsafe.Pointer(maxCollectionCount)), uintptr(unsafe.Pointer(collectDataTimeout)), uintptr(unsafe.Pointer(userName)), uintptr(maxUserNameSize), 0, 0)
+	if r1 == 0 {
+		if e1 != 0 {
+			err = errnoErr(e1)
+		} else {
+			err = syscall.EINVAL
+		}
+	}
+	return
+}
+
+func localAlloc(uFlags uint32, length uint32) (ptr uintptr) {
+	r0, _, _ := syscall.Syscall(procLocalAlloc.Addr(), 2, uintptr(uFlags), uintptr(length), 0)
+	ptr = uintptr(r0)
+	return
+}
+
+func ntCreateNamedPipeFile(pipe *syscall.Handle, access uint32, oa *objectAttributes, iosb *ioStatusBlock, share uint32, disposition uint32, options uint32, typ uint32, readMode uint32, completionMode uint32, maxInstances uint32, inboundQuota uint32, outputQuota uint32, timeout *int64) (status ntstatus) {
+	r0, _, _ := syscall.Syscall15(procNtCreateNamedPipeFile.Addr(), 14, uintptr(unsafe.Pointer(pipe)), uintptr(access), uintptr(unsafe.Pointer(oa)), uintptr(unsafe.Pointer(iosb)), uintptr(share), uintptr(disposition), uintptr(options), uintptr(typ), uintptr(readMode), uintptr(completionMode), uintptr(maxInstances), uintptr(inboundQuota), uintptr(outputQuota), uintptr(unsafe.Pointer(timeout)), 0)
+	status = ntstatus(r0)
+	return
+}
+
+func rtlNtStatusToDosError(status ntstatus) (winerr error) {
+	r0, _, _ := syscall.Syscall(procRtlNtStatusToDosErrorNoTeb.Addr(), 1, uintptr(status), 0, 0)
+	if r0 != 0 {
+		winerr = syscall.Errno(r0)
+	}
+	return
+}
+
+func rtlDosPathNameToNtPathName(name *uint16, ntName *unicodeString, filePart uintptr, reserved uintptr) (status ntstatus) {
+	r0, _, _ := syscall.Syscall6(procRtlDosPathNameToNtPathName_U.Addr(), 4, uintptr(unsafe.Pointer(name)), uintptr(unsafe.Pointer(ntName)), uintptr(filePart), uintptr(reserved), 0, 0)
+	status = ntstatus(r0)
+	return
+}
+
+func rtlDefaultNpAcl(dacl *uintptr) (status ntstatus) {
+	r0, _, _ := syscall.Syscall(procRtlDefaultNpAcl.Addr(), 1, uintptr(unsafe.Pointer(dacl)), 0, 0)
+	status = ntstatus(r0)
+	return
+}
+
+func lookupAccountName(systemName *uint16, accountName string, sid *byte, sidSize *uint32, refDomain *uint16, refDomainSize *uint32, sidNameUse *uint32) (err error) {
+	var _p0 *uint16
+	_p0, err = syscall.UTF16PtrFromString(accountName)
+	if err != nil {
+		return
+	}
+	return _lookupAccountName(systemName, _p0, sid, sidSize, refDomain, refDomainSize, sidNameUse)
+}
+
+func _lookupAccountName(systemName *uint16, accountName *uint16, sid *byte, sidSize *uint32, refDomain *uint16, refDomainSize *uint32, sidNameUse *uint32) (err error) {
+	r1, _, e1 := syscall.Syscall9(procLookupAccountNameW.Addr(), 7, uintptr(unsafe.Pointer(systemName)), uintptr(unsafe.Pointer(accountName)), uintptr(unsafe.Pointer(sid)), uintptr(unsafe.Pointer(sidSize)), uintptr(unsafe.Pointer(refDomain)), uintptr(unsafe.Pointer(refDomainSize)), uintptr(unsafe.Pointer(sidNameUse)), 0, 0)
+	if r1 == 0 {
+		if e1 != 0 {
+			err = errnoErr(e1)
+		} else {
+			err = syscall.EINVAL
+		}
+	}
+	return
+}
+
+func convertSidToStringSid(sid *byte, str **uint16) (err error) {
+	r1, _, e1 := syscall.Syscall(procConvertSidToStringSidW.Addr(), 2, uintptr(unsafe.Pointer(sid)), uintptr(unsafe.Pointer(str)), 0)
+	if r1 == 0 {
+		if e1 != 0 {
+			err = errnoErr(e1)
+		} else {
+			err = syscall.EINVAL
+		}
+	}
+	return
+}
+
+func convertStringSecurityDescriptorToSecurityDescriptor(str string, revision uint32, sd *uintptr, size *uint32) (err error) {
+	var _p0 *uint16
+	_p0, err = syscall.UTF16PtrFromString(str)
+	if err != nil {
+		return
+	}
+	return _convertStringSecurityDescriptorToSecurityDescriptor(_p0, revision, sd, size)
+}
+
+func _convertStringSecurityDescriptorToSecurityDescriptor(str *uint16, revision uint32, sd *uintptr, size *uint32) (err error) {
+	r1, _, e1 := syscall.Syscall6(procConvertStringSecurityDescriptorToSecurityDescriptorW.Addr(), 4, uintptr(unsafe.Pointer(str)), uintptr(revision), uintptr(unsafe.Pointer(sd)), uintptr(unsafe.Pointer(size)), 0, 0)
+	if r1 == 0 {
+		if e1 != 0 {
+			err = errnoErr(e1)
+		} else {
+			err = syscall.EINVAL
+		}
+	}
+	return
+}
+
+func convertSecurityDescriptorToStringSecurityDescriptor(sd *byte, revision uint32, secInfo uint32, sddl **uint16, sddlSize *uint32) (err error) {
+	r1, _, e1 := syscall.Syscall6(procConvertSecurityDescriptorToStringSecurityDescriptorW.Addr(), 5, uintptr(unsafe.Pointer(sd)), uintptr(revision), uintptr(secInfo), uintptr(unsafe.Pointer(sddl)), uintptr(unsafe.Pointer(sddlSize)), 0)
+	if r1 == 0 {
+		if e1 != 0 {
+			err = errnoErr(e1)
+		} else {
+			err = syscall.EINVAL
+		}
+	}
+	return
+}
+
+func localFree(mem uintptr) {
+	syscall.Syscall(procLocalFree.Addr(), 1, uintptr(mem), 0, 0)
+	return
+}
+
+func getSecurityDescriptorLength(sd uintptr) (len uint32) {
+	r0, _, _ := syscall.Syscall(procGetSecurityDescriptorLength.Addr(), 1, uintptr(sd), 0, 0)
+	len = uint32(r0)
+	return
+}
+
+func getFileInformationByHandleEx(h syscall.Handle, class uint32, buffer *byte, size uint32) (err error) {
+	r1, _, e1 := syscall.Syscall6(procGetFileInformationByHandleEx.Addr(), 4, uintptr(h), uintptr(class), uintptr(unsafe.Pointer(buffer)), uintptr(size), 0, 0)
+	if r1 == 0 {
+		if e1 != 0 {
+			err = errnoErr(e1)
+		} else {
+			err = syscall.EINVAL
+		}
+	}
+	return
+}
+
+func setFileInformationByHandle(h syscall.Handle, class uint32, buffer *byte, size uint32) (err error) {
+	r1, _, e1 := syscall.Syscall6(procSetFileInformationByHandle.Addr(), 4, uintptr(h), uintptr(class), uintptr(unsafe.Pointer(buffer)), uintptr(size), 0, 0)
+	if r1 == 0 {
+		if e1 != 0 {
+			err = errnoErr(e1)
+		} else {
+			err = syscall.EINVAL
+		}
+	}
+	return
+}
+
+func adjustTokenPrivileges(token windows.Token, releaseAll bool, input *byte, outputSize uint32, output *byte, requiredSize *uint32) (success bool, err error) {
+	var _p0 uint32
+	if releaseAll {
+		_p0 = 1
+	} else {
+		_p0 = 0
+	}
+	r0, _, e1 := syscall.Syscall6(procAdjustTokenPrivileges.Addr(), 6, uintptr(token), uintptr(_p0), uintptr(unsafe.Pointer(input)), uintptr(outputSize), uintptr(unsafe.Pointer(output)), uintptr(unsafe.Pointer(requiredSize)))
+	success = r0 != 0
+	if true {
+		if e1 != 0 {
+			err = errnoErr(e1)
+		} else {
+			err = syscall.EINVAL
+		}
+	}
+	return
+}
+
+func impersonateSelf(level uint32) (err error) {
+	r1, _, e1 := syscall.Syscall(procImpersonateSelf.Addr(), 1, uintptr(level), 0, 0)
+	if r1 == 0 {
+		if e1 != 0 {
+			err = errnoErr(e1)
+		} else {
+			err = syscall.EINVAL
+		}
+	}
+	return
+}
+
+func revertToSelf() (err error) {
+	r1, _, e1 := syscall.Syscall(procRevertToSelf.Addr(), 0, 0, 0, 0)
+	if r1 == 0 {
+		if e1 != 0 {
+			err = errnoErr(e1)
+		} else {
+			err = syscall.EINVAL
+		}
+	}
+	return
+}
+
+func openThreadToken(thread syscall.Handle, accessMask uint32, openAsSelf bool, token *windows.Token) (err error) {
+	var _p0 uint32
+	if openAsSelf {
+		_p0 = 1
+	} else {
+		_p0 = 0
+	}
+	r1, _, e1 := syscall.Syscall6(procOpenThreadToken.Addr(), 4, uintptr(thread), uintptr(accessMask), uintptr(_p0), uintptr(unsafe.Pointer(token)), 0, 0)
+	if r1 == 0 {
+		if e1 != 0 {
+			err = errnoErr(e1)
+		} else {
+			err = syscall.EINVAL
+		}
+	}
+	return
+}
+
+func getCurrentThread() (h syscall.Handle) {
+	r0, _, _ := syscall.Syscall(procGetCurrentThread.Addr(), 0, 0, 0, 0)
+	h = syscall.Handle(r0)
+	return
+}
+
+func lookupPrivilegeValue(systemName string, name string, luid *uint64) (err error) {
+	var _p0 *uint16
+	_p0, err = syscall.UTF16PtrFromString(systemName)
+	if err != nil {
+		return
+	}
+	var _p1 *uint16
+	_p1, err = syscall.UTF16PtrFromString(name)
+	if err != nil {
+		return
+	}
+	return _lookupPrivilegeValue(_p0, _p1, luid)
+}
+
+func _lookupPrivilegeValue(systemName *uint16, name *uint16, luid *uint64) (err error) {
+	r1, _, e1 := syscall.Syscall(procLookupPrivilegeValueW.Addr(), 3, uintptr(unsafe.Pointer(systemName)), uintptr(unsafe.Pointer(name)), uintptr(unsafe.Pointer(luid)))
+	if r1 == 0 {
+		if e1 != 0 {
+			err = errnoErr(e1)
+		} else {
+			err = syscall.EINVAL
+		}
+	}
+	return
+}
+
+func lookupPrivilegeName(systemName string, luid *uint64, buffer *uint16, size *uint32) (err error) {
+	var _p0 *uint16
+	_p0, err = syscall.UTF16PtrFromString(systemName)
+	if err != nil {
+		return
+	}
+	return _lookupPrivilegeName(_p0, luid, buffer, size)
+}
+
+func _lookupPrivilegeName(systemName *uint16, luid *uint64, buffer *uint16, size *uint32) (err error) {
+	r1, _, e1 := syscall.Syscall6(procLookupPrivilegeNameW.Addr(), 4, uintptr(unsafe.Pointer(systemName)), uintptr(unsafe.Pointer(luid)), uintptr(unsafe.Pointer(buffer)), uintptr(unsafe.Pointer(size)), 0, 0)
+	if r1 == 0 {
+		if e1 != 0 {
+			err = errnoErr(e1)
+		} else {
+			err = syscall.EINVAL
+		}
+	}
+	return
+}
+
+func lookupPrivilegeDisplayName(systemName string, name *uint16, buffer *uint16, size *uint32, languageId *uint32) (err error) {
+	var _p0 *uint16
+	_p0, err = syscall.UTF16PtrFromString(systemName)
+	if err != nil {
+		return
+	}
+	return _lookupPrivilegeDisplayName(_p0, name, buffer, size, languageId)
+}
+
+func _lookupPrivilegeDisplayName(systemName *uint16, name *uint16, buffer *uint16, size *uint32, languageId *uint32) (err error) {
+	r1, _, e1 := syscall.Syscall6(procLookupPrivilegeDisplayNameW.Addr(), 5, uintptr(unsafe.Pointer(systemName)), uintptr(unsafe.Pointer(name)), uintptr(unsafe.Pointer(buffer)), uintptr(unsafe.Pointer(size)), uintptr(unsafe.Pointer(languageId)), 0)
+	if r1 == 0 {
+		if e1 != 0 {
+			err = errnoErr(e1)
+		} else {
+			err = syscall.EINVAL
+		}
+	}
+	return
+}
+
+func backupRead(h syscall.Handle, b []byte, bytesRead *uint32, abort bool, processSecurity bool, context *uintptr) (err error) {
+	var _p0 *byte
+	if len(b) > 0 {
+		_p0 = &b[0]
+	}
+	var _p1 uint32
+	if abort {
+		_p1 = 1
+	} else {
+		_p1 = 0
+	}
+	var _p2 uint32
+	if processSecurity {
+		_p2 = 1
+	} else {
+		_p2 = 0
+	}
+	r1, _, e1 := syscall.Syscall9(procBackupRead.Addr(), 7, uintptr(h), uintptr(unsafe.Pointer(_p0)), uintptr(len(b)), uintptr(unsafe.Pointer(bytesRead)), uintptr(_p1), uintptr(_p2), uintptr(unsafe.Pointer(context)), 0, 0)
+	if r1 == 0 {
+		if e1 != 0 {
+			err = errnoErr(e1)
+		} else {
+			err = syscall.EINVAL
+		}
+	}
+	return
+}
+
+func backupWrite(h syscall.Handle, b []byte, bytesWritten *uint32, abort bool, processSecurity bool, context *uintptr) (err error) {
+	var _p0 *byte
+	if len(b) > 0 {
+		_p0 = &b[0]
+	}
+	var _p1 uint32
+	if abort {
+		_p1 = 1
+	} else {
+		_p1 = 0
+	}
+	var _p2 uint32
+	if processSecurity {
+		_p2 = 1
+	} else {
+		_p2 = 0
+	}
+	r1, _, e1 := syscall.Syscall9(procBackupWrite.Addr(), 7, uintptr(h), uintptr(unsafe.Pointer(_p0)), uintptr(len(b)), uintptr(unsafe.Pointer(bytesWritten)), uintptr(_p1), uintptr(_p2), uintptr(unsafe.Pointer(context)), 0, 0)
+	if r1 == 0 {
+		if e1 != 0 {
+			err = errnoErr(e1)
+		} else {
+			err = syscall.EINVAL
+		}
+	}
+	return
+}
+
+func bind(s syscall.Handle, name unsafe.Pointer, namelen int32) (err error) {
+	r1, _, e1 := syscall.Syscall(procbind.Addr(), 3, uintptr(s), uintptr(name), uintptr(namelen))
+	if r1 == socketError {
+		if e1 != 0 {
+			err = errnoErr(e1)
+		} else {
+			err = syscall.EINVAL
+		}
+	}
+	return
+}

vendor/github.com/containernetworking/cni/libcni/api.go

@@ -0,0 +1,497 @@
+// Copyright 2015 CNI authors
+//
+// Licensed under the Apache License, Version 2.0 (the "License");
+// you may not use this file except in compliance with the License.
+// You may obtain a copy of the License at
+//
+//     http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+package libcni
+
+import (
+	"context"
+	"encoding/json"
+	"fmt"
+	"io/ioutil"
+	"os"
+	"path/filepath"
+	"strings"
+
+	"github.com/containernetworking/cni/pkg/invoke"
+	"github.com/containernetworking/cni/pkg/types"
+	"github.com/containernetworking/cni/pkg/version"
+)
+
+var (
+	CacheDir = "/var/lib/cni"
+)
+
+// A RuntimeConf holds the arguments to one invocation of a CNI plugin
+// excepting the network configuration, with the nested exception that
+// the `runtimeConfig` from the network configuration is included
+// here.
+type RuntimeConf struct {
+	ContainerID string
+	NetNS       string
+	IfName      string
+	Args        [][2]string
+	// A dictionary of capability-specific data passed by the runtime
+	// to plugins as top-level keys in the 'runtimeConfig' dictionary
+	// of the plugin's stdin data.  libcni will ensure that only keys
+	// in this map which match the capabilities of the plugin are passed
+	// to the plugin
+	CapabilityArgs map[string]interface{}
+
+	// A cache directory in which to library data.  Defaults to CacheDir
+	CacheDir string
+}
+
+type NetworkConfig struct {
+	Network *types.NetConf
+	Bytes   []byte
+}
+
+type NetworkConfigList struct {
+	Name         string
+	CNIVersion   string
+	DisableCheck bool
+	Plugins      []*NetworkConfig
+	Bytes        []byte
+}
+
+type CNI interface {
+	AddNetworkList(ctx context.Context, net *NetworkConfigList, rt *RuntimeConf) (types.Result, error)
+	CheckNetworkList(ctx context.Context, net *NetworkConfigList, rt *RuntimeConf) error
+	DelNetworkList(ctx context.Context, net *NetworkConfigList, rt *RuntimeConf) error
+	GetNetworkListCachedResult(net *NetworkConfigList, rt *RuntimeConf) (types.Result, error)
+
+	AddNetwork(ctx context.Context, net *NetworkConfig, rt *RuntimeConf) (types.Result, error)
+	CheckNetwork(ctx context.Context, net *NetworkConfig, rt *RuntimeConf) error
+	DelNetwork(ctx context.Context, net *NetworkConfig, rt *RuntimeConf) error
+	GetNetworkCachedResult(net *NetworkConfig, rt *RuntimeConf) (types.Result, error)
+
+	ValidateNetworkList(ctx context.Context, net *NetworkConfigList) ([]string, error)
+	ValidateNetwork(ctx context.Context, net *NetworkConfig) ([]string, error)
+}
+
+type CNIConfig struct {
+	Path []string
+	exec invoke.Exec
+}
+
+// CNIConfig implements the CNI interface
+var _ CNI = &CNIConfig{}
+
+// NewCNIConfig returns a new CNIConfig object that will search for plugins
+// in the given paths and use the given exec interface to run those plugins,
+// or if the exec interface is not given, will use a default exec handler.
+func NewCNIConfig(path []string, exec invoke.Exec) *CNIConfig {
+	return &CNIConfig{
+		Path: path,
+		exec: exec,
+	}
+}
+
+func buildOneConfig(name, cniVersion string, orig *NetworkConfig, prevResult types.Result, rt *RuntimeConf) (*NetworkConfig, error) {
+	var err error
+
+	inject := map[string]interface{}{
+		"name":       name,
+		"cniVersion": cniVersion,
+	}
+	// Add previous plugin result
+	if prevResult != nil {
+		inject["prevResult"] = prevResult
+	}
+
+	// Ensure every config uses the same name and version
+	orig, err = InjectConf(orig, inject)
+	if err != nil {
+		return nil, err
+	}
+
+	return injectRuntimeConfig(orig, rt)
+}
+
+// This function takes a libcni RuntimeConf structure and injects values into
+// a "runtimeConfig" dictionary in the CNI network configuration JSON that
+// will be passed to the plugin on stdin.
+//
+// Only "capabilities arguments" passed by the runtime are currently injected.
+// These capabilities arguments are filtered through the plugin's advertised
+// capabilities from its config JSON, and any keys in the CapabilityArgs
+// matching plugin capabilities are added to the "runtimeConfig" dictionary
+// sent to the plugin via JSON on stdin.  For example, if the plugin's
+// capabilities include "portMappings", and the CapabilityArgs map includes a
+// "portMappings" key, that key and its value are added to the "runtimeConfig"
+// dictionary to be passed to the plugin's stdin.
+func injectRuntimeConfig(orig *NetworkConfig, rt *RuntimeConf) (*NetworkConfig, error) {
+	var err error
+
+	rc := make(map[string]interface{})
+	for capability, supported := range orig.Network.Capabilities {
+		if !supported {
+			continue
+		}
+		if data, ok := rt.CapabilityArgs[capability]; ok {
+			rc[capability] = data
+		}
+	}
+
+	if len(rc) > 0 {
+		orig, err = InjectConf(orig, map[string]interface{}{"runtimeConfig": rc})
+		if err != nil {
+			return nil, err
+		}
+	}
+
+	return orig, nil
+}
+
+// ensure we have a usable exec if the CNIConfig was not given one
+func (c *CNIConfig) ensureExec() invoke.Exec {
+	if c.exec == nil {
+		c.exec = &invoke.DefaultExec{
+			RawExec:       &invoke.RawExec{Stderr: os.Stderr},
+			PluginDecoder: version.PluginDecoder{},
+		}
+	}
+	return c.exec
+}
+
+func getResultCacheFilePath(netName string, rt *RuntimeConf) string {
+	cacheDir := rt.CacheDir
+	if cacheDir == "" {
+		cacheDir = CacheDir
+	}
+	return filepath.Join(cacheDir, "results", fmt.Sprintf("%s-%s-%s", netName, rt.ContainerID, rt.IfName))
+}
+
+func setCachedResult(result types.Result, netName string, rt *RuntimeConf) error {
+	data, err := json.Marshal(result)
+	if err != nil {
+		return err
+	}
+	fname := getResultCacheFilePath(netName, rt)
+	if err := os.MkdirAll(filepath.Dir(fname), 0700); err != nil {
+		return err
+	}
+	return ioutil.WriteFile(fname, data, 0600)
+}
+
+func delCachedResult(netName string, rt *RuntimeConf) error {
+	fname := getResultCacheFilePath(netName, rt)
+	return os.Remove(fname)
+}
+
+func getCachedResult(netName, cniVersion string, rt *RuntimeConf) (types.Result, error) {
+	fname := getResultCacheFilePath(netName, rt)
+	data, err := ioutil.ReadFile(fname)
+	if err != nil {
+		// Ignore read errors; the cached result may not exist on-disk
+		return nil, nil
+	}
+
+	// Read the version of the cached result
+	decoder := version.ConfigDecoder{}
+	resultCniVersion, err := decoder.Decode(data)
+	if err != nil {
+		return nil, err
+	}
+
+	// Ensure we can understand the result
+	result, err := version.NewResult(resultCniVersion, data)
+	if err != nil {
+		return nil, err
+	}
+
+	// Convert to the config version to ensure plugins get prevResult
+	// in the same version as the config.  The cached result version
+	// should match the config version unless the config was changed
+	// while the container was running.
+	result, err = result.GetAsVersion(cniVersion)
+	if err != nil && resultCniVersion != cniVersion {
+		return nil, fmt.Errorf("failed to convert cached result version %q to config version %q: %v", resultCniVersion, cniVersion, err)
+	}
+	return result, err
+}
+
+// GetNetworkListCachedResult returns the cached Result of the previous
+// previous AddNetworkList() operation for a network list, or an error.
+func (c *CNIConfig) GetNetworkListCachedResult(list *NetworkConfigList, rt *RuntimeConf) (types.Result, error) {
+	return getCachedResult(list.Name, list.CNIVersion, rt)
+}
+
+// GetNetworkCachedResult returns the cached Result of the previous
+// previous AddNetwork() operation for a network, or an error.
+func (c *CNIConfig) GetNetworkCachedResult(net *NetworkConfig, rt *RuntimeConf) (types.Result, error) {
+	return getCachedResult(net.Network.Name, net.Network.CNIVersion, rt)
+}
+
+func (c *CNIConfig) addNetwork(ctx context.Context, name, cniVersion string, net *NetworkConfig, prevResult types.Result, rt *RuntimeConf) (types.Result, error) {
+	c.ensureExec()
+	pluginPath, err := c.exec.FindInPath(net.Network.Type, c.Path)
+	if err != nil {
+		return nil, err
+	}
+
+	newConf, err := buildOneConfig(name, cniVersion, net, prevResult, rt)
+	if err != nil {
+		return nil, err
+	}
+
+	return invoke.ExecPluginWithResult(ctx, pluginPath, newConf.Bytes, c.args("ADD", rt), c.exec)
+}
+
+// AddNetworkList executes a sequence of plugins with the ADD command
+func (c *CNIConfig) AddNetworkList(ctx context.Context, list *NetworkConfigList, rt *RuntimeConf) (types.Result, error) {
+	var err error
+	var result types.Result
+	for _, net := range list.Plugins {
+		result, err = c.addNetwork(ctx, list.Name, list.CNIVersion, net, result, rt)
+		if err != nil {
+			return nil, err
+		}
+	}
+
+	if err = setCachedResult(result, list.Name, rt); err != nil {
+		return nil, fmt.Errorf("failed to set network %q cached result: %v", list.Name, err)
+	}
+
+	return result, nil
+}
+
+func (c *CNIConfig) checkNetwork(ctx context.Context, name, cniVersion string, net *NetworkConfig, prevResult types.Result, rt *RuntimeConf) error {
+	c.ensureExec()
+	pluginPath, err := c.exec.FindInPath(net.Network.Type, c.Path)
+	if err != nil {
+		return err
+	}
+
+	newConf, err := buildOneConfig(name, cniVersion, net, prevResult, rt)
+	if err != nil {
+		return err
+	}
+
+	return invoke.ExecPluginWithoutResult(ctx, pluginPath, newConf.Bytes, c.args("CHECK", rt), c.exec)
+}
+
+// CheckNetworkList executes a sequence of plugins with the CHECK command
+func (c *CNIConfig) CheckNetworkList(ctx context.Context, list *NetworkConfigList, rt *RuntimeConf) error {
+	// CHECK was added in CNI spec version 0.4.0 and higher
+	if gtet, err := version.GreaterThanOrEqualTo(list.CNIVersion, "0.4.0"); err != nil {
+		return err
+	} else if !gtet {
+		return fmt.Errorf("configuration version %q does not support the CHECK command", list.CNIVersion)
+	}
+
+	if list.DisableCheck {
+		return nil
+	}
+
+	cachedResult, err := getCachedResult(list.Name, list.CNIVersion, rt)
+	if err != nil {
+		return fmt.Errorf("failed to get network %q cached result: %v", list.Name, err)
+	}
+
+	for _, net := range list.Plugins {
+		if err := c.checkNetwork(ctx, list.Name, list.CNIVersion, net, cachedResult, rt); err != nil {
+			return err
+		}
+	}
+
+	return nil
+}
+
+func (c *CNIConfig) delNetwork(ctx context.Context, name, cniVersion string, net *NetworkConfig, prevResult types.Result, rt *RuntimeConf) error {
+	c.ensureExec()
+	pluginPath, err := c.exec.FindInPath(net.Network.Type, c.Path)
+	if err != nil {
+		return err
+	}
+
+	newConf, err := buildOneConfig(name, cniVersion, net, prevResult, rt)
+	if err != nil {
+		return err
+	}
+
+	return invoke.ExecPluginWithoutResult(ctx, pluginPath, newConf.Bytes, c.args("DEL", rt), c.exec)
+}
+
+// DelNetworkList executes a sequence of plugins with the DEL command
+func (c *CNIConfig) DelNetworkList(ctx context.Context, list *NetworkConfigList, rt *RuntimeConf) error {
+	var cachedResult types.Result
+
+	// Cached result on DEL was added in CNI spec version 0.4.0 and higher
+	if gtet, err := version.GreaterThanOrEqualTo(list.CNIVersion, "0.4.0"); err != nil {
+		return err
+	} else if gtet {
+		cachedResult, err = getCachedResult(list.Name, list.CNIVersion, rt)
+		if err != nil {
+			return fmt.Errorf("failed to get network %q cached result: %v", list.Name, err)
+		}
+	}
+
+	for i := len(list.Plugins) - 1; i >= 0; i-- {
+		net := list.Plugins[i]
+		if err := c.delNetwork(ctx, list.Name, list.CNIVersion, net, cachedResult, rt); err != nil {
+			return err
+		}
+	}
+	_ = delCachedResult(list.Name, rt)
+
+	return nil
+}
+
+// AddNetwork executes the plugin with the ADD command
+func (c *CNIConfig) AddNetwork(ctx context.Context, net *NetworkConfig, rt *RuntimeConf) (types.Result, error) {
+	result, err := c.addNetwork(ctx, net.Network.Name, net.Network.CNIVersion, net, nil, rt)
+	if err != nil {
+		return nil, err
+	}
+
+	if err = setCachedResult(result, net.Network.Name, rt); err != nil {
+		return nil, fmt.Errorf("failed to set network %q cached result: %v", net.Network.Name, err)
+	}
+
+	return result, nil
+}
+
+// CheckNetwork executes the plugin with the CHECK command
+func (c *CNIConfig) CheckNetwork(ctx context.Context, net *NetworkConfig, rt *RuntimeConf) error {
+	// CHECK was added in CNI spec version 0.4.0 and higher
+	if gtet, err := version.GreaterThanOrEqualTo(net.Network.CNIVersion, "0.4.0"); err != nil {
+		return err
+	} else if !gtet {
+		return fmt.Errorf("configuration version %q does not support the CHECK command", net.Network.CNIVersion)
+	}
+
+	cachedResult, err := getCachedResult(net.Network.Name, net.Network.CNIVersion, rt)
+	if err != nil {
+		return fmt.Errorf("failed to get network %q cached result: %v", net.Network.Name, err)
+	}
+	return c.checkNetwork(ctx, net.Network.Name, net.Network.CNIVersion, net, cachedResult, rt)
+}
+
+// DelNetwork executes the plugin with the DEL command
+func (c *CNIConfig) DelNetwork(ctx context.Context, net *NetworkConfig, rt *RuntimeConf) error {
+	var cachedResult types.Result
+
+	// Cached result on DEL was added in CNI spec version 0.4.0 and higher
+	if gtet, err := version.GreaterThanOrEqualTo(net.Network.CNIVersion, "0.4.0"); err != nil {
+		return err
+	} else if gtet {
+		cachedResult, err = getCachedResult(net.Network.Name, net.Network.CNIVersion, rt)
+		if err != nil {
+			return fmt.Errorf("failed to get network %q cached result: %v", net.Network.Name, err)
+		}
+	}
+
+	if err := c.delNetwork(ctx, net.Network.Name, net.Network.CNIVersion, net, cachedResult, rt); err != nil {
+		return err
+	}
+	_ = delCachedResult(net.Network.Name, rt)
+	return nil
+}
+
+// ValidateNetworkList checks that a configuration is reasonably valid.
+// - all the specified plugins exist on disk
+// - every plugin supports the desired version.
+//
+// Returns a list of all capabilities supported by the configuration, or error
+func (c *CNIConfig) ValidateNetworkList(ctx context.Context, list *NetworkConfigList) ([]string, error) {
+	version := list.CNIVersion
+
+	// holding map for seen caps (in case of duplicates)
+	caps := map[string]interface{}{}
+
+	errs := []error{}
+	for _, net := range list.Plugins {
+		if err := c.validatePlugin(ctx, net.Network.Type, version); err != nil {
+			errs = append(errs, err)
+		}
+		for c, enabled := range net.Network.Capabilities {
+			if !enabled {
+				continue
+			}
+			caps[c] = struct{}{}
+		}
+	}
+
+	if len(errs) > 0 {
+		return nil, fmt.Errorf("%v", errs)
+	}
+
+	// make caps list
+	cc := make([]string, 0, len(caps))
+	for c := range caps {
+		cc = append(cc, c)
+	}
+
+	return cc, nil
+}
+
+// ValidateNetwork checks that a configuration is reasonably valid.
+// It uses the same logic as ValidateNetworkList)
+// Returns a list of capabilities
+func (c *CNIConfig) ValidateNetwork(ctx context.Context, net *NetworkConfig) ([]string, error) {
+	caps := []string{}
+	for c, ok := range net.Network.Capabilities {
+		if ok {
+			caps = append(caps, c)
+		}
+	}
+	if err := c.validatePlugin(ctx, net.Network.Type, net.Network.CNIVersion); err != nil {
+		return nil, err
+	}
+	return caps, nil
+}
+
+// validatePlugin checks that an individual plugin's configuration is sane
+func (c *CNIConfig) validatePlugin(ctx context.Context, pluginName, expectedVersion string) error {
+	pluginPath, err := invoke.FindInPath(pluginName, c.Path)
+	if err != nil {
+		return err
+	}
+
+	vi, err := invoke.GetVersionInfo(ctx, pluginPath, c.exec)
+	if err != nil {
+		return err
+	}
+	for _, vers := range vi.SupportedVersions() {
+		if vers == expectedVersion {
+			return nil
+		}
+	}
+	return fmt.Errorf("plugin %s does not support config version %q", pluginName, expectedVersion)
+}
+
+// GetVersionInfo reports which versions of the CNI spec are supported by
+// the given plugin.
+func (c *CNIConfig) GetVersionInfo(ctx context.Context, pluginType string) (version.PluginInfo, error) {
+	c.ensureExec()
+	pluginPath, err := c.exec.FindInPath(pluginType, c.Path)
+	if err != nil {
+		return nil, err
+	}
+
+	return invoke.GetVersionInfo(ctx, pluginPath, c.exec)
+}
+
+// =====
+func (c *CNIConfig) args(action string, rt *RuntimeConf) *invoke.Args {
+	return &invoke.Args{
+		Command:     action,
+		ContainerID: rt.ContainerID,
+		NetNS:       rt.NetNS,
+		PluginArgs:  rt.Args,
+		IfName:      rt.IfName,
+		Path:        strings.Join(c.Path, string(os.PathListSeparator)),
+	}
+}

vendor/github.com/containernetworking/cni/libcni/conf.go

@@ -0,0 +1,268 @@
+// Copyright 2015 CNI authors
+//
+// Licensed under the Apache License, Version 2.0 (the "License");
+// you may not use this file except in compliance with the License.
+// You may obtain a copy of the License at
+//
+//     http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+package libcni
+
+import (
+	"encoding/json"
+	"fmt"
+	"io/ioutil"
+	"os"
+	"path/filepath"
+	"sort"
+)
+
+type NotFoundError struct {
+	Dir  string
+	Name string
+}
+
+func (e NotFoundError) Error() string {
+	return fmt.Sprintf(`no net configuration with name "%s" in %s`, e.Name, e.Dir)
+}
+
+type NoConfigsFoundError struct {
+	Dir string
+}
+
+func (e NoConfigsFoundError) Error() string {
+	return fmt.Sprintf(`no net configurations found in %s`, e.Dir)
+}
+
+func ConfFromBytes(bytes []byte) (*NetworkConfig, error) {
+	conf := &NetworkConfig{Bytes: bytes}
+	if err := json.Unmarshal(bytes, &conf.Network); err != nil {
+		return nil, fmt.Errorf("error parsing configuration: %s", err)
+	}
+	if conf.Network.Type == "" {
+		return nil, fmt.Errorf("error parsing configuration: missing 'type'")
+	}
+	return conf, nil
+}
+
+func ConfFromFile(filename string) (*NetworkConfig, error) {
+	bytes, err := ioutil.ReadFile(filename)
+	if err != nil {
+		return nil, fmt.Errorf("error reading %s: %s", filename, err)
+	}
+	return ConfFromBytes(bytes)
+}
+
+func ConfListFromBytes(bytes []byte) (*NetworkConfigList, error) {
+	rawList := make(map[string]interface{})
+	if err := json.Unmarshal(bytes, &rawList); err != nil {
+		return nil, fmt.Errorf("error parsing configuration list: %s", err)
+	}
+
+	rawName, ok := rawList["name"]
+	if !ok {
+		return nil, fmt.Errorf("error parsing configuration list: no name")
+	}
+	name, ok := rawName.(string)
+	if !ok {
+		return nil, fmt.Errorf("error parsing configuration list: invalid name type %T", rawName)
+	}
+
+	var cniVersion string
+	rawVersion, ok := rawList["cniVersion"]
+	if ok {
+		cniVersion, ok = rawVersion.(string)
+		if !ok {
+			return nil, fmt.Errorf("error parsing configuration list: invalid cniVersion type %T", rawVersion)
+		}
+	}
+
+	disableCheck := false
+	if rawDisableCheck, ok := rawList["disableCheck"]; ok {
+		disableCheck, ok = rawDisableCheck.(bool)
+		if !ok {
+			return nil, fmt.Errorf("error parsing configuration list: invalid disableCheck type %T", rawDisableCheck)
+		}
+	}
+
+	list := &NetworkConfigList{
+		Name:         name,
+		DisableCheck: disableCheck,
+		CNIVersion:   cniVersion,
+		Bytes:        bytes,
+	}
+
+	var plugins []interface{}
+	plug, ok := rawList["plugins"]
+	if !ok {
+		return nil, fmt.Errorf("error parsing configuration list: no 'plugins' key")
+	}
+	plugins, ok = plug.([]interface{})
+	if !ok {
+		return nil, fmt.Errorf("error parsing configuration list: invalid 'plugins' type %T", plug)
+	}
+	if len(plugins) == 0 {
+		return nil, fmt.Errorf("error parsing configuration list: no plugins in list")
+	}
+
+	for i, conf := range plugins {
+		newBytes, err := json.Marshal(conf)
+		if err != nil {
+			return nil, fmt.Errorf("Failed to marshal plugin config %d: %v", i, err)
+		}
+		netConf, err := ConfFromBytes(newBytes)
+		if err != nil {
+			return nil, fmt.Errorf("Failed to parse plugin config %d: %v", i, err)
+		}
+		list.Plugins = append(list.Plugins, netConf)
+	}
+
+	return list, nil
+}
+
+func ConfListFromFile(filename string) (*NetworkConfigList, error) {
+	bytes, err := ioutil.ReadFile(filename)
+	if err != nil {
+		return nil, fmt.Errorf("error reading %s: %s", filename, err)
+	}
+	return ConfListFromBytes(bytes)
+}
+
+func ConfFiles(dir string, extensions []string) ([]string, error) {
+	// In part, adapted from rkt/networking/podenv.go#listFiles
+	files, err := ioutil.ReadDir(dir)
+	switch {
+	case err == nil: // break
+	case os.IsNotExist(err):
+		return nil, nil
+	default:
+		return nil, err
+	}
+
+	confFiles := []string{}
+	for _, f := range files {
+		if f.IsDir() {
+			continue
+		}
+		fileExt := filepath.Ext(f.Name())
+		for _, ext := range extensions {
+			if fileExt == ext {
+				confFiles = append(confFiles, filepath.Join(dir, f.Name()))
+			}
+		}
+	}
+	return confFiles, nil
+}
+
+func LoadConf(dir, name string) (*NetworkConfig, error) {
+	files, err := ConfFiles(dir, []string{".conf", ".json"})
+	switch {
+	case err != nil:
+		return nil, err
+	case len(files) == 0:
+		return nil, NoConfigsFoundError{Dir: dir}
+	}
+	sort.Strings(files)
+
+	for _, confFile := range files {
+		conf, err := ConfFromFile(confFile)
+		if err != nil {
+			return nil, err
+		}
+		if conf.Network.Name == name {
+			return conf, nil
+		}
+	}
+	return nil, NotFoundError{dir, name}
+}
+
+func LoadConfList(dir, name string) (*NetworkConfigList, error) {
+	files, err := ConfFiles(dir, []string{".conflist"})
+	if err != nil {
+		return nil, err
+	}
+	sort.Strings(files)
+
+	for _, confFile := range files {
+		conf, err := ConfListFromFile(confFile)
+		if err != nil {
+			return nil, err
+		}
+		if conf.Name == name {
+			return conf, nil
+		}
+	}
+
+	// Try and load a network configuration file (instead of list)
+	// from the same name, then upconvert.
+	singleConf, err := LoadConf(dir, name)
+	if err != nil {
+		// A little extra logic so the error makes sense
+		if _, ok := err.(NoConfigsFoundError); len(files) != 0 && ok {
+			// Config lists found but no config files found
+			return nil, NotFoundError{dir, name}
+		}
+
+		return nil, err
+	}
+	return ConfListFromConf(singleConf)
+}
+
+func InjectConf(original *NetworkConfig, newValues map[string]interface{}) (*NetworkConfig, error) {
+	config := make(map[string]interface{})
+	err := json.Unmarshal(original.Bytes, &config)
+	if err != nil {
+		return nil, fmt.Errorf("unmarshal existing network bytes: %s", err)
+	}
+
+	for key, value := range newValues {
+		if key == "" {
+			return nil, fmt.Errorf("keys cannot be empty")
+		}
+
+		if value == nil {
+			return nil, fmt.Errorf("key '%s' value must not be nil", key)
+		}
+
+		config[key] = value
+	}
+
+	newBytes, err := json.Marshal(config)
+	if err != nil {
+		return nil, err
+	}
+
+	return ConfFromBytes(newBytes)
+}
+
+// ConfListFromConf "upconverts" a network config in to a NetworkConfigList,
+// with the single network as the only entry in the list.
+func ConfListFromConf(original *NetworkConfig) (*NetworkConfigList, error) {
+	// Re-deserialize the config's json, then make a raw map configlist.
+	// This may seem a bit strange, but it's to make the Bytes fields
+	// actually make sense. Otherwise, the generated json is littered with
+	// golang default values.
+
+	rawConfig := make(map[string]interface{})
+	if err := json.Unmarshal(original.Bytes, &rawConfig); err != nil {
+		return nil, err
+	}
+
+	rawConfigList := map[string]interface{}{
+		"name":       original.Network.Name,
+		"cniVersion": original.Network.CNIVersion,
+		"plugins":    []interface{}{rawConfig},
+	}
+
+	b, err := json.Marshal(rawConfigList)
+	if err != nil {
+		return nil, err
+	}
+	return ConfListFromBytes(b)
+}

vendor/github.com/containernetworking/cni/pkg/invoke/args.go

@@ -15,6 +15,7 @@
 package invoke
 
 import (
+	"fmt"
 	"os"
 	"strings"
 )
@@ -22,6 +23,8 @@ import (
 type CNIArgs interface {
 	// For use with os/exec; i.e., return nil to inherit the
 	// environment from this process
+	// For use in delegation; inherit the environment from this
+	// process and allow overrides
 	AsEnv() []string
 }
 
@@ -47,6 +50,9 @@ type Args struct {
 	Path          string
 }
 
+// Args implements the CNIArgs interface
+var _ CNIArgs = &Args{}
+
 func (args *Args) AsEnv() []string {
 	env := os.Environ()
 	pluginArgsStr := args.PluginArgsStr
@@ -54,14 +60,17 @@ func (args *Args) AsEnv() []string {
 		pluginArgsStr = stringify(args.PluginArgs)
 	}
 
+	// Duplicated values which come first will be overrided, so we must put the
+	// custom values in the end to avoid being overrided by the process environments.
 	env = append(env,
 		"CNI_COMMAND="+args.Command,
 		"CNI_CONTAINERID="+args.ContainerID,
 		"CNI_NETNS="+args.NetNS,
 		"CNI_ARGS="+pluginArgsStr,
 		"CNI_IFNAME="+args.IfName,
-		"CNI_PATH="+args.Path)
-	return env
+		"CNI_PATH="+args.Path,
+	)
+	return dedupEnv(env)
 }
 
 // taken from rkt/networking/net_plugin.go
@@ -74,3 +83,46 @@ func stringify(pluginArgs [][2]string) string {
 
 	return strings.Join(entries, ";")
 }
+
+// DelegateArgs implements the CNIArgs interface
+// used for delegation to inherit from environments
+// and allow some overrides like CNI_COMMAND
+var _ CNIArgs = &DelegateArgs{}
+
+type DelegateArgs struct {
+	Command string
+}
+
+func (d *DelegateArgs) AsEnv() []string {
+	env := os.Environ()
+
+	// The custom values should come in the end to override the existing
+	// process environment of the same key.
+	env = append(env,
+		"CNI_COMMAND="+d.Command,
+	)
+	return dedupEnv(env)
+}
+
+// dedupEnv returns a copy of env with any duplicates removed, in favor of later values.
+// Items not of the normal environment "key=value" form are preserved unchanged.
+func dedupEnv(env []string) []string {
+	out := make([]string, 0, len(env))
+	envMap := map[string]string{}
+
+	for _, kv := range env {
+		// find the first "=" in environment, if not, just keep it
+		eq := strings.Index(kv, "=")
+		if eq < 0 {
+			out = append(out, kv)
+			continue
+		}
+		envMap[kv[:eq]] = kv[eq+1:]
+	}
+
+	for k, v := range envMap {
+		out = append(out, fmt.Sprintf("%s=%s", k, v))
+	}
+
+	return out
+}

vendor/github.com/containernetworking/cni/pkg/invoke/delegate.go

@@ -15,39 +15,66 @@
 package invoke
 
 import (
-	"fmt"
+	"context"
 	"os"
-	"strings"
+	"path/filepath"
 
 	"github.com/containernetworking/cni/pkg/types"
 )
 
-func DelegateAdd(delegatePlugin string, netconf []byte) (*types.Result, error) {
-	if os.Getenv("CNI_COMMAND") != "ADD" {
-		return nil, fmt.Errorf("CNI_COMMAND is not ADD")
+func delegateCommon(delegatePlugin string, exec Exec) (string, Exec, error) {
+	if exec == nil {
+		exec = defaultExec
 	}
 
-	paths := strings.Split(os.Getenv("CNI_PATH"), ":")
+	paths := filepath.SplitList(os.Getenv("CNI_PATH"))
+	pluginPath, err := exec.FindInPath(delegatePlugin, paths)
+	if err != nil {
+		return "", nil, err
+	}
 
-	pluginPath, err := FindInPath(delegatePlugin, paths)
+	return pluginPath, exec, nil
+}
+
+// DelegateAdd calls the given delegate plugin with the CNI ADD action and
+// JSON configuration
+func DelegateAdd(ctx context.Context, delegatePlugin string, netconf []byte, exec Exec) (types.Result, error) {
+	pluginPath, realExec, err := delegateCommon(delegatePlugin, exec)
 	if err != nil {
 		return nil, err
 	}
 
-	return ExecPluginWithResult(pluginPath, netconf, ArgsFromEnv())
+	// DelegateAdd will override the original "CNI_COMMAND" env from process with ADD
+	return ExecPluginWithResult(ctx, pluginPath, netconf, delegateArgs("ADD"), realExec)
 }
 
-func DelegateDel(delegatePlugin string, netconf []byte) error {
-	if os.Getenv("CNI_COMMAND") != "DEL" {
-		return fmt.Errorf("CNI_COMMAND is not DEL")
-	}
-
-	paths := strings.Split(os.Getenv("CNI_PATH"), ":")
-
-	pluginPath, err := FindInPath(delegatePlugin, paths)
+// DelegateCheck calls the given delegate plugin with the CNI CHECK action and
+// JSON configuration
+func DelegateCheck(ctx context.Context, delegatePlugin string, netconf []byte, exec Exec) error {
+	pluginPath, realExec, err := delegateCommon(delegatePlugin, exec)
 	if err != nil {
 		return err
 	}
 
-	return ExecPluginWithoutResult(pluginPath, netconf, ArgsFromEnv())
+	// DelegateCheck will override the original CNI_COMMAND env from process with CHECK
+	return ExecPluginWithoutResult(ctx, pluginPath, netconf, delegateArgs("CHECK"), realExec)
+}
+
+// DelegateDel calls the given delegate plugin with the CNI DEL action and
+// JSON configuration
+func DelegateDel(ctx context.Context, delegatePlugin string, netconf []byte, exec Exec) error {
+	pluginPath, realExec, err := delegateCommon(delegatePlugin, exec)
+	if err != nil {
+		return err
+	}
+
+	// DelegateDel will override the original CNI_COMMAND env from process with DEL
+	return ExecPluginWithoutResult(ctx, pluginPath, netconf, delegateArgs("DEL"), realExec)
+}
+
+// return CNIArgs used by delegation
+func delegateArgs(action string) *DelegateArgs {
+	return &DelegateArgs{
+		Command: action,
+	}
 }

vendor/github.com/containernetworking/cni/pkg/invoke/exec.go

@@ -15,61 +15,130 @@
 package invoke
 
 import (
-	"bytes"
-	"encoding/json"
+	"context"
 	"fmt"
 	"os"
-	"os/exec"
 
 	"github.com/containernetworking/cni/pkg/types"
+	"github.com/containernetworking/cni/pkg/version"
 )
 
-func pluginErr(err error, output []byte) error {
-	if _, ok := err.(*exec.ExitError); ok {
-		emsg := types.Error{}
-		if perr := json.Unmarshal(output, &emsg); perr != nil {
-			return fmt.Errorf("netplugin failed but error parsing its diagnostic message %q: %v", string(output), perr)
-		}
-		details := ""
-		if emsg.Details != "" {
-			details = fmt.Sprintf("; %v", emsg.Details)
-		}
-		return fmt.Errorf("%v%v", emsg.Msg, details)
-	}
-
-	return err
+// Exec is an interface encapsulates all operations that deal with finding
+// and executing a CNI plugin. Tests may provide a fake implementation
+// to avoid writing fake plugins to temporary directories during the test.
+type Exec interface {
+	ExecPlugin(ctx context.Context, pluginPath string, stdinData []byte, environ []string) ([]byte, error)
+	FindInPath(plugin string, paths []string) (string, error)
+	Decode(jsonBytes []byte) (version.PluginInfo, error)
 }
 
-func ExecPluginWithResult(pluginPath string, netconf []byte, args CNIArgs) (*types.Result, error) {
-	stdoutBytes, err := execPlugin(pluginPath, netconf, args)
+// For example, a testcase could pass an instance of the following fakeExec
+// object to ExecPluginWithResult() to verify the incoming stdin and environment
+// and provide a tailored response:
+//
+//import (
+//	"encoding/json"
+//	"path"
+//	"strings"
+//)
+//
+//type fakeExec struct {
+//	version.PluginDecoder
+//}
+//
+//func (f *fakeExec) ExecPlugin(pluginPath string, stdinData []byte, environ []string) ([]byte, error) {
+//	net := &types.NetConf{}
+//	err := json.Unmarshal(stdinData, net)
+//	if err != nil {
+//		return nil, fmt.Errorf("failed to unmarshal configuration: %v", err)
+//	}
+//	pluginName := path.Base(pluginPath)
+//	if pluginName != net.Type {
+//		return nil, fmt.Errorf("plugin name %q did not match config type %q", pluginName, net.Type)
+//	}
+//	for _, e := range environ {
+//		// Check environment for forced failure request
+//		parts := strings.Split(e, "=")
+//		if len(parts) > 0 && parts[0] == "FAIL" {
+//			return nil, fmt.Errorf("failed to execute plugin %s", pluginName)
+//		}
+//	}
+//	return []byte("{\"CNIVersion\":\"0.4.0\"}"), nil
+//}
+//
+//func (f *fakeExec) FindInPath(plugin string, paths []string) (string, error) {
+//	if len(paths) > 0 {
+//		return path.Join(paths[0], plugin), nil
+//	}
+//	return "", fmt.Errorf("failed to find plugin %s in paths %v", plugin, paths)
+//}
+
+func ExecPluginWithResult(ctx context.Context, pluginPath string, netconf []byte, args CNIArgs, exec Exec) (types.Result, error) {
+	if exec == nil {
+		exec = defaultExec
+	}
+
+	stdoutBytes, err := exec.ExecPlugin(ctx, pluginPath, netconf, args.AsEnv())
 	if err != nil {
 		return nil, err
 	}
 
-	res := &types.Result{}
-	err = json.Unmarshal(stdoutBytes, res)
-	return res, err
+	// Plugin must return result in same version as specified in netconf
+	versionDecoder := &version.ConfigDecoder{}
+	confVersion, err := versionDecoder.Decode(netconf)
+	if err != nil {
+		return nil, err
+	}
+
+	return version.NewResult(confVersion, stdoutBytes)
 }
 
-func ExecPluginWithoutResult(pluginPath string, netconf []byte, args CNIArgs) error {
-	_, err := execPlugin(pluginPath, netconf, args)
+func ExecPluginWithoutResult(ctx context.Context, pluginPath string, netconf []byte, args CNIArgs, exec Exec) error {
+	if exec == nil {
+		exec = defaultExec
+	}
+	_, err := exec.ExecPlugin(ctx, pluginPath, netconf, args.AsEnv())
 	return err
 }
 
-func execPlugin(pluginPath string, netconf []byte, args CNIArgs) ([]byte, error) {
-	stdout := &bytes.Buffer{}
-
-	c := exec.Cmd{
-		Env:    args.AsEnv(),
-		Path:   pluginPath,
-		Args:   []string{pluginPath},
-		Stdin:  bytes.NewBuffer(netconf),
-		Stdout: stdout,
-		Stderr: os.Stderr,
+// GetVersionInfo returns the version information available about the plugin.
+// For recent-enough plugins, it uses the information returned by the VERSION
+// command.  For older plugins which do not recognize that command, it reports
+// version 0.1.0
+func GetVersionInfo(ctx context.Context, pluginPath string, exec Exec) (version.PluginInfo, error) {
+	if exec == nil {
+		exec = defaultExec
 	}
-	if err := c.Run(); err != nil {
-		return nil, pluginErr(err, stdout.Bytes())
+	args := &Args{
+		Command: "VERSION",
+
+		// set fake values required by plugins built against an older version of skel
+		NetNS:  "dummy",
+		IfName: "dummy",
+		Path:   "dummy",
+	}
+	stdin := []byte(fmt.Sprintf(`{"cniVersion":%q}`, version.Current()))
+	stdoutBytes, err := exec.ExecPlugin(ctx, pluginPath, stdin, args.AsEnv())
+	if err != nil {
+		if err.Error() == "unknown CNI_COMMAND: VERSION" {
+			return version.PluginSupports("0.1.0"), nil
+		}
+		return nil, err
 	}
 
-	return stdout.Bytes(), nil
+	return exec.Decode(stdoutBytes)
+}
+
+// DefaultExec is an object that implements the Exec interface which looks
+// for and executes plugins from disk.
+type DefaultExec struct {
+	*RawExec
+	version.PluginDecoder
+}
+
+// DefaultExec implements the Exec interface
+var _ Exec = &DefaultExec{}
+
+var defaultExec = &DefaultExec{
+	RawExec: &RawExec{Stderr: os.Stderr},
 }

vendor/github.com/containernetworking/cni/pkg/invoke/find.go

@@ -30,18 +30,14 @@ func FindInPath(plugin string, paths []string) (string, error) {
 		return "", fmt.Errorf("no paths provided")
 	}
 
-	var fullpath string
 	for _, path := range paths {
-		full := filepath.Join(path, plugin)
-		if fi, err := os.Stat(full); err == nil && fi.Mode().IsRegular() {
-			fullpath = full
-			break
+		for _, fe := range ExecutableFileExtensions {
+			fullpath := filepath.Join(path, plugin) + fe
+			if fi, err := os.Stat(fullpath); err == nil && fi.Mode().IsRegular() {
+				return fullpath, nil
+			}
 		}
 	}
 
-	if fullpath == "" {
-		return "", fmt.Errorf("failed to find plugin %q in path %s", plugin, paths)
-	}
-
-	return fullpath, nil
+	return "", fmt.Errorf("failed to find plugin %q in path %s", plugin, paths)
 }

vendor/github.com/containernetworking/cni/pkg/invoke/os_unix.go

@@ -1,4 +1,4 @@
-// Copyright 2015 CNI authors
+// Copyright 2016 CNI authors
 //
 // Licensed under the Apache License, Version 2.0 (the "License");
 // you may not use this file except in compliance with the License.
@@ -12,20 +12,9 @@
 // See the License for the specific language governing permissions and
 // limitations under the License.
 
-package ip
+// +build darwin dragonfly freebsd linux netbsd openbsd solaris
 
-import (
-	"io/ioutil"
-)
+package invoke
 
-func EnableIP4Forward() error {
-	return echo1("/proc/sys/net/ipv4/ip_forward")
-}
-
-func EnableIP6Forward() error {
-	return echo1("/proc/sys/net/ipv6/conf/all/forwarding")
-}
-
-func echo1(f string) error {
-	return ioutil.WriteFile(f, []byte("1"), 0644)
-}
+// Valid file extensions for plugin executables.
+var ExecutableFileExtensions = []string{""}

vendor/github.com/containernetworking/cni/pkg/invoke/os_windows.go

@@ -0,0 +1,18 @@
+// Copyright 2016 CNI authors
+//
+// Licensed under the Apache License, Version 2.0 (the "License");
+// you may not use this file except in compliance with the License.
+// You may obtain a copy of the License at
+//
+//     http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+package invoke
+
+// Valid file extensions for plugin executables.
+var ExecutableFileExtensions = []string{".exe", ""}

vendor/github.com/containernetworking/cni/pkg/invoke/raw_exec.go

@@ -0,0 +1,62 @@
+// Copyright 2016 CNI authors
+//
+// Licensed under the Apache License, Version 2.0 (the "License");
+// you may not use this file except in compliance with the License.
+// You may obtain a copy of the License at
+//
+//     http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+package invoke
+
+import (
+	"bytes"
+	"context"
+	"encoding/json"
+	"fmt"
+	"io"
+	"os/exec"
+
+	"github.com/containernetworking/cni/pkg/types"
+)
+
+type RawExec struct {
+	Stderr io.Writer
+}
+
+func (e *RawExec) ExecPlugin(ctx context.Context, pluginPath string, stdinData []byte, environ []string) ([]byte, error) {
+	stdout := &bytes.Buffer{}
+	c := exec.CommandContext(ctx, pluginPath)
+	c.Env = environ
+	c.Stdin = bytes.NewBuffer(stdinData)
+	c.Stdout = stdout
+	c.Stderr = e.Stderr
+	if err := c.Run(); err != nil {
+		return nil, pluginErr(err, stdout.Bytes())
+	}
+
+	return stdout.Bytes(), nil
+}
+
+func pluginErr(err error, output []byte) error {
+	if _, ok := err.(*exec.ExitError); ok {
+		emsg := types.Error{}
+		if len(output) == 0 {
+			emsg.Msg = "netplugin failed with no error message"
+		} else if perr := json.Unmarshal(output, &emsg); perr != nil {
+			emsg.Msg = fmt.Sprintf("netplugin failed but error parsing its diagnostic message %q: %v", string(output), perr)
+		}
+		return &emsg
+	}
+
+	return err
+}
+
+func (e *RawExec) FindInPath(plugin string, paths []string) (string, error) {
+	return FindInPath(plugin, paths)
+}

vendor/github.com/containernetworking/cni/pkg/ip/cidr.go

@@ -1,51 +0,0 @@
-// Copyright 2015 CNI authors
-//
-// Licensed under the Apache License, Version 2.0 (the "License");
-// you may not use this file except in compliance with the License.
-// You may obtain a copy of the License at
-//
-//     http://www.apache.org/licenses/LICENSE-2.0
-//
-// Unless required by applicable law or agreed to in writing, software
-// distributed under the License is distributed on an "AS IS" BASIS,
-// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-// See the License for the specific language governing permissions and
-// limitations under the License.
-
-package ip
-
-import (
-	"math/big"
-	"net"
-)
-
-// NextIP returns IP incremented by 1
-func NextIP(ip net.IP) net.IP {
-	i := ipToInt(ip)
-	return intToIP(i.Add(i, big.NewInt(1)))
-}
-
-// PrevIP returns IP decremented by 1
-func PrevIP(ip net.IP) net.IP {
-	i := ipToInt(ip)
-	return intToIP(i.Sub(i, big.NewInt(1)))
-}
-
-func ipToInt(ip net.IP) *big.Int {
-	if v := ip.To4(); v != nil {
-		return big.NewInt(0).SetBytes(v)
-	}
-	return big.NewInt(0).SetBytes(ip.To16())
-}
-
-func intToIP(i *big.Int) net.IP {
-	return net.IP(i.Bytes())
-}
-
-// Network masks off the host portion of the IP
-func Network(ipn *net.IPNet) *net.IPNet {
-	return &net.IPNet{
-		IP:   ipn.IP.Mask(ipn.Mask),
-		Mask: ipn.Mask,
-	}
-}

vendor/github.com/containernetworking/cni/pkg/ip/ipmasq.go

@@ -1,66 +0,0 @@
-// Copyright 2015 CNI authors
-//
-// Licensed under the Apache License, Version 2.0 (the "License");
-// you may not use this file except in compliance with the License.
-// You may obtain a copy of the License at
-//
-//     http://www.apache.org/licenses/LICENSE-2.0
-//
-// Unless required by applicable law or agreed to in writing, software
-// distributed under the License is distributed on an "AS IS" BASIS,
-// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-// See the License for the specific language governing permissions and
-// limitations under the License.
-
-package ip
-
-import (
-	"fmt"
-	"net"
-
-	"github.com/coreos/go-iptables/iptables"
-)
-
-// SetupIPMasq installs iptables rules to masquerade traffic
-// coming from ipn and going outside of it
-func SetupIPMasq(ipn *net.IPNet, chain string, comment string) error {
-	ipt, err := iptables.New()
-	if err != nil {
-		return fmt.Errorf("failed to locate iptables: %v", err)
-	}
-
-	if err = ipt.NewChain("nat", chain); err != nil {
-		if err.(*iptables.Error).ExitStatus() != 1 {
-			// TODO(eyakubovich): assumes exit status 1 implies chain exists
-			return err
-		}
-	}
-
-	if err = ipt.AppendUnique("nat", chain, "-d", ipn.String(), "-j", "ACCEPT", "-m", "comment", "--comment", comment); err != nil {
-		return err
-	}
-
-	if err = ipt.AppendUnique("nat", chain, "!", "-d", "224.0.0.0/4", "-j", "MASQUERADE", "-m", "comment", "--comment", comment); err != nil {
-		return err
-	}
-
-	return ipt.AppendUnique("nat", "POSTROUTING", "-s", ipn.String(), "-j", chain, "-m", "comment", "--comment", comment)
-}
-
-// TeardownIPMasq undoes the effects of SetupIPMasq
-func TeardownIPMasq(ipn *net.IPNet, chain string, comment string) error {
-	ipt, err := iptables.New()
-	if err != nil {
-		return fmt.Errorf("failed to locate iptables: %v", err)
-	}
-
-	if err = ipt.Delete("nat", "POSTROUTING", "-s", ipn.String(), "-j", chain, "-m", "comment", "--comment", comment); err != nil {
-		return err
-	}
-
-	if err = ipt.ClearChain("nat", chain); err != nil {
-		return err
-	}
-
-	return ipt.DeleteChain("nat", chain)
-}

vendor/github.com/containernetworking/cni/pkg/ip/link.go

@@ -1,153 +0,0 @@
-// Copyright 2015 CNI authors
-//
-// Licensed under the Apache License, Version 2.0 (the "License");
-// you may not use this file except in compliance with the License.
-// You may obtain a copy of the License at
-//
-//     http://www.apache.org/licenses/LICENSE-2.0
-//
-// Unless required by applicable law or agreed to in writing, software
-// distributed under the License is distributed on an "AS IS" BASIS,
-// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-// See the License for the specific language governing permissions and
-// limitations under the License.
-
-package ip
-
-import (
-	"crypto/rand"
-	"fmt"
-	"net"
-	"os"
-
-	"github.com/containernetworking/cni/pkg/ns"
-	"github.com/vishvananda/netlink"
-)
-
-func makeVethPair(name, peer string, mtu int) (netlink.Link, error) {
-	veth := &netlink.Veth{
-		LinkAttrs: netlink.LinkAttrs{
-			Name:  name,
-			Flags: net.FlagUp,
-			MTU:   mtu,
-		},
-		PeerName: peer,
-	}
-	if err := netlink.LinkAdd(veth); err != nil {
-		return nil, err
-	}
-
-	return veth, nil
-}
-
-func makeVeth(name string, mtu int) (peerName string, veth netlink.Link, err error) {
-	for i := 0; i < 10; i++ {
-		peerName, err = RandomVethName()
-		if err != nil {
-			return
-		}
-
-		veth, err = makeVethPair(name, peerName, mtu)
-		switch {
-		case err == nil:
-			return
-
-		case os.IsExist(err):
-			continue
-
-		default:
-			err = fmt.Errorf("failed to make veth pair: %v", err)
-			return
-		}
-	}
-
-	// should really never be hit
-	err = fmt.Errorf("failed to find a unique veth name")
-	return
-}
-
-// RandomVethName returns string "veth" with random prefix (hashed from entropy)
-func RandomVethName() (string, error) {
-	entropy := make([]byte, 4)
-	_, err := rand.Reader.Read(entropy)
-	if err != nil {
-		return "", fmt.Errorf("failed to generate random veth name: %v", err)
-	}
-
-	// NetworkManager (recent versions) will ignore veth devices that start with "veth"
-	return fmt.Sprintf("veth%x", entropy), nil
-}
-
-// SetupVeth sets up a virtual ethernet link.
-// Should be in container netns, and will switch back to hostNS to set the host
-// veth end up.
-func SetupVeth(contVethName string, mtu int, hostNS ns.NetNS) (hostVeth, contVeth netlink.Link, err error) {
-	var hostVethName string
-	hostVethName, contVeth, err = makeVeth(contVethName, mtu)
-	if err != nil {
-		return
-	}
-
-	if err = netlink.LinkSetUp(contVeth); err != nil {
-		err = fmt.Errorf("failed to set %q up: %v", contVethName, err)
-		return
-	}
-
-	hostVeth, err = netlink.LinkByName(hostVethName)
-	if err != nil {
-		err = fmt.Errorf("failed to lookup %q: %v", hostVethName, err)
-		return
-	}
-
-	if err = netlink.LinkSetNsFd(hostVeth, int(hostNS.Fd())); err != nil {
-		err = fmt.Errorf("failed to move veth to host netns: %v", err)
-		return
-	}
-
-	err = hostNS.Do(func(_ ns.NetNS) error {
-		hostVeth, err := netlink.LinkByName(hostVethName)
-		if err != nil {
-			return fmt.Errorf("failed to lookup %q in %q: %v", hostVethName, hostNS.Path(), err)
-		}
-
-		if err = netlink.LinkSetUp(hostVeth); err != nil {
-			return fmt.Errorf("failed to set %q up: %v", hostVethName, err)
-		}
-		return nil
-	})
-	return
-}
-
-// DelLinkByName removes an interface link.
-func DelLinkByName(ifName string) error {
-	iface, err := netlink.LinkByName(ifName)
-	if err != nil {
-		return fmt.Errorf("failed to lookup %q: %v", ifName, err)
-	}
-
-	if err = netlink.LinkDel(iface); err != nil {
-		return fmt.Errorf("failed to delete %q: %v", ifName, err)
-	}
-
-	return nil
-}
-
-// DelLinkByNameAddr remove an interface returns its IP address
-// of the specified family
-func DelLinkByNameAddr(ifName string, family int) (*net.IPNet, error) {
-	iface, err := netlink.LinkByName(ifName)
-	if err != nil {
-		return nil, fmt.Errorf("failed to lookup %q: %v", ifName, err)
-	}
-
-	addrs, err := netlink.AddrList(iface, family)
-	if err != nil || len(addrs) == 0 {
-		return nil, fmt.Errorf("failed to get IP addresses for %q: %v", ifName, err)
-	}
-
-	if err = netlink.LinkDel(iface); err != nil {
-		return nil, fmt.Errorf("failed to delete %q: %v", ifName, err)
-	}
-
-	return addrs[0].IPNet, nil
-}

vendor/github.com/containernetworking/cni/pkg/ip/route.go

@@ -1,47 +0,0 @@
-// Copyright 2015 CNI authors
-//
-// Licensed under the Apache License, Version 2.0 (the "License");
-// you may not use this file except in compliance with the License.
-// You may obtain a copy of the License at
-//
-//     http://www.apache.org/licenses/LICENSE-2.0
-//
-// Unless required by applicable law or agreed to in writing, software
-// distributed under the License is distributed on an "AS IS" BASIS,
-// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-// See the License for the specific language governing permissions and
-// limitations under the License.
-
-package ip
-
-import (
-	"net"
-
-	"github.com/vishvananda/netlink"
-)
-
-// AddDefaultRoute sets the default route on the given gateway.
-func AddDefaultRoute(gw net.IP, dev netlink.Link) error {
-	_, defNet, _ := net.ParseCIDR("0.0.0.0/0")
-	return AddRoute(defNet, gw, dev)
-}
-
-// AddRoute adds a universally-scoped route to a device.
-func AddRoute(ipn *net.IPNet, gw net.IP, dev netlink.Link) error {
-	return netlink.RouteAdd(&netlink.Route{
-		LinkIndex: dev.Attrs().Index,
-		Scope:     netlink.SCOPE_UNIVERSE,
-		Dst:       ipn,
-		Gw:        gw,
-	})
-}
-
-// AddHostRoute adds a host-scoped route to a device.
-func AddHostRoute(ipn *net.IPNet, gw net.IP, dev netlink.Link) error {
-	return netlink.RouteAdd(&netlink.Route{
-		LinkIndex: dev.Attrs().Index,
-		Scope:     netlink.SCOPE_HOST,
-		Dst:       ipn,
-		Gw:        gw,
-	})
-}

vendor/github.com/containernetworking/cni/pkg/ipam/ipam.go

@@ -1,68 +0,0 @@
-// Copyright 2015 CNI authors
-//
-// Licensed under the Apache License, Version 2.0 (the "License");
-// you may not use this file except in compliance with the License.
-// You may obtain a copy of the License at
-//
-//     http://www.apache.org/licenses/LICENSE-2.0
-//
-// Unless required by applicable law or agreed to in writing, software
-// distributed under the License is distributed on an "AS IS" BASIS,
-// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-// See the License for the specific language governing permissions and
-// limitations under the License.
-
-package ipam
-
-import (
-	"fmt"
-	"os"
-
-	"github.com/containernetworking/cni/pkg/invoke"
-	"github.com/containernetworking/cni/pkg/ip"
-	"github.com/containernetworking/cni/pkg/types"
-
-	"github.com/vishvananda/netlink"
-)
-
-func ExecAdd(plugin string, netconf []byte) (*types.Result, error) {
-	return invoke.DelegateAdd(plugin, netconf)
-}
-
-func ExecDel(plugin string, netconf []byte) error {
-	return invoke.DelegateDel(plugin, netconf)
-}
-
-// ConfigureIface takes the result of IPAM plugin and
-// applies to the ifName interface
-func ConfigureIface(ifName string, res *types.Result) error {
-	link, err := netlink.LinkByName(ifName)
-	if err != nil {
-		return fmt.Errorf("failed to lookup %q: %v", ifName, err)
-	}
-
-	if err := netlink.LinkSetUp(link); err != nil {
-		return fmt.Errorf("failed to set %q UP: %v", ifName, err)
-	}
-
-	// TODO(eyakubovich): IPv6
-	addr := &netlink.Addr{IPNet: &res.IP4.IP, Label: ""}
-	if err = netlink.AddrAdd(link, addr); err != nil {
-		return fmt.Errorf("failed to add IP addr to %q: %v", ifName, err)
-	}
-
-	for _, r := range res.IP4.Routes {
-		gw := r.GW
-		if gw == nil {
-			gw = res.IP4.Gateway
-		}
-		if err = ip.AddRoute(&r.Dst, gw, link); err != nil {
-			// we skip over duplicate routes as we assume the first one wins
-			if !os.IsExist(err) {
-				return fmt.Errorf("failed to add route '%v via %v dev %v': %v", r.Dst, gw, ifName, err)
-			}
-		}
-	}
-
-	return nil
-}

vendor/github.com/containernetworking/cni/pkg/ns/README.md

@@ -1,31 +0,0 @@
-### Namespaces, Threads, and Go
-On Linux each OS thread can have a different network namespace.  Go's thread scheduling model switches goroutines between OS threads based on OS thread load and whether the goroutine would block other goroutines.  This can result in a goroutine switching network namespaces without notice and lead to errors in your code.
-
-### Namespace Switching
-Switching namespaces with the `ns.Set()` method is not recommended without additional strategies to prevent unexpected namespace changes when your goroutines switch OS threads.
-
-Go provides the `runtime.LockOSThread()` function to ensure a specific goroutine executes on its current OS thread and prevents any other goroutine from running in that thread until the locked one exits.  Careful usage of `LockOSThread()` and goroutines can provide good control over which network namespace a given goroutine executes in.
-
-For example, you cannot rely on the `ns.Set()` namespace being the current namespace after the `Set()` call unless you do two things.  First, the goroutine calling `Set()` must have previously called `LockOSThread()`.  Second, you must ensure `runtime.UnlockOSThread()` is not called somewhere in-between.  You also cannot rely on the initial network namespace remaining the current network namespace if any other code in your program switches namespaces, unless you have already called `LockOSThread()` in that goroutine.  Note that `LockOSThread()` prevents the Go scheduler from optimally scheduling goroutines for best performance, so `LockOSThread()` should only be used in small, isolated goroutines that release the lock quickly.
-
-### Do() The Recommended Thing
-The `ns.Do()` method provides control over network namespaces for you by implementing these strategies. All code dependent on a particular network namespace should be wrapped in the `ns.Do()` method to ensure the correct namespace is selected for the duration of your code.  For example:
-
-```go
-targetNs, err := ns.NewNS()
-if err != nil {
-    return err
-}
-err = targetNs.Do(func(hostNs ns.NetNS) error {
-	dummy := &netlink.Dummy{
-		LinkAttrs: netlink.LinkAttrs{
-			Name: "dummy0",
-		},
-	}
-	return netlink.LinkAdd(dummy)
-})
-```
-
-### Further Reading
- - https://github.com/golang/go/wiki/LockOSThread
- - http://morsmachine.dk/go-scheduler