Merge pull request #55684 from wu-qiang/kms-plugin-grpc-api

Automatic merge from submit-queue (batch tested with PRs 58437, 59490, 55684). If you want to cherry-pick this change to another branch, please follow the instructions <a href="https://github.com/kubernetes/community/blob/master/contributors/devel/cherry-picks.md">here</a>.

gRPC-based KMS plugin service

**What this PR does / why we need it**:
Implement for issue https://github.com/kubernetes/kubernetes/issues/51965
**Which issue(s) this PR fixes** *(optional, in `fixes #<issue number>(, fixes #<issue_number>, ...)` format, will close the issue(s) when PR gets merged)*:
Fixes #51965

**Special notes for your reviewer**:
@destijl @sakshamsharma @deads2k @ericchiang
The implementation based on the document https://docs.google.com/document/d/1S_Wgn-psI0Z7SYGvp-83ePte5oUNMr4244uanGLYUmw/edit
**Release note**:

```release-note
Implement envelope service with gRPC, so that KMS providers can be pulled out from API server.
```

hack/.golint_failures

@@ -594,6 +594,7 @@ staging/src/k8s.io/apiserver/pkg/storage/storagebackend
 staging/src/k8s.io/apiserver/pkg/storage/testing
 staging/src/k8s.io/apiserver/pkg/storage/tests
 staging/src/k8s.io/apiserver/pkg/storage/value
+staging/src/k8s.io/apiserver/pkg/storage/value/encrypt/envelope/v1beta1
 staging/src/k8s.io/apiserver/pkg/util/feature
 staging/src/k8s.io/apiserver/pkg/util/flag
 staging/src/k8s.io/apiserver/pkg/util/proxy

hack/update-generated-kms-dockerized.sh

@@ -0,0 +1,62 @@
+#!/bin/bash
+
+# Copyright 2018 The Kubernetes Authors.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+set -o errexit
+set -o nounset
+set -o pipefail
+
+KUBE_ROOT=$(dirname "${BASH_SOURCE}")/..
+KUBE_KMS_GRPC_ROOT="${KUBE_ROOT}/staging/src/k8s.io/apiserver/pkg/storage/value/encrypt/envelope/v1beta1/"
+source "${KUBE_ROOT}/hack/lib/init.sh"
+
+kube::golang::setup_env
+
+BINS=(
+	vendor/k8s.io/code-generator/cmd/go-to-protobuf/protoc-gen-gogo
+)
+make -C "${KUBE_ROOT}" WHAT="${BINS[*]}"
+
+if [[ -z "$(which protoc)" || "$(protoc --version)" != "libprotoc 3."* ]]; then
+  echo "Generating protobuf requires protoc 3.0.0-beta1 or newer. Please download and"
+  echo "install the platform appropriate Protobuf package for your OS: "
+  echo
+  echo "  https://github.com/google/protobuf/releases"
+  echo
+  echo "WARNING: Protobuf changes are not being validated"
+  exit 1
+fi
+
+function cleanup {
+	rm -f ${KUBE_KMS_GRPC_ROOT}/service.pb.go.bak
+}
+
+trap cleanup EXIT
+
+gogopath=$(dirname $(kube::util::find-binary "protoc-gen-gogo"))
+
+PATH="${gogopath}:${PATH}" \
+  protoc \
+  --proto_path="${KUBE_KMS_GRPC_ROOT}" \
+  --proto_path="${KUBE_ROOT}/vendor" \
+  --gogo_out=plugins=grpc:${KUBE_KMS_GRPC_ROOT} ${KUBE_KMS_GRPC_ROOT}/service.proto
+
+# Update boilerplate for the generated file.
+echo "$(cat hack/boilerplate/boilerplate.go.txt ${KUBE_KMS_GRPC_ROOT}/service.pb.go)" > ${KUBE_KMS_GRPC_ROOT}/service.pb.go
+sed -i".bak" "s/Copyright YEAR/Copyright $(date '+%Y')/g" ${KUBE_KMS_GRPC_ROOT}/service.pb.go
+
+# Run gofmt to clean up the generated code.
+kube::golang::verify_go_version
+gofmt -l -s -w ${KUBE_KMS_GRPC_ROOT}/service.pb.go

hack/update-generated-kms.sh

@@ -0,0 +1,29 @@
+#!/bin/bash
+
+# Copyright 2018 The Kubernetes Authors.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+set -o errexit
+set -o nounset
+set -o pipefail
+
+KUBE_ROOT=$(dirname "${BASH_SOURCE}")/..
+
+# NOTE: All output from this script needs to be copied back to the calling
+# source tree.  This is managed in kube::build::copy_output in build/common.sh.
+# If the output set is changed update that function.
+
+${KUBE_ROOT}/build/run.sh hack/update-generated-kms-dockerized.sh "$@"
+
+# ex: ts=2 sw=2 et filetype=sh

hack/verify-generated-kms.sh

@@ -0,0 +1,45 @@
+#!/bin/bash
+
+# Copyright 2018 The Kubernetes Authors.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+set -o errexit
+set -o nounset
+set -o pipefail
+
+KUBE_ROOT=$(dirname "${BASH_SOURCE}")/..
+KUBE_KMS_GRPC_ROOT="${KUBE_ROOT}/staging/src/k8s.io/apiserver/pkg/storage/value/encrypt/envelope/v1beta1/"
+source "${KUBE_ROOT}/hack/lib/init.sh"
+
+kube::golang::setup_env
+
+function cleanup {
+	rm -rf ${KUBE_KMS_GRPC_ROOT}/_tmp/
+}
+
+trap cleanup EXIT
+
+mkdir -p ${KUBE_KMS_GRPC_ROOT}/_tmp
+cp ${KUBE_KMS_GRPC_ROOT}/service.pb.go ${KUBE_KMS_GRPC_ROOT}/_tmp/
+
+ret=0
+KUBE_VERBOSE=3 "${KUBE_ROOT}/hack/update-generated-kms.sh"
+diff -I "gzipped FileDescriptorProto" -I "0x" -Naupr ${KUBE_KMS_GRPC_ROOT}/_tmp/service.pb.go ${KUBE_KMS_GRPC_ROOT}/service.pb.go || ret=$?
+if [[ $ret -eq 0 ]]; then
+    echo "Generated KMS gRPC is up to date."
+    cp ${KUBE_KMS_GRPC_ROOT}/_tmp/service.pb.go ${KUBE_KMS_GRPC_ROOT}/
+else
+    echo "Generated KMS gRPC is out of date. Please run hack/update-generated-kms.sh"
+    exit 1
+fi