Merge pull request #62152 from smarterclayton/use_client_store

Automatic merge from submit-queue (batch tested with PRs 63001, 62152, 61950). If you want to cherry-pick this change to another branch, please follow the instructions <a href="https://github.com/kubernetes/community/blob/master/contributors/devel/cherry-picks.md">here</a>.

When bootstrapping a client cert, store it with other client certs

The kubelet uses two different locations to store certificates on
initial bootstrap and then on subsequent rotation:

* bootstrap: certDir/kubelet-client.(crt|key)
* rotation:  certDir/kubelet-client-(DATE|current).pem

Bootstrap also creates an initial node.kubeconfig that points to the
certs. Unfortunately, with short rotation the node.kubeconfig then
becomes out of date because it points to the initial cert/key, not the
rotated cert key.

Alter the bootstrap code to store client certs exactly as if they would
be rotated (using the same cert Store code), and reference the PEM file
containing cert/key from node.kubeconfig, which is supported by kubectl
and other Go tooling. This ensures that the node.kubeconfig continues to
be valid past the first expiration.

Example:

```
bootstrap:
  writes to certDir/kubelet-client-DATE.pem and symlinks to certDir/kubelet-client-current.pem
  writes node.kubeconfig pointing to certDir/kubelet-client-current.pem
rotation:
  writes to certDir/kubelet-client-DATE.pem and symlinks to certDir/kubelet-client-current.pem
```

This will also allow us to remove the wierd "init store with bootstrap cert" stuff, although I'd prefer to do that in a follow up.

@mikedanese @liggitt as per discussion on Slack today

```release-note
The `--bootstrap-kubeconfig` argument to Kubelet previously created the first bootstrap client credentials in the certificates directory as `kubelet-client.key` and `kubelet-client.crt`.  Subsequent certificates created by cert rotation were created in a combined PEM file that was atomically rotated as `kubelet-client-DATE.pem` in that directory, which meant clients relying on the `node.kubeconfig` generated by bootstrapping would never use a rotated cert.  The initial bootstrap certificate is now generated into the cert directory as a PEM file and symlinked to `kubelet-client-current.pem` so that the generated kubeconfig remains valid after rotation.
```

Kubernetes-commit: 939c0783e191a940d009399f7d6d00251feb025a

util/cert/io.go

@@ -17,7 +17,11 @@ limitations under the License.
 package cert
 
 import (
+	"crypto"
+	"crypto/ecdsa"
+	"crypto/rsa"
 	"crypto/x509"
+	"encoding/pem"
 	"fmt"
 	"io/ioutil"
 	"os"
@@ -101,6 +105,27 @@ func LoadOrGenerateKeyFile(keyPath string) (data []byte, wasGenerated bool, err
 	return generatedData, true, nil
 }
 
+// MarshalPrivateKeyToPEM converts a known private key type of RSA or ECDSA to
+// a PEM encoded block or returns an error.
+func MarshalPrivateKeyToPEM(privateKey crypto.PrivateKey) ([]byte, error) {
+	switch t := privateKey.(type) {
+	case *ecdsa.PrivateKey:
+		derBytes, err := x509.MarshalECPrivateKey(t)
+		if err != nil {
+			return nil, err
+		}
+		privateKeyPemBlock := &pem.Block{
+			Type:  ECPrivateKeyBlockType,
+			Bytes: derBytes,
+		}
+		return pem.EncodeToMemory(privateKeyPemBlock), nil
+	case *rsa.PrivateKey:
+		return EncodePrivateKeyPEM(t), nil
+	default:
+		return nil, fmt.Errorf("private key is not a recognized type: %T", privateKey)
+	}
+}
+
 // NewPool returns an x509.CertPool containing the certificates in the given PEM-encoded file.
 // Returns an error if the file could not be read, a certificate could not be parsed, or if the file does not contain any certificates
 func NewPool(filename string) (*x509.CertPool, error) {

util/certificate/certificate_store.go

@@ -46,6 +46,15 @@ type fileStore struct {
 	keyFile        string
 }
 
+// FileStore is a store that provides certificate retrieval as well as
+// the path on disk of the current PEM.
+type FileStore interface {
+	Store
+	// CurrentPath returns the path on disk of the current certificate/key
+	// pair encoded as PEM files.
+	CurrentPath() string
+}
+
 // NewFileStore returns a concrete implementation of a Store that is based on
 // storing the cert/key pairs in a single file per pair on disk in the
 // designated directory. When starting up it will look for the currently
@@ -64,7 +73,7 @@ func NewFileStore(
 	certDirectory string,
 	keyDirectory string,
 	certFile string,
-	keyFile string) (Store, error) {
+	keyFile string) (FileStore, error) {
 
 	s := fileStore{
 		pairNamePrefix: pairNamePrefix,
@@ -79,6 +88,11 @@ func NewFileStore(
 	return &s, nil
 }
 
+// CurrentPath returns the path to the current version of these certificates.
+func (s *fileStore) CurrentPath() string {
+	return filepath.Join(s.certDirectory, s.filename(currentPair))
+}
+
 // recover checks if there is a certificate rotation that was interrupted while
 // progress, and if so, attempts to recover to a good state.
 func (s *fileStore) recover() error {