mirror of
https://github.com/k3s-io/kubernetes.git
synced 2025-07-23 11:50:44 +00:00
Merge pull request #26908 from pmorie/pv-dynprov
Automatic merge from submit-queue dynamic provisioning proposal Proposal for dynamic provisioning using storage classes; supercedes #17056 @kubernetes/sig-storage
This commit is contained in:
k8s-merge-robot
GitHub
commit
cdb4df8ea5
260
docs/proposals/volume-provisioning.md
Normal file
260
docs/proposals/volume-provisioning.md
Normal file
@ -0,0 +1,260 @@
|
||||
<!-- BEGIN MUNGE: UNVERSIONED_WARNING -->
|
||||
|
||||
<!-- BEGIN STRIP_FOR_RELEASE -->
|
||||
|
||||
<img src="http://kubernetes.io/img/warning.png" alt="WARNING"
|
||||
width="25" height="25">
|
||||
<img src="http://kubernetes.io/img/warning.png" alt="WARNING"
|
||||
width="25" height="25">
|
||||
<img src="http://kubernetes.io/img/warning.png" alt="WARNING"
|
||||
width="25" height="25">
|
||||
<img src="http://kubernetes.io/img/warning.png" alt="WARNING"
|
||||
width="25" height="25">
|
||||
<img src="http://kubernetes.io/img/warning.png" alt="WARNING"
|
||||
width="25" height="25">
|
||||
|
||||
<h2>PLEASE NOTE: This document applies to the HEAD of the source tree</h2>
|
||||
|
||||
If you are using a released version of Kubernetes, you should
|
||||
refer to the docs that go with that version.
|
||||
|
||||
Documentation for other releases can be found at
|
||||
[releases.k8s.io](http://releases.k8s.io).
|
||||
</strong>
|
||||
--
|
||||
|
||||
<!-- END STRIP_FOR_RELEASE -->
|
||||
|
||||
<!-- END MUNGE: UNVERSIONED_WARNING -->
|
||||
|
||||
## Abstract
|
||||
|
||||
Real Kubernetes clusters have a variety of volumes which differ widely in
|
||||
size, iops performance, retention policy, and other characteristics.
|
||||
Administrators need a way to dynamically provision volumes of these different
|
||||
types to automatically meet user demand.
|
||||
|
||||
A new mechanism called 'storage classes' is proposed to provide this
|
||||
capability.
|
||||
|
||||
## Motivation
|
||||
|
||||
In Kubernetes 1.2, an alpha form of limited dynamic provisioning was added
|
||||
that allows a single volume type to be provisioned in clouds that offer
|
||||
special volume types.
|
||||
|
||||
In Kubernetes 1.3, a label selector was added to persistent volume claims to
|
||||
allow administrators to create a taxonomy of volumes based on the
|
||||
characteristics important to them, and to allow users to make claims on those
|
||||
volumes based on those characteristics. This allows flexibility when claiming
|
||||
existing volumes; the same flexibility is needed when dynamically provisioning
|
||||
volumes.
|
||||
|
||||
After gaining experience with dynamic provisioning after the 1.2 release, we
|
||||
want to create a more flexible feature that allows configuration of how
|
||||
different storage classes are provisioned and supports provisioning multiple
|
||||
types of volumes within a single cloud.
|
||||
|
||||
### Out-of-tree provisioners
|
||||
|
||||
One of our goals is to enable administrators to create out-of-tree
|
||||
provisioners, that is, provisioners whose code does not live in the Kubernetes
|
||||
project. Our experience since the 1.2 release with dynamic provisioning has
|
||||
shown that it is impossible to anticipate every aspect and manner of
|
||||
provisioning that administrators will want to perform. The proposed design
|
||||
should not prevent future work to allow out-of-tree provisioners.
|
||||
|
||||
## Design
|
||||
|
||||
This design represents the minimally viable changes required to provision based on storage classe configuration. Additional incremental features may be added as a separte effort.
|
||||
|
||||
We propose that:
|
||||
|
||||
1. For the base impelementation storage class and volume selectors are mutually exclusive.
|
||||
|
||||
2. An api object will be incubated in extensions/v1beta1 named `storage` to hold the a `StorageClass`
|
||||
API resource. Each StorageClass object contains parameters required by the provisioner to provision volumes of that class. These parameters are opaque to the user.
|
||||
|
||||
3. `PersistentVolume.Spec.Class` attribute is added to volumes. This attribute
|
||||
is optional and specifies which `StorageClass` instance represents
|
||||
storage characteristics of a particular PV.
|
||||
|
||||
During incubation, `Class` is an annotation and not
|
||||
actual attribute.
|
||||
|
||||
4. `PersistentVolume` instances do not require labels by the provisioner.
|
||||
|
||||
5. `PersistentVolumeClaim.Spec.Class` attribute is added to claims. This
|
||||
attribute specifies that only a volume with equal
|
||||
`PersistentVolume.Spec.Class` value can satisfy a claim.
|
||||
|
||||
During incubation, `Class` is just an annotation and not
|
||||
actual attribute.
|
||||
|
||||
6. The existing provisioner plugin implementations be modified to accept
|
||||
parameters as specified via `StorageClass`.
|
||||
|
||||
7. The persistent volume controller modified to invoke provisioners using `StorageClass` configuration and bind claims with `PersistentVolumeClaim.Spec.Class` to volumes with equivilant `PersistentVolume.Spec.Class`
|
||||
|
||||
8. The existing alpha dynamic provisioning feature be phased out in the
|
||||
next release.
|
||||
|
||||
### Controller workflow for provisioning volumes
|
||||
|
||||
1. When a new claim is submitted, the controller attempts to find an existing
|
||||
volume that will fulfill the claim.
|
||||
|
||||
1. If the claim has non-empty `claim.Spec.Class`, only PVs with the same
|
||||
`pv.Spec.Class` are considered.
|
||||
|
||||
2. If the claim has empty `claim.Spec.Class`, all existing PVs are
|
||||
considered.
|
||||
|
||||
All "considered" volumes are evaluated and the
|
||||
smallest matching volume is bound to the claim.
|
||||
|
||||
2. If no volume is found for the claim and `claim.Spec.Class` is not set or is
|
||||
empty string dynamic provisioning is disabled.
|
||||
|
||||
3. If `claim.Spec.Class` is set the controller tries to find instance of StorageClass with this name. If no
|
||||
such StorageClass is found, the controller goes back to step 1. and
|
||||
periodically retries finding a matching volume or storage class again until
|
||||
a match is found. The claim is `Pending` during this period.
|
||||
|
||||
4. With StorageClass instance, the controller finds volume plugin specified by
|
||||
StorageClass.ProvisionerType.
|
||||
|
||||
5. All provisioners are in-tree; they implement an interface called
|
||||
`ProvisionableVolumePlugin`, which has a method called `NewProvisioner`
|
||||
that returns a new provisioner.
|
||||
|
||||
6. The controller calls volume plugin `Provision` with ProvisionerParameters from the `StorageClass` configuration object.
|
||||
|
||||
7. If `Provision` returns an error, the controller generates an event on the
|
||||
claim and goes back to step 1., i.e. it will retry provisioning periodically
|
||||
|
||||
8. If `Provision` returns no error, the controller creates the returned
|
||||
`api.PersistentVolume`, fills its `Class` attribute with `claim.Spec.Class`
|
||||
and makes it already bound to the claim
|
||||
|
||||
1. If the create operation for the `api.PersistentVolume` fails, it is
|
||||
retried
|
||||
|
||||
2. If the create operation does not succeed in reasonable time, the
|
||||
controller attempts to delete the provisioned volume and creates an event
|
||||
on the claim
|
||||
|
||||
Existing behavior is un-changed for claims that do not specify `claim.Spec.Class`.
|
||||
|
||||
### `StorageClass` API
|
||||
|
||||
A new API group should hold the API for storage classes, following the pattern
|
||||
of autoscaling, metrics, etc. To allow for future storage-related APIs, we
|
||||
should call this new API group `storage` and incubate in extensions/v1beta1.
|
||||
|
||||
Storage classes will be represented by an API object called `StorageClass`:
|
||||
|
||||
```go
|
||||
package storage
|
||||
|
||||
// StorageClass describes the parameters for a class of storage for
|
||||
// which PersistentVolumes can be dynamically provisioned.
|
||||
//
|
||||
// StorageClasses are non-namespaced; the name of the storage class
|
||||
// according to etcd is in ObjectMeta.Name.
|
||||
type StorageClass struct {
|
||||
unversioned.TypeMeta `json:",inline"`
|
||||
ObjectMeta `json:"metadata,omitempty"`
|
||||
|
||||
// ProvisionerType indicates the type of the provisioner.
|
||||
ProvisionerType string `json:"provisionerType,omitempty"`
|
||||
|
||||
// Parameters for dynamic volume provisioner.
|
||||
ProvisionerParameters map[string]string `json:"provisionerParameters,omitempty"`
|
||||
}
|
||||
|
||||
```
|
||||
|
||||
`PersistentVolumeClaimSpec` and `PersistentVolumeSpec` both get Class attribute
|
||||
(the existing annotation is used during incubation):
|
||||
|
||||
```go
|
||||
type PersistentVolumeClaimSpec struct {
|
||||
// Name of requested storage class. If non-empty, only PVs with this
|
||||
// pv.Spec.Class will be considered for binding and if no such PV is
|
||||
// available, StorageClass with this name will be used to dynamically
|
||||
// provision the volume.
|
||||
Class string
|
||||
...
|
||||
}
|
||||
|
||||
type PersistentVolumeSpec struct {
|
||||
// Name of StorageClass instance that this volume belongs to.
|
||||
Class string
|
||||
...
|
||||
}
|
||||
```
|
||||
|
||||
Storage classes are natural to think of as a global resource, since they:
|
||||
|
||||
1. Align with PersistentVolumes, which are a global resource
|
||||
2. Are administrator controlled
|
||||
|
||||
### Provisioning configuration
|
||||
|
||||
With the scheme outlined above the provisioner creates PVs using parameters specified in the `StorageClass` object.
|
||||
|
||||
### Provisioner interface changes
|
||||
|
||||
`struct volume.VolumeOptions` (containing parameters for a provisioner plugin)
|
||||
will be extended to contain StorageClass.ProvisionerParameters.
|
||||
|
||||
The existing provisioner implementations will be modified to accept the StorageClass configuration object.
|
||||
|
||||
### PV Controller Changes
|
||||
|
||||
The persistent volume controller will be modified to implement the new
|
||||
workflow described in this proposal. The changes will be limited to the
|
||||
`provisionClaimOperation` method, which is responsible for invoking the
|
||||
provisioner and to favor existing volumes before provisioning a new one.
|
||||
|
||||
## Examples
|
||||
|
||||
### AWS provisioners with distinct QoS
|
||||
|
||||
This example shows two storage classes, "aws-fast" and "aws-slow".
|
||||
|
||||
```
|
||||
apiVersion: v1
|
||||
kind: StorageClass
|
||||
metadata:
|
||||
name: aws-fast
|
||||
provisionerType: kubernetes.io/aws-ebs
|
||||
provisionerParameters:
|
||||
zone: us-east-1b
|
||||
type: ssd
|
||||
|
||||
|
||||
apiVersion: v1
|
||||
kind: StorageClass
|
||||
metadata:
|
||||
name: aws-slow
|
||||
provisionerType: kubernetes.io/aws-ebs
|
||||
provisionerParameters:
|
||||
zone: us-east-1b
|
||||
type: spinning
|
||||
```
|
||||
|
||||
# Additional Implementation Details
|
||||
|
||||
0. Annotation `volume.alpha.kubernetes.io/storage-class` is used instead of `claim.Spec.Class` and `volume.Spec.Class` during incubation.
|
||||
|
||||
1. `claim.Spec.Selector` and `claim.Spec.Class` are mutually exclusive. User can either match existing volumes with `Selector` XOR match existing volumes with `Class` and get dynamic provisioning by using `Class`. This simplifies initial PR and also provisioners.
|
||||
|
||||
# Cloud Providers
|
||||
|
||||
Since the `volume.alpha.kubernetes.io/storage-class` is in use a `StorageClass` must be defined to support provisioning. No default is assumed as before.
|
||||
|
||||
<!-- BEGIN MUNGE: GENERATED_ANALYTICS -->
|
||||
[]()
|
||||
<!-- END MUNGE: GENERATED_ANALYTICS -->
|
||||
Loading…
Reference in New Issue
Block a user