Dynamic provisioning

    The OKD persistent volume framework enables this functionality and allows administrators to provision a cluster with persistent storage. The framework also gives users a way to request those resources without having any knowledge of the underlying infrastructure.

    Many storage types are available for use as persistent volumes in OKD. While all of them can be statically provisioned by an administrator, some types of storage are created dynamically using the built-in provider and plug-in APIs.

    OKD provides the following provisioner plug-ins, which have generic implementations for dynamic provisioning that use the cluster’s configured provider’s API to create new storage resources:

    Any chosen provisioner plug-in also requires configuration for the relevant cloud, host, or third-party provider as per the relevant documentation.

    The Cluster Storage Operator might install a default storage class depending on the platform in use. This storage class is owned and controlled by the operator. It cannot be deleted or modified beyond defining annotations and labels. If different behavior is desired, you must define a custom storage class.

    The following sections describe the basic definition for a StorageClass object and specific examples for each of the supported plug-in types.

    The following resource shows the parameters and default values that you use to configure a storage class. This example uses the AWS ElasticBlockStore (EBS) object definition.

    Sample StorageClass definition

    1(required) The API object type.
    2(required) The current apiVersion.
    3(required) The name of the storage class.
    4(optional) Annotations for the storage class.
    5(required) The type of provisioner associated with this storage class.
    6(optional) The parameters required for the specific provisioner, this will change from plug-in to plug-in.

    Storage class annotations

    To set a storage class as the cluster-wide default, add the following annotation to your storage class metadata:

    1. storageclass.kubernetes.io/is-default-class: "true"

    For example:

    1. apiVersion: storage.k8s.io/v1
    2. kind: StorageClass
    3. metadata:
    4.   annotations:
    5.     storageclass.kubernetes.io/is-default-class: "true"
    6. ...

    This enables any persistent volume claim (PVC) that does not specify a specific storage class to automatically be provisioned through the default storage class.

    The beta annotation storageclass.beta.kubernetes.io/is-default-class is still working; however, it will be removed in a future release.

    To set a storage class description, add the following annotation to your storage class metadata:

    1. kubernetes.io/description: My Storage Class Description

    For example:

    1. apiVersion: storage.k8s.io/v1
    2. kind: StorageClass
    3. metadata:
    4.   annotations:
    5.     kubernetes.io/description: My Storage Class Description
    6. ...

    RHOSP Cinder object definition

    cinder-storageclass.yaml

    1. kind: StorageClass
    2. apiVersion: storage.k8s.io/v1
    3. metadata:
    4.   name: gold
    5. provisioner: kubernetes.io/cinder
    6. parameters:
    7.   type: fast  (1)
    8.   availability: nova (2)
    9.   fsType: ext4 (3)

    Once installed, the OpenStack Manila CSI Driver Operator and ManilaDriver automatically create the required storage classes for all available Manila share types needed for dynamic provisioning.

    AWS Elastic Block Store (EBS) object definition

    aws-ebs-storageclass.yaml

    1. kind: StorageClass
    2. apiVersion: storage.k8s.io/v1
    3. metadata:
    4.   name: slow
    5. parameters:
    6.   type: io1 (1)
    7.   iopsPerGB: "10" (2)
    8.   encrypted: "true" (3)
    9.   kmsKeyId: keyvalue (4)
    10.   fsType: ext4 (5)
    1(required) Select from io1, , sc1, st1. The default is gp2. See the for valid Amazon Resource Name (ARN) values.
    2(optional) Only for io1 volumes. I/O operations per second per GiB. The AWS volume plug-in multiplies this with the size of the requested volume to compute IOPS of the volume. The value cap is 20,000 IOPS, which is the maximum supported by AWS. See the AWS documentation for further details.
    3(optional) Denotes whether to encrypt the EBS volume. Valid values are true or false.
    4(optional) The full ARN of the key to use when encrypting the volume. If none is supplied, but encypted is set to true, then AWS generates a key. See the for a valid ARN value.
    5(optional) File system that is created on dynamically provisioned volumes. This value is copied to the fsType field of dynamically provisioned persistent volumes and the file system is created when the volume is mounted for the first time. The default value is ext4.

    Azure Disk object definition

    azure-advanced-disk-storageclass.yaml

    1Using WaitForFirstConsumer is strongly recommended. This provisions the volume while allowing enough storage to schedule the pod on a free worker node from an available zone.
    2Possible values are Shared (default), Managed, and Dedicated.

    Red Hat only supports the use of kind: Managed in the storage class.

    With Shared and Dedicated, Azure creates unmanaged disks, while OKD creates a managed disk for machine OS (root) disks. But because Azure Disk does not allow the use of both managed and unmanaged disks on a node, unmanaged disks created with Shared or Dedicated cannot be attached to OKD nodes.

    3Azure storage account SKU tier. Default is empty. Note that Premium VMs can attach both Standard_LRS and Premium_LRS disks, Standard VMs can only attach Standard_LRS disks, Managed VMs can only attach managed disks, and unmanaged VMs can only attach unmanaged disks.

    1. If kind is set to Shared, Azure creates all unmanaged disks in a few shared storage accounts in the same resource group as the cluster.

    2. If kind is set to Managed, Azure creates new managed disks.

    3. If kind is set to Dedicated and a storageAccount is specified, Azure uses the specified storage account for the new unmanaged disk in the same resource group as the cluster. For this to work:

      • The specified storage account must be in the same region.

      • Azure Cloud Provider must have write access to the storage account.

    4. If kind is set to Dedicated and a storageAccount is not specified, Azure creates a new dedicated storage account for the new unmanaged disk in the same resource group as the cluster.

    Procedure

    1. Define a ClusterRole object that allows access to create and view secrets:

      1. apiVersion: rbac.authorization.k8s.io/v1
      2. kind: ClusterRole
      3. metadata:
      4. #  name: system:azure-cloud-provider
      5.   name: <persistent-volume-binder-role> (1)
      6. rules:
      7. - apiGroups: ['']
      8.   resources: ['secrets']
      9.   verbs:     ['get','create']
      1The name of the cluster role to view and create secrets.

    2. Add the cluster role to the service account:

      1. $ oc adm policy add-cluster-role-to-user <persistent-volume-binder-role>

      Example output

      1.  system:serviceaccount:kube-system:persistent-volume-binder

    3. Create the Azure File StorageClass object:

      1. kind: StorageClass
      2. apiVersion: storage.k8s.io/v1
      3. metadata:
      4.   name: <azure-file> (1)
      5. provisioner: kubernetes.io/azure-file
      6. parameters:
      7.   location: eastus (2)
      8.   skuName: Standard_LRS (3)
      9.   storageAccount: <storage-account> (4)
      10. reclaimPolicy: Delete
      11. volumeBindingMode: Immediate

    Considerations when using Azure File

    The following file system features are not supported by the default Azure File storage class:

    • Symlinks

    • Hard links

    • Extended attributes

    • Sparse files

    Additionally, the owner user identifier (UID) of the Azure File mounted directory is different from the process UID of the container. The uid mount option can be specified in the StorageClass object to define a specific user identifier to use for the mounted directory.

    The following StorageClass object demonstrates modifying the user and group identifier, along with enabling symlinks for the mounted directory.

    1. kind: StorageClass
    2. apiVersion: storage.k8s.io/v1
    3. metadata:
    4.   name: azure-file
    5. mountOptions:
    6.   - uid=1500 (1)
    7.   - gid=1500 (2)
    8.   - mfsymlinks (3)
    9. provisioner: kubernetes.io/azure-file
    10. parameters:
    11.   location: eastus
    12.   skuName: Standard_LRS
    13. reclaimPolicy: Delete
    14. volumeBindingMode: Immediate
    1Specifies the user identifier to use for the mounted directory.
    2Specifies the group identifier to use for the mounted directory.
    3Enables symlinks.

    GCE PersistentDisk (gcePD) object definition

    gce-pd-storageclass.yaml

    1. apiVersion: storage.k8s.io/v1
    2. kind: StorageClass
    3. metadata:
    4.   name: standard
    5.   annotations:
    6.     storageclass.kubernetes.io/is-default-class: "true"
    7. provisioner: kubernetes.io/gce-pd
    8. parameters:
    9.   type: pd-standard (1)
    10.   replication-type: none
    11. volumeBindingMode: WaitForFirstConsumer
    12. allowVolumeExpansion: true
    13. reclaimPolicy: Delete
    1Select either pd-standard or pd-ssd. The default is pd-standard.

    VMware vSphere object definition

    vsphere-storageclass.yaml

    1For more information about using VMware vSphere with OKD, see the VMware vSphere documentation.
    2diskformat: thin, zeroedthick and eagerzeroedthick are all valid disk formats. See vSphere docs for additional details regarding the disk format types. The default value is thin.

    Use the following process to change the default storage class. For example you have two defined storage classes, gp2 and standard, and you want to change the default storage class from gp2 to standard.

    1. List the storage class:

      1. $ oc get storageclass

      Example output

      1. NAME                 TYPE
      2. gp2 (default)        kubernetes.io/aws-ebs (1)
      3. standard             kubernetes.io/aws-ebs
      1(default) denotes the default storage class.

    2. Change the value of the storageclass.kubernetes.io/is-default-class annotation to false for the default storage class:

      1. $ oc patch storageclass gp2 -p '{"metadata": {"annotations": {"storageclass.kubernetes.io/is-default-class": "false"}}}'

    3. Make another storage class the default by setting the storageclass.kubernetes.io/is-default-class annotation to true:

      1. $ oc patch storageclass standard -p '{"metadata": {"annotations": {"storageclass.kubernetes.io/is-default-class": "true"}}}'

    4. Verify the changes:

      1. $ oc get storageclass

      Example output

      1. NAME                 TYPE
      3. standard (default)   kubernetes.io/aws-ebs