Persistent Volumes and Storage Classes

A PersistentVolume (PV) is a piece of storage in the cluster that has been provisioned by an administrator or dynamically provisioned using storage classes. PVs are volume plugins like volumes, but have a lifecycle independent of any individual Pod that uses the PV. PVs can be provisioned either statically or .

A PersistentVolumeClaim (PVC) is a request for storage by a user. It is similar to a Pod. Pods consume node resources and PVCs consume PV resources.

KubeSphere supports dynamic volume provisioning based on storage classes to create PVs.

A provides a way for administrators to describe the classes of storage they offer. Different classes might map to quality-of-service levels, or to backup policies, or to arbitrary policies determined by the cluster administrators. Each storage class has a provisioner that determines what volume plugin is used for provisioning PVs. This field must be specified. For which value to use, please read the official Kubernetes documentation or check with your storage administrator.

The table below summarizes common volume plugins for various provisioners (storage systems).

You need a user granted a role including the permission of Cluster Management. For example, you can log in to the console as directly or create a new role with the permission and assign it to a user.

Click Platform in the upper-left corner and select Cluster Management.
If you have enabled the with member clusters imported, you can select a specific cluster. If you have not enabled the feature, refer to the next step directly.
On the Cluster Management page, go to Storage Classes under Storage, where you can create, update, and delete a storage class.
In KubeSphere, you can create storage classes for QingCloud-CSI, GlusterFS, and Ceph RBD. Alternatively, you can also create customized storage classes for other storage systems based on your needs. Select a type and click Next.

Some settings are commonly used and shared among storage classes. You can find them as dashboard parameters on the console, which are also indicated by fields or annotations in the StorageClass manifest. You can see the manifest file in YAML format by clicking Edit YAML in the upper-right corner.

Here are parameter descriptions of some commonly used fields in KubeSphere.

Parameter	Description
Volume Expansion	Specified by `allowVolumeExpansion` in the manifest. When it is set to `true`, PVs can be configured to be expandable. For more information, see Allow Volume Expansion.
Reclaim Policy	Specified by `reclaimPolicy` in the manifest. For more information, see .
Storage System	Specified by `provisioner` in the manifest. It determines what volume plugin is used for provisioning PVs. For more information, see Provisioner.
Access Mode	Specified by `metadata.annotations[storageclass.kubesphere.io/supported-access-modes]` in the manifest. It tells KubeSphere which is supported.
Volume Binding Mode	Specified by `volumeBindingMode` in the manifest. It determines what binding mode is used. Delayed binding means that a volume, after it is created, is bound to a volume instance when a Pod using this volume is created. Immediate binding means that a volume, after it is created, is immediately bound to a volume instance.

For other settings, you need to provide different information for different storage plugins, which, in the manifest, are always indicated under the field parameters. They will be described in detail in the sections below. You can also refer to Parameters in the official documentation of Kubernetes.

QingCloud CSI

QingCloud CSI is a CSI plugin on Kubernetes for the storage service of QingCloud. Storage classes of QingCloud CSI can be created on the KubeSphere console.

Prerequisites

QingCloud CSI can be used on both public cloud and private cloud of QingCloud. Therefore, make sure KubeSphere has been installed on either of them so that you can use cloud storage services.
QingCloud CSI Plugin has been installed on your KubeSphere cluster. See for more information.

Settings

For more information about storage class parameters, see .

GlusterFS is an in-tree storage plugin on Kubernetes, which means you don’t need to install a volume plugin additionally.

Prerequisites

The GlusterFS storage system has already been installed. See for more information.

Settings

Parameter	Description
REST URL	Heketi REST URL that provisions volumes, for example, <Heketi Service cluster IP Address>:<Heketi Service port number>.
Cluster ID	Gluster cluster ID.
REST Authentication	Gluster enables authentication to the REST server.
REST User	Username of Gluster REST service or Heketi service.
Secret Namespace/Secret Name	Namespace of the Heketi user secret.
Secret Name	Name of the Heketi user secret.
Minimum GID	Minimum GID of the volume.
Maximum GID	Maximum GID of the volume.
Volume Type	Type of volume. The value can be none, replicate:<Replicate count>, or disperse:<Data>:<Redundancy count>. If the volume type is not set, the default volume type is replicate:3.

For more information about storage class parameters, see .

Ceph RBD

As hyperkube images were , in-tree Ceph RBD may not work without hyperkube. Nevertheless, you can use rbd provisioner as a substitute, whose format is the same as in-tree Ceph RBD. The only different parameter is provisioner (i.e Storage System on the KubeSphere console). If you want to use rbd-provisioner, the value of provisioner must be ceph.com/rbd (Enter this value in Storage System in the image below). If you use in-tree Ceph RBD, the value must be kubernetes.io/rbd.

Prerequisites

The Ceph server has already been installed. See Ceph Installation Documentation for more information.
Install the plugin if you choose to use rbd-provisioner. Community developers provide that you can use to install rbd-provisioner by helm.

Settings

For more information about StorageClass parameters, see .

You can create custom storage classes for your storage systems if they are not directly supported by KubeSphere. The following example shows you how to create a storage class for NFS on the KubeSphere console.

NFS Introduction

NFS (Net File System) is widely used on Kubernetes with the external-provisioner volume plugin . You can create the storage class of nfs-client by clicking Custom.

Note

It is not recommended that you use NFS storage for production (especially on Kubernetes version 1.20 or later) as some issues may occur, such as failed to obtain lock and input/output error, resulting in Pod CrashLoopBackOff. Besides, some apps may not be compatible with NFS, including Prometheus.

Prerequisites

An available NFS server.
The volume plugin nfs-client has already been installed. Community developers provide charts for nfs-client that you can use to install nfs-client by helm.

Common Settings

Parameter	Description
Volume Expansion	Specified by `allowVolumeExpansion` in the manifest. Select `No`.
Reclaim Policy	Specified by `reclaimPolicy` in the manifest. The value is by default.
Storage System	Specified by `provisioner` in the manifest. If you install the storage class by charts for nfs-client, it can be `cluster.local/nfs-client-nfs-client-provisioner`.
Access Mode	Specified by `.metadata.annotations.storageclass.kubesphere.io/supported-access-modes` in the manifest. `ReadWriteOnce`, `ReadOnlyMany` and `ReadWriteMany` are all selected by default.
Volume Binding Mode	Specified by `volumeBindingMode` in the manifest. It determines what binding mode is used. Delayed binding means that a volume, after it is created, is bound to a volume instance when a Pod using this volume is created. Immediate binding means that a volume, after it is created, is immediately bound to a volume instance.

Parameters

Storage class details page

After you create a storage class, click the name of the storage class to go to its details page. On the details page, click Edit YAML to edit the manifest file of the storage class, or click More to select an operation from the drop-down menu:

Set as Default Storage Class: Set the storage class as the default storage class in the cluster. Only one default storage class is allowed in a KubeSphere cluster.
Volume Management: Manage volume features, including: Volume Clone, Volume Snapshot, and Volume Expansion. Before enabling any features, you should contact your system administrator to confirm that the features are supported by the storage system.
Delete: Delete the storage class and return to the previous page.

On the Volumes tab, view the volumes associated to the storage class.

Once the storage class is created, you can create volumes with it. You can list, create, update and delete volumes in Volumes under Storage on the KubeSphere console. For more details, please see .

Log in to KubeSphere web console as admin. Click Platform in the upper-left corner, select Cluster Management, and click Volumes under Storage.
Click the Volume Instances tab on the Volumes page to view the volume instance list page that provides the following information:
- Name: Name of the volume instance. It is specified by the field .metadata.name in the manifest file of the volume instance.
- Status: Current status of the volume instance. It is specified by the field .status.phase in the manifest file of the volume instance, including:
  - Available: The volume instance is available and not yet bound to a volume.
  - Bound: The volume instance is bound to a volume.
  - Failed: The volume instance is unavailable.
- Capacity: Capacity of the volume instance. It is specified by the field .spec.capacity.storage in the manifest file of the volume instance.
- Access Mode: Access mode of the volume instance. It is specified by the field .spec.accessModes in the manifest file of the volume instance, including:
  - RWO: The volume instance can be mounted as read-write by a single node.
  - ROX: The volume instance can be mounted as read-only by multiple nodes.
  - RWX: The volume instance can be mounted as read-write by multiple nodes.
- Recycling Strategy: Recycling strategy of the volume instance. It is specified by the field .spec.persistentVolumeReclaimPolicy in the manifest file of the volume instance, including:
  - Retain: When a volume is deleted, the volume instance still exists and requires manual reclamation.
  - Delete: Remove both the volume instance and the associated storage assets in the volume plugin infrastructure.
  - Recycle: Erase the data on the volume instance and make it available again for a new volume.
- Creation Time: Time when the volume instance was created.
Click on the right of a volume instance and select an operation from the drop-down menu:
- Edit: Edit the YAML file of a volume instance.
- View YAML: View the YAML file of the volume instance.
- Delete: Delete the volume instance. A volume instance in the Bound status cannot be deleted.

Volume instance details page

Click the name of a volume instance to go to its details page.
On the details page, click Edit Information to edit the basic information of the volume instance. By clicking More, select an operation from the drop-down menu:
- View YAML: View the YAML file of the volume instance.
- Delete: Delete the volume instance and return to the list page. A volume instance in the Bound status cannot be deleted.
Click the Resource Status tab to view the volumes to which the volume instance is bound.
Click the Metadata tab to view the labels and annotations of the volume instance.