22 Management and Practical Implementation of Storage Objects Pv, Pvc, Storage Classes

22 Management and Practical Implementation of Storage Objects PV, PVC, Storage Classes #

When it comes to managing storage objects in Kubernetes, most readers use Local and NFS storage types the most. Mounting based on local volumes rarely encounters problems and doesn’t require much effort to learn and organize. However, starting from my perspective, I want to lead readers to explore deeper into the implementation details of objects and the cloud-native storage operations and management, and see how we can effectively manage these resources.

Understanding PV, PVC, StorageClass #

StorageClass is a way to describe storage classes. Different types may map to different service quality levels, backup strategies, or any policies defined by cluster administrators. Kubernetes itself is not aware of what each class represents. This concept is sometimes referred to as “configuration files” in other storage systems.

Each StorageClass contains the provisioner, parameters, and reclaimPolicy fields, which are used when the StorageClass needs to dynamically allocate PersistentVolumes.

The naming of StorageClass objects is important. Users use this name to request a specific class. When creating a StorageClass object, administrators set the name and other parameters of the StorageClass object, and once the object is created, it cannot be updated. An example is as follows:

apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: standard
provisioner: kubernetes.io/aws-ebs
parameters:
  type: gp2
reclaimPolicy: Retain
allowVolumeExpansion: true
mountOptions:
  - debug
volumeBindingMode: Immediate

PersistentVolume (PV) is a piece of storage in the cluster. It can be provisioned by administrators in advance or dynamically provisioned using a StorageClass. PersistentVolumes are global cluster resources, just like Nodes, without the concept of Namespace isolation. PVs are implemented using volume plugins, just like ordinary Volumes. The only difference is that PVs have their own lifecycle. This API object describes the implementation details of the storage, whether it is NFS, iSCSI, or storage systems specific to cloud platforms.

PersistentVolumeClaim (PVC) represents a user’s request for storage. It is conceptually similar to a Pod. Pods consume node resources, while PVCs consume PV resources. Pods can request a specific amount of resources (CPU and memory); similarly, PVCs can request a specific size and access mode for the PVs.

Although PersistentVolumeClaim allows users to consume abstract storage resources, it is common for multiple PVs to be provided with different properties (e.g., performance) to meet different user needs. Cluster administrators need to be able to provide PVs with different characteristics, and the differences between these PVs are not limited to volume size and access mode. At the same time, the details of how the volumes are implemented should not be exposed to the users. To meet this requirement, the StorageClass resource is introduced.

To summarize, for storage resources, we are referring to the “iron triangle” API objects by default: StorageClass, PersistentVolume, and PersistentVolumeClaim.

Understanding CSI #

Starting from Kubernetes v1.13, CSI has entered a stable and usable phase, so it is necessary for users to understand this container storage interface. CSI volume types are externally referenced CSI volume plugins used to interact between Pods and externally running CSI volume drivers on the same node. After deploying a CSI-compliant volume driver, users can use CSI as the volume type to mount the storage provided by the driver.

19-1-csi-arch

Until now, the Kubernetes community has been responsible for testing, maintaining, and other matters related to storage plugins. Even if contributors provide collaborations, it is not easy to merge and release them into the main branch. In addition, storage plugins need to be released with Kubernetes. If there are problems with the storage plugins, it may affect the normal operation of other Kubernetes components.

In view of this, Kubernetes and CNCF decided to abstract container storage and move the storage part outside the container orchestration system in the form of a standard interface. The purpose of CSI is to define an industry standard that allows storage vendors to implement, maintain, and deploy their own storage plugins. These storage plugins run on Kubernetes in the form of Sidecar Containers and provide stable storage services for container platforms.

In the above CSI design diagram:

  • The light green color represents the components that are abstracted from the Kubernetes community and can be reused. They are responsible for connecting to the CSI plugin (on the right) and interacting with the Kubernetes cluster:

    • Driver-registrar: A sidecar container that uses Kubelet to register CSI drivers and adds the NodeId (retrieved through the GetNodeID call to the CSI endpoint) to the annotation of the Kubernetes Node API object.
    • External-provisioner: A sidecar container that listens to Kubernetes PersistentVolumeClaim objects and triggers CreateVolume and DeleteVolume operations on the CSI endpoint.
    • External-attacher: A sidecar container that listens to Kubernetes VolumeAttachment objects and triggers ControllerPublish and ControllerUnPublish operations, responsible for attaching/detaching volumes to Node nodes.

  • The light gray color represents third-party implementation storage plugin drivers, which include three services:

  • CSI identify: Identifies the plugin service and maintains the health status of the plugin.

  • CSI Controller: Handles functions such as creating/deleting, attaching/detaching, and snapshots.

  • CSI Node: Handles functions such as attach/mount, umount/detach.

When integrating with common storage systems, you can use the built-in solutions without any modifications. These solutions are ready to use, but they have clear limitations. They only support a limited number of storage types, and their extensibility is poor, often limited by version restrictions. Additionally, the official announcement states that no new features will be added to these solutions in the future. In contrast, using CSI allows for decoupling from Kubernetes core components and supports a wider range of storage types and advanced features. Therefore, it is recommended to use CSI as a preferred supply method. Due to its non-intrusive plugin deployment for orchestration systems, it is favored by storage service providers.

Using Volume Snapshots for Backups #

Similar to the API resources PersistentVolume and PersistentVolumeClaim, VolumeSnapshotContent and VolumeSnapshot API resources are provided for creating volume snapshots for users and administrators.

VolumeSnapshotContent represents snapshots obtained from volumes configured by administrators in the cluster. It is a resource in the cluster, similar to PersistentVolume.

VolumeSnapshot represents a user’s request for a volume snapshot. It is similar to PersistentVolumeClaim.

VolumeSnapshotClass allows you to specify different attributes for VolumeSnapshots. These attributes may differ for snapshots of the same volume on different storage systems, so the same StorageClass used for PersistentVolumeClaim cannot be used to express them.

Volume snapshots provide Kubernetes users with a standardized backup and recovery method that allows them to copy the contents of a volume at a specific point in time without creating an entirely new volume. For example, a database administrator can use this feature to back up a database before making any editing or deletion changes.

When using this feature, users should note the following:

  • The API objects VolumeSnapshot, VolumeSnapshotContent, and VolumeSnapshotClass are Custom Resource Definitions (CRD) and not part of the core API. VolumeSnapshot is only supported for CSI drivers.
  • As part of the VolumeSnapshot beta deployment process, the Kubernetes team provides a snapshot controller that deploys to the control plane, as well as a helper container called csi-snapshotter that is deployed alongside CSI drivers. The snapshot controller monitors VolumeSnapshot and VolumeSnapshotContent objects and is responsible for the dynamic provision and deletion of VolumeSnapshotContent objects. The csi-snapshotter helper container monitors VolumeSnapshotContent objects and triggers CreateSnapshot and DeleteSnapshot actions on the CSI endpoint.
  • CSI drivers may or may not have implemented volume snapshot functionality. CSI drivers that support volume snapshots may utilize csi-snapshotter. For more information, refer to the CSI driver documentation.
  • The installation of CRDs and the snapshot controller is the responsibility of the Kubernetes distribution.

Lifecycle of VolumeSnapshots and VolumeSnapshotContents #

VolumeSnapshotContents are resources within the cluster. VolumeSnapshots are requests for these resources. The interaction between VolumeSnapshotContents and VolumeSnapshots follows this lifecycle:

  1. Provisioning of Volume Snapshots

    There are two ways to configure snapshots: pre-provisioned or dynamic.

  2. Pre-Provisioning

    Cluster administrators create some VolumeSnapshotContents, which carry the details of the actual snapshots on the storage system for consumption by cluster users. They exist in the Kubernetes API and are available for use.

  3. Dynamic

    You can request dynamic snapshots from a PersistentVolumeClaim instead of using pre-existing snapshots. VolumeSnapshotClass specifies specific parameters for the storage provider to use when fetching the snapshot.

  4. Binding

    The snapshot controller handles the binding of VolumeSnapshot objects to the appropriate VolumeSnapshotContent objects, both in the pre-provisioning and dynamic supply cases. The binding is a one-to-one mapping.

    In the case of pre-provisioning binding, the VolumeSnapshot will remain unbound until the requested VolumeSnapshotContent object is created.

  5. Protection of PersistentVolumeClaim as the Source of Snapshots

    This protection ensures that the in-use PersistentVolumeClaim API objects are not removed from the system while they are being snapshot, as it would result in data loss. When a snapshot of a PersistentVolumeClaim is taken, the PersistentVolumeClaim is in use. If you delete a PersistentVolumeClaim API object that is being used as a source for a snapshot, the PersistentVolumeClaim object will not be immediately deleted. Instead, the deletion of the PersistentVolumeClaim object will be deferred until the snapshot is ready or aborted.

6. Deletion

Deletion is triggered by deleting the VolumeSnapshot object and will follow the DeletionPolicy. If the DeletionPolicy is set to Delete, the underlying storage snapshot will be deleted along with the VolumeSnapshotContent object. If the DeletionPolicy is set to Retain, both the underlying snapshot and the VolumeSnapshotContent will be preserved.

VolumeSnapshots #

Each VolumeSnapshot contains a specification and a status:

apiVersion: snapshot.storage.k8s.io/v1beta1
kind: VolumeSnapshot
metadata:
  name: new-snapshot-test
spec:
  volumeSnapshotClassName: csi-hostpath-snapclass
  source:
    persistentVolumeClaimName: pvc-test

The persistentVolumeClaimName is the name of the PersistentVolumeClaim data source for the snapshot. This field is required for dynamic provisioned snapshots.

A volume snapshot can request a specific class by specifying the name of the VolumeSnapshotClass using the volumeSnapshotClassName property. If nothing is set, the default class will be used if available.

For pre-existing snapshots, you need to specify a volumeSnapshotContentName as the source of the snapshot, as shown in the following example. For pre-existing snapshots, the volumeSnapshotContentName source field is required.

apiVersion: snapshot.storage.k8s.io/v1beta1
kind: VolumeSnapshot
metadata:
  name: test-snapshot
spec:
  source:
    volumeSnapshotContentName: test-content

Volume Snapshot Contents #

Each VolumeSnapshotContent contains a specification and status.The common controller in dynamic provisioning creates the VolumeSnapshotContent object. Here’s an example:

apiVersion: snapshot.storage.k8s.io/v1beta1
kind: VolumeSnapshotContent
metadata:
  name: snapcontent-72d9a349-aacd-42d2-a240-d775650d2455
spec:
  deletionPolicy: Delete
  driver: hostpath.csi.k8s.io
  source:
    volumeHandle: ee0cfb94-f8d4-11e9-b2d8-0242ac110002
  volumeSnapshotClassName: csi-hostpath-snapclass
  volumeSnapshotRef:
    name: new-snapshot-test
    namespace: default
    uid: 72d9a349-aacd-42d2-a240-d775650d2455

volumeHandle is the unique identifier for the volume created by the storage backend, returned by the CSI driver during volume creation. This field is required for dynamic provisioned snapshots. It specifies the source volume for the snapshot.

For pre-existing snapshots, the creation of VolumeSnapshotContent objects is the responsibility of the cluster administrator, as shown below:

apiVersion: snapshot.storage.k8s.io/v1beta1
kind: VolumeSnapshotContent
metadata:
  name: new-snapshot-content-test
spec:
  deletionPolicy: Delete
  driver: hostpath.csi.k8s.io
  source:
    snapshotHandle: 7bdd0de3-aaeb-11e8-9aae-0242ac110002
  volumeSnapshotRef:
    name: new-snapshot-test
    namespace: default

snapshotHandle is the unique identifier for the volume snapshot created by the storage backend. This field is required for pre-existing snapshots. It specifies the CSI snapshot ID on the storage system that VolumeSnapshotContent represents.

Restoring Volume from Snapshot #

You can restore data from a snapshot by providing a new volume using the dataSource field in the PersistentVolumeClaim object. Here’s an example:

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: restore-pvc
spec:
  storageClassName: csi-hostpath-sc
  dataSource:
    name: new-snapshot-test
    kind: VolumeSnapshot
    apiGroup: snapshot.storage.k8s.io
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: 10Gi

Summary #

With the widespread adoption of stateful applications, the use of stateful storage object resources has become more common in the Kubernetes ecosystem. One of the most important aspects is the backup and restore of objects. Currently, the concept of VolumeSnapshot included in the latest API objects is still in the testing phase and not directly available in production. Also, note the design of the external plugin architecture for CSI volume drivers, which is currently production-ready, so please refrain from using built-in drivers to mount volumes. Depending on the implementation of storage drivers, managing NFS storage remains the most important part. Mastering the creation, backup, and restore operations is already sufficient for 90% of storage object usage skills. For distributed storage like Ceph, it can be extensively used in development and testing environments, and it can be widely adopted when the Ceph driver reaches maturity.