When working with StatefulSet
s in Kubernetes, you can use volumeClaimTemplates
to have K8s dynamically provision Persistent Volumes for you. In the case of Azure Kubernetes Service, these end up in the MC_resouce-group-name_aks-name_region resource group together with the other automatically provisioned resources like VMs and Load Balancers.
Statically provisioned disks
For persistent data (in its broader meaning, not K8s terms), you may not want to tie your data storage lifecycle to your Kubernetes cluster. Instead you may wish to create you storage – such as Azure Managed disks – outside of K8s (say, Terraform) and then map them to your pods.
The Azure Kubernetes Service documentation contains a simple example of how this can be achieved for a single pod. Andy Zhang, working with Kubernetes at Microsoft, has a GitHub repository with more detailed examples of static provisioning.
In short, you create your Persistent Volume
apiVersion: v1 kind: PersistentVolume metadata: name: my-statically-provisioned-disk spec: capacity: storage: 1Gi accessModes: - ReadWriteOnce persistentVolumeReclaimPolicy: Retain azureDisk: kind: Managed diskName: {diskName} diskURI: /subscriptions/{subscriptionId}/resourcegroups/{resourceGroupName}/providers/Microsoft.Compute/disks/{diskName} fsType: ext4 readOnly: false cachingMode: ReadOnly
and then you can reference it either directly from your pod, or via an intermediate PersistentVolumeClaim.
StatfulSet
When you want to use statically provisioned disks as the persistent volumes of a StatefulSet
, you could create PersistentVolumeClaim
s with the same names as those that the StatefulSet
would have created for dynamic provisioning (claimname-podname-N), before you create the StatefulSet
.
A more elegant solution however, is to use match labels to identify the PVs to use for your StatefulSet
pods, as per this Stackoverflow anser.
apiVersion: v1 kind: PersistentVolume metadata: name: my-statically-provisioned-disk labels: app: influxdb # Used by volumeClaimTemplates spec: capacity: storage: 1Gi accessModes: - ReadWriteOnce persistentVolumeReclaimPolicy: Retain azureDisk: kind: Managed diskName: {diskName} diskURI: /subscriptions/{subscriptionId}/resourcegroups/{resourceGroupName}/providers/Microsoft.Compute/disks/{diskName} fsType: ext4 readOnly: false cachingMode: ReadOnly --- apiVersion: apps/v1 kind: StatefulSet ... spec: ... template: ... spec: ... volumeClaimTemplates: - metadata: name: influxdata spec: selector: matchLabels: app: influxdb # The labels that the PersistentVolumes must have to be used to fulfil this claim accessModes: - ReadWriteOnce resources: requests: storage: 1Gi
Automatic volume expansion
Kubernetes supports automatic expansion of dynamically provisioned disks, and we can actually make this work for statically provisioned disks as well.
First, regardless of dynamic vs static provisioning you must have a StorageClass
with allowVolumeExpansion: true
. As per the Azure documentation, this is not the case for the built in storage classes. Instead you will need to create your own StorageClass
, for example
apiVersion: storage.k8s.io/v1 kind: StorageClass metadata: name: expandable-managed-premium provisioner: kubernetes.io/azure-disk reclaimPolicy: Delete allowVolumeExpansion: true # Change compared to built in managed-premium parameters: storageaccounttype: Premium_LRS kind: Managed
Now, make sure to reference this storage class from your volumeClaimTemplates
and, in the case of static provisioning, your PersistentVolume
.
In the case of statically provisioned disks, I also suggest you add the pv.kubernetes.io/provisioned-by: kubernetes.io/azure-disk
annotation. (I have yet to verify whether this is required or not.)
So we end up with something like this
apiVersion: v1 kind: PersistentVolume metadata: name: my-statically-provisioned-disk labels: app: influxdb # Used by volumeClaimTemplates annotations: pv.kubernetes.io/provisioned-by: kubernetes.io/azure-disk finalizers: - kubernetes.io/pv-protection spec: capacity: storage: 1Gi accessModes: - ReadWriteOnce persistentVolumeReclaimPolicy: Retain azureDisk: kind: Managed diskName: {diskName} diskURI: /subscriptions/{subscriptionId}/resourcegroups/{resourceGroupName}/providers/Microsoft.Compute/disks/{diskName} fsType: ext4 readOnly: false cachingMode: ReadOnly storageClassName: expandable-managed-premium # Support volume expansion volumeMode: Filesystem --- apiVersion: apps/v1 kind: StatefulSet ... spec: ... template: ... spec: ... volumeClaimTemplates: - metadata: name: influxdata spec: selector: matchLabels: app: influxdb # The labels that the PersistentVolumes must have to be used to fulfil this claim accessModes: - ReadWriteOnce resources: requests: storage: 1Gi storageClassName: expandable-managed-premium # Support volume expansion
Automatic volume expansion for StatefulSet
At the time of this writing, changing the resources.requests.storage
of a StatefulSet
volumeClaimTemplates
is not supported by Kubernetes. There are issues on GitHub both for the main project and the Enhancement Tracking and Backlog project. (The latter also has a PR.)
Here is my recommended approach to work around this until properly supported, which works for dynamically as well as statically provisioned disks:
- Prepare your Kubernetes (or Helm) file with the new storage size.
- For each pod (P) in your
StatefulSet
, repeat- Delete the
StatefulSet
without cascadingkubectl delete sts --cascade=false your-stateful-set
- Delete the pod
kubectl delete pod/my-pod-P
- Edit the
PersistentVolumeClaim
of the podkubectl edit pvc my-volume-my-pod-P
Set the new storage size under
resources.requests.storage
. - Re-create the
StatefulSet
using kubectl apply or Helm as applicable - Wait for the volume to expand. Use
kubectl describe pvc
to see the progress. Sometimes another pod restart is required for changes to be applied.
- Delete the
References
- This GitHub issue contains some tips for migrating from dynamically to statically provisioned disks. You can also use disk snapshots.
- Andy Zhang also has some tips on growing disks on GitHub.