Kubernetes Volumes in Action: Managing Data the Right Way

Life for Kubernetes admins would be simple if all workloads were “stateless” and never required a way to store data – in which case there would be no need for Kubernetes volumes. But life, alas, is not that simple. Many apps are “stateful,” meaning they need access to data storage resources.

Hence the importance of learning to work with Kubernetes volumes, which this article covers by explaining how volumes work in Kubernetes, how they compare to other data storage solutions (like persistent volumes), how to deploy volumes, and how to monitor for and troubleshoot performance issues related to Kubernetes volumes.

What are Kubernetes volumes?

Kubernetes volumes are a way for pods to access data stored on an external file system. This means that when you create a volume for a pod, the pod is able to read and write from a location that is independent of the individual containers running inside the pod. In this way, volumes make it possible for all of the containers in a pod to share a common storage location.

Volumes typically work by mapping a directory on a node (meaning the server that hosts a pod) to the pod, then allowing containers within the pod to read and write from that directory. The way that the node storage directory is configured and shared with the pod varies depending on which type of volume you choose; we’ll say more on that in a bit, when we explain the main types of Kubernetes volumes and associated use cases.

Kubernetes volumes vs. persistent volumes vs. persistent volume claims

Importantly, Kubernetes volumes are not the only way of exposing external data storage resources to applications running in a Kubernetes cluster. The other main method is to use a persistent volume, which works in a different way from a standard volume.

The main difference between a volume and a persistent volume is that a volume is linked to a specific, individual pod, whereas a persistent volume is a cluster-level resource that can be made available to multiple pods. As a result, a volume is dependent on the lifecycle of just one pod, and any data stored in the volume will disappear when the pod shuts down. For this reason, you may sometimes hear people refer to Kubernetes volumes as “ephemeral volumes,” as a way of distinguishing them from persistent volumes. Persistent volumes are different because data hosted in them is independent of any individual pod and persists across pods restarts.

Volumes are useful in situations (such as making configuration data available to containers within the pod) where you need to provide temporary storage to just one pod. In contrast, you would use persistent volumes in scenarios (like hosting a database) where you need to share data between multiple pods, and/or where you want data to remain intact even if a pod shuts down or restarts.

To make a persistent volume available to a pod, Kubernetes admins have to set up a persistent volume claim (PVC). PVCs are the method that makes it possible to share a persistent volume with some pods in a cluster but not others.

When you’re working with volumes instead of persistent volumes, there is no need to create a “claim” because, again, each volume is always linked to just one pod – so the pod doesn’t have to “claim” it.

Summary: Main differences in volumes vs. persistent volumes vs. persistent volume claims

Why are Kubernetes volumes important for stateful applications?

Kubernetes volumes are useful for running certain types of stateful applications – although it’s important to understand that volumes are not the right solution for all stateful apps.

Let’s unpack that statement a bit by diving deeper into what constitutes a stateful application. A stateful app is any type of application that needs to be able to preserve its “state” for some period of time by writing data to a storage location. There are two basic types of stateful apps:

• Temporary stateful apps, which only need to maintain session-specific data (like information about users who are currently logged in). Preserving this data indefinitely is not important for a temporary stateful app, and it’s okay if the data disappears when the app shuts down.
• Persistent stateful apps, which need to be able to retain data (like information about users who have logged out) for extended periods of time, and to ensure that the data remains available even if the app restarts.

Kubernetes volumes are important for running the first type of stateful app – one that requires only temporary persistent storage. They’re less effective for persistent stateful apps because volumes don’t provide a way of guaranteeing that data will persist if a pod shuts down (you’d want to use a persistent volume if that is your goal).

Kubernetes volume types and use cases

There are four main types of ephemeral volumes in Kubernetes:

1. EmptyDir: Creates a temporary volume when a pod is placed on a node. EmptyDir volumes are typically used for creating storage locations that are initially empty, and that can be used by the containers inside a pod to store data temporarily.
2. HostPath: Makes a directory on the pod’s host node available to pods. Any data that is stored in the directory on the host will be available to the pod. HostPath volumes can be used to share data between pods and a node to support use cases like writing log data to the host. A major caveat, however, is that HostPath volumes are tied to individual nodes, and if a pod is reassigned to a new node, it won’t be able to access HostPath volumes from a previous node, which is why this type of volume doesn’t guarantee persistent storage that will be available across pod restarts.
3. ConfigMap: A type of volume that makes a ConfigMap available to a pod. This volume type is designed specifically for sharing ConfigMap variables with pods at runtime.
4. Secret: Similar to a ConfigMap volume, a Secret volume makes secrets available to a pod at runtime.

How to work with Kubernetes volumes

To work with volumes in Kubernetes, you must first define a volume as part of a pod specification. You do this by including a volumes: section when defining a pod.

Note, however, that defining a volume for a pod doesn’t actually make the volume available to containers inside the pod. To do that, you must include volumeMounts inside the pod’s spec to specify which containers should have access to the volume.

For example, here’s a pod spec that creates an EmptyDir volume named shared-volume and makes it available to containers named writer and reader. This spec also defines mountPaths (/data, in this example) for each container, which determine where the volume will be mounted inside each container’s internal file system:

apiVersion: v1
kind: Pod
metadata:
  name: shared-volume-pod
spec:
  containers:
  - name: writer
    image: busybox
    volumeMounts:
    - name: shared-data
      mountPath: /data
  - name: reader
    image: busybox
    volumeMounts:
    - name: shared-data
      mountPath: /data
  volumes:
  - name: shared-data
    emptyDir: {}

After defining a pod spec like this, you’d save it to a file and deploy the pod with a command such as:

kubectl apply -f /path/to/pod-spec.yml

The volume will be created automatically when the pod launches, and the containers will be able to access the volume per the definition in the pod spec.

Access modes and storage policies

In general, ephemeral volumes in Kubernetes don’t support configuration options related to access modes or storage policies. As a result, there is no way in most cases to say that a volume should be read-only, for instance, or to restrict how many times a pod can write to a volume. These options are generally supported only when using persistent volumes.

We say “in general” because in certain cases, the underlying container storage interface (CSI) drivers that enable access to storage may support options for configuring access modes and storage policies. Support varies between specific CSIs. It’s also possible (but not recommended) to impose access restrictions at the node level, such as by mounting a partition on a node’s file system in read-only mode, then exposing that partition’s mount path as a HostPath volume.

That all said, if you want to configure access modes or storage policies for volumes, it’s best to use persistent volumes instead of ephemeral volumes. The former provides access mode and storage policy support natively through Kubernetes, making these options easier and more reliable to configure.

Common challenges in managing Kubernetes volumes

While Kubernetes volumes are a relatively straightforward type of resource, you may nonetheless run into common issues like the following when working with volumes:

• Volume not available: If a volume that you defined as part of a pod’s spec isn’t available when the pod runs, check to make sure that the path you specified for the volume actually exists on the host (if you used a HostPath). Be sure as well that there are no typos in mountPaths defined for sharing a volume with containers.
• Insufficient storage: Kubernetes can’t magically create storage if no available storage resources exist on a host node. As a result, you may run into issues if the node that hosts your pod is short on available disk resources – in which case moving the pod to a different node is one way to resolve the problem.
• Can’t write to volume: If a volume exists but your containers can’t write to it, the likeliest cause is that node-level settings (such as a directory mounted on a node in read-only mode) prevent writing data to it.
• Data in volume sometimes disappears after pod restarts: If you create a HostPath volume, data stored in the volume will remain available across pod restarts so long as the pod remains on the same node. But if it moves to a different node, the data will no longer be available (unless you manually copy it from the old node to the new one). This behavior can have the effect of making it seem like data inconsistently persists between restart events. To avoid this frustration, use a persistent volume rather than an ephemeral volume if you need a guarantee of persistence.

Best practices for Kubernetes volume management

To get the greatest value from Kubernetes volumes, consider these best practices.

Know when – and when not – to use ephemeral volumes

As we’ve explained, volumes are handy when you need to run an app that requires temporary stateful storage. But you’ll set yourself up for disappointment and frustration if you try to use volumes for apps that require persistent stateful storage. For those, use persistent volumes instead.

Don’t rely on HostPath for persistent storage

Along similar lines, avoid any temptation to treat HostPath volumes as a way to store data persistently. You may be able to pull this off if you set up a HostPath volume and use a node feature like NodeSelectors to force the pod to run consistently on the same node. But that’s a clunky solution, and it would be better simply to use a persistent volume.

Create different volumes for distinct use cases

Instead of trying to use a single volume to meet all of the storage requirements of your pod, define multiple volumes if you need to support multiple use cases. For example, one volume could serve as a “scratch” space where containers can write temporary data, while another volume hosts a ConfigMap.

Assign the minimum necessary level of volume access

To minimize security risks – and to keep your configuration simpler and easier to manage – make volumes available only to the specific containers that need them. Sharing a volume with all containers in a pod is not a best practice unless each container actually has a reason to use the volume.

Monitoring and troubleshooting Kubernetes volumes

The first step in monitoring volumes and troubleshooting issues associated with them is to get a list of existing volumes. Unlike for persistent volumes, there is no simple command in Kubernetes for viewing ephemeral volumes. What you can do, however, is use kubectl describe to get details about running pods using a command like the following:

kubectl describe pod my-pod

The output will include information about any volumes associated with the pod my-pod. If you see that volume exists but your containers can’t actually access it, check for typos in the volume and volumeMount configurations.

You can also use external tools to monitor the overall health of node and cluster storage. Running the df command directly on a node will show how much of a node’s storage resources are currently being used, which can be helpful in troubleshooting issues where a volume is not working because the host node is out of space. Similarly, the command ls -l displays details about the permissions settings of a directory or file system, and can be helpful if node-level access restrictions are creating problems with a volume.

Simplifying Kubernetes volumes with groundcover observability

While command-line tools can be useful for getting a quick status-check on volume state or storage health, the best way to maintain comprehensive visibility into the state of all of your volumes and storage resources is to use an observability solution like groundcover.

With groundcover, you get continuous, granular visibility into your entire cluster. In this way, groundcover makes it easy to identify and troubleshoot issues with volumes. You can also quickly understand how a problem with a volume impacts specific pods or other resources within your cluster.

Getting more value from your volumes

Ephemeral volumes aren’t the only way to store data persistently in Kubernetes – and depending on what you’re trying to do, they may not be the best way. But for simple stateful applications that require temporary storage, volumes are usually the solution you need. Just be sure to choose the right volume type, and to navigate the various configuration issues (like host-level file access restrictions) that can prevent volumes from working properly.

FAQ

What are the security risks associated with Kubernetes volumes?

The main security risk linked to ephemeral volumes in Kubernetes is the exposure of sensitive data. This can occur in situations where a volume stores sensitive information (such as a secret or private user data) and the data can be read by containers that should not be able to access it.

In addition, when using volume types like HostPath, there is a risk that sensitive data stored in a volume could be read by users or applications who have access to the volume path through the host node.

The best way to avoid data exposure risks is to ensure that volumes are shared only with trusted containers that have a reason to access them. In addition, to prevent node-level data exfiltration, avoid using HostPath volumes for storing sensitive data. It’s generally better to use persistent volumes if you need to store sensitive information, since these provide more options for restricting access control.

A second security risk to consider is that any containers that can write to a volume could potentially inject malicious data as a means of manipulating application behavior. For example, they could modify configuration variables stored in a volume in a way that enables unauthorized access to an application. Here again, the best way to prevent this risk is to avoid making volumes accessible to untrusted containers. For similar reasons, avoid using HostPath volumes for storing configuration data, since anyone with access to the HostPath directory on the host node could modify the data.

What strategies help optimize costs when using Kubernetes volumes?

The first step toward optimizing costs associated with volumes in Kubernetes is avoiding unnecessary data storage. Ensure that any applications that write to volumes remove outdated data (such as obsolete log files). Compressing data before writing it to a volume can also help to use storage space more efficiently.

In addition, monitoring disk usage is important for identifying inefficiencies that may lead to wasted money when deploying volumes. You can track disk usage on individual nodes using operating system utilities like df. Monitoring for signs of disk pressure will also alert you to situations where storage resources are running low, possibly due to inefficient volume usage.

How does observability improve reliability for Kubernetes volumes?

Observability improves the reliability of Kubernetes volumes in two key ways. First, it helps alert you to problems that could disrupt a volume, such as a shortage of space or a failed volume deployment.

Second, observability provides the context necessary to understand what caused a volume issue. For instance, observability tools can help you determine which node is hosting a pod that is experiencing a problem with a volume. They can also report on the storage health of the node. With this contextual information, you can more quickly identify and resolve the root cause of the issue.