Kubernetes Garbage Collection: Challenges & Best Practices

Kubernetes is like a messy teenager in one key respect: It tends to produce a lot of junk. Just as teens have a habit of dumping dirty clothes in the hallway or leaving half-eaten food on the couch, Kubernetes resources produce various types of “garbage” – like unused container images and persistent volumes – in the course of normal operations.

Fortunately, unlike teenagers, Kubernetes is pretty good at cleaning up after itself. Thanks to a collection of features known as garbage collection, Kubernetes automatically finds and deletes most of the junk that, if left untended, would quickly fill a cluster with garbage.

Unfortunately, though, Kubernetes garbage collection doesn’t always happen completely automatically. Plus, there may be circumstances where you’d want to modify a cluster’s default garbage collection behavior. Hence the importance of understanding exactly how garbage collection works, how to configure or customize garbage collection routines, and how to monitor operations to ensure Kubernetes actually cleans up all of its junk.

What is Kubernetes garbage collection?

Kubernetes garbage collection is a set of built-in Kubernetes features for removing resources that are no longer needed.

The “garbage” that Kubernetes collects comes in many forms. Examples include:

• Container images that a cluster downloaded to run an application, but that are no longer necessary.
• Persistent volumes that have ceased to be active.
• Pods that have reached a termination state but have not actually shut down.
• Data associated with a Kubernetes Job that ran to completion.

Importantly, garbage collection does not apply to internal resources that are generated or managed by an application – such as data that an application writes to memory and no longer needs, or outdated log files created by a container. Cleanup of resources like these must be managed at the application level. Kubernetes garbage collection applies only to resources that are directly managed by Kubernetes, as opposed to those managed by applications running on Kubernetes.

Why Kubernetes garbage collection matters for cluster health

Garbage collection in Kubernetes is important for a straightforward reason: If unused objects aren’t cleaned up, they’ll waste cluster resources. By extension, they’ll leave fewer resources available for active workloads, negatively impacting cluster health.

The types of resources that Kubernetes garbage uses can vary depending on the nature of the garbage. Some objects, like unnecessary container images, only waste node disk space. Others, like terminated pods, can waste CPU and memory, since they’re still running even though they’re not supposed to be active.

Admins could, of course, clean up unnecessary resources on their own to avoid waste. But by automating the deletion of most types of garbage, Kubernetes garbage collection makes the lives of admins much simpler, while also helping to ensure that cluster resources are always used efficiently.

Types of Kubernetes garbage collection

As we mentioned, garbage collection isn’t a single feature; it’s a collection that includes multiple mechanisms for cleaning up garbage. Each type of garbage collection applies to a different category of resource. Here’s a look at the main types of Kubernetes garbage collection.

Pods and ReplicaSets

Various circumstances can arise in which Pods and ReplicaSets remain active but should be deleted. Pods could reach a termination state (meaning they have either succeeded or failed) without shutting down, for instance. Or, a Pod may reference a ReplicaSet (a set of identical Pods) that is no longer active, but the Pod is still running.

In cases like these, garbage collection identifies and deletes the unused Pod or ReplicaSet resources.

Containers and images

When Kubernetes deploys a container for the first time, it typically downloads an image for the container and stores it locally on the node that will host the container. If the container is no longer active (because, for example, the Pod of which it was part was deleted), there is no need to keep the image on hand – so garbage collection would delete it, freeing up storage resources on the node.

Secrets and ConfigMaps

Kubernetes will delete Secrets and ConfigMaps automatically if it determines that the resource that was using them (which is specified using the ownerReference field in the Secret or ConfigMap definition) no longer exists.

Note, however, that this type of garbage collection only occurs if the Secret or ConfigMap contains an ownerReference specification. Kubernetes won’t automatically delete other Secrets or ConfigMaps, even those that are not being actively used by any workloads.

Nodes

Nodes can store various types of resources, like container images. When they’re no longer used, Kubernetes garbage collection will delete them automatically. Note, however, that Kubernetes garbage collection won’t automatically remove unused or idle nodes. You’d need to implement a node autoscaler to achieve that goal.

Volumes

Garbage collection for Persistent Volumes (PVs), which provide storage resources to Kubernetes workloads, may occur automatically when a PV’s Persistent Volume Claim (PVC) is removed. We say “may” because this only happens if the PV has a Reclaim Policy defined as Delete. If you set a Retain Reclaim Policy, Kubernetes will keep the PV on hand even if it is no longer linked to a PVC.

How Kubernetes garbage collection works

Garbage collection in Kubernetes works based on the following key principles and capabilities:

• Comparing desired to actual state: Kubernetes constantly monitors the “desired” state of the cluster (meaning the one admins define via declarative configurations) to the actual state. Objects that are not part of the desired state, but that exist in actuality, are considered garbage and marked for deletion.
• Cleanup utilities: To perform deletion, Kubernetes relies primarily on Kubelet, the Kubernetes agent that runs locally on each node and is responsible for removing resources from those nodes. The cluster API server also plays a role in garbage collection because it needs to update the cluster state to ensure that deleted objects are no longer referenced.
• Garbage collection frequency: The cluster performs garbage collection every five minutes for most types of objects (like unused containers). Some (such as unused container images) are deleted every minute.
• Cascading deletion: When garbage collection occurs, it applies not only to objects that Kubernetes has identified as garbage, but also to any objects that are “owned” by the garbage objects. Typically, ownership is established via an ownerReference declaration. This principle is known as cascading deletion because it involves the removal of all of an object’s dependents, not just the object itself.

Configuring garbage collection in Kubernetes

The collection of most types of unused resources occurs automatically and by default in Kubernetes. Admins don’t need to configure it explicitly if they’re happy with the default behavior.

However, you can optionally configure certain aspects of garbage collection using the following fields and variables:

• TTL: When configuring a Kubernetes Job, you can define a time to live (TTL) parameter. This tells Kubernetes to keep objects associated with a Job on hand for a certain period of time after the Job itself has completed. This can be helpful in cases where you want to keep certain types of data (like a log file that tracks Job completion) on hand so you can copy it somewhere persistent before the Job objects are deleted.
• Reclaim Policy: As noted above, when creating a Persistent Volume (PV), you can set a Reclaim Policy that tells Kubernetes not to delete a PV if the associated Persistent Volume Claim (PVC) is removed.
• OwnerReference: As we also mentioned above, you can configure ownerReference options to associate objects with each other. If you do, Kubernetes will garbage-collect all of the associated objects at the same time.
• Garbage collection thresholds: For kube-controller-manager, a --terminated-pod-gc-threshold flag can be defined that tells Kubernetes to retain a certain number of inactive Pods before starting to delete them. Similar thresholds can be defined for container and container image deletion by Kubelet using the --image-gc-high-threshold, --image-gc-low-threshold, --maximum-dead-containers-per-container, and --maximum-dead-containers flags.

Common challenges in Kubernetes garbage collection

The main challenges you’re likely to encounter with Kubernetes garbage collection fall into two categories:

1. Objects aren’t garbage-collected when expected: If you think Kubernetes should have removed an unused resource automatically but it did not, make sure that there are no configuration options in place that prevent the object’s deletion (refer to the preceding section for details on those options). It’s also possible that garbage deletion is failing because the control node(s) can’t reach a certain worker node that needs to clean up resources, so ensure that there are no network connectivity issues.
2. Objects are deleted that shouldn’t be: If Kubernetes automatically deletes objects that you expected it to retain, make sure you understand how garbage collection works and confirm that the resource in question should not have been garbage-collected. Note, too, that garbage collection behavior can vary a bit between Kubernetes distributions, so check your distribution’s documentation to see if any unique features triggered the unexpected garbage collection. You can also use the garbage collection configuration options described above to prevent deletion of certain types of objects.

Best practices for Kubernetes garbage collection

To keep garbage collection routines as efficient as possible, consider these best practices:

• Stick with default behavior where appropriate: The more custom garbage collection options you set, the harder it becomes to predict and monitor garbage collection behavior. Stick with the defaults unless you need a custom option.
• Avoid external tools: It’s possible to try to perform garbage collection in Kubernetes using external software, such as a script that runs on each node and searches for unused container images. However, this is clunky and adds resource overhead. It’s better to use the built-in garbage collection capabilities in most cases.
• Manage application-specific garbage collection: As we mentioned, Kubernetes won’t handle internal garbage collection for workloads. It’s on you (or your application developers) to ensure that garbage produced by individual applications is cleaned up as needed.

Monitoring Kubernetes garbage collection effectively

Since garbage collection failures can waste resources, it’s important to monitor garbage collection and ensure it proceeds properly.

Unfortunately, Kubernetes doesn’t automatically generate alerts or reports related to garbage collection. You can, however, monitor data such as node CPU, memory, and disk metrics – which, if they’re high, could be an indication that garbage collection is not occurring as it should. You can also monitor for terminated Pods or containers that have not been deleted, another sign that garbage collection may be failing (unless you configured a threshold for terminated resources and it hasn’t been passed yet).

How groundcover simplifies Kubernetes garbage collection monitoring

By providing detailed, comprehensive observability into the state of all cluster resources, groundcover clues admins in quickly to garbage collection problems. Kubernetes on its own won’t tell you when you have a problem like a node that is bloated with unused container images or an unwieldy collection of terminated Pods, but groundcover’s monitoring and data visualization features make it easy to detect and troubleshoot problems like these.

Keeping a tidy house – or cluster – with garbage collection

While Kubernetes garbage collection can get complicated in the sense that it involves multiple variables and options, the good news is that it’s ultimately easier than keeping your house clean (especially, but not only, when you live with teenagers). 

FAQ

How does Kubernetes garbage collection impact overall cluster performance?

Garbage collection typically improves overall cluster performance by removing unused objects, which in turn frees up resources.

However, the total impact of garbage collection on cluster performance depends on which types of resources each object is using and how much spare capacity the cluster has. Garbage collection is more important in a cluster whose nodes are close to being maxed out on CPU, memory, and/or disk space, as compared to one that has a great deal of extra resources available.

What metrics and logs are most important to track for garbage collection issues?

The best way to track garbage collection and monitor for issues is to measure node CPU, memory, and disk usage. High usage could be a sign that unused objects are not being cleaned up. In addition, monitor the total number of terminated Pods, terminated containers, terminated Jobs, and unclaimed Persistent Volumes within your cluster. These should also normally be deleted via garbage collection (unless you’ve chosen configuration options that tell Kubernetes to retain them).

Can groundcover detect and alert on inefficient garbage collection?

Yes, groundcover can help detect and alert about garbage collection problems. They do this primarily by monitoring overall cluster health and performance, while also granularly tracking the state and health of each node, Pod, container, and so on. With this insight, admins can easily recognize anomalies (like a node whose disk space is maxed out because it’s storing unnecessary container images, or a Pod that has been stuck in the terminated state for a prolonged period) that could reflect inefficient garbage collection.