Kubernetes Runtime Security: Key Risks, Controls & Best Practices

There are many steps you can take to help prevent Kubernetes security vulnerabilities before you actually deploy workloads. You can carefully scan your container images for vulnerabilities. You can harden your nodes with security frameworks like SELinux. You can carefully configure Pod networking to close vulnerable connections by default.

And yet, none of these measures will necessarily guarantee Kubernetes runtime security - meaning the ability to secure Kubernetes infrastructure and workloads that are actively running. There is always a chance that vulnerabilities and other security risks will slip through and affect live Kubernetes clusters.

Hence, the importance of deploying runtime security protections for Kubernetes, which you’ll learn all about in the following article.

What is runtime security in Kubernetes?

In Kubernetes, runtime security is the practice of mitigating threats that affect live, running Kubernetes environments.

For example, Kubernetes runtime security can assist with the following challenges:

• Detecting a running container that has been infected by malware and should be shut down and redeployed based on a new, secure image.
• Detecting and blocking malicious network traffic that is able to slip through firewalls and is reaching live workloads.
• Identifying a Kubernetes node that has been compromised by threat actors, who may, in turn, be able to take control of containers and Pods hosted on the node.

Runtime security is distinct from static security, which deals with addressing risks that impact non-running code or configurations. That’s important, too, for establishing a strong security posture by default (which practices like container image scanning and network configuration analysis help to do). But no matter how hard you work to mitigate threats before you actually deploy applications, there’s no way to guarantee that risks won’t slip past your initial defenses.

Why runtime security in Kubernetes is critical for modern workloads

Runtime security is the only way of detecting risks that impact live, running applications.

Indeed, runtime security is all the more important in Kubernetes because Kubernetes itself includes no built-in controls for detecting and blocking threats. Kubernetes will happily keep running containers that contain malware or are using compromised nodes, for example, unless admins discover these risks and take steps to mitigate them.

It’s worth noting as well that the complexity of Kubernetes can make runtime threats harder to detect than they often are in simpler environments. It’s easy enough to scan for signs of active malware when all of your applications reside on just one server, for instance, because then you only have to monitor just one set of processes for anomalies.

But in a Kubernetes cluster that includes dozens or hundreds of nodes, detecting risks becomes much more challenging. Not only do you need to monitor the behavior of a multitude of applications and services, but you also have to unpack the complicated relationships between them. If a Pod’s CPU utilization has spiked, for example, is it because it’s running crypto mining malware? Or is it simply because of poorly configured requests and limits? These are the types of issues that make Kubernetes runtime security especially important - but also especially complex.

Common runtime security threats in Kubernetes

Major runtime threats and risks that commonly impact Kubernetes clusters can be broken into four basic categories: Container-level threats, host-level threats, cluster-level threats, and network security risks.

1. Container-level threats

Container runtime threats are those that impact individual containers.

One of the most common types of runtime threats in this category is malware within a container, which can occur either because malware made its way into the image used to launch the container or because attackers managed to install malware on a previously clean container after it starts.

Container-level threats can also result from security misconfigurations, such as allowing containers to run as root or failing to apply a proper Pod Security Standards policy. Mistakes like these don’t necessarily lead to breaches, but they can make attacks easier to execute or escalate by removing access controls that would otherwise hinder the activity of attackers.

2. Host-level threats

Host-level threats refer to security issues on the nodes (or host servers) that host the Kubernetes cluster plane, as well as Pods and containers.

Runtime threats that affect hosts often result from malware running on the hosts. Malicious users with accounts on the host, or accounts with excess privileges, can also cause threats. So can unpatched software libraries or applications, which could contain vulnerabilities that threat actors can exploit.

3. Cluster-level threats

Cluster-level runtime threats stem from vulnerabilities that affect the overall Kubernetes cluster.

A common cause is outdated control plane software. If you don’t update your control plane, it may contain unpatched vulnerabilities. Configuration issues, like failure to deploy RBAC controls to restrict user privileges within the cluster, can also lead to cluster-level runtime threats.

4. Network-level threats

At the network level, the main type of runtime threat is malicious traffic that is able to reach vulnerable workloads.

As a best practice, Kubernetes workloads should only be reachable over the network by other applications or services that have a reason to connect to them. For instance, a Pod that only needs to talk to other Pods within the cluster would have no reason to be exposed to external network connections. But oversights in networking configurations could expose the Pod accidentally, allowing threat actors to access the workload remotely.

Another type of network-level runtime risk is Denial-of-Service attacks, which occur when attackers flood Kubernetes workloads or services with illegitimate requests as a way of causing them to slow down or crash. The best way to mitigate this risk is to block unnecessary connections in the first place. Monitoring for Denial-of-Service attempts and filtering out malicious or unusual traffic also helps to prevent network service disruption.

Kubernetes runtime security vs. traditional container security

The concept of runtime security is certainly not new to the world of containers. Since the launch of Docker in 2013, there has been plenty of talk about the importance of protecting container runtimes.

The main difference between traditional security measures for container runtimes and Kubernetes runtime security, however, is that securing Kubernetes at runtime is more complex, mainly because there are more components and layers to protect. With traditional containers, you only had to secure the runtime engine that executed your containers and monitor your containers once they were running, along with the container host. But with Kubernetes, you have many more runtime components to protect - containers, Pods, nodes, services, and so on.

Kubernetes runtime security tools and approaches

As we mentioned, Kubernetes itself provides no built-in tools for detecting runtime security threats. But it does provide some tooling that can help to harden runtime environments and mitigate risks, including:

• RBAC: Role-based access controls within Kubernetes can restrict the resources that users and workloads can access.
• Pod Security Standards: Pod Security Standards can restrict the privileges available to Pods, another way of helping to prevent unauthorized activity and stop privilege escalation attacks.

But when it comes to detecting and responding to runtime threats, you need to deploy external tools. Key types of runtime solutions in this regard include:

• Observability software: By continuously monitoring and observing workloads, you can detect anomalies (like sudden changes in resource utilization or the start of unexpected processes) that could be the sign of a runtime breach.
• Runtime security enforcement: Various third-party tools exist for actively blocking runtime security threats on Kubernetes. They generally focus on different types of runtime risks; for example, frameworks like SELinux and AppArmor can help with host-level runtime security for nodes, while KubeArmor is a runtime security engine that focuses on Kubernetes workloads.

Best practices for runtime security in Kubernetes

To minimize the risk of runtime threats and maximize your ability to respond to them, consider the following best practices:

• Start with prevention: It’s a cliché that an ounce of prevention equals a pound of cure, but it’s also true in this case. The best way to mitigate runtime threats is to invest in controls (like container image scanning and workload configuration auditing) that help you detect and mitigate risks before workloads start.
• Establish a baseline: Detecting anomalies is not really possible unless you have a baseline to compare them against. To that end, determine what your cluster looks like (in terms of observability metrics, process behavior, and so on) under normal, secure circumstances. This will serve as your baseline during runtime security monitoring.
• Automate response: Runtime threats can quickly escalate. To maximize your ability to mitigate minor issues before they turn into major problems, consider automation tooling, like the runtime security enforcement solutions mentioned in the previous section. These allow you to block threats without waiting for a human to review the problem.
• Make an incident response plan: While automations can help to remediate one-off or small-scale runtime threats, you’ll want an incident response plan for coordinating handling of major incidents, such as a malware breach that affects your control plane or a Denial-of-Service attack that targets multiple Pods.

Operational challenges in achieving strong runtime security in Kubernetes

While runtime security for Kubernetes is critical, it can also be challenging to implement. The biggest issue in this respect - as we mentioned above - is the sheer complexity of Kubernetes. There are many runtime components to monitor and secure, and many variables to weigh and correlate. This is why automation and comprehensive monitoring are important for operationalizing runtime security.

Lack of built-in tooling for enforcing runtime security in Kubernetes or detecting threats is also a challenge. But again, you can leverage external tools to fill these gaps.

How groundcover strengthens runtime security in Kubernetes

Speaking of external runtime security tools, one key type of solution is an observability platform, like groundcover.

groundcover continuously monitors all facets and layers of your cluster - containers, Pods, nodes, the control plane, and the network - and delivers comprehensive observability data through highly customizable dashboards. That means you can quickly detect and investigate anomalies to determine whether they’re the result of mundane issues (like a spike in legitimate traffic causing a node’s CPU utilization to increase) or security threats.

Paired with other runtime tools that help to respond to threats, groundcover provides the essential visibility that security teams need to take a proactive approach to protecting Kubernetes at runtime.

Get a running start against Kubernetes runtime threats

No Kubernetes security strategy is complete without runtime protections. While static security is important as well, an amount of pre-deployment security protections will guarantee a secure runtime environment – which is why continuous visibility into runtime security, and proactive response to runtime threats, is absolutely essential for protecting Kubernetes clusters and workloads.

FAQ

What types of threats are most commonly detected through runtime security in Kubernetes?

The most common type of runtime security threat in Kubernetes is malware, which can infect both containers and hosts. Threats involving privilege escalation attacks, such as containers running as the root user, are also a frequently detected type of runtime security issue.

How does runtime security differ from static scanning or CI/CD security in Kubernetes?

The key difference between runtime security and static or CI/CD security is that runtime security focuses on threats that affect live, running workloads. In contrast, static security deals with identifying risks before workloads are deployed.

Static security is important as a way of mitigating risks proactively. However, there is always a chance that static protections won’t detect all security issues, which is why runtime security is essential as a second layer of defense.

How does groundcover provide runtime threat detection without adding cluster performance overhead?

groundcover assists with runtime threat detection by comprehensively collecting observability data. Using this information, admins can identify anomalies that may be signs of a breach.

And because groundcover collects data using eBPF, a hyper-efficient framework that is built into the Linux kernel, it is able to assist in runtime threat detection with minimal performance overhead - unlike many conventional security monitoring tools, which rely on resource-hungry agents that can reduce cluster performance.