Kubernetes Certificate Rotation: Methods, Risks & Best Practices

Kubernetes certificates, unlike diamonds, are not forever. They expire periodically – and if you don’t rotate certificates out before expiration occurs, you risk disrupting workloads that depend on having a valid certificate present. What’s more, failure to rotate certificates can heighten security risks by making it easier for attackers to abuse stolen certificates.

Hence why it’s critical to have a Kubernetes certificate rotation strategy in place. Read on for tips and details as we unpack what Kubernetes certificate rotation means, methods for rotating certificates in Kubernetes, potential challenges and drawbacks of the rotation process, and best practices for rotating certificates efficiently and reliably.

What is Kubernetes certificate rotation?

Kubernetes certificate rotation is the process of renewing and replacing the digital certificates that Kubernetes uses to authenticate and encrypt communication between cluster components. These certificates have expiration dates, and if they are not renewed before they expire, critical services such as the API server, kubelets, controllers, and Etcd may be unable to communicate securely, potentially causing cluster outages.

To understand fully what certificate rotation means, let’s step back a bit and talk about certificates in Kubernetes: Kubernetes relies on Public Key Infrastructure (PKI) and CA certificates (sometimes also referred to in this context as root certificates) to establish trust between components. Examples include API server certificates, kubelet client certificates, Etcd certificates and certificates used for communication between control plane services. By pairing public certificates with private keys, Kubernetes resources can reliably authenticate each other. For context, it’s worth noting that Kubernetes mainly uses self-signed certificates for internal operations; public CA certificates are only used for public-facing services hosted in a Kubernetes cluster.

When a certificate expires, it means that it’s no longer valid, and components that were using the certificate can no longer communicate. Certificate rotation addresses this issue by updating the credentials before they expire, in order to maintain secure authentication and encrypted connections throughout the cluster.

Why certificate rotation is critical for Kubernetes security

In addition to mitigating the risk of service outages, certificate rotation also plays an important role in Kubernetes security.

This is because there is a risk that attackers can harvest certificates, typically by circumventing access controls and exfiltrating certificates from the servers that host them. The attackers can then use the certificates to establish trusted connections to Kubernetes resources.

Rotating certificates regularly doesn’t completely prevent the risk of certificate theft or abuse, but it reduces it because the shorter the time that a certificate remains active, the smaller the window during which attackers can abuse stolen certificate data. If the same certificate remains in place for years, there’s a higher chance that threat actors will steal and misuse it than there would be if the certificate were only valid for a few months. 

Types of certificates in Kubernetes that require rotation

Various types of certificates exist in Kubernetes that should be rotated periodically. Here’s a look at key examples:

• API server certificates: Secure communication between the Kubernetes API server and clients, including kubectl, controllers, and cluster components. These certificates must be rotated before expiration to maintain secure access to the control plane.
• Kubelet certificates: Used by kubelets to authenticate with the API server and establish encrypted communications. Regular rotation helps ensure that node identities remain valid and reduces the risk associated with compromised credentials.
• Client certificates for administrators and users: Authenticate human users, automation tools, and service accounts that access the cluster through certificate-based authentication. Rotating these certificates limits long-term exposure if credentials are leaked or stolen.
• Controller manager certificates: Enable the kube-controller-manager to securely communicate with the API server and perform cluster management tasks. Rotation prevents service interruptions caused by expired control plane credentials.
• Scheduler certificates: Allow the kube-scheduler to authenticate with the API server when making workload placement decisions. These certificates require periodic renewal to maintain uninterrupted scheduling operations.

Kubernetes certificate rotation methods and approaches

There are several ways to go about actually rotating certificates in Kubernetes – some of which are manual and some automated.

Manual Kubernetes certificate rotation using kubeadm

Manual certificate rotation with kubeadm is common within self-managed Kubernetes clusters where administrators maintain direct control over control plane components. Kubeadm provides commands such as kubeadm certs check-expiration to identify certificates approaching expiration and kubeadm certs renew to generate new certificates. After renewal, administrators typically restart affected control plane components or nodes to ensure the updated certificates are loaded into memory.

While manual rotation offers precise control over the timing and scope of certificate updates, it requires careful planning and operational oversight to avoid service disruptions caused by expired or improperly deployed certificates.

Automatic Kubernetes certificate rotation with kubelet and upgrades

Kubernetes supports automatic certificate rotation for kubelet instances via built-in certificate signing request (CSR) workflows. When this feature is enabled, kubelets automatically generate new certificate requests before their existing certificates expire, and the cluster can approve and issue replacements without administrator intervention. In addition, Kubernetes version upgrades often include mechanisms that renew or regenerate certain cluster certificates as part of the upgrade process.

Automated rotation reduces administrative burden, minimizes the risk of certificate expiration, and helps maintain secure node authentication, although organizations must still monitor certificate status and verify that automated renewal processes are functioning correctly.

Kubernetes certificate rotation in managed services (EKS, GKE, AKS)

Managed Kubernetes platforms such as Amazon Web Services Elastic Kubernetes Service (EKS), Google Cloud Google Kubernetes Engine (GKE), and Microsoft Azure Kubernetes Service (AKS) automate much of the certificate lifecycle management process. In these environments, cloud providers can handle the rotation of control plane certificates, internal service certificates, and other platform-managed credentials behind the scenes, reducing operational complexity for cluster operators.

Note, however, that customers remain responsible for managing certain certificates, such as application TLS certificates, ingress certificates, and credentials associated with external integrations. Understanding the shared responsibility model is important because even in managed environments, failures to rotate customer-managed certificates can still result in outages, authentication failures, or security risks.

Kubernetes certificate rotation process step by step

No matter which approach you use for rotating certificates, the process boils down to the following key steps (which can be performed either manually or automatically depending on the tooling you leverage).

1. Back up Kubernetes certificates and cluster state

First, back up the current certificates so that you can fall back to them in case a problem arises with a new certificate.

To do this, you’ll need to create backups of certificate files, kubeconfig files and the Etcd datastore before making changes. Common approaches include copying files from /etc/kubernetes/pki and creating an etcd snapshot using the etcdctl snapshot save command.

2. Renew certificates using kubeadm commands

Using a kubeadm command like the following, generate new certificates to replace the existing ones:

kubeadm certs renew all

Alternatively, you can select an individual certificate to renew by running:

sudo kubeadm certs renew <certificate-name>

3. Restart control plane components after certificate rotation

Restart affected control plane components so they load the new certificates. On kubeadm-managed clusters, this may involve restarting the kubelet with systemctl restart kubelet or allowing static pods for the API server, controller manager, and scheduler to restart automatically after manifest changes are detected.

4. Validate cluster health after certificate rotation

Finally, confirm that cluster services are functioning correctly by checking node and pod status with commands such as kubectl get nodes, kubectl get pods -A and kubectl cluster-info.

You can also review logs and metrics data using observability tools to verify Kubernetes component connectivity and ensure that the new certificates are being used successfully.

How to check Kubernetes certificate expiration and status

To determine whether a certificate should be rotated, you can check when it’s set to expire using the command:

kubeadm certs check-expiration

This provides a summary of control plane certificates, their expiration dates, and the remaining validity period.

Alternatively, admins can use Linux CLI tools like OpenSSL to inspect certificates directly, without working via native Kubernetes tooling.

Either approach provides details about certificates’ current status and upcoming expiration.

Monitoring and alerting for Kubernetes certificate expiration

As noted above, you can use the command kubeadm certs check-expiration to check manually whether a specific certificate is due to expire soon and should be rotated. But to automate and scale the process, you’d typically want to use an observability platform that can detect and alert you to certificates in need of rotation. This is the most efficient way to ensure that certificates are rotated regularly within large-scale clusters.

How eBPF-based observability improves Kubernetes certificate rotation visibility

When it comes to managing certificates automatically with help from observability platforms, eBPF – a framework built into the Linux kernel – provides a critical advantage.

That’s because observability tools can use eBPF to collect data in a hyper-efficient way that avoids wasting resources. Other, more traditional approaches to observability involve deploying agents, which bloat environments by consuming comparatively large amounts of CPU and memory.

With eBPF, it’s possible to track certificate status efficiently, without resource waste. The result is continuous visibility into certificate status, combined with efficient use of cluster resources.

Common risks and failures in Kubernetes certificate rotation

While rotating Kubernetes certificates is important, it can also be a somewhat fraught process. Problems may arise due to issues like the following:

• Failure to apply changes: Typically, processes must be restarted for certificate changes to take effect. There is a risk that admins will renew certificates but forget to restart processes, with the result that changes never take effect.
• Connectivity issues: If Kubernetes components lose connectivity during the certificate renewal process (for example, if a kubelet instance can no longer connect to the cluster), renewals may not take effect.
• Hardcoded configurations: In some cases, certificates contain or are generated based on hardcoded data (like IP addresses). Changes to this information may not be factored into certificate renewals, leading to certificates that contain inaccurate data and may not work.
• Forgotten certificates: As noted above, Kubernetes contains many types of certificates, spread across disparate resources. Given this sprawl, it can be easy to overlook certificates during the renewal process. 

Best practices for Kubernetes certificate rotation and expiry management

To manage certificate rotations as effectively and reliably as possible, consider the following best practices:

• Automate certificate rotation whenever possible: Enable built-in certificate rotation features and use automation tools to reduce manual effort, minimize human error, and ensure certificates are renewed before they expire.
• Continuously monitor certificate expiration dates: Using observability platforms, implement monitoring and alerting for all Kubernetes certificates, including control plane, node, ingress and application certificates, so teams receive advance notice of upcoming expirations.
• Maintain a complete certificate inventory: Track the location, purpose, owner, and expiration date of every certificate in the cluster to ensure no credentials are overlooked during rotation planning.
• Test rotation procedures regularly: Validate certificate renewal and recovery processes in non-production environments to confirm that rotations can be performed without causing authentication failures, service disruptions, or loss of cluster access.

Real-time Kubernetes certificate visibility and alerting with groundcover

When it comes to ensuring the visibility necessary to ensure certificates are kept up-to-date, groundcover has you covered. By comprehensively monitoring the status of every resource across Kubernetes clusters, groundcover clues admins into certificates that are close to expiring, failed renewals, or problems (like broken connectivity) that could cause rotation processes to fail.

What’s more, because groundcover uses eBPF to collect data, it provides this visibility without depriving workloads of the resources they need to operate at peak efficiency. It’s a best-of-both-worlds approach to certificate visibility.

Rotating certificates may not be the most glorious aspect of Kubernetes cluster management. But it’s an essential one for ensuring service reliability, while also mitigating security risks related to certificate abuse. Hence the value of investing in tools and processes for rotating certificates, monitoring certificate status, and confirming rotation success.