Kubernetes Logging: A Complete Guide to Efficient Management

Part of the beauty of Kubernetes is that you can tell it how you want your containerized applications to operate, and Kubernetes then attempts to manage them automatically according to your specifications.

However, just because Kubernetes tries to do what you want doesn’t mean that it always actually does what you request. As with any complex system, a variety of things can go wrong in Kubernetes, and you need to monitor the status of Kubernetes continuously to stay on top of those challenges.

That’s why Kubernetes logging is important for troubleshooting, security, and compliance, especially due to the ephemeral nature of pods and containers. Kubernetes pods are ephemeral, so local logs are lost when pods or containers are deleted, making centralized log aggregation essential. Logging in Kubernetes helps preserve crucial information, since logs from containers are ephemeral and disappear when pods die, making it hard to investigate the root cause of problems. By understanding which log files are available in Kubernetes, which data they reveal and how to analyze that data, you can gain critical visibility into what’s happening with your applications. Logs provide insights into system health and application behavior. You can also troubleshoot problems effectively when they arise.

When dealing with log files and data, it's important to aggregate logs from cluster logs, container logs, and node logs to ensure all the logs are collected for comprehensive monitoring and compliance. Collecting all the logs, including audit logs, is essential for effective monitoring, troubleshooting, and meeting compliance requirements. Effective logging supports compliance with security and privacy regulations by enabling auditing and tracking of access to applications and data.

With those goals in mind, let’s walk through everything K8s admins need to know about logging. This article breaks down the Kubernetes logging architecture, explains which log files are available in a Kubernetes cluster and walks through the process of managing logs using a logging agent. Logs should be stored securely to protect sensitive information and support compliance.

Introduction to Kubernetes Cluster

A Kubernetes cluster is the foundational unit of a Kubernetes environment, bringing together a group of machines—called nodes—to run and manage containerized applications at scale. Each node can be a physical server or a virtual machine, and clusters can span on-premises data centers or cloud platforms. The orchestration and management of these nodes are handled by the Kubernetes control plane, which is made up of several critical components, including the API server and the controller manager. The API server acts as the central communication hub for the cluster, processing requests and updates, while the controller manager ensures that the desired state of the cluster is maintained by managing controllers that handle tasks like node health and application scaling.

Kubernetes logging is a vital aspect of managing a Kubernetes cluster. By capturing logs from the control plane, nodes, and containerized applications, administrators gain deep insights into the behavior, performance, and security of the entire cluster. Effective logging enables proactive monitoring, rapid troubleshooting, and informed decision-making, ensuring that both the infrastructure and the applications running on it remain healthy and resilient.

Types of logs in a Kubernetes cluster

The first step in mastering Kubernetes logging is to understand which logs are generated in a Kubernetes cluster and how they can be categorized.

Logs generated in Kubernetes can be categorized into four main types: cluster logs, node logs, container logs, and Kubernetes events.

That said, at a high level, Kubernetes logs can be broken down into the following categories:

• ‍System logs: These logs provide information about the state of the core components of Kubernetes itself, such as the API server and scheduler.‍
• Application logs: These are logs that provide insight into the status and health of applications running inside containers. For example, you can find application-level error messages here.‍
• Audit logs: Audit logs record actions taken by human users as well as system processes inside Kubernetes. They’re valuable for researching changes that took place before a problem occurred. They can also help to identify potential security risks.

System component logs are crucial for monitoring and troubleshooting core cluster components.

By collecting and analyzing all three of these types of logs, and correlating them with top Kubernetes metrics for visibility and control as well as Kubernetes events for monitoring and troubleshooting, you can achieve comprehensive visibility into all major components of your Kubernetes environment – the control plane, the nodes that host applications and the applications themselves.

Cluster logs provide an overview of cluster performance, including logs from the API server, scheduler, and controller manager. Node logs offer insights into the health of individual worker nodes and include logs from Kubernetes components such as kubelet and kube-proxy. Container logs are essential for troubleshooting application issues. Logs generated by these components are used for troubleshooting and monitoring.

The Kubernetes logging architecture

One factor that complicates Kubernetes logging is that most components of Kubernetes are constantly changing, and some are not persistent. For example, when containers shut down, any data stored inside them, including logs, will go away with them. Centralized aggregation is often required in production environments to prevent log loss.

To solve this challenge, Kubernetes uses a logging architecture – known as cluster-level logging – that decouples log storage and lifecycles from nodes, Pods and containers. Cluster-level logging provides a separate backend to store, analyze and query logs from various sources within Kubernetes. Log forwarders such as Fluentd are commonly used to aggregate logs and forward log data from nodes and containers to a centralized logging system. Forwarding logs to a central location is essential for troubleshooting and monitoring, as it ensures comprehensive visibility across the infrastructure. Kubernetes logging requires a separate strategy for persistence due to the dynamic and ephemeral nature of the environment.

That said, Kubernetes doesn’t provide a native storage solution for hosting log data. You have to implement that on your own, using a third-party logging solution that integrates with Kubernetes.

As for actually generating log data, Kubernetes does that by writing logs from control plane components directly. In addition, a container runtime handles and redirects output generated by applications to stdout and strderr streams, which can then be turned into logs. Different container runtimes handle these streams in somewhat different ways (for example, the integration with kubelet uses the CRI logging format), but they all make it possible to log data about the status of applications as long as those applications can expose data to stdout and stderr.

Kubernetes log structures

Because there are so many different types of logs in Kubernetes, the exact structure of Kubernetes log files varies. However, choosing the right log format, especially structured logging formats like JSON, is crucial for consistent log analysis and management. But in general, logs typically include the following structural components:

• ‍Timestamp: Timestamps record the time at which each log entry was generated, usually in the format of *YYYY-MM-DD HH:MM:SS.microseconds *‍
• Log level: Log level identifies the severity of the log entries. For example, entries could be categorized at the info, warningerror levels.‍
• Component: This identifies the component or process that generated the log entry. The component could be the Kubernetes API server, a specific pod or a container.‍
• Message: The message contains the actual content of the log entry, which may include details about the event or error that occurred.‍
• Additional fields: Depending on the logging driver and configuration, Kubernetes log files may also include additional fields such as the Pod metadata and names, namespace, or container ID.

Using structured logging formats like JSON simplifies the search, filtering, and analysis of logs, making it easier to identify issues quickly. Structured logging also allows centralized systems to index specific fields such as severity, user_id, or request_id for more precise querying.

In most cases, this data is sufficient for understanding not just what happened, but also for investigating the context that caused it to happen – and, by extension, for troubleshooting and remediating problems.

Kubernetes log example

As an example of a Kubernetes log file entry, here’s a sample log that records data about a kubelet event.

{
  "timestamp": "2024-01-15T10:23:45.123Z",
  "severity": "INFO",
  "component": "kubelet",
  "node": "node-1",
  "message": "Started container my-container",
  "object": "pod/my-pod",
  "namespace": "default"
}

As you can see, this entry tells us that the container named my-container started at a particular time. It also identifies the Kubernetes component (kubelet) associated with the container. For more detailed troubleshooting, you can access kubelet logs directly using the kubelet API or by querying logs on Linux and Windows nodes.

If the container were to fail to become ready, this log entry might be useful because it would tell us that the container at least tried to start. With that information, we could rule out some potential causes of failure, such as Kubernetes not trying to schedule the container at all. If we wanted more context on why the container wasn’t achieving the ready state, we could look at the logs from the container itself, correlate them with Kubernetes liveness probe configuration and other Kubernetes health checks and probes, and investigate common failure patterns such as CrashLoopBackOff errors in Kubernetes. When using kubectl logs with pods that have multiple containers, remember to specify the container name to view the correct logs. 

Node Logs

Node logs are a crucial source of information for understanding the health and performance of the nodes within your Kubernetes cluster. These logs are generated by the underlying operating system, the container runtime, and other system components running on each node. Node logs can reveal issues related to resource usage, system errors, or failures in the container runtime, making them invaluable for troubleshooting issues that affect the stability of your cluster and the applications it hosts.

To access node logs, you can use the `kubectl logs` command, which allows you to retrieve logs from containers running on specific nodes. For more comprehensive log management, organizations often deploy a centralized logging system using Kubernetes logging tools such as Fluentd or Logstash. These logging agents can be installed on each node to collect, aggregate, and forward node logs to a centralized logging solution, where logs remain accessible for analysis and long-term storage. This approach streamlines log aggregation across the cluster, making it easier to identify patterns, diagnose problems, and maintain the overall health of your Kubernetes environment.

Kubernetes Events

Kubernetes events are lightweight records that capture significant activities and changes within your cluster, such as the creation, deletion, or update of pods and other resources. These events provide real-time insights into the operational state of your cluster and can help you quickly identify issues or unexpected behaviors affecting your applications or infrastructure.

You can retrieve Kubernetes events using the ` kubectl get events` command, which displays a chronological list of recent events in your cluster. Collecting and storing event log data is considered one of the Kubernetes logging best practices, as it allows you to correlate events with other log data for a more complete understanding of cluster activity. By monitoring events alongside system and application logs, you can more effectively troubleshoot issues, ensure compliance, and maintain the reliability and security of your Kubernetes environment.

Application-Level Logging Configuration

Configuring logging at the application level is essential for gaining detailed visibility into the behavior of your containerized applications running in a Kubernetes cluster. Application-level logging configuration involves setting parameters such as log levels (e.g., info, warning, error), log formats, and log destinations. These settings can typically be managed through configuration files, environment variables, or other mechanisms provided by the application itself.

To ensure that application logs are easily accessible and actionable, Kubernetes logging tools like logging agents and log aggregation platforms can be used to collect and forward log messages to a centralized logging solution. Implementing structured logging—where log messages are formatted in a consistent, machine-readable way—further enhances your ability to analyze and troubleshoot application logs. Structured logging makes it easier to search, filter, and correlate log entries, enabling faster root cause analysis and more effective monitoring across your Kubernetes cluster. By leveraging these logging tools and best practices, you can ensure that your application logs provide maximum value for both operational monitoring and long-term analysis.

Log rotation and archiving Kubernetes logs

In most cases, you don’t want your Kubernetes logs to live forever. If you keep outdated logs, you waste storage space. Managing log volumes is important to prevent excessive resource usage and maintain cluster stability. In addition, if log files become too large, it grows difficult to search and analyze them efficiently. High-volume logging can overwhelm storage and degrade performance in a Kubernetes environment, so log rotation and archiving are essential. Establishing a retention policy for logs, including log rotation, helps manage disk space and ensures logs are available for auditing and troubleshooting purposes.

That’s why it’s important to rotate and archive Kubernetes logs. Log rotation is the process of moving older log files to a different storage location or deleting them entirely. Archiving means retaining a copy of log files for long-term storage, typically using a lower-cost and more scalable storage solution than the storage integrated into your cluster itself, which is especially relevant when you run Kubernetes on-premises with local storage constraints.

Kubernetes offers several methods for rotating logs, including the logrotate utility. Logrotate is a Linux tool that you can configure to rotate logs according to a set schedule – such as daily, weekly or monthly. Logrotate can also rotate logs automatically when log files reach a specified size or when available disk space crosses a certain threshold.

Alternatively, you can rotate Kubernetes logs using a logging tool designed to support Kubernetes, such as Fluentd or Logstash. These tools can also rotate Kubernetes logs based on factors like log file size, log file age or a combination of both.

Using logging agents to achieve Kubernetes logging efficiency

An important aspect of Kubernetes logging that admins should understand is that although Kubernetes provides a cluster-level logging architecture, it doesn’t offer a native tool for performing the actual cluster-level logging. Instead, admins have to implement their own method of collecting logs. Basic logging in Kubernetes involves configuring applications to write logs to standard output (stdout) and standard error (stderr streams), which are then automatically captured and managed by the system for analysis and integration.

There are several common approaches to this challenge:

• ‍Node-level logging: You can install a logging agent on every node in your cluster and use it to collect logs, typically using a Kubernetes DaemonSet to ensure the agent runs on each node. Although installing logging agents on each node can be tedious, this is a relatively efficient way to gain access to all available logs.‍
• Sidecar containers: Sidecar containers in Kubernetes can host logging agents inside application Pods. The sidecars collect logs and stream or aggregate them to a preconfigured location. Using a sidecar container allows for capturing and processing logs independently from the main application, enabling sophisticated log data routing and greater flexibility. This approach is easier to implement than node-level logging because it doesn’t require you to install logging agents on each node. The downside is that sidecar containers can increase the resource overhead of your cluster, due to the resources required to run the sidecar containers.‍
• Pushing logs from the backend: It’s possible to push logs directly from within an application, provided your application includes logic for this purpose. Application-level logging configuration allows developers to customize logging mechanisms, enabling structured logs and metadata tagging for better analysis. For distributed systems, following microservices logging best practices helps ensure these application-level logs remain consistent, correlated and scalable. This approach is the most complex to implement because it requires you to configure logging directly within your application, but it’s also efficient because it doesn’t require you to run standalone logging agents.

Log security is a crucial aspect of Kubernetes logging. Logs may inadvertently contain sensitive data, so it is essential to implement security measures such as access control (e.g., RBAC), redacting sensitive information before transmission, and avoiding logging secrets like API keys or PII to ensure logs are stored securely and compliance is maintained.

Kubernetes log collectors

To add more efficiency to Kubernetes logging and help centralize the process of collecting logs from the various Kubernetes components, you may want to use third-party tools such as Fluentd, ELK Stack, Sematext, Logstash, or a cloud-native system like Grafana Loki for K8s logging at scale. These tools are log collectors that can be configured to collect log data from multiple independent sources, then push it all to a central location. Fluentd is a popular open-source log aggregator that allows you to collect various logs from your Kubernetes cluster, process them, and then ship them to a data storage backend of your choice. The ELK Stack (Elasticsearch, Logstash, and Kibana) is a popular open-source tool used for logging in Kubernetes, often deployed alongside Fluentd for log aggregation. In general, log collectors are easier to deploy and manage than individual node-based or sidecar-based logging agents that you manage manually.

To deploy a log collector, you simply create a Deployment that tells Kubernetes how to run it. For example, you can deploy a logging tool using the following command or configuration. Here’s a Fluentd Deployment written in YAML:

As you can see, this configuration tells Kubernetes to run Fluentd as a container. It also configures storage volumes that will serve as a location for aggregating the log data. Tools like Sematext enable you to aggregate and analyze log data from various layers within Kubernetes environments.

The log collector approach improves the efficiency of Kubernetes logging because it allows you to collect and store logs from multiple locations using a centralized log stream. From there, you can easily find, search, and analyze any logs that are relevant to you. Structured logging facilitates log analysis, making it easier to analyze logs for troubleshooting and monitoring. In addition, you get features like built-in filtering and alerting functionality, which you can leverage to help automate the process of monitoring Kubernetes logs.

Log collectors also help to facilitate a scalable log architecture for Kubernetes. As your Kubernetes clusters grow, so will the number of logs it generates. As long as you have log collectors configured, you can add the new log files to your log stream to handle the growth.

Get by with a little help from K8s logs

In conclusion, Kubernetes logging is essential for troubleshooting, security, and maintaining visibility within your cluster. Implementing Role-Based Access Control (RBAC) enhances log security by controlling access to logs and preventing unauthorized access in a Kubernetes environment. Monitoring logs and setting up alerts for specific events, such as error rates exceeding a threshold, is crucial for proactive issue resolution. Additionally, Kubernetes logging requires a separate strategy for persistence due to the dynamic nature of the environment.

From control plane components, to applications, to security events and beyond, Kubernetes’s logging architecture provides the opportunity for gaining comprehensive visibility into what’s happening inside your cluster. But the key word there is opportunity. Kubernetes doesn’t actually collect or analyze the logs for you; it expects you to do that using one of the logging methods described above.

As you consider which Kubernetes logging strategy to adopt, evaluate factors such as how efficient you need log collection to be, how your logging needs may (or may not) scale over time and how often you need to rotate or archive logs. When you get these things right, Kubernetes logs become a K8s admin’s best friend by helping to deliver actionable Kubernetes observability into all facets of the Kubernetes architecture.