Kubernetes structured logging best practices

Unstructured log lines make volume harder to control, because every consumer downstream needs a regex that breaks on the next format change. Kubernetes structured logging replaces free-text messages with machine-readable key-value output, JSON in practice, so your pipeline parses once at the source instead of guessing at every hop.

First, define Kubernetes structured logging. Next, trace logs from klog through node collection and enable JSON output. Then compare collection architectures, configure parsing and storage, tune operations, plan migration, and evaluate how groundcover fits that pipeline.

What is Kubernetes structured logging

Kubernetes structured logging means emitting every log entry as a fixed message plus named key-value pairs, so machines extract fields without pattern matching. In Kubernetes itself, the implementation is klog’s structured API and the JSON output format defined in KEP-1602.

The difference shows up in the call site. An unstructured call interpolates values into free text:

klog.Infof("Pod %s status updated to %s", podName, status)

The structured equivalent separates the message from its data:

klog.InfoS("Pod status updated", "pod", klog.KObj(pod), "status", "ready")

With --logging-format=json, that call produces output like the official example:

{
  "ts": 1580306777.04728,
  "v": 4,
  "msg": "Pod status updated",
  "pod": {"name": "nginx-1", "namespace": "default"},
  "status": "ready"
}

The nested pod object comes from KObj, not from the key-value pair alone. Passing a plain string instead ("pod", "kubedns") serializes as "pod": "kubedns" — still queryable, but without the name-and-namespace structure that makes references consistent across components.

The first form forces every downstream consumer to write a regex that breaks when someone rewords the message. The second form lets you query pod.name="nginx-1" in any backend, and KEP-1602 chose JSON specifically for its broad backend support (Elasticsearch, Stackdriver, BigQuery, Splunk) and ad-hoc tooling like jq.

Two feature states matter here, and the official docs track them separately. Structured logging — the InfoS/ErrorS API and its default text serialization — has been beta since v1.23. The JSON output format is still marked alpha, since v1.19, and neither has a GA milestone. A KEP tracking update merged in October 2025 noted that no graduation was planned for 1.35; check the KEP for the current cycle before assuming that still holds. The docs also warn that migration to structured messages is unfinished, so parsers must handle non-JSON lines. Your pipeline needs a fallback path.

How Kubernetes structured logging works

Structured logs pass through four layers: the klog API in the component, the JSON serializer, the container runtime’s capture of stdout and stderr, and the node filesystem that collection agents read.

The stdout and stderr logging contract

The container runtime captures stdout and stderr before any logging agent sees a byte. The logging architecture docs explain that the kubelet sends log location and rotation instructions to the runtime over CRI, and the runtime writes files under /var/log/pods by default (configurable via podLogsDir). Agents tail the container log paths under /var/log/containers/*.log.

The runtime wraps every line in its own envelope. Docker wraps lines in JSON; CRI runtimes write a text format that marks partial lines with a P tag and complete lines with F, which is why Fluent Bit ships built-in docker and cri multiline parsers. Your application’s JSON is therefore a string inside the runtime’s envelope: every parser configuration in this article unwraps the envelope first, then parses the inner JSON where that second stage is configured.

One more contract detail matters for debugging: kubectl logs returns only the contents of the latest log file, so at default rotation settings you can retrieve at most 10MiB per container that way.

JSON log format for Kubernetes components

JSON output from Kubernetes components reserves four keys, and the system-logs documentation plus KEP-1602 define them exactly:

Do not pass these as key arguments in your own InfoS or ErrorS calls. Two more keys appear in practice. The contextual logging blog explains that WithName() adds logger, with multiple calls concatenated by dots. The caller key also shows up in real output (for example "caller":"main.go:92" in klog issue #294) though KEP-1602 does not list it as reserved.

The format carries no API stability guarantee. Field names and JSON serialization are subject to change between releases, output always goes to stderr regardless of format, and process startup can produce non-JSON lines. Treat the schema as stable enough to parse, not stable enough to hardcode.

The klog library: InfoS and ErrorS

InfoS and ErrorS are the migration targets for any Go component or custom controller that should emit structured output. klog.go declares both signatures, confirmed at pkg.go.dev/k8s.io/klog/v2:

func InfoS(msg string, keysAndValues ...interface{})
func ErrorS(err error, msg string, keysAndValues ...interface{})

ErrorS keys its error argument as err. Verbosity-gated variants exist too: klog.V(4).InfoS(...) only emits at verbosity 4 or higher, which becomes your primary volume-control lever later.

The contributor migration guide sets the key naming conventions:

• lowerCamelCase keys (containerName, not container_name)
• Alphanumeric characters only
• err, never error, to match the ErrorS key
• Plural keys for multiple indistinguishable values (pods, not podList)
• The object kind as the key for Kubernetes objects (pod, node, deployment)
• No implementation detail in key names (path, not directory)

Consistent keys are what make cross-component queries possible.

Kubernetes object references with KObj and KRef

klog v2 exposes KObj and KRef helpers for referencing Kubernetes objects, listed in the klog v2 package reference. KObj takes an object that exposes name and namespace and emits both; KRef builds the same reference when you only hold the namespace and name as strings rather than the object itself.

Both return an ObjectRef, which Kubernetes JSON output serializes as a nested object — the "pod": {"name": "nginx-1", "namespace": "default"} shape in the earlier example. In text format the same reference renders as namespace/name. Combined with the kind-as-key convention, every component that logs about the same pod produces the same queryable structure.

Contextual logging

Contextual logging (KEP-3077) attaches a logger with pre-set key-value pairs to a context.Context, so every function in a workflow logs the same correlation fields without repeating them. A reconcile loop that calls LoggerWithValues once with the object reference gets those pairs on every log line downstream, which is what makes cross-workflow correlation queries work.

Retrieval is one call:

logger := klog.FromContext(ctx)

LoggerWithValues and LoggerWithName wrap the underlying logr functionality. NewContext provides the context wrapper, and all three become no-ops when the ContextualLogging feature gate is off.

Kubernetes added the infrastructure in v1.24 without changing components, and the gate reached beta with default true in v1.30. It remains beta and enabled by default as of v1.36, with no GA milestone. Kubernetes migrated kube-scheduler and kube-controller-manager in v1.29; the kubelet conversion is still in progress. Performance is a non-issue at production settings: the 1.29 migration measured no slowdown for kube-scheduler at -v3 or lower, with a 28% slowdown in some test cases only at debug verbosity.

The API-to-serialization path defines what the collection layer receives. The next step is turning JSON output on.

Enabling JSON logs on control plane components

The system-logs documentation lists --logging-format=json support for kubelet, kube-apiserver, kube-controller-manager, and kube-scheduler. Defaults are identical across the three control plane components; only the kubelet needs the config file:

Use this rollout:

1. For kube-apiserver, kube-controller-manager, and kube-scheduler, add --logging-format=json to each component’s command line (static pod manifest or systemd unit, depending on how your control plane runs).
2. For the kubelet, set the logging format in the KubeletConfiguration file passed via --config rather than on the command line, since the kubelet reference marks the CLI flag deprecated.
3. Restart each component and confirm stderr now emits JSON lines with ts, v, and msg keys.

JSON output depends on the LoggingBetaOptions feature gate, which defaults to true in v1.36.0. One version wrinkle: the v1.36.0 component reference pages list only text as a permitted --logging-format value, while the v1.33 kubelet reference documented json explicitly and the system-logs page continues to document --logging-format=json as active. Test the flag against your exact patch version before rolling it across the control plane.

Known limitations to plan around, all from the official documentation:

• --vmodule is unsupported: Per-file verbosity overrides only work with the text format, and the KubeletConfiguration API says so in the field description.
• Startup lines stay unstructured: Process startup can emit non-JSON lines.
• Output is always stderr: Use kube-log-runner to redirect when a shell or systemd is unavailable.
• Removed klog flags: Kubernetes deprecated --log-dir, --log-file, --logtostderr, and eight other klog flags in v1.23 and removed them in v1.26, so file-based redirection belongs to the runtime and collection layer now.

With the control plane and your applications emitting JSON, the architectural decision is what collects it.

Node-level vs cluster-level logging architectures

The Kubernetes logging architecture documentation defines node-level collection and application-managed collection. Application-managed collection includes a sidecar in the application pod or direct application push to a backend. The patterns trade resource overhead against per-pod flexibility:

The DaemonSet pattern wins by default because node-level logging runs one agent per node and needs no changes to the applications on that node, and OpenTelemetry’s Kubernetes guidance likewise lists the DaemonSet as preferred and the sidecar as advanced configuration. The streaming sidecar earns its place when an application writes to files instead of stdout, though the docs note that writing to a file and then streaming to stdout can double node storage, and recommend /dev/stdout as the destination when the app writes a single file. Reserve the full logging-agent sidecar for workloads whose parsing or routing needs cannot be expressed in node-level configuration, and accept losing kubectl logs for them.

OpenAI, Lockheed Martin, and CNCF contributors documented the DaemonSet’s production failure modes. In a CNCF case study, OpenAI ran Fluent Bit as one DaemonSet and separate DaemonSets for an OTel Collector plus a Datadog agent on each node; busy hosts hit Linux CFS throttling and dropped logs, and switching Fluent Bit from inotify to stat-based polling cut its CPU usage 50% and returned 30,000 CPU cores to production. A CNCF engineering blog documents unscoped OTel DaemonSet collectors multiplying metrics 20–40× and OOM-killing 2GB nodes despite memory_limiter, with 4GB minimums and nodeAffinity constraints as the fix. Lockheed Martin reported that running a separate agent per target left half its cluster resources dedicated to compliance work like logging, before it consolidated to a single collector.

Those reports support two operating rules: run one collector per node, because the collector duplication guidance warns that multiple collectors on the same node produce duplicate data, and budget the collector’s CPU and memory as production capacity, not overhead.

Configuring the aggregation pipeline and backends

Each agent configuration below unwraps the runtime envelope. The Fluent Bit example also merges the application’s inner JSON into queryable fields. The shown Fluentd and OpenTelemetry configurations leave the inner application JSON in the message body, so they require an additional JSON parsing stage when you need application fields upstream of the backend.

Fluent Bit

Fluent Bit tails /var/log/containers, handles both Docker and CRI envelopes via multiline.parser, and enriches records with pod metadata through the Kubernetes filter. The YAML form (supported for all settings since v3.2):

pipeline:
  inputs:
    - name: tail
      tag: kube.*
      path: /var/log/containers/*.log
      multiline.parser: docker, cri

  filters:
    - name: kubernetes
      match: 'kube.*'
      kube_url: https://kubernetes.default.svc:443
      kube_ca_file: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
      kube_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token
      kube_tag_prefix: kube.var.log.containers.
      merge_log: on
      merge_log_key: log_processed

merge_log: on parses the application’s JSON body into fields under log_processed, which is the step that makes per-field backend queries work. Two lifecycle notes: Fluent Bit has scheduled classic-mode deprecation for the end of 2026, so write new configuration in YAML, and the fluent/fluent-bit-kubernetes-logging manifests repo is no longer maintained; deploy via the Fluent Bit Helm chart instead, and pin whichever version is current when you deploy.

Fluentd

Fluentd’s tail source uses the <parse> directive, and the parser type depends on the runtime envelope. For Docker JSON logs, @type json; for CRI/containerd, the cri parser shipped in the Fluentd DaemonSet images since v1.12.0-xxx-1.1:

<source>
  @type tail
  path /var/log/containers/*.log
  pos_file /var/log/fluentd-containers.log.pos
  tag kubernetes.*
  <parse>
    @type cri
  </parse>
</source>

This source parses the CRI envelope only. Add a separate JSON parsing stage if you need Fluentd to extract fields from the application’s inner JSON before forwarding the record. The flat format key is deprecated in Fluentd v1.x; use the <parse> block form.

OpenTelemetry Collector Filelog Receiver

collector-contrib v0.101.0 introduced the container operator and collapsed what used to be a roughly 69-line multi-operator chain into one block. Run v0.102.1 or later, which fixed metadata placement so Kubernetes fields land on the resource rather than attributes:

receivers:
  filelog:
    include:
      - /var/log/pods/*/*/*.log
    exclude:
      - /var/log/pods/*/otel-collector/*.log
    start_at: end
    include_file_path: true
    include_file_name: false
    operators:
      - type: container
        id: container-parser

The container operator auto-detects Docker and the supported CRI formats, including CRI-O and containerd. It recombines CRI partial lines and writes k8s.pod.name, k8s.pod.uid, k8s.container.name, and k8s.namespace.name to the entry resource from the file path. The operator unwraps the container envelope but does not parse an application’s inner JSON, so add a JSON parser when you need those application fields as attributes.

start_at: end controls where the receiver begins reading a file it has no stored offset for: at the end, skipping everything already written. That keeps a fresh collector from replaying every existing log file on startup, but it also means you lose lines written before the collector came up, including across restarts if the offset database is not persisted. Use start_at: beginning when losing that window matters more than the replay cost.

Pair the receiver with the Kubernetes attributes processor for deployment and node metadata. Recent collector-contrib releases refer to this component as k8s_attributes; check the README for the release you are running, since the configuration key has changed and older k8sattributes examples are still widespread. The processor requires a ClusterRole with get, watch, and list on pods, namespaces, nodes, and workload resources.

Mapping fields to backends

The same parsed field lands in a different place depending on the backend:

In Grafana Loki, structured metadata carries fields like pod and trace_id that Loki extracts automatically and you query with label filters. The feature was experimental in 2.9.0 and became generally available in 2.9.4. It requires both the tsdb index type and schema v13 — Loki 3.0 enables structured metadata by default and will not start without that combination. The | json parser extracts everything else at query time.

Operational tuning for SREs

The kubelet, not your collection agent, decides how much log data survives on the node, and its defaults set the ceiling everything else works inside.

Log rotation and disk exhaustion

The kubelet owns rotation via CRI, and the KubeletConfiguration defaults are consistent across v1.32 through v1.36:

That is a 50 MiB ceiling per container.

• GKE enforces that the product of size and file count cannot exceed 1% of total node storage, a sensible bound anywhere.
• The kubelet does not rotate logs a component writes to a hostPath volume.
• The kubelet does not track disk usage if logs sit on a separate filesystem from /var.
• When rotation loses the race, node-pressure eviction takes over at nodefs.available below 10%, with no grace period: garbage collection and image cleanup run first, then pods get killed.

For high-throughput nodes, raise containerLogMaxWorkers above 1 and shorten containerLogMonitorInterval.

Verbosity and cost

klog’s -v level and V(level).InfoS gating are your volume controls, and KEP-1602 measured that migrating to structured format alone increases log volume by around 4%. Structured fields turn volume control from a sampling problem into a routing problem: you can drop v>=5 records or route debug namespaces to lower-cost storage by field, instead of sampling blind.

Multi-line stack traces

A Java or Go stack trace arrives as dozens of runtime-wrapped lines. The collection agent must recombine those lines into one record. Fluent Bit ships built-in go, java, python, and ruby multiline parsers, applied through the multiline filter with multiline.key_content log. Apply these constraints from the official docs:

• The multiline filter must be the first filter in the chain, because records re-enter the head of the pipeline and earlier filters would run twice.
• Never define two multiline filters matching the same tags, which creates an infinite loop.
• The buffer defaults to 2MB, after which the record flushes with multiline_truncated: true.

On Fluentd, fluent-plugin-concat handles CRI partial records, with one sharp edge: incomplete multiline buffers that hit flush_interval route to @ERROR, and without an explicit timeout_label block the final lines of a crashing pod, often the ones you need, are silently dropped.

JSON serialization overhead

The KEP-1602 microbenchmark measured JSON InfoS at 319 ns/op against 2455 ns/op for text InfoS. Read that gap with care: it dates from the 2020 proof of concept, and the JSON path used zap while the text path used klog’s own formatter, so it compares two implementations as much as two formats. A kubelet replay benchmark found writing JSON more than 2x faster than structured text, while cluster-level ClusterLoader2 testing found no clear winner between formats. The safe conclusion is narrower than “JSON is faster”: serialization is not where your logging budget goes, since logging accounts for under 2% of overall Kubernetes CPU by KEP-1602’s estimate. Volume and rotation determine pipeline performance. Which format you standardize on decides how much of that volume you can route rather than sample.

Best practices and migration

Format choice, field hygiene, and query rewrites decide whether a structured logging migration sticks.

logfmt vs JSON

Logfmt, the key=value convention described in Brandur Leach’s write-up, is denser and more human-readable in a terminal, but it has no formal specification, no nesting, no arrays, and no types; everything decodes as a text token. JSON supports typed values plus nested objects and arrays, and it is the only structured format Kubernetes control plane components emit via --logging-format. Tooling tilts the same way: Loki and Fluent Bit parse both natively, but Fluent Bit’s Kubernetes filter auto-merge is JSON-only (logfmt needs an explicit Merge_Parser or pod annotation), and the OTel Collector’s key_value_parser extracts logfmt-like pairs as strings with no type inference. Standardize on JSON for Kubernetes pipelines; keep logfmt only where humans tail flat, shallow application logs directly.

PII and sensitive field masking

Mask at the collection layer, before storage, so nothing sensitive ever leaves the node in cleartext. Four mechanisms cover the common cases:

• Fluent Bit allowlisting: The record modifier filter uses allowlist_key to remove every field not explicitly listed, a fail-closed posture, while the modify filter’s Remove_regex strips fields by pattern.
• OTel redaction processor: The redaction processor deletes attributes off an allowlist and masks values matching blocked patterns; an empty allowed_keys removes all attributes.
• Body-level rewriting: Attribute rules never touch the free-text body field, so use the transform processor’s replace_pattern statements for secrets embedded in message text.
• Deterministic masking: For low-entropy values like IP addresses, HMAC functions beat plain hashes, and salted HMAC masking keeps cross-record correlation intact because the same input yields the same token.

Two failure modes to design against: an unmatched pattern produces no error, so unanticipated data shapes ship in cleartext, and every OTTL statement runs against every log record on the node, so ten patterns means ten regex evaluations per line at full volume.

Phased migration checklist

Sequence the cutover so parsing never breaks mid-migration:

1. Inventory current formats per namespace by sampling /var/log/containers output.
2. Enable --logging-format=json on a staging control plane first, and confirm your parsers handle the unstructured startup lines the format’s alpha status all but guarantees.
3. Standardize application output on JSON to stdout, one format per stream; the Kubernetes docs explicitly recommend against writing different formats to the same log stream.
4. Configure the two-step parse (runtime envelope, then inner JSON) and place multiline handling first in the filter chain.
5. Add masking filters between parsing and the forward output.
6. Rewrite queries against the new fields before cutover, not after.
7. Watch volume and rotation for the first weeks: the ~4% structured-format increase plus any verbosity changes land here.

Querying structured logs

How structured logging fits with groundcover

Complete telemetry access determines how much value structured logging delivers. The Flora eBPF sensor deploys as a single DaemonSet, one pod per node, and captures logs, metrics, traces, and Kubernetes events directly from the Linux kernel with zero application instrumentation. One sensor per node avoids stacking a log shipper, a metrics exporter, and an APM agent on every host while keeping those signals available in one platform.

groundcover stores logs in ClickHouse inside the customer’s environment. Under BYOC (Bring Your Own Cloud, where the data plane runs in your cloud account), compute, storage, ingestion, and telemetry processing remain in that account. The split-plane architecture keeps the telemetry data plane in your environment while groundcover manages the control plane for the hosted UI, APIs, authentication, and orchestration.

Flat per-node pricing adds an economic consequence to that architecture. groundcover has no per-GB ingestion charges, so log volume does not change the per-node unit price; neither verbosity level nor field cardinality changes it. Teams can retain the full-fidelity structured logs they built the pipeline to collect instead of dropping debug levels or stripping fields to control ingestion charges.

If you want to validate the architecture on your own cluster, the free plan includes BYOC and requires no credit card. Deploy Flora on one cluster and evaluate full-cluster visibility within hours.