Kubernetes Secrets: How They Work, Types, and Best Practices

Kubernetes Secrets let you store and manage sensitive data, like passwords, tokens, and certificates, securely within your cluster. When not managed properly, Secrets can expose your system to serious security risks.

That, at least, is a high-level take on why Kubernetes Secrets management is important. For the details, read on as we explain how Kubernetes Secrets work, which types exist, how to manage Secrets, and best practices for protecting Secrets without disrupting the efficiency of your cluster.

What are Kubernetes Secrets, and why do they matter?

In Kubernetes, Secrets are a type of object that represents sensitive information, such as passwords or certificates. Secrets store this data in a way that makes it possible to grant access to it under controlled circumstances.

Secrets simplify the deployment and management of workloads that require authentication, authorization or encryption. Without Secrets, it would be more difficult to perform tasks that require the sharing of sensitive data between two or more applications or services. You’d have to store the data in plain text (which would be insecure), or use an external tool to secure the data and share it as needed (which would add administrative complexity). Kubernetes Secrets mitigate this challenge by providing a secure way to store and manage sensitive information as a native feature.

How Kubernetes Secrets Work

Kubernetes Secrets work based on the following core principles:

• Key-value pairs: Every Secret in Kubernetes takes the form of a key-value pair. The key is usually a string (such as a username), and the value is sensitive information associated with that string (like a password).
• Storage in Etcd: Secrets are stored in Etcd, the key-value store system that houses most configuration data for Kubernetes clusters.
• Declarative management: Like other types of Kubernetes resources, Secrets are managed declaratively. This means admins declare which Secrets they want to create, usually by defining them within YAML code. They can also declare (again, via YAML) which resources should have access to which Secrets. For instance, when defining a Pod, admins can specify Secrets that should be available to the Pod.

Encoding: Kubernetes Secrets are encoded using Base64. This makes it possible to transmit Secrets data in a consistent way. Importantly, encoding is not the same as encryption since encoded Secrets can be decoded, and Kubernetes doesn’t encrypt Secrets by default. We’ll say more about this when we discuss Secrets management best practices.

Types of Kubernetes Secrets

Kubernetes Secrets come in many forms. Here’s a look at the most common types.

Opaque Secrets

Opaque Secrets are the most flexible type of Secret, as they can store any type of data as key-value pairs. They’re typically used for managing authentication and authorization data, like passwords and API keys.

Basic authentication Secrets

Basic authentication Secrets store usernames and passwords as key-value pairs. Unlike opaque Secrets, however, which are open-ended in that they can store any type of data, basic authentication Secrets can only manage usernames and passwords. The most common use case for this type of Secret is HTTP authentication.

TLS Secrets

TLS Secrets house TLS certificates and their associated private keys. They’re most often used for encrypting network traffic, including both internal cluster communications and traffic flowing to or from external endpoints.

Registry Secrets

Registry Secrets are another special type of Secret designed (as you might surmise) for storing the authentication data used to connect to container registries. You could use opaque Secrets for this purpose as well, but it’s simpler to do this using registry Secrets because they store authentication data in the form registries expect.

Bootstrap token Secrets

Bootstrap token Secrets are a special type of Secret that store the information nodes need to authenticate with a cluster. They’re used to automate the process of joining new nodes to a cluster.

Creating and managing Kubernetes Secrets

Now that we’ve explained what Kubernetes Secrets are, let’s explore the process for working with them.

Creating Secrets

The most common way to create a Kubernetes Secret is to define it using YAML. For example, here’s a YAML manifest for a Secret that stores a username and password:

apiVersion: v1
        kind: Secret
        metadata:
          name: username-password-secret
        type: Opaque
        data:
          username: user
          password: password

Note that in this example, the key and the value (i.e., the username and the password) are not encoded within the YAML file. Kubernetes will automatically encode the value (the password) if it’s not encoded. But you can also enter the value as an encoded string within your YAML manifest, if you wish.

After defining the secret, save the YAML manifest as a file and apply it to your cluster using a kubectl command such as:

kubectl apply -f username-password-secret.yaml

Kubernetes will then automatically create the Secret.

It’s also possible to create Secrets using a single kubectl command by defining the Secret values on the command line. For example, the following command creates a Secret that stores a username and password:

kubectl create secret generic username-password-secret --from-literal=username=user --from-literal=password=password

Viewing Secrets

You can view a list of available Secrets using kubectl:

kubectl get secrets

To get details on a particular Secret, use kubectl describe:

kubectl describe secret secret-name

(Replace secret-name in the code above with the name of your actual secret.)

Note that by default, Kubernetes will display Secrets values in encoded form. If you want to view them as plain text, you need to pipe the secret into the base64 utility (which must be available in your shell environment), such as in the following example:

kubectl get secret username-password-secret -o jsonpath='{.data.username}' | base64 --decode

How to use Secrets in Pods and controllers

So far, we’ve looked at how to create and manage Secrets. But how do you actually use them in an application? The answer is to make Secrets available to a Pod when defining it. You can go about this in two basic ways.

First, you can expose Secrets as environment variables by including code like the following in your Pod spec:

env:
          - name: USERNAME
            valueFrom:
              secretKeyRef:
                name: username-password-secret
                key: username

This will make the value of the Secret named username-password-secret (i.e., the password) available as an environment variable named USERNAME within the Pod. Applications can reference that variable to get the value.

Second, you can expose Secrets as a mounted volume using code such as the following:

volumes:
          - name: secret-volume
            secret:
              secretName: username-password-secret
        volumeMounts:
          - name: secret-volume
            mountPath: /etc/secrets

In this case, the data defined in the secret username-password-secret would be available to the Pod at the path /etc/secrets.

Accessing Secrets for controllers

In addition to sharing Secrets with Pods, it’s possible for Kubernetes controllers (meaning the control loops that manage cluster state) to access Secrets through the Kubernetes API. This is helpful for use cases such as accessing a TLS certificate to encrypt traffic.

Controllers can access Secrets as long as the Service Account associated with the controller has permission to do so. You can grant permission by creating an RBAC rule such as the following:

apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  name: secret-reader
  namespace: my-namespace
rules:
  - apiGroups: [""]
    resources: ["secrets"]
    verbs: ["get", "list", "watch"]

This grants permission to get, list, and watch Secrets in the namespace my-namespace. If you also created a RoleBinding that links this Role to a Service Account, the Service Account would have these permissions.

Limitations and alternatives to Kubernetes Secrets

While Secrets are a powerful feature in Kubernetes, they’re also subject to some limitations. Notable challenges include:

• Lack of encryption: By default, Kubernetes stores Secrets in unencrypted form within Etcd – which means that anyone who can access Etcd (such as anyone with a user account and sufficient permissions on the Kubernetes control plane node, which hosts Etcd) can read Secrets, unless admins turn on encryption.
• Access control complexity: Managing access configurations for Secrets can be challenging because you have to define access permissions for each resource separately. This becomes unwieldy when you have hundreds of thousands of Secrets to manage.
• Secrets injection complexity: Exposing Secrets to applications as environment variables or volumes doesn’t always work well. For instance, it can be problematic in situations where applications run in restricted shells and lack the ability to read environment variables or access storage volumes.
• Static Secrets: Kubernetes Secrets are static, meaning that they exist indefinitely until you delete them. This is less secure than dynamic Secrets generation (the ability to generate Secrets on-demand) or temporary Secrets, neither of which Kubernetes supports natively.

Because of limitations like these, some admins choose to use external Secrets management solutions in Kubernetes, such as HashiCorp Vault or AWS Secrets Manager. These tools store Secrets outside of Kubernetes and expose them to applications using methods such as sidecar containers (in which case a secrets manager agent running within Pods provides access to the externally stored Secrets) or by turning externally managed Secrets into native Kubernetes objects.

Alternative Kubernetes Secrets managers require more effort to set up than relying on native Kubernetes Secrets alone, but they provide additional features (like the ability to generate Secrets dynamically or encrypt them by default) that can make the added trouble worth it.

Kubernetes Secrets best practices

No matter which tool you use to manage Kubernetes Secrets, the following best practices can help to streamline the process while improving security:

• Enable encryption: To protect Secrets at rest, enable encryption in Etcd. The way to do this varies a bit between Kubernetes distributions, but in most cases, it requires creating an encryption-provider-config file that includes encryption keys.
• Use the right Secrets type: Different types of Secrets serve different purposes – and although you can usually find a way to achieve most tasks using opaque Secrets, it’s a best practice to use a type of Secret designed for your specific use case. Use opaque Secrets only if no dedicated Secret type exists.
• Remove outdated Secrets: Kubernetes won’t automatically remove Secrets that you no longer need. You have to remove them manually, or use external tools to automate the process.
• Update Secrets: Similarly, update Secrets periodically to reduce the risk that secret leaks (meaning situations where unauthorized parties gain access to Secrets) can enable breaches.

Monitoring and alerting for Kubernetes Secrets

From a monitoring and alerting perspective, the best way to keep track of Secrets is to use Kubernetes’s audit logging feature to monitor for events related to Secrets. Here’s an example policy that will log whenever someone creates or updates a Secret:

apiVersion: audit.k8s.io/v1
    kind: Policy
    rules:
      - level: Metadata
        resources:
          - group: ""
            resources: ["secrets"]
        verbs: ["get", "list", "create"]
      - level: Request
        resources:
          - group: ""
            resources: ["secrets"]
        verbs: ["create", "update"]

You can then view audit logs to track Secrets changes. You could also scrape the logs and monitor Secrets events using an external observability tool.

Troubleshooting Kubernetes Secrets: Common errors and fixes

The most common problem that you’re likely to face when managing Kubernetes Secrets is a Secret that can’t be found. Here are troubleshooting steps you can perform:

1. Check Secrets: Use the command kubectl get secrets to list Secrets and verify that the Secret in question exists.
2. Check YAML: Check your YAML code to make sure there are no typos in your Secret reference.
3. Review RBAC configurations: Make sure that RBAC policies grant access to the Secret by the Pod or Service Account. Check in particular that the RBAC policies include the right access verbs.
4. Check cluster state: If all of the above fails, verify that your cluster is operating normally, and that no connectivity problems exist between the control plane and worker nodes.

How groundcover improves visibility for Kubernetes Secrets

At groundcover, we can’t manage Secrets for you – we leave that to tools that specialize in secrets management.

We can, however, provide holistic visibility into your Kubernetes cluster so that if something goes wrong with a Secret or the workload accessing it, you can get to the root of the problem quickly. We do this by continuously tracking metrics and performance trends for all resources across your cluster – Pods, nodes, Services, and more.

When it comes to Secrets troubleshooting, context is everything, and groundcover provides the context you need to manage Secrets efficiently and effectively.

The secret to great Secrets management in Kubernetes

Secrets are a fact of life – for humans and for applications alike. That’s why, as a Kubernetes admin, your job is to ensure that your cluster has access to the Secrets your workloads need to do their jobs, and that you keep those Secrets secure.

FAQ

What’s the safest approach to manage Kubernetes Secrets in GitOps?

If you use GitOps (meaning you store all of your configuration code in Git), the best way to manage Secrets securely is to use an external secrets manager, like HashiCorp Vault or AWS Secrets Manager. You can then manage Secrets stored in Git as custom resources within Kubernetes.

When should I use an external secrets store over native Kubernetes Secrets?

While it’s fine to use Kubernetes’s native Secrets tooling for simpler use cases, external secrets managers provide additional features, like the ability to rotate Secrets automatically or revoke Secrets that are no longer being used.

How does groundcover detect and alert on misuse of Kubernetes Secrets?

By monitoring Kubernetes events, groundcover can display alerts for issues like missing Secrets. It can also detect workload performance problems, such as the failure of a Pod to start due to the inability to access Secrets properly.

Are Kubernetes Secrets encrypted by default?

No. Kubernetes encodes Secrets with Base64 but does not encrypt them at rest. To protect sensitive data, you must enable encryption in Etcd manually or use an external secrets manager.