The adoption of Kubernetes technology has been impressive in recent years. The reason is simple: Kubernetes has become the critical ingredient to how countless enterprises are choosing to transform their businesses. It is also the de facto standard for automating deployment, scaling, and management of containerized applications. Even with its massive deployment statistics, it’s important to remember that these are still early days for Kubernetes. It’s a relatively young technology, and many organizations continue to struggle with the widespread complexities of adopting Kubernetes initiatives. The challenges take time and patience to overcome, but obviously are worth the effort because it’s proven that Kubernetes is the driving force that empowers digital and business transformation initiatives.

Why do mainstream technology enterprises widely use Kubernetes? Simple, it’s an extendable, lightweight, open-source container orchestration platform that boasts an ever-expanding ecosystem of security tools, support, and services. This has made it the go-to platform for businesses that are looking to better manage their container assignments and services.

The popularity of Kubernetes is beneficial for its users: the development of a widespread community offers support to developers. However, the community support that offers so many advantages also brings risk into the conversation. That risk happens because the same things that are Kubernetes benefits for users are also advantages that bad actors can use to exploit vulnerabilities. Kubernetes containers have several security risks, including runtime threats, vulnerabilities, exposures, and failed compliance audits. How and why is Kubernetes a double edged sword that is both beneficial but has led to mainstream security incidents such as Capital One, Docker Hub, Microsoft Azure, Tesla, Jenkins and others?

The reason security continues to be the No. 1 concern for organizations using Kubernetes has little to do with the inherent unsafety of Kubernetes itself. Rather, it has a lot to do with how new and complex Kubernetes actually is, and the fact that even skilled cloud-native developers often have a hard time finding their way around. That steep learning curve and an ongoing shortage of skilled IT labor lead to Kubernetes exposures due to misconfigurations. Those misconfigurations are the number one cause for Kubernetes security incidents.

Credentials Are Secrets, Secrets Grant Access to Confidential Information

Handling secrets in a cloud-native environment is a challenge for many organizations. Virtually all applications require some type of secret, such as a database password, a service token, or a certificate, or the ability to establish secure connections. The growing popularity of Kubernetes in the last few years means that many security administrators are now searching for the capabilities to properly handle secrets in Kubernetes services. However, built-in secret capabilities of Kubernetes are limited: they lack certain functions that are essential for securely handling secrets. And if you don’t manage the secrets correctly, risks are certain to follow such as secrets leaked in source code, in application logs, etc., and we all know that leaky security means that critical information can fall into the wrong hands with potentially disastrous results.

Explaining Kubernetes Secrets

Kubernetes provides a Secret object to securely store secret values. There are two types of secrets in Kubernetes:

• Built-in secrets – Kubernetes Service Accounts automatically generate credentials that are used to securely access the Kubernetes API.
• Custom secrets – You can create a Secret object yourself to store sensitive data.

Kubernetes secrets can be created in one of the following ways:

• Automatically – Existing files with sensitive data can be stored as Secret objects with the kubectl command:

kubectl create secret my-secret --from-file=./credentials.txt

• Manually – You can author a Secret manifest in either JSON or YAML:

apiVersion: v1

kind: Secret

metadata:

  name: my-secret

data:

  username: bXktdXNlcm5hbWUK

  password: bXktcGFzc3dvcmQK

type: Opaque

This object is then created based on the manifest using kubectl apply.

Limitations of Kubernetes secrets

Kubernetes Secrets is readily available, unfortunately making it the first choice for many who are new to k8s or new to secrets management best practices. Kubernetes Secrets on its own has several security limitations vulnerabilities.

First, as the sensitive values in the example manifest above are base64 encoded – and thus not encrypted – it is easy to decode these values with the following command:

echo bXktdXNlcm5hbWUK | base64 --decode

my-username

Therefore, the secret’s manifest cannot be stored beside any other manifests (Deployments, Services, and so on) in Git. You don’t want to store a plain-text sensitive value in your repository!

Second, Kubernetes uses etcd as the storage for the secrets. Etcd is a distributed key/value storage known for its great performance. However, it lacks some features that are vital for sensitive data, such as an audit log, insights into key age, and automatic key rotation.

In addition, by default, Kubernetes stores secrets in etcd unencrypted, which means that anyone with access to the etcd cluster has access to the secrets.

Third, anyone with root access to the Kubernetes nodes can access all secrets by impersonating kubelet. Kubernetes currently shares all secrets with all nodes. Preventing privilege escalation is, therefore, essential for blocking access to secrets.

Fourth, Kubernetes can make secrets available to pods either through exposing them as environment variables or by mounting them as files containing plain-text secrets. Environment variables can easily expose sensitive values through, for example, debug logs. Plain-text files on disk are available to everyone with access to the pod.

Finally, Kubernetes’ RBAC functionality provides only get and set permissions for secrets. When you get a secret, you can only get its decrypted value. A more secure zero-trust setup would allow a developer to set a secret and then only retrieve the encrypted value for consumption. This is not available natively in Kubernetes.

Secrets Management Best Practices for Kubernetes

As discussed, Kubernetes security relies on solid secrets management, so now that we know the problem, how do organizations not just accidentally leave the keys in the car and remain vigilant about not allowing their secrets to be discovered?

Some common approaches to getting more secure secret management on Kubernetes is to introduce an external secret management solution, such as Hashicorp Vault, AWS Secrets Manager, Azure Key Vault, or Google Secret Manager. While these approaches resolve some issues inherent in Kubernetes, other – more general – secret management challenges persist. With a third-party solution, it is still virtually impossible to ensure that all holes will be plugged.

Because Kubernetes clusters are distributed and dynamic, they’re vulnerable to attacks and are challenging to secure. To secure our Kubernetes environments, DevSecOps need to use various security tools and techniques throughout an application lifecycle, from building to deploying and runtime.

To keep Kubernetes secure, developers should have a platform that firmly secures and authenticates containers and manages secrets to ensure they’re allowing access to only specific applications, tools, and information. Tools like Conjur integrate with Kubernetes — and most DevOps tools — to effortlessly manage secrets as part of your overall Kubernetes security plan.

When you’re ready to go on the offensive to find security vulnerabilities before hackers arrive, try open-source security tools for yourself.