Author: Upkar Lidder (LinkedIn)
Bio: Upkar Lidder is senior product manager at Tenable with 10+ years in IT development including team management, functional and technical leadership roles with a deep experience in full stack technology. Upkar is currently focused on Security and DevSecOps in ShiftLeft, Containers and Cloud Native environment.

Putting the right pieces together to ensure reliable and secure Kubernetes infrastructure.

The benefits of adopting cloud-native practices have been talked about by industry professionals ad nauseam, with everyone extolling its ability to lower costs, easily scale and fuel innovation like never before.

Easier said than done.

Companies want to go cloud-native, but they’re still trying to solve the metaphorical “security puzzle” that’s at the heart of Kubernetes management. They start with people and processes, shift corporate culture to match cloud-native security best-practices, and then sort of hit a wall. They understand they need a way to embed security into the standard developer workflows and cluster deployments, but creating continuous and secure GitOps is – in a word – hard.

So, what makes Kubernetes security so difficult?

For starters, Kubernetes is usually managed by developers. They can spin up clusters quickly, but they may not be responsible for managing or securing them. Enter the site-reliability engineers (SREs), who feel the need to slow down developer velocity in order to better manage clusters and make sure they know what’s going into production. Though not ideal, slowing things down is often seen as a reasonable tradeoff for eliminating unmanaged clusters and the kinds of risks that make IT security professionals shudder.

Another challenge of Kubernetes security is that its default configurations aren’t very secure. Kubernetes is built for speed, performance, and scalability. Though each piece of the Kubernetes puzzle has security capabilities built-in, they’re often not turned on by default. That usually means developers forgo security features to move faster.

As if that weren’t challenging enough, Kubernetes has complex privilege-management features that can easily become everyone’s greatest pain point. Kubernetes comes with its own role-based access controls (RBAC), but they add yet another layer of complexity. Any changes made to the Kubernetes puzzle need to be reflected in the others.

The four tenets of Kubernetes security

Although all these challenges roll up to something pretty daunting, all hope is not lost. Enter the four tenets of Kubernetes security: Easy-to-digest best-practices you can implement – today – that will make your Kubernetes and cloud-native infrastructure more secure, and help you manage your cyber exposure.

Tenet #1: Manage Kubernetes misconfigurations with solid policies

Applying policies and associated industry benchmarks consistently throughout the development lifecycle is an essential first step in Kubernetes security. For example, you might have a policy that says you’re not allowed to run any containers with root privileges in your clusters, or that your Kubernetes API server cannot be accessible to the public.

If your policies are violated, that means you have misconfigurations that will lead to unacceptable security risks. Leveraging policy frameworks like OPA and industry benchmarks like CIS can help harden Kubernetes environments and prevent misconfigurations from going into production. Set your policies first, and don’t just park them on a shelf. Revisit them on a regular schedule to ensure they continue to suit your needs.

Tenet #2: Implement security guardrails

Kubernetes security starts in the development process. Most teams developing for cloud-native platforms are using infrastructure-as-code (IaC) to provision and configure their systems. They should expand that to policy-as-code. That means you and your team can apply the same security policies across the software-development lifecycle.

For example, developers can scan their code for security problems on their workstations, in CI/CD pipelines, in container image registries, and in the Kubernetes environment itself. Leveraging developer-friendly tools can help seamlessly integrate security into the development process. Open-source IaC static code-scanners, like Terrascan, can help ensure only secure code enters your environment.

Tenet #3: Understand and remediate container image vulnerabilities

Remediating container image vulnerabilities can be a challenge because it can be hard to see what’s actually running in your environment. Kubernetes dashboards don’t tell you much; you need to know what your images contain and how they’re built. Many developers use common base images and call it good. Unfortunately, unscanned base images can leave you vulnerable, and the risks are compounded at the registry and host levels.

Developers usually skip this sort of image-scanning because it slows them down. The fact is, if they’re not in the habit of being on the lookout for outdated OS images, misconfigured settings, permission issues, embedded credentials and secrets, deprecated or unverified packages, unnecessary services, and exposed ports, they’re just handing off the problem to someone else. Yes, their work may go more quickly, but their practices are slowing the entire software-delivery process.

Some common security risks:

• In images: Outdated packages, embedded secrets, use of untrusted images
• In Registries: Stale images, lax authentication, lack of testing
• In Hosts: Lax network policies, weak access controls, unencrypted data volumes and storage, insecure host infrastructure

Tenet #4: Exposure Management

Once you’ve implemented the first three tenets, it’s time to start looking at your infrastructure holistically. Not all policies apply in all cases, so how are you applying your exclusions? Not every vulnerability is critical, so how do you prioritize fixes and automate remediation? These questions can guide you as you work toward becoming less reactive and more proactive.

At the end of the day, visibility is central to managing security in Kubernetes and cloud-native environments. You need to be able to recognize when your configurations have drifted from your secure baselines, and readily identify failing policies and misconfigurations. Only then can you get the full picture of your attack surface.

Where do you go from here? The journey to comprehensive cloud security

You might be feeling a little overwhelmed by the sheer number of attack vectors in cloud environments. But, with a solid game-plan and taking to heart the best-practices described above, cloud-native security isn’t just about the destination – your runtimes – but the journey (the entire development lifecycle). Shifting left to catch vulnerabilities using policy-as-code is a great first step toward ensuring the reliability and security of your cloud assets – and wrangling all the pieces of the Kubernetes puzzle.

• Link to video demo