Author: John Walsh, Principal Software Development Security Expert, CyberArk
Bio: John Walsh is a distinguished authority in software security, with over 15 years of experience in various roles within enterprise security companies. From software developer to product manager, product marketing, and open-source community manager, his diverse background has equipped him with a comprehensive understanding of securing the entire software supply chain. This unique perspective enables him to communicate the intricacies of enterprise software security with clarity and depth, catering to both technical and non-technical audiences.

The recent breach at CircleCI underscores the dangers of keeping secrets in various locations like private code repositories (GitHub), scripts, configuration files, encrypted files at rest, CI/CD pipeline code, or even Kubernetes Secrets, where they are not easily rotatable, auditable, authenticated, and secured. CircleCI promptly advised its customers to “promptly change any and all secrets stored within CircleCI” and to “inspect internal logs for any signs of unauthorized access.” Moreover, they suggested customers avoid storing any “long-lasting” credentials in CircleCI.

CircleCI is not an isolated case. Several notable breaches related to unsecured secrets have been in the headlines lately:

• During the same week as the CircleCI breach, Slack announced that a “malicious actor downloaded private code repositories” following the theft and misuse of “a limited number of Slack employee tokens.” Fortunately, no “downloaded repositories contained customer data [or] means to access customer data.” However, it emphasizes that source code is a decidedly insecure place to store secrets.
• The Uber breach disclosed on September 19, 2022, permeated the enterprise due to hard-coded secrets within a PowerShell script, enabling the attacker to achieve elevated access and escalate privileges. This instance again reinforces why embedding secrets in code is not recommended.

Understanding Kubernetes Secrets

In Kubernetes, secrets are of two types: Built-in and custom.

• Built-in secrets are automatically generated by Kubernetes service accounts and are coupled with containers, as well as API credentials. These can be disabled or overridden if necessary.
• Custom secrets allow you to define your confidential data and establish a custom secret for its storage.

Despite Kubernetes Secrets being safer and more adaptable than direct deployment in the Pod or Docker image creation, there are several disadvantages. Kubernetes Secrets store usernames and passwords as base-64 encoded strings. Although not immediately readable, text encoding is not secure, as acknowledged in the official Kubernetes Secret’s documentation.

Furthermore, stored secrets are only accessible within the cluster where they are stored, which is not optimal for numerous cloud deployment scenarios. Organizations frequently have applications running in containers in Kubernetes and also have other applications constructed using serverless functions like AWS Lambda and other cloud-native tools.

There’s a prevailing misbelief that encrypting secrets at rest automatically ensures security. However, this still necessitates manual rotation of the secrets, which, as evidenced by the CircleCI breach, can rapidly become an overwhelming and vital issue in urgent secret rotation scenarios.

Best Practices for Managing Secrets in Kubernetes

Developers can undertake various crucial steps to enhance the security of their Kubernetes environment, such as:

• Grasping what authentication and authorization secrets are necessary to control and their optimum location in your Kubernetes cluster.
• Learning how secrets can be most securely configured, stored, and managed for your deployed pods.

Always ensure to meticulously plan which containers require access to each of your secrets and avoid sharing secrets where they are not strictly required.

Employing a centralized secrets management solution facilitates easier auditing, access control, and secrets management, providing organizations a unified view of their overall Kubernetes security environment. This aids in comprehending and controlling how individual containers and services communicate with each other.

In order to restrict unauthorized access and demonstrate compliance, critical system access should be audited regularly. Central audit trails offer insight into vital security events.

Integrate role-based access control (RBAC) and multifactor authentication, such as certificates. While RBAC authorizes secret access based on a security policy, outlining a role, time, and task-based repository of access management, it’s insufficient protection if the secrets provider authenticates a secret to an imposter. Therefore, strong multifactor authentication using several attributes, only available to trusted containers, such as certificates, is essential to authenticate containers requesting secrets.

A frequently neglected simple guideline: Regularly rotate and alter secrets.

The Need for Centralized Secrets Management

In the current climate of persistent cybersecurity threats, enterprises must concentrate on controllable elements to diminish risk and potential exposure. Given the absence of a well-defined IT perimeter, adopting a zero-trust security approach becomes imperative. Protecting non-human/machine identities and their corresponding secrets is crucial to safeguarding against the subsequent attack. Any static credentials or secrets must be eradicated from all scripts and source code. Security teams should manage access across the entire enterprise comprehensively, without segmentation or oversight gaps, and possess the capability to auto-rotate credentials when necessary. A centralized secrets management system emerges as the optimal choice for authenticating, authorizing, and auditing non-human access as it enables organizations to thoroughly comprehend who has access to what and to auto-rotate or revoke access as required.

Join us at KubeCon + CloudNativeCon North America this November 6 – 9 in Chicago for more on Kubernetes and the cloud-native ecosystem.