Creating a Secure Kubernetes Deployment? Five Ways the New NSA Kubernetes Hardening Guide Can Help

From IC Insider Rancher Government Solutions

By: Andy Clemenko, Field Engineer, Rancher Government Solutions

andy.clemenko@rancherfederal.com

Twitter: @clemenko

With more than 20 years of experience as a System Administrator, I can confidently assert that the lack of technical validation for systems has been an issue, especially when it comes to security.

In recent years, STIGs and the CIS benchmarks have helped tremendously, but they still don’t address big-picture concepts like securing Kubernetes deployments, which is where I spend the majority of my time as a field engineer with Rancher Government Solutions (RGS). Every day, my colleagues and I work together to address the unique security and operational needs of the U.S. Government and military around application modernization, containers and, of course, Kubernetes.

So, it’s not a surprise that we celebrated earlier this year when the NSA and CISA released an updated Kubernetes Hardening Guide. The guide is designed as minimum standard for hardening Kubernetes against some common attack vectors – namely supply chain, malicious threat actors, and insider threats. In other words, the work we do every day.

The guide covers the best practices for preventing and mitigating such attacks, and it’s an incredible resource for teams looking to create a secure strategy for any Kubernetes deployment. Moreover, it can help existing Kubernetes teams ensure their deployments meet the guide’s standard and, if not, that their partners are equipped to help get them there.

The guide describes “the security challenges associated with setting up and securing a Kubernetes cluster. It includes strategies for system administrators and developers of National Security Systems, helping them avoid common misconfigurations and implement recommended hardening measures and mitigations when deploying Kubernetes.”

Wait. What is Kubernetes?

If you’re asking yourself, “What the heck is Kubernetes?” don’t despair.

The Kubernetes Hardening Guide begins with a little level-setting, noting that Kubernetes is “an open-source system that automates the deployment, scaling, and management of applications run in containers, and is often hosted in a cloud environment.”

 In our experience at RGS, Kubernetes helps boost productivity, reduces cost and risk, and moves organizations closer to achieving their hybrid cloud goals.

The guide is important because, as I noted above, there’s always been a disconnect between the documentation and the practical, secure, implementation of software – any software. The Kubernetes ecosystem is no stranger to this problem. And since Kubernetes is open source, there are about a dozen different distributions. Each distribution makes strategic decisions on the deployment details. Some vendors make good choices. Some vendors make bad choices.

The guide is intended to bridge the gap from the initial install to a hardened and secure Kubernetes cluster. It is essential to have a guide like this when your adversary never quits. This guide will also keep the dozen or so vendors honest when it comes to securing their Kubernetes distribution.

If only Tesla had this guide before their Kubernetes cluster got “hacked.” Reporting from the tech news site Ars Technica  at the time noted that, “The initial point of entry for the Tesla cloud breach … was an unsecured administrative console for Kubernetes, an open source package used by companies to deploy and manage large numbers of cloud-based applications and resources.”

Meaning, had the guide been available then, the system administrators, hopefully, would not have exposed an un-authenticated dashboard. This dashboard gave the attackers full control of the cluster. The attackers were then able to escalate privileges to gain access to customer databases themselves.

This is just one example – of many – showing why this guide is so important.

What’s In the Guide?

The Kubernetes Hardening Guide covers topics from a couple of major categories. Among them: Kubernetes Pod Policy, Network Separation and Hardening, Authentication and Authorization, Audit Logging and Threat Detection and Upgrading and Application Security Practices.

The guide does a great job diving deep and giving technical examples across topics, again, providing a strong, secure technical foundation in the process. This article hits the high points in the guide, but readers should check out the Key Points section of the guide for more technical information.

Are All Kubernetes Are Created Equal?

No.

Again, Kubernetes is open source and free to use by anyone. This is good and bad. Take a look at how many Kubernetes distribution vendors, and therefore choices, there are.

However, Kubernetes has a slight complexity problem. When vendors take the underlying code and package it, they inherently make tradeoffs. Some vendors lean toward having a complete experience at the cost of security. Some lean toward lock-in of an entire ecosystem. One vendor builds additional object types on top of Kubernetes. Meaning, it is very difficult to change distributions if you rely on that object. Rancher, and our fully conformant RKE2 distribution, leans hard toward security.

This isn’t a sales pitch. It’s a technical reality that influences decision making within our customers’ organizations.

Ok. What Else Do I Need to Know?

Easy. Five things:

1. Kubernetes Pod Policy

First, when it comes to Kubernetes Pod Policy, the guide covers everything from building secure images, rootless containers, and the underlying container engine. Pod policy is a foundational element of Kubernetes and is a logical place for the guide to start.

Because it was specifically designed for government, RKE2 is shipped with all of this in place in ContainerD, but it is replaceable with the engine of choice and it’s essential that teams are aware of how to get their pod policy in line with the updated NSA standard.

2. Network Separation and Hardening

The guide also outlines how to use network policies and firewalls to separate and isolate resources.  RKE2 uses Canal as the default. However, there are several to choose from and picking the best Container Network Infrastructure (CNI) that fits your environment is important.

The guide goes on to talk about how to secure the control plane. Ideally, the entire cluster would be behind a firewall that can used for isolation. Adding Rancher as an additional management layer allows for a single point of entry for management tasks. Rancher then can apply all the authentication and authorization to the underlying RKE2. This approach provides a decreased surface area for the cluster and getter control.

It’s important to encrypt both traffic and sensitive data (such as Secrets) at rest. RKE2 supports Secret Encryption out of the box, and Rancher provides the ability to single-click install Istio, which provides additional network isolation capabilities as a service mesh.

3. Authentication and Authorization

The key points in the guide around authentication and authorization revolve around actually implementing it. One of the key features that Rancher added to RKE2 is centralized user authentication. Several popular authentication methods include OIDC and SAML2. Rancher supports a large number of methods. In addition, Rancher can manage RBAC for not just one cluster, but many.

4. Audit Logging and Threat Detection

The guide points to enabling audit logs and using third-party security tools. Keep in mind, when it comes to audit logging, it’s very important to understand who is interacting with the cluster. There are several ways to implement this with RKE2. First, you can enable the audit log during the initial install. You can also enable Rancher’s API audit during the install.

As for higher levels of threat detection, we recommend another open-source tool called Neuvector because of its incredible ability to actively defend against unknown or unexpected behavior. Most products in this category require pre-written policies, but Neuvector has the ability to baseline and block anything outside the normal behavior.

5. Upgrading and Application Security Practices

As the guide notes, it’s exceptionally important to ensure that all the components in a Kubernetes environment are upgraded in a timely manner. Ask Equifax about the consequence of not patching Apache Struts.

All infrastructure needs to be updated periodically. RKE2 helps with this by providing simple installation methods that make keeping the cluster updated very easy and automatable. We also include a built-in Continuous Delivery (CD) tool called Fleet that allows a team to apply GitOPs methods to managing an application’s lifecycle. Simply put, you can use Git to push updates to the applications being deployed. Also known as Infrastructure as Code, or (IaC).

So, What Did We Learn?

Without question, there is a lot of valuable information within The Kubernetes Hardening Guide. As noted previously, guides like this go a long way toward setting standards and best practices and enabling teams to ensure their environments are as secure as they can be or that they have the right support partners in place to get them there.

Another key takeaway is this: These guidelines are going to be updated all the time. As they should. There is so much innovation and discovery in this space – and so many bad actors too. Keeping pace with innovation while staying one-step ahead of cybercriminals is so important. There is no place for a “set it and forget it” mentality.

Anyone working in the defense and intelligence community – or in the technology space – has the privilege of doing important and exciting work. My colleagues at RGS and I are fortunate enough to be able to do both. It’s my sincere hope that these tips helped to demystify both Kubernetes and the hardening guide as well.

About RGS

Rancher Government Solutions is specifically designed to address the unique security and operational needs of the US Government and military as it relates to application modernization, containers and Kubernetes.

Rancher is a complete open source software stack for teams adopting containers. It addresses the operational and security challenges of managing multiple Kubernetes clusters at scale, while providing DevOps teams with integrated tools for running containerized workloads.

RGS supports all Rancher products with US based American citizens with the highest security clearances who are currently supporting programs across the Department of Defense, Intelligence Community and civilian agencies.

About IC Insiders

IC Insiders is a special sponsored feature that provides deep-dive analysis, interviews with IC leaders, perspective from industry experts, and more. Learn how your company can become an IC Insider.