Kubernetes Security 101: 10 Best Practices and Fundamentals

What is Kubernetes security?

Kubernetes, also known as K8s, is a popular open source container orchestration platform. Kubernetes security provides protection for the entire cluster during its entire lifecycle, from the control plane to APIs to pods and more.

Deploying defense-in-depth strategies to secure containers and orchestration infrastructure is a way to create strong protections from threat actors. Use security policies, access control, encryption, and more to keep K8s containers protected from data leakage and malicious activity.

Best practices to secure Kubernetes piece by piece

Perhaps the simplest way to approach Kubernetes security best practices is to think about how to secure each part of the Kubernetes cluster, follow fundamental security principles from Kubernetes and OWASP, and identify security tools and resources available.

Alongside traditional security efforts like container isolation, use of verified container images, and keeping Kubernetes patched and updated, implement these 10 Kubernetes security best practices.

1. Use TLS for Kubernetes control plane security

The control plane serves as the management component of the Kubernetes cluster. It includes:

• kube-apiserver: Server that exposes HTTP API and validates commands and information sent to it.
• etcd: Key-value store for secrets management.
• kube-scheduler: Assigns all pods to nodes that will best fit based upon resources and rules.
• kube-controller-manager: Controllers for K8s API that troubleshoot issues with nodes, endpoints, and more.
• cloud-controller-manager: Used to integrate with cloud providers for tasks like configuring network routes, creating load balancers, and more.

Implement Transport Layer Security (TLS) for data encryption both for data in transit and data at rest. This is for data stored in the control plane and sent to and from control plane clients. Use separate encryption for workload data at rest.

2. Encrypt all other Kubernetes data

For data outside of Kubernetes, which is the majority as only non-persistent data lives inside running pods and log data, implement strong data encryption. Typically, the data that your clusters create and/or access will live in some kind of external storage system that interfaces with Kubernetes through a storage plug-in.

To secure data associated with Kubernetes, you should follow the best practices that you would use to secure data inside any large-scale storage system. Encrypt data at rest wherever possible. Use access control policies to restrict who can access data. Ensure that the servers that manage your storage pools are properly locked down. Back up data to help protect yourself against data theft or ransomware attacks.

Follow the same security measures to protect any data that lives natively inside Kubernetes pods and nodes, because Kubernetes does not offer any special tools for this.

Ensure that your organization has tools and policies to secure etcd. As etcd serves as a key-value store, organizations need to limit read or write access. Implement role-based access control (RBAC) and etcd access control lists (ACL) to keep account access limited to only those who absolutely need it. 

Follow the principle of least privilege (POLP) to grant permissions to only a few admin-level accounts and use strong credentials for access between etcd and the Kubernetes API. Lastly, consider isolating etcd behind firewalls and other network security for a layered defense approach.

3. Implement RBAC for Kubernetes API security

The Kubernetes API is what binds the various pieces of a cluster together. As such, it’s one of the most important resources in Kubernetes to secure.

The Kubernetes API is designed to be secure by default. It will only respond to requests that it can properly authenticate and authorize.

That said, API authentication and authorization are governed by RBAC policies that you configure. Thus, the API is only as secure as your RBAC policies. Creating secure RBAC policies that enforce POLP and assign permissions on a granular basis are basic best practices for ensuring Kubernetes API security.

You can further enhance API security by taking advantage of admission controllers. Admission controllers evaluate requests after the API server has already authenticated and authorized them. In this way, admission controllers provide an optional secondary layer of defense against requests that should not be allowed. By enabling and configuring admission controllers, you can enforce security rules related to API requests.

Finally, API requests can be secured at the network level by configuring secure certificates and requiring the API server to serve requests on a secure port rather than on localhost.

4. Harden node security

Nodes are the servers that comprise Kubernetes clusters. In most cases, nodes run some version of Linux, although worker nodes may run Windows. Nodes can be virtual machines or bare-metal servers, but the difference doesn’t really matter from a security perspective.

You should adopt the same security strategies for securing Kubernetes nodes that you would use to secure any type of server.

Kubernetes node security strategies include:

• Remove extraneous applications, libraries, and other components of the operating system in order to minimize your attack surface.
• Provision nodes with minimalist Linux distributions, such as Alpine Linux.
• Eliminate unnecessary user accounts.
• Ensure that nothing runs as root unless strictly necessary.
• Deploy operating system-hardening frameworks, like AppArmor or SELinux.
• Collect and analyze OS logs to detect possible breaches.

If you have experience securing servers at the OS level in any type of environment, you likely already know how to handle Kubernetes node security. At the node level, security considerations aren’t really any different when you’re dealing with nodes running Kubernetes than they are for any type of server.

One important aspect of hardening node security is ensuring the master node is locked down. While there are no fundamental differences between securing a master node and securing a worker node, master node security is more important because a breach could cause more damage to your cluster. Still, the procedures for securing the OS on a master node are the same as for a worker node.

Guide to OWASP Top 10 for Kubernetes

LEARN MORE

5. Isolate workloads and use network policies

Kubernetes network security is similar to pod security in that it starts with following the best practices that you would use to secure any network.

• Isolate workloads: You should make sure to create network architecture that isolates workloads from the public internet unless they need to interface with it. This includes deploying firewalls at the gateway level to block traffic from offending hosts and monitor network traffic for signs of a breach. Additionally, you can use third-party security tools, such as a service mesh.
• Use network policies: Kubernetes offers a limited amount of native tooling for securing networking resources. This tooling comes in the form of network policies. While network policies aren’t a security feature per se, admins can use them to control how traffic flows within a Kubernetes cluster.

You can create Kubernetes network policies to do things like isolate pods from each other at the network level or filter incoming traffic. Network policies aren’t a substitute for securing networking configurations outside of Kubernetes; instead, think of them as an additional resource that complements the security rules that you build into your overall network architecture.

6. Secure pods to protect applications

In Kubernetes, a pod is a container or set of containers used to run an application. In order to secure your applications, you need to secure your pods.

Some aspects of pod security require practices external to Kubernetes:

• Perform security tests on the application before deployment.
• Scan container images before you run them.
• Collect logs from pods and analyze them to detect potential breaches or malicious behavior.

Kubernetes does provide some native capabilities to harden pod security once pods are running. These include:

• RBAC policies: Use these to manage access to pods by users and services within the cluster.
• Security contexts: Use contexts to define the privilege level at which pods run.
• Network policies: Use to isolate pods at the network level.
• Admission controllers: Use to enforce additional security rules that run alongside RBAC.

The types of pod security tools that you use and the way you configure them will depend on the nature of your workloads. There is no one-size-fits-all approach to pod security. Some pods can be entirely isolated from each other at the network level, for example, while others need to be able to communicate.

Whatever your specific requirements, you should evaluate the resources available to secure Kubernetes pods and make sure that you use them to their full extent.

7. Use audit logs

Kubernetes can optionally keep granular records of which actions were performed in a cluster, who performed them, and what the results were. Using these Kubernetes audit logs, you can comprehensively audit clusters to detect potential security issues in real time as well as conduct forensic investigations after an incident.

To use audit logs, you must first create an audit policy, which defines how Kubernetes records events. Kubernetes documentation includes full details on establishing audit policies.

In addition, because Kubernetes doesn’t provide tools to help you analyze audit logs at scale, you’ll likely want to stream audit logs to an external monitoring or observability platform that will help you detect anomalies and alert you to breaches. Otherwise, you can only monitor audit events manually, which is not practical at scale.

8. Isolate workloads with separate namespaces

In Kubernetes, namespaces can be used to isolate different workloads from each other.

While you can run everything inside a single namespace, a Kubernetes security best practice is to create different namespaces for each team and/or type of workload in your cluster. 

You may want to separate your dev/test environment from production using different namespaces, for example.

Managing multiple namespaces does increase the administrative complexity of Kubernetes to a certain extent because you’ll need to create different role-based access control (RBAC) policies for each namespace. However, the extra effort is well worth it because it minimizes the potential impact of a data breach.

9. Secure the Kubernetes dashboard

Kubernetes cluster owners install the dashboard separately to manage clusters and can lead to the creation of service accounts with elevated privileges.

Some ways to protect the Kubernetes dashboard include:

• Require additional authentication if anyone requests access outside of your LAN.
• Use RBAC to keep service account access limited and prevent accidental elevated privileges.
• Implement Kubernetes network policies to block requests to the dashboard.
• Deploy multi-factor authentication and an authenticating reverse proxy.
• Use kubectl for secure connections without needing to provide credentials.

10. Adopt third-party Kubernetes security tools for extra protection

Although Kubernetes provides certain types of security capabilities to help harden resources running within clusters, Kubernetes is not designed to help security teams detect or manage security incidents.

To manage Kubernetes security at scale, you’ll most likely need to leverage external security tools. These Kubernetes security tools can perform several important security functions, including:

• Scanning your RBAC policies, security contexts, and other configuration data to identify misconfigurations that could create security issues.
• Provide application and container image scanning functionality, which you can use to build an automated security pipeline that feeds into your Kubernetes clusters.
• Collect, aggregate, and analyze application logs and audit logs to help you detect anomalies that may signal a breach.

There are a variety of third-party security tools available to protect Kubernetes clusters. Open source options include kube-bench and Falco. kube-bench runs security checks based upon Center for Internet Security (CIS) Benchmarks. Falco provides cloud-native runtime threat detection for Kubrenetes by analyzing Kubernetes audit logs.

Get cloud-native visibility with Sysdig

Container threats move fast, so organizations need cloud-native security solutions to keep pace. Not all Kubernetes security tools offer complete visibility into container and orchestration infrastructure.

Use Sysdig for container and Kubernetes security and get an 80% reduction in time to remediation, 50% of audit costs and time avoided, and get back over 10 hours a month previously dedicated to security and compliance.

With Sysdig, organizations get risk prioritization, vulnerability management, real-time threat detection, Kubernetes security posture management, and improved incident response.