Security Considerations for Container Orchestration

Orchestration platforms are enabling organizations to scale their applications at an unparalleled rate which is why many technology centric companies are surging to move applications onto a distributed datacenter wide platform that enables them to scale at a click of a button.

One orchestration platform that is rapidly growing in popularity is a distributed datacenter wide operating system running on top of the Apache Mesos kernel. A traditional operating system manages resources for a single server, whereas a distributed operating system seamlessly manages resources for multiple servers acting as one shared pool of resources. Once Mesos agents have been established on servers throughout a datacenter, a cluster is formed, and Mesos jobs can be distributed from a Mesos master to servers with available resources within the cluster for processing.

A risk associated with the Mesos master and slave model revolves around the authentication of Mesos services. Many Mesos deployments do not require authentication by default – if an attacker can communicate with either the Mesos master service (default TCP port 5050) or a Mesos slave service (default TCP port 5051), the attacker may be able to easily gain remote code execution (RCE) on another server within the cluster.

Frameworks are commonly installed within DC/OS environments to provide datacenter wide services. For example, the Marathon Framework is commonly used within these environments to perform container orchestration and help ensure that a specific number of instances of a container are running persistently within the DC/OS cluster. The Chronos Framework is also commonly deployed to provide fault tolerant distributed job scheduling throughout the cluster.

Today many frameworks do not by default enforce authentication on both management web UIs and the API interfaces. Therefore, if an attacker can communicate with the framework, they may be able to gain remote code execution (RCE) on servers within the cluster.

One hurdle associated with the exploitation of framework services, is that many implementations will deploy framework services to random high TCP ports on arbitrary servers within the cluster, making it slightly more difficult for an unaware attacker to find the services within the datacenter. This hurdle can be overcome by leveraging a few built in services within the DC/OS environment. First, many times a unique top level domain (TLD) is established for the Mesos cluster which can be used by services within the cluster to locate frameworks (e.g. ping -c3 marathon.mesos). Second, if the Mesos master service can be queried, a complete list of Mesos slaves can be obtained potentially reducing the attack space. Lastly, many times a Mesos DNS service is established that will enable a remote attacker to perform an enumerate API call. This is the functional equivalent of performing a DNS zone transfer – it will provide a detailed map of all services and random high ports back to the attacker.

Defenders can easily take quite a few steps to help prevent the above exploitation paths, including:

  • Enabling authentication on all Mesos Masters and Agents
  • Enabling authentication on all Framework Web UIs and APIs services
  • Disabling the enumerate API call for the Mesos DNS service
  • Logging authentication requests and the execution of jobs for detection of suspicious events

As the popularity of orchestration platforms grows, attackers will continue to spend more resources building tools and techniques to exploit these frameworks and services. Organizations leveraging these technologies would be wise to spend the extra cycles up front to put reasonable security controls in place before using these platforms to host production applications.

Bryce Kunz
Sr. Lead Security Engineer, Digital Marketing

Comments are closed.