Assembling A Container Security Program: Securing The Build

As we mentioned in the last post, most people don’t seem to consider the build environment when thinking about container security, but it’s an important aspect of it. Traditionally, the build environment is the domain of developers, and they don’t share a lot of the details with outsiders (i.e. operations folks). But this is beginning to change With Continuous Integration (CI) and more automated deployments – if not outright Continuous Deployment (CD) – the build environment is more likely to go straight into production. This means that operations, quality assurance, release management and other groups are having to cooperate on building out automation scripts and working more closely with one another. Collaboration means a more complex, distributed working environment with more stakeholders having access. And as the speed in which DevOps principles are beginning to break down the barriers between these groups, and even security teams contributing test scripts and configuration updates, better controls are needed to restrict who can alter the build environment, who can update code, and an audit process to validate who did what.

Let’s not forget why containers are so attractive to developers. First, a container simplifies building and packaging of application code – abstracting the app from it’s physical environment – so developers can worry about the application and not the supporting systems. Second, the container model promotes lightweight services model, whereby big applications are broken down to small pieces, making it easier to modify and scale, especially in cloud and virtual environments. Finally, a very practical consideration is container start-up times are nearly instant, allowing agile scale-up or scale down according to demand. It’s important to keep these facets in mind when considering security controls because any security control that detracts from one of these core advantages will not be considered.

I break build environment security into two basic areas. The first is access and usage of the basic tools that form the build pipeline, including source code control, build tools, the build controller, container management facilities and runtime access. We at Securosis often call this the ‘management plane’ as these interfaces — be it API or GUI – are how we set access policies, automated behaviors and audit activity. Second is security testing of your code and the container, and validating it conforms to security and operational practices. In this post we focus on the former.

Securing The Build

Here we discuss steps needed to protect your code, and more specifically, to protect the build systems so they implement the build process you intended. Conceptually this is very simple, but as there are a lot of pieces in this puzzle, implementation can be a little complex.

Others may refer to this as Secure Software Delivery, or Software Supply Chain Management, or simply’ ‘Build Server Security’; take your pick. It’s the assemblage of tools that oversee and implement your process. But for the purposes of this discussion, these terms are synonymous.

The following are a list of recommendations for securing the platforms in the build environment to ensure secure container construction. We include those Docker and non-Docker supplied tools – source code control, build controller, docker engine and repository tools – that provide automation and orchestration. For each tool in the chain, you will employ a combination of identity management, roles, platform segregation, secure storage for sensitive data, network encryption and event logging to provide security controls. That said, there are some subtleties with each platform which are discussed as follows.

* **Source Code Control**: Stash, Git, Github, or some variant are common. And source code control is one of the tools that will have wide access because it is now common that security, operations, and quality assurance will also contribute code, tests and configuration data. Distributed access means all traffic should be run over SSL or VPN connections. User roles and access levels will be essential at controlling who can do what, but we recommend requiring token or certificate based authentication with two factor authentication _at a minimum_ for all administrative access. 
* **Build Tools and Controllers**: The vast majority of development teams we spoke with used build controllers like Bamboo and Jenkins, with these platforms becoming an essential part of their automated build processes. As these provide many pre-, post- and intra- build options, and can link to a myriad of other facilities, security is somewhat complex. We suggest full network segregation of the build controller system(s), and lock network connections down to source code controller and docker services. If you can, deploy build servers as on-demand containers; this ensures standardization of build environment and consistency of new containers. We recommend that as few team members responsible for the build have access the build controllers as possible, and leverage built-in features to restrict capabilities when developers need access. We recommend locking down configuration and control data to ensure build controller behavior cannot be tampered with. We recommend keeping any sensitive data, such as ssh keys, API access keys, database credential and the like in a secure database or data repository (_e.g._: key manager, DMG file, Vault) and pull credentials as needed to ensure sensitive data is not sitting on disk. Finally, enable log facilities/add-ons available with the build controller, and stream output to a secured location for auditing. 
* **Docker**: You'll use Docker as a tool for pre-production as well as production, building the build environment and test environments used to vet new containers. As with build controllers like Jenkins, you'll want to limit Docker access in the build environment to specific container admin accounts and limit network access to accept content from the build controller system(s), and whatever trusted repository or registry you use. 

In our next post we will discuss validation of individual containers and the contents held within.

- Adrian Lane (0) Comments Subscribe to our daily email digest

from Assembling A Container Security Program: Securing The Build