Assembling a Container Security Program: Monitoring and Audit

In our last post of this series we have split out two key areas: Monitoring and Audit. We want to draw additional attention here in the first case because most development and security teams were not aware of these options, and in the later case as most teams help a lot of misconceptions and fear on the subject. Here we dig into these two areas as they are essential to container security programs.

Monitoring

Every security control we have discussed thus far has to do with preventative security. Essentially these are security efforts that remove vulnerabilities from the environment, or make it hard from anyone to exploit these defects. Basically we address known attack vectors with understood responses like patching, configuration or encryption. But vulnerability scans only take you so far. What about issues your not expecting? Those cases where a new attack variant that gets by your security controls, or a trusted employee makes a mistake? This is where monitoring comes in. It’s how you discover the unexpected stuff. Monitoring is critical to security programs as it’s how you find out what is effective, track what’s really happening in your environment, and detect what’s broken.

For container security it’s no less important, but today it’s not something you’re going to get from Docker or any other container provider.

Monitoring tools work by first collecting events, and then examining those events in relation to security policies. The events may be requests for hardware resources, IP-based communications, API requests to other services, or sharing of information with other containers. Policy types are varied. We may have deterministic policies, such as which users or groups can terminate resources, or which containers are not allowed to make external HTTP requests, or what services a container is allowed to run. Or we may have dynamic – sometimes called behavioral – policies, such as containers calling undocumented ports, using 50% more memory resources than typical, or alerting when runtime parameters are passed that are not commonly associated with a certain container types. Combining deterministic white/black list policies with dynamic behavioral detection provides the best of both worlds, where you can detect cases where there are simple violations of policy, or detect when something out of the ordinary has occurred.

We strongly recommend that monitoring container activity be part of your security program. There are a couple monitoring products on the market today offer by third party security vendors specializing in container security. Common evaluation criteria to differentiate suitability of one platform vs. another is as follows:

• Deployment Model: How does the product collect events? What events and API calls can it collect for inspection? Typically there are two models for deployment; agent atop host computer, and a contianer based agent running atop the Docker engine. How hard/easy is it to deploy the collectors? Can they be deployed as easily as containers or is there a lot of manual work to perform?
• Policy Management: You’ll need to evaluate how easy it is to build new – or modify existing – policies within the tool. You will want to see a standard set of security policies from the vendor to help speed up deployment, but over the lifetime of the product, you’ll stand up and manage your own policies, so ease of management is a factor in your long term happiness with the tool.
• Behavioral Analysis: What, if any, behavioral analysis capabilities are available? How flexible, meaning what types of input data, can be used for making policy decisions? As behavior based analysis requires some system monitoring to determine normal behavior, and thus be able to flag ‘abnormal’ events, you’re often limited to a couple sets of indicators, like user ID or IP address, to base your policies on. The more you have available – system calls, ports, resource usage, image ID, inbound and outbound connectivity – the more granular your controls can be.
• Activity Blocking: Does the vendor provide the capability to block requests or activity? It’s beneficial to be able to block policies violations and ensure that container behave the way they are supposed to. Careful use of blocking is require, as these policies sometimes disrupt containers with new functionality, leading to friction between development and security teams, but the feature is highly effective for security to maintain control over what containers can do.
• Platform Support: You’ll need to verify that monitoring tool supports the OS platforms you use (CentOS, CoreOS, SUSE, Red Hat) and the orchestration tool (Swarm, Kubernetes, Mesos, ECS) of your choice.

Audit And Compliance

What happened with the last build? Did we remove SSHD from the container? Did we add the new security tests to Jenkins? Was is the current build in the repository?

Like many of you reading this, you may not know the answer of the top of your head, but you know where to get it: The log files. GIT, Jenkins, JFrog, Docker and just about every tool used in development create go files and it’s how we figure out what happened and, quite often, what went wrong. There are people outside of development, namely the security and compliance teams, that have similar security related questions about what is going on with the container environment and if security controls are in place. Logs are how you get these external teams the answers they need.

Most of the earlier sections in this research work, like build environment and runtime security, will have some compliance requirements associated with them. Be it externally mandated requirements like PCI-DSS or GLBA, or internal security requirements from internal audit and security teams, they will want to see that security controls are in place and are working. And no, they won’t just take your word for it; they will want to see audit reports for specific event types relevant to their audit. Similarly, if your company has a Security Operations Center, in order to investigate alerts or determine if a breach has occurred, they are going to want to see all system and activity logs over a period of time to reconstruct events. You really don’t want to get too deep into this stuff, just get them the data they want and let them worry about specifics.

The good news most of what you need is already in place. We did not speak with any firm during our research that did not have Splunk, Log storage or SIEM on premise, and in many cases all three were available. What’s more is the vast majority of code repositories, build controllers and container management systems – more specifically Docker runtime and Docker Trusted Registry – produce event logs, and do so in a format that can be consumed by the various log management or Security Information and Event Management (SIEM) systems. And too do most of the third party security tools for image validation and monitoring as well. What you’ll need to evaluate is how easy this is to accomplish. Some simply dump syslog formatted information into a directory and it’s up to you to drop this into a Splunk, S3 bucket, Loggly or SIEM tool. In other cases — most cases actually — you can specify CEF, JSON or other format and the tools can automatically link to the SIEM of your choice, and send events as they occur.

This concludes our research on Building a Container Security Program. We covered a ton of different aspects — both production and non-production issues. As such we tried to cover in sufficient depth as to be helpful, but not take overwhelm in details. As such, if we missed something you feel is important, or you have questions unanswered, please leave a question and we will address in the comments and in the research paper if appropriate. It’s your feedback that helps make these better so please help us — and other readers — out.

- Adrian Lane (0) Comments Subscribe to our daily email digest

from Assembling a Container Security Program: Monitoring and Audit