The TENTH Annual Disaster Recovery Breakfast: Are you F’ing Kidding Me?

Posted under: General

What was the famous Bill Gates quote? “We always overestimate the change that will occur in the next two years and underestimate the change that will occur in the next ten.” Well, we at Securosis actually can gauge that accurately given this is the TENTH annual RSA Conference Disaster Recovery Breakfast.

I think pretty much everything has changed over the past 10 years. Except that stupid users still click on things they shouldn’t. And auditors still give you a hard time about stuff that doesn’t matter. And breaches still happen. But we aren’t fighting for budget or attention much anymore. If anything, they beat a path to your door. So there’s that. It’s definitely a “be careful what you wish for” situation. We wanted to be taken seriously. But probably not this seriously.

We at Securosis are actually more excited for the next 10 years, and having been front and center on this cloud thing we believe over the next decade the status quo of both security and operations will be fundamentally disrupted. And speaking of disruption, we’ll also be previewing our new company – DisruptOPS at breakfast, if you are interested.

We remain grateful that so many of our friends, clients, and colleagues enjoy a couple hours away from the insanity that is the RSAC. By Thursday it’s very nice to have a place to kick back, have some quiet conversations, and grab a nice breakfast. Or don’t talk to anyone at all and embrace your introvert – we get that too.

The DRB happens only because of the support of CHEN PR, LaunchTech, CyberEdge Group, and our media partner Security Boulevard. Please make sure to say hello and thank them for helping support your recovery.

As always the breakfast will be Thursday morning (April 19) from 8-11 at Jillian’s in the Metreon. It’s an open door – come and leave as you want. We will have food, beverages, and assorted non-prescription recovery items to ease your day. Yes, the bar will be open. You know how Mike likes the hair of the dog.

Please remember what the DR Breakfast is all about. No spin, no magicians (since booth babes were outlawed) and no plastic light sabers (much to Rich’s disappointment) -– it’s just a quiet place to relax and have muddled conversations with folks you know, or maybe even go out on a limb and meet someone new. We are confident you will enjoy the DRB as much as we do.

To help us estimate numbers, please RSVP to rsvp (at) securosis (dot) com.

- Mike Rothman
(0) Comments
Subscribe to our daily email digest

The post The TENTH Annual Disaster Recovery Breakfast: Are you F’ing Kidding Me? appeared first on Security Boulevard.

from The TENTH Annual Disaster Recovery Breakfast: Are you F’ing Kidding Me?

Evolving to Security Decision Support: Data to Intelligence

Posted under: Research and Analysis

As we kicked off the Evolving to Security Decision Support series, the point we needed to make is the importance of enterprise visibility to the success of your security program. Given all the moving pieces in your environment, including the usage of various clouds (SaaS and IaaS), mobile devices, containers, and eventually IoT devices – it’s increasingly hard to really know where your critical data is and how it’s being used.

Though enterprise visibility is necessary, but not sufficient. You still have to figure out if/how you are being attacked and if/how data and/or apps are being misused. Ultimately no one gets any credit for knowing where you can be attacked. You get credit for stopping attacks and protecting critical data. Ultimately that’s all that matters. The good news is that many organizations already do extensive security data collection (thanks compliance!), so you have a base to work with. It’s really just a matter of turning all of that security data into actual intelligence that you can use for security decision support.

The History of Security Monitoring

Let’s start by providing some historical perspective on how we got here, and why many organizations already do extensive security data collection. It all started in the early 2000s with deployment of the first SIEM, which was really deployed to make sense of the avalanche of alerts coming from firewalls and intrusion detection gear. You remember those days, right?

SIEM evolution was driven by the need to gather logs and generate reports to substantiate controls (thanks again compliance!). Thus the SIEM products focused on storing and gathering data than actually making sense of it. You could generate alerts on things you knew to look for, which typically meant you got pretty good at finding attacks you have already seen. But you were pretty limited in the ability to detect the attacks you haven’t seen.

SIEM technology continues to evolve, but mostly to add scale and data sources to keep up with the number of devices and activity that need to be monitored. Yet, that doesn’t really help address the issue that many organizations don’t want more alerts, they want better alerts. In order to get those better alerts, two separate capabilities have come together in an interesting way:

1. Threat Intelligence: Since SIEM rules were based on looking for what you’ve seen before, you were limited in what you could look for – unless you’ve seen it. What if you could leverage attacks other companies have seen and then look for those attacks, so you could basically know what’s coming? That’s the underlying concept for external threat intelligence.

2. Security Analytics: The other capability isn’t exactly new, it’s using advanced math to look at the security data you’ve already collected to profile normal behaviors, and then look for stuff that isn’t normal and may be malicious in nature. Call it anomaly detection, machine learning, or whatever – the concept is the same. Gather a bunch of security data, build mathematical profiles of activity patterns, then look for activity that isn’t normal.

Let’s go into both of these capabilities to gain a better understanding how they work and then we’ll be able to show how powerful integrating the two can be to generating those better alerts.

Threat Intel Identifies What Could Happen

Culturally for most of the past 20 years, security folks were the kids that didn’t play well in the sandbox. No one wanted to appear vulnerable, so data breaches and successful attacks were the dirty little secret of security. Sure it happened, but not to us. Yeah right. Still there would be very high profile issues (like SQL*Slammer) that couldn’t be swept under the rug, but they hit everyone so it wasn’t that big of a deal.

Yet over the past 5 years a shift has occurred within security circles, and it was borne out of necessity as most things are. Security practitioners realized that no one was perfect and that we collectively could improve our ability to defend ourselves if we would share information about adversary tactics and the specific indicators being used in those attacks. This is something that we dubbed “benefiting from the misfortune of others” a few years ago. Everyone benefits because one of us was attacked and we all learn about that attack and can look for it. Thus the modern threat intelligence market emerged.

In terms of the current state of threat intel, you typically see the following types of data shared within commercial services, industry groups/ISACs, and open source communities:

• Bad IP Addresses: IP addresses that behave badly, for instance participating in a botnet or acting as a spam relay, should probably be blocked at your egress filter, since you know that no good will come from communicating with that network. You can buy a black list of bad IP addresses, and this is probably the lowest hanging fruit in the threat intel world.
• Malware Indicators: Basically next generation attack signatures can be gathered and shared to look for the activity that is representative of typical attacks. You know these indicate an attack, so being able to look for them within your security monitors help keep your defenses current.

The key value of threat intel is to accelerate the human as we described in our Introduction to Threat Operations research. But what does that even mean? To illustrate a bit let’s consider retrospective search. Basically this involves being notified of a new attack via a threat intel feed and using those indicators to mine your existing security data to see if you’ve seen this attack before you knew to look for it – ergo retrospective search. Of course, it would be better to detect the attack when it happens, but the ability to go back and search for new indicators in old security data clearly shortens the detection window.

Another use of threat intel is to refine your hunting process. This involves having the hunter learn about a specific adversary’s tactics and then undertake a hunt for that adversary. It’s not like the adversary is going to send out a memo detailing its primary TTPs, so threat intel is the way to figure out what they are likely to do. This makes the hunter much more efficient (and yes, accelerates the human) by focusing on the typical tactics used by likely adversaries.

Much of the threat intel available today is focused on data meant to be pumped into traditional controls, like SIEMs and egress filters. There is an emerging need for intel on the new areas of exposure like cloud, IoT, and mobility. As more attacks leverage these new attack vectors, more data becomes available, making us all smarter. But in the meantime, there is a clear gap in the data to address these emerging technologies.

Yet, effectively leveraging threat intel does not realize the full potential of Security Decision Support. Knowing what could happen is very helpful. But you still end up with a long list of stuff to triage and potentially remediate and little real context to prioritize those efforts. That’s where analytics comes in.

Analytics Identifies What is Happening

The SIEM has historically been algorithmically challenged because its correlation engine was designed to alert on activity you knew to look for. It doesn’t help much with stuff you haven’t seen. Thus, as happens in markets like security, new approaches emerged to fill the gaps of the incumbent technologies. Security analytics is a case in point.

In concept, security analytics is pretty simple. Use advanced math to establish a baseline of activity in the morass of collected security data. Then look for situation that could indicate malicious activity or misuse of critical systems/data. In reality, it’s anything but simple.

The general technology underlying security analytics tools is anomaly detection. You remember that, right? The security analytics vendors can come up with fancy terms, but at its core, this isn’t new. In fact, we’ve been looking for anomalies in our security data for over a decade. Remember network-based anomaly detection (NBAD)? We certainly do (being security historians and all) and that was the first “security analytics” offering we can remember.

To be clear, NBAD worked and still does. The technology has been wrapped up into a variety of different offerings, so there aren’t really stand-alone NBAD companies anymore, but the approach has evolved and morphed into what we now call security analytics. So what’s different now? Firstly, you can analyze a lot more data, a lot more efficiently. As opposed to just looking at network flow records (like in the NBAD days), you can look at detailed network packet data, endpoint telemetry, logging activity from pretty much all of the devices and applications in use, and even potentially transaction data and then build the baseline to learn what is normal activity in your environment.

Most enterprise networks are pretty complicated (and getting more so), so building a baseline across a number of seemingly unrelated security data sources is challenging. So to simplify things a bit, you can start by looking at different use cases to chunk up the universe of activity into a manageable set that give you a reasonable place to start.

For example, you may want to find compromised devices. One way to do that is to look for devices that are doing strange things that could indicate misuse. Typically someone in your finance group shouldn’t be doing recon on devices in the engineering area. Or vice-versa. That’s not normal, and as such should probably be investigated. So paying attention to device behavior to detect malware is certainly a common use for security analytics.

Along with device behavior, you could also expand the purview of the analysis to look at broader insider threats by building a baseline for how specific employees use their devices (called user behavior analytics - UBA). Then when the employee does something funky, like connecting to the finance system from a tablet at home, you can flag that as something to look at.

You could extend that use case to a broader analysis by adding data from physical access systems (to see when the employee is in the office), as well as HR records (an employee is under investigation and has become a flight risk). Or you could look at the usage by employees of specific applications, especially the ones accessing critical (and proprietary) corporate data. From all this data you can profiling the employee’s usage and transaction patterns. That gives you the baseline to look for activity that should be investigated.

Thus security analytics provides you not with a list of things you should look for (like threat intel or threat modeling), rather a list of things that are happening in your environment, providing more actionable alerts that most likely warrant investigation.

Yet security analytics on its own also doesn’t rise to the standard of Security Decision Support. The analytics can identify things you need to look for, but you still don’t have a sense of importance or prioritization relative to the other things the analytics platform is alerting on. So now we need to ask a few more questions of the system to really understand how to make better decisions.

Driving Security Decisions

Now that you have an understanding of which attacks are being used in the wild (via threat intel) and which systems/users/applications are potentially being misused, the next step is to use that context to drive action. But what action? How do you prioritize amongst all of the things (both internal and external) that should be looked at? Basically you need to balance the following to be able to effectively decide on what actions will be most impactful on your environment:

• Asset/Data Value: There are corporate systems and data that are pretty important to your company. When this kind of stuff is compromised, heads roll – probably yours. So obviously you are going to prioritize potential situations with these systems or users above most things. This is a subjective measure of value and requires a bunch of discussion with senior management to understand the value of the systems. But if you want to stay employed, you have to factor asset value into the mix since you don’t have the resources or time to get everything done.
• Confidence: False positives hurt you by wasting time on stuff that turns out to be nothing. Reducing this wasted time is done by tracking the confidence you have in your data/intel sources and analytical techniques. Obviously if a specific intel source sends you crap, then you should maybe not jump when it alerts. Likewise if a bunch of alerts based on a user’s mobile activity turn out to be much ado about nothing, then you should minimize that source over time. Yet, if you do find something via retrospective search that indicates an attack, then that should be pretty high priority since you know the attack is legit and you know it’s happening in your environment.
• Internal Skills/Resources: The industry is making progress in automating some security activities, but there still seems to be an infinite amount of work to do. You also need to balance the reality of the skills at your disposal to prioritize. If you are weak at Tier 1 of your response because the front line staffers keep getting poached by consulting firms, then you may want to send alerts that may be urgent directly to Tier 2. Similarly, if you know the Tier 3 folks get into deep water on file-less attacks, then you may want to just quarantine those devices until your external forensics team can take a look.

This kind of process requires the ability to measure effectiveness of both threat intel and analytics over time. A gut feel isn’t really what you should be using to determine which sources work best. So the sooner you start working on quantifying value, the better. Not just from being able to prioritize better, but also to save money. If you are spending money on sources or analytics platforms that don’t provide value, maybe you should stop doing that.

By leveraging threat intel and more advanced security analytics, we’ve narrowed the aperture of all of the things you could look at, and helped you identify what you need to look at. We aren’t going to claim that this makes your to-do list manageable, but every bit helps. By focusing on likely attacks (based on threat intel) on those devices that are acting abnormally while considering the value of the asset being attacked and the confidence you have in the data, it’s far more likely you’ll focus on the attacks that matter.

This is what Security Decision Support is all about. Enabling you to make the best use of your available resources by working smarter, leveraging technology and external resources to improve the effectiveness of your security team.

As we wrap up the series, our next piece will focus on how better, more contextual alerts can favorably impact security operations and what integrations are required to make that vision a reality.

- Mike Rothman
(0) Comments
Subscribe to our daily email digest

The post Evolving to Security Decision Support: Data to Intelligence appeared first on Security Boulevard.

from Evolving to Security Decision Support: Data to Intelligence

Best Practices, Unintended Consequences, Negative Outcomes

Posted under: Research and Analysis

Information Security is a profession. We have job titles, recognized positions in nearly every workplace, professional organizations, training, and even some early degree programs. I mean none of that sarcastically, but I wouldn’t necessarily say we are a mature profession. We still have a lot to learn about ourselves. This isn’t unique to infosec, it’s part of any maturing profession, and we can learn their lessons.

As I go through the paramedic re-entry process I realized, much to my surprise, that I have been a current or expired paramedic for over half the lifetime of the profession. Although I kept my EMT up, I haven’t really stayed up to date with paramedic practices (the EMT level is basically advanced first aid; paramedics get to use drugs, electricity, and all sorts of interesting… tubes). Paramedics first appeared in the 1970’s and when I started in the early 1990’s we were just starting to rally behind national standards and introduce real science of the prehospital environment into protocols and standards. Now the training has increased from about 1000 hours in my day to 1500-1800 hours, in many cases with much higher pre-training requirements (typically college level anatomy and physiology). Catching back up and seeing the advances in care is providing the kind of perspective that an overly-analytical type like myself is inexorably drawn towards, and provides powerful parallels to the less-mature information security profession.

One great example of a deeper understanding of the consequences of the science is how we treat head injuries. I don’t mean the incredible, and tragic, lessons we are learning about Traumatic Brain Injury (TBI) from the military and NFL, but something simpler, cleaner, and more face-palmy.

Back in my active days we used to hyperventilate head injuries with increased intercranial pressure (ICP, because every profession has to have their TLAs). In layman terms… hit head, go boom, brain swells like anything else you smash into a hard object (in this case, the inside of your own skull), but in this case it is swelling inside a closed container with a single exit (which involves squeezing the brain through the base of your skull and pushing the brain stem out of the way, oops). We would intubate the patients and bag them at an increased rate with 100% oxygen for two reasons — to increase the oxygen in their blood and try and get more O2 to the brain cells, and because hyperventilation reduced brain swelling. Doctors could literally see a brain in surgery shrink when they hyperventilated their patients. More O2? Less swelling? Cool!

But outcomes didn’t seem to match the in-your-face visual feedback of a shrinking brain. Why? It turns out that the brain shrinks because when you hyperventilate a patient you reduce the amount of CO2 in their blood. This changes the pH balance, and also triggers something called vasoconstriction. The brain sank because the blood vessels feeding the brain were providing less blood to the brain.

Well, darn. THAT probably isn’t good.

I treated a lot of head injuries in my day, especially as one of the only mountain rescue paramedics in the country. I likely caused active harm to these patients, even though I was following the best practices and standards of the time. They don’t haunt me, I did my job as best I could with what we knew at the time, but I certainly am glad to provide better care today.

Let’s turn back to information security and focus on passwords. Without going into the history our password standards, in most cases, no longer match the risk profile. In fact, we see these often causing active harm.

Requiring someone to come up with a password with a bunch of strange characters and rotate it every 90 days no longer improves security. Blocking password managers from filling in password fields? Beyond inane.

We originally came up with our password rules due to peculiarities with hashing algorithms and password storage in Windows. Length is a pretty good one to put into place, and advising people not to use things that are easy to guess. But we threw in strange characters due to rainbow tables and hash matching. Forced password rotations due to letting people steal or databases and having time to brute force things.

However, if we use modern password hashing algorithms and good seeds we dramatically reduce the chances of brute force attacks even if someone steals the password. The 90 day and strange character requirements really aren’t overly helpful. They are, in fact, more likely harmful since users will forget their passwords and rely on weaker password reset mechanisms. Think the name of your first elementary school is hard to find? Let’s just say it ain’t as hard to spot as a unicorn.

Blocking password managers from filling fields? In a day and age when they are included in most browsers and operating systems? If you hate your users that much just dox them yourselves and get over it.

The parallel to treatment protocols for head injuries is pretty damn direct here. We made decisions with the best evidence at the time, but the times changed. Now the onus on us is to update our standards to reflect the current science.

Block the 1234 passwords and require a decent minimum length, but let the users pick what they want and focus more on your internal security and storage, seeds, and hashing. Support an MFA option appropriate to the kind of data you are working with, and build in a hard to fake password reset/recovery option. Actually, that last area is one that’s ripe for more research and better options.

We shouldn’t codify negative outcomes into our standards of practice. And when we do, we should recognize and change. That’s the mark of a true, continuously evolving profession.

- Rich
(0) Comments
Subscribe to our daily email digest

The post Best Practices, Unintended Consequences, Negative Outcomes appeared first on Security Boulevard.

from Best Practices, Unintended Consequences, Negative Outcomes

Evolving to Security Decision Support: Visibility is Job 1

Posted under: Research and Analysis

To be masters of the obvious, it’s not getting any easier to detect attacks. Not that it was ever really easy, but at least you knew what tactics the adversaries would use and you’d have a general idea of where they would end up because you knew where your important data was and largely had a single type of device that accessed it – the PC. Hard to believe we’re longing for the days of early PCs and centralized data repositories.

That is not today’s world. You face professional adversaries (and possibly nation-states) that use agile methods to develop and test attacks. They have means of obfuscating who they are and what they are trying to do to further complicate detection. They prey upon perpetually gullible employees who click on anything to gain a foothold in your environment. Further complicating matters is the inexorable march towards cloud services, which moves both unstructured content to cloud storage, outsources back office functions to a variety of service providers, and moves significant portions of your technology environment to the public cloud. And these movements are accelerating, seemingly exponentially.

There has always been a playbook to deal with attackers when we knew what they were trying to do. Whether you effectively executed on that playbook notwithstanding, the fundamentals were understood. As we mentioned in the Future of Security series, the old ways don’t work anymore and that puts practitioners behind the 8 ball. The playbook has changed and the old security architectures are rapidly becoming obsolete. For instance, it’s increasingly difficult to insert inspection bottlenecks into your cloud environment without adversely impacting the efficiency of the technology stack. Moreover, sophisticated adversaries have the ability to use exploits that aren’t going to be caught by traditional assessment and detection technologies (even if they don’t have to use it frequently).

This means we need a better way to assess the security posture of your organization, detect attacks, and determine applicable methods to work around and eventually remediate exposures in your environment. As much as the industry whinges about adversary innovation, the security industry has made progress in improving your ability to assess and detect these attacks. We’ve written a lot about threat intelligence and security analytics over the past few years. Those are the cornerstone technologies to deal with adversary’s improved capabilities.

But these technologies and capabilities cannot stand alone. Just pumping some threat intel into your SIEM is not going to help you understand the contextual relevance of the information. And doing advanced analytics on the scads of security data you collect is not enough either because you may be missing a totally new attack vector.

Ultimately what you need is a better way to assess your organizational security posture, determine when you are under attack, and figure out how to make the pain stop. This involves not just technology, but also process changes and a clear understanding of how your technology infrastructure is evolving towards the cloud. This is no longer just assessment or analytics, it’s something bigger. It’s what we are now calling Security Decision Support (SDS). Snazzy, huh?

In this blog series “Evolving to Security Decision Support”, we’ll delve into these concepts and show you how to gain both the visibility and context to understand what you have to do and why. Security Decision Support provides a means of prioritizing the thousands of things you can do allowing you to hone in on the few things that you must do.

As with all of Securosis’ research developed using our Totally Transparent methodology, we don’t mention specific vendors or products rather focusing on the solution architecture and decision points that will help you practically leverage our research. Yet, we still have to pay the bills, so we’ll take a moment to thank Tenable, who has agreed to license the content when it’s complete.

Visibility in the Olden Days

Securing pretty much anything starts with visibility. You can’t manage what you can’t see, and a zillion other overused adages all illustrate the same point. If you don’t know what’s on your network and where your critical data is, you don’t have much of a chance to protect it.

In the olden days, you know back in the early 2000s, visibility was pretty straight forward. First you had your data on mainframes in the data center. Even when you started using LANs to connect everything, the data was still over the raised floor or in a pretty simple email system. Early client/server systems started complicating things a bit, but everything was still on networks you controlled in data centers that you had the keys to. You could scan your address space and figure out where everything was and see what vulnerabilities needed to be dealt with.

That worked pretty well for a long time. There were issues of scale and the need/desire to scan higher into the technology stack, so you started seeing first stand-alone and then integrated application scanners. Once rogue devices started appearing on your network, it was no longer sufficient to just scan your address space every couple of weeks, so passive network monitoring allowed you to watch the traffic and flag (and assess) unknown devices.

Those were the good old days, when things were relatively simple. OK, maybe not simple, but you could size the problem. That’s no longer the case.

Visibility Challenged

We use a pretty funny meme in many of our presentations. It shows a man from the 1870s remembering blissfully the good old days when they knew where their data was. That image always gets a lot of laughs from the audience. But it’s laughter brought on by pain because everyone on the room knows it’s true. Nowadays you don’t really know where your data is, and that really complicates your capability to determine the security posture of the systems with access to it.

These challenges are a direct result of a number of key technology innovations:

• SaaS: Securosis talks about the fact that SaaS is the New Back Office, and that has pretty drastic ramifications on visibility. Many organizations have deployed a CASB just to figure out what SaaS services are in use because it’s not like your business folks come an ask permission to use a business-oriented service. This isn’t a problem that’s going away. If anything, more of your business processes will be moving to SaaS.
• IaaS: Speaking of cloudy stuff, you have teams that are using Infrastructure as a Service (IaaS), either moving existing systems out of your data centers or building new systems for the cloud. IaaS really messes with how you assess your environment. Scanning is a lot harder and some of the “servers” (now called instances) will only live for a few hours. The network addressing is different and you can’t really implement taps to see the traffic. It’s a different world for sure, and it’s one where you are pretty much blind.
• Containers: Another foundational technology to allow more portability and flexibility in how you build and deploy application components is containers. Without going into any detail about why containers are cool, suffice it to say that your developers are likely working with them as they architect new applications, especially in the cloud. But these containers provide some challenges in visibility and security because they are short-lived (they spin up when you need them), self-contained (usually not externally addressable) and don’t provide access for a traditional scan. Thus containers pretty much break your existing discovery and assessment processes.
• Mobility: It seems kind of old hat to even be mentioning the reality that you have critical data on smart devices (phones and tablets), but it expands your attack surface and makes it hard to really understand both where your data is and how those devices are configured.
• IoT: A little further into the horizon is the idea of the Internet of Things (IoT). Some would argue it’s here today and with the number of sensors being deployed and smart systems that are network connected, those folks may be right. Regardless, if you look even just a year or two into the future, you can bet there will be a lot more network connected devices accessing your data and expanding your attack surface. And that means that you’ll need to be able to find them and assess them.

And we are just getting started. It won’t be long before the next discontinuous innovation makes it harder to figure out where critical data resides and what’s happening with it. To put a bow on the discussion of the challenges you face, we’ll talk about some reasonable bets to make. We’re pretty confident that there will be more cloud in use tomorrow than there is today. We’re equally confident that there will be more devices accessing your stuff tomorrow than today. And that’s pretty much all you need to know to understand the extent and magnitude of the problem.

Challenge Accepted

To again be masters of the obvious, it’s hard to be a security professional nowadays. We get it. Yet crawling up into the fetal position on your data center floor isn’t really an option. First of all, you probably don’t even have a data center anymore. And if you do, it’s either being repurposed as warehouse space or has been sold off to a cloud provider. Secondly, that won’t really solve any problems.

So what to do? Remember that you can’t manage or protect what you can’t see, so first we need to focus on visibility as the first step on the path to Security Decision Support. Visibility across the enterprise. Wherever your data resides. On whatever platform. That means discovery and assessment of all your stuff.

We’re pretty sure you haven’t been able to totally shut off your data centers and move everything to SaaS and IaaS (even though you may want to), so that means you’ll need to make sure you aren’t missing anything within your traditional infrastructure. Thus, you’ll need to continue your existing vulnerability management program.

• Network, security, databases and systems: You already scan your network and security devices, all of the servers you control and probably your databases as well (thanks compliance mandates!), so you’ll keep doing this. Hopefully you’ve been evolving your vulnerability management environment and have some means of prioritizing all of the stuff in your environment.
• Applications: You are also likely scanning your web applications as well. That’s a good thing. Keep doing that. And working with the developers to ensure they are fixing the issues you find before something is deployed to millions of customers. Obviously as the developers continue to adapt agile methods of building software, you’ll still need to evangelize the need to identify the issues with the application stacks and given the velocity of software changes, fix those issues faster.

That’s the stuff that you should already be doing. Maybe not as well as you should (there is always room for improvement, right?), but at least for compliance purposes you are already doing something. Where it gets interesting is how discovery and assessment works for a lot of these new environments and innovations that you need to grapple with. Let’s look at the innovations we described above and get a sense for how things change in this new world.

SaaS

As we mentioned, many of you have deployed a CASB (cloud access security broker) to look at your egress network traffic and figure out what SaaS services are actually in use. It’s always entertaining to hear the anecdotes of how some vendors will ask a customer how many SaaS services they think are being used, and they say maybe a couple dozen. And then the vendor (with great dramatic effect) drops the report on the deck and it’s closer to 1500.

To be clear, you don’t have to use a purpose-built device or service to figure out in-use SaaS as many of the secure web gateways offer this kind of visibility, as well as DLP solutions (focused on controlling exfiltration). So one method of discovery is looking at egress traffic.

Another means of discovery and assessment is through the SaaS provider’s API (application programming interface). The more mature SaaS companies understand that visibility is a problem, so they offer reasonably granularity about usage and activity via their API. You can pull down this information and integrate it with your other security data for analysis. We’ll dig into the analysis aspect of Security Decision Support in the next post.

IaaS

As your organization moves existing systems and builds new applications for the cloud, you’ll need to take a more proactive measure to get a sense of what resources actually live in the cloud. Unlike SaaS, where someone is presumably connecting to a service from inside your organization (and you can presumably see that), an egress filter isn’t going to provide much detail about what lies within the public cloud service.

In this case, the API really is your friend. Any tools that focus on visibility will need to poll the cloud provider’s API to learn what systems are running in an environment and then to assess them. One mention of caution are the API limitations of some cloud providers. You cannot make infinite API calls to any cloud provider (for obvious reasons), so you’ll need to build your IaaS environment with this in mind.

We favor cloud architectures that use multiple accounts per application for lots of reasons. Overcoming API limitations is one, as well as minimizing blast radius of an attack using better functional isolation between applications. Yet, that’s a much larger discussion for a different day. But you can check out our recent video about this very concept, if you are so inclined.

Befriend the Accountants

A point we’ll make about cloud services is general should be familiar from many contexts. Follow the money. For both SaaS and IaaS, the only thing we can be sure of is that someone is getting paid for any services that you use. And that means that whoever pays the bill should be able to let you know what services are in use and for what.

So another recommendation we have is to make sure that you are friendly with the accounting team. Take them out to lunch from time to time. Support their charitable causes. Whatever it takes to keep them on your side and responsive to your requests for accounting records for cloud services.

To be clear, a report from accounting is not a replacement for pulling information from APIs or monitoring your egress traffic. Attackers move fast and can do a lot of damage in the time it takes for the provider to bill you and then for accounting to receive the bill and process it. So you are likely 4 - 6 weeks behind what’s really happening. So you’ll want to use this kind of information to verify what you should already know. And to identify the stuff that you should know about, but maybe don’t.

Containers

Given that containers encapsulate micro-services which are not usually persistent and cannot really be accessed or scanned from external entities (like a vuln scanner), the model of a separate capability to discover and assess these containers doesn’t really work. Thus, you’ll have to build discovery and assessment into the container system. First, you’ll want to make sure that any of the containers you build are not vulnerable, so you’ll integrate assessment into the container build process. Thus, any container spun up will be built using an image that is not vulnerable.

Then you’ll also want to track the usage of the containers and make sure that nothing drifts, which means inserting some kind of technology (an agent or API call) into the build/deploy as containers spin up. That technology will track the container through its lifecycle (and report back to a central repository) and watch for signs of an attack on the component. Let’s reiterate the fact that this isn’t something you can bolt on after the fact (like most of security). So right when you are done buying pizza for the accounting team, you may want to have a happy hour with the developers. Without their participation, you’ll have precious little visibility into your container environment.

Mobility

It’s been a while since you could stick your head in the sand and hope that mobile devices were a passing fad. Nowadays they are full participants in your IT environment and there are new, innovative applications being rolled out to gain business advantage from the flexibility of these devices. But with ubiquity of a class of technology, lots of solutions emerge to address the common problems.

In terms of visibility and assessment of mobile devices, there are dozens of solutions (even after consolidation). In order to access corporate data or install purpose-built mobile apps, the device will need to be registered with the corporations mobile device management (MDM) environment. These platforms can provide an inventory of not just the device, but what is installed on the device. More sophisticated offerings now have the capability to block certain apps from running or stop the device from accessing some networks, based on the configuration and assessment.

So that’s the good news. Where there is still work to do is in integrating that information into the rest of the Security Decision Support stack. You’ll want to be able to pull the telemetry from the MDM environment and use that as part of your security analytics strategy. For example, figuring out that a certain person’s mobile device is being used to access cloud data stores they aren’t authorized to look at, while that same user’s computer is doing recon on the finance network could be an indication of a successful compromise. You’d like your analytics environment to connect the two data points, and highlights the importance of enterprise visibility. But let’s not get ahead of ourselves yet, we’ll get into analytics in the next post.

IoT

The thing about many IoT devices is that they aren’t your standard, run of the mill, PC or mobile device. They likely don’t have an API that you can poll to figure out what’s going on, nor does it allow you to install an agent to monitor activity. Additionally, these devices can appear on many network segments that may not be as highly monitored or protected, such as the shop floor or security video network.

Figuring out the presence of these devices, assessing security, and then looking for potential misuse requires a different approach, one that is largely passive in nature. So your best bet will be to monitor those networks, profile the devices on each network, baseline their typical traffic patterns and then look for situations where the devices aren’t acting normally. Yet, it is a little more challenging than collecting a bunch of NetFlow records on a shop floor network. These IoT devices may use proprietary, non-standard protocols further complicating the discovery and assessment process. As you factor these types of devices into your Security Decision Support strategy, you’ll need to weigh the complexity of identifying and assessing these devices against the risk of attack.

(Re)Visitation

Of course, we’ll need to put a few caveats around these concepts. First, emerging technologies are moving targets. Let’s just take IaaS as an example. The Cloud providers are rapidly introducing APIs and other mechanisms to provide a view into what’s running in their environments. You could make similar points for every class of technology. It seems most device makers (regardless of the type of device) realize that customers want to be able to manage their technology as part of a bigger system, so many (but not all, sigh) are providing better access to its innards in more flexible ways. That’s the kind of progress you like to see.

Yet, tomorrow’s promise doesn’t solve today’s problem. You have to build a process and implement tooling based on what’s available today. Thus, for many of these emerging environments, you’ll want to build a periodic revisitation of your strategy into your SDS process, similar to how you (should) revisit your malware detection approaches periodically.

Yes, revisiting your enterprise visibility approaches can be time consuming, and expensive if something needs to change. Reversing course on decisions you made over the past year can be frustrating. But that’s the world you live in, and resisting it will just make you cranky. Or more accurately, more cranky. If you expect to revisit all of these decisions and at times toss some tools and embrace others, it makes it much easier to handle. Even more importantly, managing expectations on the part of management that this could (or more likely would) be an eventuality, it will go a long ways to maintaining your current employment status.

To summarize, the first step toward Security Decision Support is enterprise visibility and understanding the exposure of those assets and data, wherever it is. Next we’ll dig into figuring out what’s really at risk by integrating the external view of the security world (threat intel) and doing more sophisticated analytics on both the internal security data you collect.

- Mike Rothman
(0) Comments
Subscribe to our daily email digest

The post Evolving to Security Decision Support: Visibility is Job 1 appeared first on Security Boulevard.

from Evolving to Security Decision Support: Visibility is Job 1

Firestarter: Architecting Your Cloud with Accounts

Posted under:

We are taking over our own Firestarter and kicking off a new series of discussions on cloud security… from soup to nuts (whatever that means). Each week for the next few months we will cover, in order, how to build out your cloud security program. We are taking our assessment framework and converting it into a series of discussions talking about what we find and how to avoid issues. This week we start with architecting your account structures, after a brief discussion of the impact of the Meltdown and Spectre vulnerabilities since they impact cloud (at least for now) more than your local computer.

Watch or listen:

---

- Rich
(0) Comments
Subscribe to our daily email digest

The post Firestarter: Architecting Your Cloud with Accounts appeared first on Security Boulevard.

from Firestarter: Architecting Your Cloud with Accounts

Wrangling Backoffice Security in the Cloud Age: Part 2

Posted under: Research and Analysis

This is the second part in a two part series/paper on managing your increased use and reliance on SaaS for traditionally back office applications. [Part 1 is here]. This will also be including in a webcast with Box on March 6 and you can [register here]

Where to start

Moving your back office applications to the cloud is the classic frog in a frying pan scenario. Sure, there are a few orgs out there that plan everything out ahead of time, but for most of the companies and agencies we work with it tends to be far less controlled. Multiple business units run into the cloud on their own, especially since all you need for SaaS is a web browser and a credit card, and the next thing you know your cloud footprint is WAY bigger than expected.

This is a challenge for security teams who are often tasked with fixing one cloud at a time as the requests come in, without having the time or support to take a step back and build out a program to support the transition. We don’t recommend putting the brakes on and pissing everyone off, but we do recommend the first step is to build a program instead of just blocking and tackling. Here’s how to pull that off when things are already in motion.

Build an “embrace and extend” program

The first step isn’t so much “do this” as “adopt this way of thinking”. It’s also probably the most important piece of advice we have for you.

There are two ways to approach the problem of enforcing your security needs onto an external platform. Wedge in a standard stack of security controls across the board, or evaluate the cloud provider, embrace their security capabilities, extend them where you can, and wedge in controls where you can’t.

The first option seems best on the surface since you gain the appearance of consistency, but the practical reality is the only way to pull that off is to break some of the cloud’s functionality, and it’s more an illusion anyway due to the large underlying technical difference between platforms.

We recommend a dual-path approach. Where possible build security controls and management you can extend to the cloud while embracing the cloud platform’s security capabilities, but also have a wedge stack (usually a CASB in Man in the Middle/Proxy mode) available for those providers that don’t offer effective security capabilities.

SaaS is the Wild West of the cloud with a mixture of some amazing, best of breed security combined with providers that will set your hair on fire with their negligence.

Start with updating your rise assessment process

The next step is to put some rigor into which cloud providers you support. The objective is to have a supported SaaS platform for each major back office application category, which minimizes the odds of employees trying to use unsanctioned and insecure providers. There is no need to rip apart your existing risk assessment process for new tools and technologies, but you do need to tune it with a few specifics to handle SaaS:

• Build a registry of sanctioned applications in major categories, such as file storage and collaboration, CRM, ERP, HR, communications, etc.
• Assessing applications can be tough, but usually involves:
• See if it supports our [recommended Critical Security Capabilities for Cloud Providers]. This list is a good starting point for the components you need to integrate the cloud provider into your security program.
• Know your compliance requirements and check the provider’s compliance certifications. You may end up only approving some providers for some kinds of data.
• Obtain the cloud provider’s security and compliance documentation. Many now post this information in the [Cloud Security Alliance’s STAR Registry] and use the standardized CSA Common Assessment Initiative Questionnaire (CAIQ) so you can compare apples to apples.
• Pull and review the provider’s security documentation and validate features and capabilities.
• If you use a CASB (Cloud Access and Security Broker) many of them include internal risk ratings you can use to help your selection process.
• Once you pick the provider then document which kinds of data it is approved for in the registry. Ideally you only want one major provider per application category and new requests can be steered in that direction. However, be open to diverging business unit needs that might require a second provider in the same category.
• Include fast and slow assessment paths, with the fast path for providers that won’t have any sensitive data (e.g. marketing related without PII). You don’t want to slow the business down if you can avoid it or you just might learn the limits of your control and popularity.

Build a federated identity management program

Few things push you towards full federated identity and single sign on than cloud. It’s pretty much the only way to operate.

Although you can handle things with direct federation to your directory servers, in our experience this is an area where a commercial tool is a big help. We recommend build around a federated identity broker and including three key pieces in your plan:

• For every cloud provider have a non-federated administrative account. That way when your federated identity broker has an issue you can still get into the cloud.
• Don’t hide your entire back office behind a single appliance. Use a high availability service (and push hard to get real uptime numbers) or multiple on-premise appliances. You do NOT want to be the one answering the help desk when the entire organization loses access to every back office application because your broker borked on an update.
• Require MFA for at least all administrative users, and ideally all users. When possible, further enforce MFA by requiring it as an attribute for authentication on the cloud platform (the cloud will look for an MFA attribute from the identity broker, this isn’t a separate MFA).

Create a SaaS security program

Notice that we only get into the security meat in our third step. That’s because your security program will be crippled without starting with a good process for selecting providers and solid identity management to handle your users.

While specific technologies and options change constantly and vary greatly across SaaS providers and security toolsets there are some consistencies we can build a program around:

• Use a CASB or something similar for your security management console for consistency, but not to the point where you fail to leverage inherent controls in your SaaS platform. CASBs can be great to provide multi-platform visibility and implement some controls, but this abstraction layer means you might miss some powerful capabilities in your platform itself. For your most critical cloud providers don’t rely just on the CASB, also make sure you understand the full native security capabilities. Fortunately, this is usually a short list (handful) of key providers, and the CASB will work for most of the rest.
• Always try to avoid having to proxy (Man in the Middle) your connections to the cloud provider just to integrate your CASB. If you follow our Critical Security Capabilities you will tend to choose providers the CASB can integrate with using APIs so it doesn’t weaken your encryption and trust model.
• Adjust your security requirements based on the type of data. Yes, these seems blindingly obvious but we have to mention it since we see a lot of programs that assume everything in the cloud should be treated the same.
• Have a documented incident response process for at least your major SaaS providers… and test it. The main scenarios are unapproved public data, account takeover, loss of your federation connection, and employee abuse. More than anything else, know who to call in an incident and make sure you know the provider’s process to authenticate that you are a valid customer contact for incidents.
• Build a security log management strategy for SaaS. You will need to account for a variety of sources, connection methods, and file formats.
• Once you gain visibility and monitoring, then determine what compromises a security incident and how you can detect them. This could be from your log feeds, external monitoring, the CASB, or some internal event trigger in the cloud provider.
• Go back and make sure you keep up to date on your providers and their security capabilities. Embrace new security features. You may not use them all, but you certainly need to know they are there and if they would bring you value.

Additional tips for success

Our recommendations are really only a starting point and don’t cover the vast majority of the technical specifics once you get into the weeds. This research focuses more on the high-level program components than the details, but here are some additional tidbits that may help as you move forward:

• Learn to love automation. Especially when you have the API support you need from the provider, just as with IaaS you should consider writing some of your own automation tools to validate controls, implement common processes, and speed up response.
• Audit the controls you set in the SaaS platform as well as in any management tool (CASB). Don’t wait until the auditor calls to find out someone disabled a key control (like making sensitive data public) and it didn’t trigger an alarm.
• Encryption is one of the most complex, messiest aspects of SaaS with wildly divergent security models and implementation specifics. Our general recommendation is to use providers with robust encryption support and avoid the proxy models.
• Keep track of your providers, at least the most important ones. They are constantly adding new capabilities and some of these may bring significant security benefits. Someone in security should be on the mailing list and it’s worth having at least an annual capabilities review.

Enterprise adoption of Software as a Service is nothing new, but there is a big difference between the ad-hoc adoption of a few services, to moving essentially every back office application into the cloud. Building out your security program to support this rapid evolution will reduce risk and support agility without killing the security team as they learn new skills and techniques.

- Rich
(0) Comments
Subscribe to our daily email digest

The post Wrangling Backoffice Security in the Cloud Age: Part 2 appeared first on Security Boulevard.

from Wrangling Backoffice Security in the Cloud Age: Part 2

Wrangling Backoffice Security in the Age of Cloud

Posted under: Research and Analysis

Over a year ago we first published our series on Tidal Forces: The Trends Tearing Apart Security As We Know It where we identified three key mega-trends in technology with deep, lasting impact on the practice of security:

• Endpoints are different, often more secure, and frequently less open. If we look at the hardening of operating systems, especially exemplified by the less-open-but-more-secure model of Apple’s iOS, the cost of exploiting endpoints is trending towards being much higher. At least it was before Meltdown and Spectre, but fortunately those are (big) blips, not a permanent destination.
• Software as a Service (SaaS) is the new back office. Organizations continue to push more and more of their supporting applications into SaaS, especially things like document management, CRM, and ERP that aren’t core to their mission.
• Infrastructure as a Service (IaaS) is the new data center. The growth of public IaaS has exceeded even our aggressive expectations. It’s the home for most new applications being developed, and a large number of organizations are shifting existing application stacks to IaaS even when it doesn’t necessarily make sense.

The fundamental precept of the “Tidal Forces” concept is that these trends act like gravity wells. We are all pulled inexorably towards them, at a rate that increases the closer we get, until we are internally ripped apart as some parts of the organization move more quickly, others are left behind, and teams like infrastructure and security are forced to support both ends of the spectrum.

Since publication nothing has dissuaded us from believing these trends will only continue to accelerate and increase internal pressures.

There are practical security implications of this migration of the back office into an ever-growing menagerie of remote services. It’s more than losing physical control; different services have different capabilities and all demand new security management models, tools, and techniques. The more you try and force the lessons of the past into the future, the more painful the transition. It isn’t that we throw all our knowledge and skills away, it’s that we need to translate them to apply security to the new environments.

This short paper will highlight some of the top ways security operations are affected, then highlight recommendations to manage the problem over time.

How the SaaS transition impacts security

Moving your most sensitive data to an outside provider quickly shatters the illusion that physical control matters anymore. But such a shift also doesn’t abrogate you of overall security accountability. Depending on how you manage the transition you create both advantages and challenges.

The biggest challenge with Software as a Service is the sheer range of capabilities across even similar-looking providers. There are some top-notch SaaS providers that recognize major security incidents are potentially existential events for their business, and thus invest heavily in both security capabilities and features. Other companies are fast-moving startups that care more about customer acquisition than customer safety (they’ll learn, painfully). Aside from inherent security, these are effectively remote applications and each has their own internal security models and capabilities that need to be managed. Risk assessment and platform knowledge become high priorities for security teams managing SaaS.

It also doesn’t help that, by nature, these platforms are all Internet accessible. Meaning so is your data, potentially, if you fail to configure them properly. Nearly all of the services default to secure options, but the news is filled with examples of… exceptions.

Existing tools and techniques rarely apply directly to cloud. You don’t manage a firewall, you need to use federation for identity management, and pretty much every traditional monitoring tool breaks. Take, for example, log management for monitoring and incident response. You generally only have access to the logs the cloud platform provides, if they offer them at all, and they are most likely in a custom format and are only accessible as API calls, within the cloud provider’s user interface, or as data dumps.

Planning on just sniffing the traffic? Aside from that providing you nearly no context, ongoing adoption of TLS 1.3 forces you to drop to less secure encryption options (if they are even supported) to capture the traffic. Or you engage in a man-in-the-middle attack against your own users and reduce security to improve monitoring.

Lastly, and for some of you most importantly, is compliance. You are fully reliant on the SaaS provider’s compliance, and then need to ensure you configure and use everything correctly. With IaaS we can sometimes get around some of these restrictions, but with SaaS that usually isn’t an option. When a provider has baseline compliance with a regulation or standard we call it compliance inheritance, but that only means they provide a compliant baseline, and if you decide to make all your PII records publicly shareable… good luck with the auditors.

Every new technology comes with tradeoffs. And, in the end, our job as security is to decide if we have a net gain or loss in risk, and how to best mitigate that risk to the level our organization desires. Cloud comes with tremendous potential security benefits; outsourcing our applications and data to a set of providers with far more incentive to keep it secure than we have, but this is only true if we select the right provider, and use the right configure, and the right security processes and tools to manage it.

- Rich
(0) Comments
Subscribe to our daily email digest

The post Wrangling Backoffice Security in the Age of Cloud appeared first on Security Boulevard.

from Wrangling Backoffice Security in the Age of Cloud

Container Security 2018: Logging and Monitoring

Posted under: Research and Analysis

We close out this research paper with two key areas: Monitoring and Auditing. We want to draw attention to them because they are essential to security programs, but have received only sporadic coverage in security blogs and the press. When we go beyond network segregation and network policies for what we allow, the ability to detect misuse is extremely valuable, which is where monitoring and logging come in. Additionally, most Development and Security teams are not aware of the variety of monitoring options available, and we have seen a variety of misconceptions and outright fear of the volume of audit logs to capture, so we need to address these issues.

Monitoring

Every security control discussed so far can be classed as preventative security. These efforts remove vulnerabilities or make them hard to exploit. We address known attack vectors with well-understood responses such as patching, secure configuration, and encryption. But vulnerability scans can only take you so far. What about issues you are not expecting? What if a new attack variant gets by your security controls, or a trusted employee makes a mistake? This is where monitoring comes in: it is how you discover unexpected problems. Monitoring is critical to any security program – it’s how you learn what works, track what’s really happening in your environment, and detect what’s broken.

Monitoring is just as important for container security, but container providers don’t offer it today.

Monitoring tools work by first collecting events, then comparing them to security policies. Events include requests for hardware resources, IP-based communication, API requests to other services, and sharing information with other containers. Policy types vary widely. Deterministic policies address areas such as which users and groups can terminate resources, which containers are disallowed from making external HTTP requests, and which services a container is allowed to run. Dynamic (also called ‘behavioral’) policies address issues such as containers connecting to undocumented ports, using more memory than normal, or exceeding runtime thresholds. Combining deterministic white and black lists with dynamic behavior detection offers the best of both worlds, enabling you to detect both simple policy violations and unexpected variations from the ordinary.

We strongly recommend you include monitoring container activity in your security program. A couple container security vendors offer monitoring tools. Popular evaluation criteria include:

• Deployment Model: How does the product collect events? What events and API calls can it collect for inspection? Typically these products use one of two models for deployment: either an agent embedded in the host OS, or a fully privileged container-based monitor running in the Docker environment. How difficult are collectors to deploy? Do host-based agents require a host reboot to deploy or update? You need to assess what types of events can be captured.
• Policy Management: You need to evaluate how easy it is to build new policies or modify existing ones. You will want a standard set of security policies from the vendor to speed deployment, but you will also stand up and manage your own policies, so ease of management is key to long-term happiness.
• Behavioral Analysis: What, if any, behavioral analysis capabilities are available? How flexible are they – what types of data are available for use in policy decisions? Behavioral analysis starts with system monitoring to determine ‘normal’ behavior. The pre-built criteria for detecting aberrations are often limited to a few sets of indicators, such as user ID or IP address, but more advanced tools offer a dozen or more choices. The more you have available – such as system calls, network ports, resource usage, image ID, and inbound and outbound connectivity – the more flexible your controls can be.
• Activity Blocking: Does the vendor offer blocking of requests or activity? Blocking policy violations helps ensure containers behave as intended. Care is required because such policies can disrupt new functionality, causing friction between Development and Security, but blocking is invaluable for maintaining Security’s control over what containers can do.
• Platform Support: You need to verify your monitoring tool supports your OS platforms (CentOS, CoreOS, SUSE, Red Hat, Windows, etc.) and orchestration tool (Swarm, Kubernetes, Mesos, or ECS).

Audit and Compliance

What happened with the last build? Did we remove sshd from that container? Did we add the new security tests to Jenkins? Is the latest build in the repository? You may not know the answers off the top of your head, but you know where to get them: log files. Git, Jenkins, JFrog, Docker, and just about every development tool creates log files, which we use to figure out what happened – and all too often, what went wrong. There are people outside Development – namely Security and Compliance – with similar security-related questions about what is going on in the container environment, and whether security controls are functioning. Logs are how you get answers for these teams.

Most of the earlier sections in this paper, covering areas such as build environments and runtime security, carry compliance requirements. These may be externally mandated like PCI-DSS or GLBA, or internal requirements from internal audit or security teams. Either way, auditors will want to see that security controls are in place and working. And no, they won’t just take your word for it – they will want audit reports for specific event types relevant to their audit. Similarly, if your company has a Security Operations Center, they will want all system and activity logs some time period to reconstruct events, and, investigate alerts, and/or determine whether a breach occurred. You really don’t want to get too deep into that stuff – just get them the data and let them worry about the details.

CIS offers benchmarks and security checklists for container security, orchestration manager security, and most compliance initiatives. These are a good starting point for conducting basic security and compliance assessments of your container environment. In addition ‘vendors’ – both open source teams and cloud service providers – offer security deployment and architecture recommendations to help produce dependable environments. Finally, we see configuration checkers arriving in the market – both open source and commercial tools – a quick search is likely to offer a couple options relevant to your environment, whatever it may be.

Container security programs are quite complex, due to the large number of areas which require attention. The good news is that most of what you need is already in place. During our investigation for this series we did not speak with any firms which did not already have Splunk, log storage, or SIEM on-premise, and in many cases all three were available. Additionally the vast majority of code repositories, build controllers, and container management systems – specifically the Docker runtime and Docker Trusted Registry – produce event logs in formats which can be consumed by various log management and SIEM systems without modification. As do most third-party image validation and monitoring security tools. You will need to determine how easy this is to leverage. Some simply dump syslog formatted information into a directory, at which point it’s up to you to drop this into Splunk, an S3 bucket, Loggly, or your SIEM tool. In other cases – most, actually – you can specify CEF, JSON, or some other format, and the tools can automatically link to your SIEM of choice, sending events as they occur.

- Adrian Lane
(0) Comments
Subscribe to our daily email digest

The post Container Security 2018: Logging and Monitoring appeared first on Security Boulevard.

from Container Security 2018: Logging and Monitoring

Container Security 2018: Runtime Security Controls

Posted under: Research and Analysis

Unlike the tools and processes discussed in previous sections, here we focus on containers in production systems. This includes which images are moved into production repositories, selecting and running containers, and the security of underlying host systems.

Runtime Security

• The Control Plane: Our first order of business is ensuring the security of the control plane — the platforms for managing host operating systems, the scheduler, the container client, engine(s), the repository, and any additional deployment tools. Again, as we advised for container build environment security, we recommend limiting access to specific administrative accounts: one with responsibility for operating and orchestrating containers and another for system administration (including patching and configuration management). We recommend network and physical segregation (on-premise) or logical segregation (for cloud and virtual systems). The good news is there are several third-party tools which offer full identity and access management, LDAP/AD integration, and token-based SSO (i.e.: SAML) across systems.
• Resource Usage Analysis: Many readers are familiar with this concept from a performance standpoint, but it offers insigt into basic code security as well. Does the container allow Port 22 (i.e.: Admin) access? Does the container try to update itself? What external systems and utilities does it depend upon? All external resource usage is a potential attack point for attackers so it’s good hygiene to limit these ingress and egress points. To manage the scope of what containers can access, third-party tools can monitor runtime access to environment resources both inside and outside the container. Usage analysis is basically automated review of resource requirements. This is useful in a number of ways — especially for firms moving from a monolithic architecture to microservices. They help developers understand what references they can remove from their code, and help Operations narrow down roles and access privileges.
• Selecting the Right Image: We recommend establishing a trusted image repository and ensuring that your production environment can only pull containers from that trusted source. Ad hoc container management is a good way to facilitate engineers bypassing security controls, so we recommend establishing a central, trusted repositories where production images are stored. We also recommend scripting the process to avoid manual
  intervention and ensure that the latest certified container is always selected. This means checking application signatures in your scripts before to putting containers into production, elimiating any manual overhead for verification. Trusted repository and registry services can help by rejecting containers which are not properly signed. Fortunately many options are available, so find one you like. Keep in mind that if you build many containers each day, a manual process will quickly break down. It is okay to have more than one image repository — if you are running across multiple cloud environments, there are advantages to leveraging the native registry in each.
• Immutable Images: Developers often leave shell access to container images, so once in production they can log into the container. The motivation is often for debugging and changing code on the fly, which is a bad thing for consistency and for security. Immutable containers – not allowing SSH connections – prevents changes in runtime. If forces developers to fix code in the development pipeline, and it takes away one of the priciple attack paths attackers use. Attackers looking to take over a container and use it to attack the undelying host or other containers scan for this. We strongly suggest use of immutable containers that do not offer this sort of ‘port 22’ access, and making sure containers are changed in the build process, not in production.
• Input Validation: At startup containers accept parameters, configuration files, credentials, JSON, and scripts. In some more aggressive scenarios, ‘agile’ teams shove new code segments into a container as input variables, making existing containers behave in fun new ways. Validate that input data is suitable and satisfy policy, either manually or using a third-party security tool. You must ensure that each container receives the correct user and group IDs to map to the assigned view at the host layer. Taken together, this prevents someone from forcing a container to misbehave, or simply prevent developers from making dumb mistakes.
• Container Group Segmentation: One of the principle benefits of conatiner management systems is to help scale tasks across pools of shared servers. Each manager platform offers a modular architecture, with scaling performed on node/minion/slave sub-groups, which in turn host a set of containers. Each node forms it’s own logical subnet, limiting network addressibility between sets of containers. This segregation provides a form of ‘blast radius’ to limit what resources a container can communicate with. It is up to the application architects and security teams to leverage this construct to improve security. You can enforce this with network policies of the conatiner manager service, or with network security controls provided by your cloud vendor. Over and above this orchestration manager feature, third party container security tools – running as an agent in the container or as part of the underlying operation system – can provide a type of logical network segmentation further limiting network connections between grouos of containers. Taken together this offers fine grained isolation of the conatiner and the container groups from one another.
• Blast Radius: For those of you running containers in a cloud environment, is to run different containers under different cloud user accounts. This limits the resources available to any given container. If an account or container set is compromised the same cloud service restrictions which prevent tenants from interfering with each other limit damage between accounts and projects. For more information see our reference material on limiting blast radius with user accounts.

Platform Security

Until recently, when someone talked about container security, what they were really talking about is how to secure the hypervisor and unde;ying operating systems. And because of that most articles and presentations you find on container security focuses on this single – albeit important – facet. We believe that runtime security needs to be more than that, and we break the challenge into three areas; host OS hardening, isolation of namespaces, and segregation of workloads by trust level. The following section discusses these three areas.

• Host OS/Kernel Hardening: Hardening a host operating system is how to protect that OS from attacks or misuse. It typically starts with the selection of a hardened variant of the operating system you wish to use. But while these versions come with both secure variants of libraries and features, there will still be work to be donw with baseline configuration and the removal of unneeded features. At a minimun you’ll want to ensure user authentication and roles for access are set, that permissions for binary file access are properly set, logging for audit data is enabled, and that the base OS bundle is fully patched. A review of patching and configuration of the virtualization libraries (such as libcontainer, libvirt, and LXC) that the container engine relies upon to protect itself are fully patched.
• Resource Isolation and Allocation: A critical element to container security is limiting container access to underlying operating system resources, specifically so a container cannot snoop on – or steal – data from other containers. The first step is making sure container priviledges are assigned to a role. While the container engine must run at the root user, you containers must not, so you want too set up user roles for the container groups. Next up is the resource isolation model for containers, which is built atop two concepts; cgroups and namepsaces. Namespaces creates a virtual map of resources any given task is to be provided. It maps specific users and groups to subsets of resources (e.g.: networks, files, IPC, etc.) within their Namespace. We recommend default deny on all inbound requests and only allow only those containers that need to comminicate an open network channel. And it is essential that you not mix container and non-container services on the same machine. You will create specific user IDs for containers and/or group IDs for different classes of containers, and then assign IDs to containers at runtime. A container is then limited to how much of a resouce it is allotted by a Control Group (i.e.: cgroup). The cgroup provides a mechanism to partition tasks into hierarchical groups, and control how much of any particular resouce (e.g.: memory, CPU) a task can use. This helps protect one group of containers from being resource starved by another.
• Segregation of Workloads: We discussed resource isolation at the kernel level, but you should also isolate container engine/OS groups and their containers at the network layer. For container isolation we recommend mapping groups of mutually trusted containers to separate machines and/or network security groups. For containers running critical services or management tools, consider running a limited number of containers per VM and group them by trust level/workload or grouping them into into a dedicated cloud VPC to limit attack surface and minimize an attacker’s ability to pivot should a service or container be compromised. As we mentioned above in Container Group Segregation item above, there are 3rd party and orchesrtration manager features to aid with segmentation. In extreme cases you can consider one container per VM or physical server for on-premise applications, but this defeats some benefits of using containers atop virtual infrastructure.

Platform security and container isolation are both huge fields of study, and we have only scratched the surface. If you want to learn more, OS platform providers, Docker, and many third-party security providers offer best practices, research papers, and blogs with in great detail, often detailing issues with specific operating systems.

Orchestration Manager Security

This research effort is focused on container security, but any discussion of container security now comes from the perspective of securing containers within a specific orchestration management framework. There are many orchestration managers out there: Kubernetes, Mesos, Swarm, as well as cloud native container management systems from AWS, Azure and GCP. Kubernetes is the dominant tool for managing clusters of containers, and with it’s rapid rise in popularity comes many additional concerns for any container security program, both because of the added complexity of the environment, but also the default security of Kubernetes is generously described as ‘poor’. There are publicly demonstrable attacks which show it is possible to gain root access of nodes, escalate priviledges, bypass identity checks, and exfiltrate code, keys and credentials. Our point here is that many of the container managers need lots of tuning to be secure.

We have already discussed security aspects like OS hardening, image safety, Namespaces and network isolation to reduce ‘blast radius’. And we have already discussed hardening contianer code, use of trusted image repositories to keep admins from accidentally running malicious containers, and use of immutable container images to disallow direct shell access. Here we cover specific aspects you should consider to secure you orchestration manager.

• Management Plane Security: Cluster management, regardless if you’re using Swarm, Kubernetes or Apache Meso, will be handle via command line APIs. For example, ‘etcd’ key-value store and ‘kubectl’ controller are fundamental components to managing a Kubernetes cluster, and these tools can be misused in a variety of ways by an attacker. In fact the graphical user interface on some platforms do not require user authentication, so disabling their use is a common best practice. You’ll want to limit who has access to administrative features, but as command line tools such as this can also be imported by developers or attackers, simple access controls are not enough! Network isolation will help protect the master management server, and help control where administrative commands can be run. A combination of network isolation, leveraging more recent IAM services built into the cluster manager (RBAC for Kubernetes), and setting up least privleges for service accounts on nodes.
• Segregation of workloads: We have already reviewed Namepsaces and network isolation, but there are more basic controls that should be put in place. With both on-premise Kubernetes and cloud container deployments, it is common for us to find a flat network architecture. We also find that developers and QA personnel have direct access to production accounts and servers. We strongly recommend segregation of development and production resources in general, and within the production orchestration systemsm to segregate sentive workloads to different nodes or even cluster instances. Second, setting up network security policies or security groups (in AWS parlace) to ‘default deny’ on inbound connections at as a good starting point, and only add specific exceptions that are needed for applications to run. This is the default network policy for most cloud services and is effective at reducing attack surface. Default-deny also helps reduce the liklihood containers try to auto-update themselves from external sources, and deny attackers the ability to upload new attack tools should they gain a foothold in your environment.
• Limiting Discovery: Cluster management tools collect lots of meta data on cluster configuration, containers and nodes within the system. This data is essential for the cluster management server to run, but it is also a map for attackers when probing your system. Limiting what services an users can access meta-data, and ensuring requsting parties are fully authorized helps reduce the attack service. Many of the platforms offer meta-data proxies to help filter and validate requests.
• Upgrade and patch: The engineering efforts behind most container managers have reponded well to known security issues, and as a result, newer versions of cluster managers tend to be far more secure. With virtualization a key element of any cluster, and as these platforms have redumdamcy built in, you can leverage cluster management feature to quickly patch an replace both cluster services as well as containers.
• Logging: We recommend collecting logs from all containers and all nodes. As many attacks focus on privledge escalation and obtaining certificates, we recommend monitoring all identity modification API calls and all failures to detect attacks.
• Test yourself: There are secuirty checkers and CIS security benchmarks for containers and container orchestration managers, which youy can use to get an idea of how well your baseline security stacks up. These are a good initial step when validating cluster security. As the default configurations for container managers tends to be insecure, and most admins are not fully aware of all of the features any given cluster offers, these checkers are a great way to get up to speed on appropriate security controls.

Keep in mind these are very basic recommendations, and we really cannot do this topic justice within the scope of this research paper. That said we really want to raise the readers awareness that there are existing and proven attacks on all of the open source container management systems and there is a considerable amount of work needed to get a cluster security. Beyond the basics, each container manager has it’s own set of specific security issues and nuances on how to protect the cluster from specific types of attack.

Secrets Management

When you start up a container, or an orchestration manager for that matter, it will need permissions to communicate with other containers, nodes, databases and other network accessible resources. In highy modular, service oriented architectures, a container without credentials to connect to APIs, encrypt data, or prove it’s identity to other services won’t get much work done. What we don’t want is engineers hard coding secrets into the container, nor do we really want these secrets sitting in files on the server. But provisioning machine identities is tricky; we need to securely pass sensitive data to ephemeral instances as they start up.

The new class of products that address this issue are called ‘Secrets Management’ platforms. These products securely store encryption keys, API certificates, identity tokens, SSL certificates and passwords. These secrets can be shared across groups of trusted services and users, leveraging existing directory services to determine who has access to what. Solutions are readuly available; there are commercial tools available, and many of the orchestration managers and container ecosystem providers (i.e.: Docker) offer a secrets management capability built in.

We cannot fully discuss this topic in the scope of this research paper, so if you need more information please see our complete research work on Secrets Management.

In our last post we will discuss logging and monitoring.

- Adrian Lane
(0) Comments
Subscribe to our daily email digest

The post Container Security 2018: Runtime Security Controls appeared first on Security Boulevard.

from Container Security 2018: Runtime Security Controls

Container Security 2018: Securing Container Contents

Posted under: Research and Analysis

Testing the code and supplementary components which will execute within a container, and verifying that both conform to security and operational practices is core to any container security effort. One of the major advances over the last year or so is the introduction of security features for the software supply chain, from container engine providers like Docker, Rocket, OpenShift and so on. And we are seeing a number of third-party vendors help validate conatiner content both before and after deployment. Each of these solutions focus on slightly different threats to container construction; for example Docker provides tools to certify that a container has gone through your process without alteration through use of digital sigantures and container repositories. Third-party tools focus on security benefits outside of what the engine providers do, such as examining libraries for known flaws. So while things like process controls, digital signing services to verify chain of custody, and creation of a bill of materials based on known trusted libraries are all important, you’re going to need more than what is packaged with the base container management platforms. You will want to examine the third-party tools which help harden the container inputs, analyze resource usage, perform static code analysis, analyze the composition of libraries, and check against known malware signatures. In a nutshell, you’ll need to look for more that what comes with the base platform you choose.

Container Validation and Security Testing

• Runtime User Credentials: We could go into great detail here about user IDs and Namespace views and resource allocation, but instead let’s focus on the most important thing: don’t run the container processes as root, as that provides attackers access to the underlying kernel and a path to attack other containers or the Docker engine itself. We recommend using specific user ID mappings with restricted permissions for each class of container. We understand that roles and permissions change over time, which requires some work to keep kernel views up to date, but this provides a failsafe to limit access to OS resources and virtualization features underlying the container engine.
• Security Unit Tests: Unit tests are a great way to run focused test cases against specific modules of code — typically created as your development teams find security and other bugs — without needing to build the entire product every time. They cover things such as XSS and SQLi testing of known attacks against test systems. Additionally, the body of tests grows over time, providing a regression testbed to ensure that vulnerabilities do not creep back in. During our research we were surprised to learn that many teams run unit security tests from Jenkins. Even though most are moving to microservices, fully supported by containers, they find it easier to run these tests earlier in the cycle. We recommend unit tests somewhere in the build process to help validate the code in containers is secure.
• Code Analysis: A number of third-party products perform automated binary and white box testing, rejecting the build if critical issues are discovered. We are also seeing several new tools that are plug-ins to common Integrated Development Environments (IDE), where code is checked for security issues prior to check-in. We recommend you implement some form of code scans to verify the code you build into containers is secure. Many newer tools have full RESTful API integration within the software delivery pipeline. These tests usually take a bit longer to run but still fit within a CI/CD deployment framework.
• Composition Analysis: A useful security technique is to check libraries and supporting code against the CVE (Common Vulnerabilities and Exposures) database to determine whether you are using vulnerable code. Docker and a number of third parties – including some open source distributions – provide tools for checking common libraries against the CVE database, and they can be integrated into your build pipeline. Developers are not typically security experts, and new vulnerabilities are discovered in common tools weekly, so an independent checker to validate components of your container stack is both simple and essential.
• Hardening: Over and above making sure what you use is free of known vulnerabilities, there are other tricks for securing containers before deployment. Hardening in this context is similar to OS hardenng (which we discuss in the following section); by removing libraries and unneeded packages to reduce attack surface. There are several ways to check for unused contents of the container, and then work with the Development team to remove items which are unused or unnecessary. Another approach to hardening is to check for hard-coded passwords, keys, or other sensitive items in the container — these breadcrumbs makes things easy for developers, but much easier for attackers. Some firms use manual scans for this, while others leverage tools to automate scanning.
• Container Signing and Chain of Custody: How do you know where a container came from? Did it go through your build process? The problem here is what is called image to container drift, where unwanted additions are added to the image. You want to ensure that the entire process was followed, and that somewhere along the way some well-intentioned developer did not subvert the process with untested code. You accomplish this by creating a cryptographic digest of all image contents as a unique ID, and then track it though the lifecycle - ensuring that no unapproved images are run in the environment. Digests and digital fingerprints help you detect code changes and identify where the container came from. Some of the conatiner management platfroms provide tools to digitially fingerprint code at each phase of the development process, along with tools to validate the signature chain. But these capabilities are seldom used, and the platforms such as Docker make only optionally produce signatures. While the code should be checked prior to being placed into a registry or container library, the work of signing images and code modules happens during build. You will need to create specific keys for each phase of the build, sign code snippets on test completion but before code is sent on to the next step in the process, and — most importantly — keep these keys secured so attackers cannot create their own trusted code signatures. This gives you some assurance that the vetting process proceeded as intended.
• Bill Of Materials: What’s in the container? What code is running in your production environment? How long ago did you build this container image? These are common questions when something goes awry. In case of container compromise, a very practical question is: how many containers are currently running this software bundle? One recommendation — especially for teams which don’t perform much code validation during the build process — is to leverage scanning tools to check pre-built containers for common vulnerabilities, malware, root account usage, bad libraries, and so on. If you keep containers around for weeks or months, it is entirely possible that a new vulnerability has since been discovered, so the container is now suspect. Second, we recommend using the Bill of Materials capabilities available in some scanning tools to catalog container contents. This helps you identify other potentially vulnerable containers, and to scope remediation efforts.

In the next section we will talk about how to proetct comtainers when they are in production.

- Adrian Lane
(0) Comments
Subscribe to our daily email digest

The post Container Security 2018: Securing Container Contents appeared first on Security Boulevard.

from Container Security 2018: Securing Container Contents