Joe Barrett bio photo

Joe Barrett

INFOSEC Professional

Email Twitter

This blog post is directly inspired by this tweet. Thanks to @haydnjohnson for getting me thinking about putting this down on paper.

I’ll admit, there are over half a dozen different terms that people use to describe the services that we offer. And if you ask any three consultants, you’ll probably get ten different answers as to what each one means. I’m not at all trying to be the definitive source on this, but I am hoping this can highlight why I’m using the terms that I do. Questions? Disagreements? I’d be surprised if you all did agree with me!

Phrasing! Image courtesy of one of the best shows ever

Vulnerability Assessment

This is what most people have traditionally thought of as the entry level service offering. The point of a vulnerability assessment is to identify and categorize the vulnerabilities on a system or network. This can be done by manually examining systems, running various automated tools, or simply through questioning and documentation. The line is typically drawn at proving the vulnerability by exploiting it. In this offering, clients don’t want that. Vulnerability Assessments can be done unprivileged, but you get much more complete results with full privileges to all systems. Either way, these aren’t really adversarial engagements since you’re not there to actually take over any systems.

Intrusion Detection

This can be called a Compromise Assessment in some circles as well. These are friendly engagements like the VA, but with the goal of coming in and hunting across the entire network looking for evidence that an attacker is actively compromising the client. Typically tends to combine some variety of network- and host-based analyses as well as a mix of signature/IOC-driven analysis as well as other techniques like frequency analysis. They can also include some amount of DFIR, although traditionally only in pursuit of findings that the other methods turn up. However, this can very easily turn into a formal Incident Response engagement if you find a systemic malware infection.

Blue Team

A fairly unheard of concept in the commercial world, this comes from my government background. A Blue Team is typically thought of as a vulnerability assessment on steroids. It’s meant to highlight and provide recommendations around systemic issues, giving you a much better idea of how to resolve the problems on your network in a meaningful fashion.

For example, a VA might give you an 800-page report listing all of the missing patches on your network. Anyone who’s ever gotten one of those reports knows that they’re completely worthless. A Blue Team report might inform you that a certain Organizational Unit in Active Directory has blocked inheritance, which is why all 430 computers inside that OU structure haven’t been patched in nine months. In our Blue Team assessments, we also included intrusion detection as part of the service offering since we had full access to all internal hosts already.

Penetration Test

This is where most people start getting wrapped around the axle. Most of what the industry offers to clients as a penetration test is simply a vulnerability assessment combined with “proof of exploitation” of key and critical vulnerabilities. In a derogatory phrase, if the exploit isn’t available in Metasploit, it’s not verified on the client network. Lots of organizations offer far more sophisticated services under the penetration testing name, but I think the industry average is probably along the lines of “Run Nessus+nmap, import into Metasploit, and see what pops shells.” As a result, the phrase has lost a lot of credibility.

Red Team

Having worked alongside a variety of military and government red teams, I’d like to think I have a fairly solid understanding of what this consists of. Red Team engagements are the full spectrum warfare of security assessments. In a red team engagement, the consultants attack the client organization using physical means, social engineering, and technological avenues. They normally last several months to over a year and are focused around showing all of the different ways that an organization may be breached by any means.

Most importantly, you can’t red team yourself - there are too many conflicts of interest and too much tunnel vision to properly explore all attack avenues. Recently, some organizations have been rebranding penetration tests as red team exercises in order to get more market share and command higher prices. At the same time, I’ve had prospective clients tell us that “No no, we red team ourselves all of the time. I’ve got five guys running Nessus and analyzing the results every week, we don’t need your services.” Despite trying to explain that what they’re doing isn’t Red Team, the message doesn’t always get through.

All that being said, even the “Red Team” phrase is misappropriated far too much that it’s not clear what this service provides.

Adversarial Emulation

Also termed things like “Threat Emulation” or “Adversary Simulation”, this has also been referred to as “red team lite.” The purpose of this isn’t to bring a full court press against a target organization, but rather to model your attack after an actual threat vector that you believe could reasonably target the organization. Your goal is to help test (and strengthen) the organization’s defenses by testing their capabilities against the adversarial techniques. You accomplish this by breaking into the network, gaining access to their most important resources, and demonstrating impact. Most importantly, you are hacking to get caught. However, this takes not only different tools from traditional penetration testing, but also a different mindset and a serious emphasis on impact.

Unfortunately, most people don’t have the slightest idea what this offering is or why it would matter to them. Also, and perhaps most importantly, there’s no catchy phrase like “pentest” or “red team” that describes this offering - it’s quite a lot to say.

Purple Team

This is a concept I’ve heard a few times, typically spun as “We have both a Red and Blue Team in house and they compete!” Well, aside from the aforementioned fact that you can’t red team yourself, that’s an excellent idea. Now, you’ve got to have an organization large enough to have both a competent defensive team and a competent attack team (and that attack team typically operates as an Adversarial Emulation team), but if you have the manpower it’s a great way to get realistic feedback on your defenses on a regular basis.

So Which Do I Want?

That’s a very good question.

If you really don’t have a clue how well secured your organization is, if you haven’t had security assessments before, you probably want to start off with something non-adversarial like a vulnerability assessment (which you could do yourself by just buying Nessus) or a Blue Team engagement. It’s also probably worth having someone do an intrusion detection / compromise assessment to ensure that you aren’t already hacked while you’re trying to spin up a network defense program.

If you feel like you’re doing security fairly well, if you think your SOC can detect threats, then you should hire someone to conduct an adversarial emulation. The results of that will probably be pretty startling, but it will give you a great set of recommendations on how to better watch for realistic threats throughout the network. Once you’re satisfied that you’re truly unhackable, bring on the red team to humble you once again.