In a recent security webinar I was listening to, the topic of black box vs. gray box vs. white box testing came up. If you’re not familiar with the terms:
- Black Box Testing – The penetration testers (AKA pen testers, red team, red teamers) have no previous knowledge of the system(s) being tested. They have to discover everything.
- Gray Box Testing – The pen testers have some information to use for the penetration test. For instance, they may be given limited details on how a given application works such as the general architecture and maybe some of the technologies used, but they don’t have access to everything.
- White Box Testing – The pen testers have access to everything that can be provided. This could even be source code access.
If your organization is spending money, then meaningful results are a must. Pen testing isn’t supposed to be a pat on the back. It’s supposed to be a tool to assess defenses so you can make adjustments. It’s a critical feedback loop for cybersecurity. So a report back of, “We didn’t find anything,” is useless. This is why I am not a big fan of pure black box testing. If the penetration testing team doesn’t find anything there are several possibilities:
- The team wasn’t very good.
- The team was just unlucky in the path it proceeded down.
- The parameters of the engagement limited what they could discover too greatly. For instance, the duration of the pen test was too short.
- Your organization is truly secure.
The last, your organization is truly secure, is the least likely. That means your org has spent money and gotten nothing for it. That’s bad. So what do I prefer? First, check the ego. I’ve been on the defensive side and of course when I first started out, I wanted a complete win. But as I matured as a security professional, my attitude changed. I don’t want a complete win any more because I believe that’s a false positive. There are vulnerabilities. We haven’t covered everything. And I sure want the pen testers to find these issues instead of a hostile actor. If the pen test team finds the problems, then we can plan solutions, we can prioritize, we can be efficient, we aren’t working in emergency mode. So I want to progressively enable the pen testers with more and more information (within the scope of the engagement).
To start with, I prefer a predefined period of time for black box testing. This often provides insight into what a knowledgeable adversary can discover and breach within a short time. After that set period, the team transitions into gray box testing. The organization should have insight as to what is important and what is potentially vulnerable. Letting a pen test team know that you want them to spend more time looking at a particular application or especially at a particular tier of an application helps the pen test team focus their efforts to what is going to provide the organization the most value from the engagement.
And I have seen cases where gray box testing proceeded into white box testing because the pen test team found something significant of grave concern. In order to assess the scope of the issue and try to find mechanisms to protect the org until the root cause can be remediated, the pen test team works with the org’s technical team(s) to provide a workable solution. This may mean going to code. Again, the point is to get value out of the engagement.