Posted by In 2002, Donald Rumsfeld, who was the US Secretary of Defense at the time, was asked a question about the lack of evidence of Iraqi weapons of mass destruction. His response was one of the classic statements ever given in a press conference:
Reports that say that something hasn’t happened are always interesting to me, because as we know, there are known knowns; there are things we know we know. We also know there are known unknowns; that is to say we know there are some things we do not know. But there are also unknown unknowns—the ones we don’t know we don’t know. And if one looks throughout the history of our country and other free countries, it is the latter category that tend to be the difficult ones.
Donald Rumsfeld, 2002
Unknown unknowns are indeed the difficult ones. They are the things you don’t know you don’t know. And they are the most important thing for security organizations.
Let’s explain using the figure below. The x-axis represents whether an alert was generated. The y-axis represents whether an attack actually happened. The green boxes can be thought of as “known knowns.” Starting on the bottom left quadrant, if there is no attack and no alert, this is an expected condition. Similarly, in the upper right quadrant, when true attacks occur, they should be alerted.
The bottom right quadrant represents false alarms. These are alerts that need to be chased down, but are benign. Security organizations spend a lot of time and effort to tune out false alarms to avoid wasting time on non-issues. Managed Security Service Providers (MSSPs) need to limit false alarms to meet their service level agreements and not overwhelm their customers.
Now we come to the most important part of the figure below – what Rumsfeld might call the unknown unknowns. Missed detections (top left quadrant) are true attacks that are not alerted. There may be many reasons – insufficient security tools, poorly configured tools, poorly tuned tools (e.g., from over-aggressively tuning out false positives). The problem is that missed detections are the most important problems for security operations and they have traditionally been the hardest problems to identify. Until now.
Breach and Attack Simulation (BAS) is a way to simulate real attacks in your network to determine missed detections. BAS is used in the operational network. This means the attacks occur in the “fog of war” – that is, among the noise that naturally occurs in your network. We call this “true positive” data, because BAS tells you the attack technique used and the exact time it occurred. It is the only systematic way to constantly and consistently test security controls against missed detections. Anyone who has done machine learning algorithms knows that the training data must include true positive data. Even if you have not implemented machine learning yet, you still need to test with true positive data.
What about techniques such as purple teaming? Purple team testing is great, but it is generally not consistent nor is it comprehensive. Networks change constantly and security configurations must also be constantly tested.
Get a handle on the unknown unknowns. If you are not using regular BAS testing, your security operations – people, process, and technology – are not being properly verified.
Security Economics 