Have you heard the phrase ‘screaming into the void?’ If you ever want to know what that feels like, ask a room full of security professionals what key performance indicators (KPIs) they use to report on their security program.

On two separate occasions (a year and a half apart), I asked on LinkedIn what people’s KPIs are in security. I ask at roundtables, conferences, speaking engagements, dinners, and happy hours, and scream it into the ether. The reverberating silence echoing my question back to me is always the answer I receive.

Well, either that or, “mean time to discover/repair.”

This makes me wonder not only what people are using as security KPIs, but whether people are using KPIs at all..

The Problems with KPIs

One problem with security KPIs is that it’s hard to report zero. And if you have zero incidents and continue to, how do you make that number relevant to your executive team, who want to feel that the money they are giving you is valuable?

Another concern is being too boring or, alternatively, being too intense and creating fear, uncertainty and doubt (FUD—also known as security melodrama). If I start talking about all the alerts we received in a month I am liable to do both of these. At one company I worked with, the monthly reporting metrics listed how many alerts all of the alerting tools gave. Just that –  a number. 

We had 11,375 alerts this month. 

As a security leader, I would have no idea what to do with this information by itself. I cannot imagine non-technical, non-security executives trying to understand and consider it. 

But even with context, you can do too much. I know it’s hard to imagine someone being bored by an intense reenactment of my security operations center triaging a Cozy Bear or Mazafaka attack. But it is possible, when at the end, the punchline is, “and we are fine.” Especially if that keeps being the conclusion month over month.

KPIs are useful and crucial for storytelling. When contextualized, they can help you measure where you are, where you’re going, and how you’re doing over time. Understanding what’s important to you, what you‘re trying to achieve, and what your executives care about is how to start thinking about KPIs instead of just selecting random measurements or telling the same story over and over again.

The Importance of Measuring What Matters

Just as doctors measure height during annual checkups, even when growth is no longer expected, in cybersecurity, we often continue to measure certain metrics out of habit. If you’re not getting taller after 20, why bother measuring—unless you fell down a rabbit hole and ate a piece of cake, those height checks are a bit pointless. Similarly, just as height measurements might be relevant at certain life stages, certain KPIs might be more pertinent at different stages of a company’s security maturity.

The Challenge of Measuring “Lack of Breaches”

We’re all familiar with the Zen koan, “If a tree falls in a forest and no one is around, does it make a sound?” But what about, “If you have not had an incident, are you secure?”

One of the most significant challenges in cybersecurity is measuring the absence of events, specifically, the “lack of breaches.” Unlike more tangible metrics, the absence of breaches doesn’t provide a straightforward data point. However, it’s arguably one of the most critical indicators of a successful security program.

Think of it like trying to measure the lack of illness. If you didn’t catch a cold this year, it might be due to good hygiene, a healthy diet, and regular exercise. Or it might be because you were bitten by a radioactive spider that enhanced your immune system, and now you have a sudden urge to crawl up walls and solve crime in your city. But there’s no simple metric to count the colds you didn’t catch. Similarly, the absence of security breaches indicates that your preventive measures are effective, but quantifying this absence requires indirect metrics.

Quantifying the lack of breaches requires a combination of indirect metrics. Some useful numbers might be (your environment will make some of these more relevant than others):

• Incident Response Times: How quickly and effectively your team can respond to potential threats.
• Preventative Measures: The robustness of your security protocols and how effectively they prevent incidents.
• Historical Data: Comparing past breach incidents with current data to demonstrate improvement.

Communicating this to executives and boards can be challenging, as it involves explaining that the lack of breaches is a result of effective security measures rather than sheer luck. Emphasizing the potential cost savings and risk mitigation achieved through robust security practices can help illustrate the value of this absence.

Contextualizing Security KPIs

KPIs serve a critical function in reporting to executives and boards. They demonstrate the security leader’s knowledge, understanding, and effectiveness and justify the investment in security tools and measures. However, the effectiveness of a KPI depends on the context and the organization’s specific needs.

Therefore, along with the above metrics, I strongly recommend the Verizon Data Breach Incident Report (DBIR) and IBM Ponemon Cost of a Data Breach. These two annual reports are wonderful for an over-time understanding of the types of threats, the impact of successful breaches, and the industry-specific costs. This can help contextualize what the security program at your company is doing or needs to do relative to the industry you are in.

One company I worked at, in my first week, one of the executives said, “I don’t really know why you’re here. We’ve never had an issue.” I respectfully explained to her that she had no idea if that was accurate as they had no way to measure that information, no alerting, no monitoring, no prevention. I am so grateful she said that to me; it immediately helped me understand what was important to show the executives.

If your company is facing a surge in attacks, tracking the number of alerts and responses might be relevant. But without context, it’s just noise. Instead of stating, “We had 200 positive alerts this month,” explain the significance: “We had 200 positive alerts this month, with 6 of them being ransomware attempts that were successfully thwarted, potentially saving us up to $3 million if measured against the average cost of a breach in our industry.”

Or, if you have a lot of phishing attempts, tracking the number of phishing emails blocked by your email security system isn’t enough. Contextualize it by adding: “Our email security system blocked 1,000 phishing emails this month. Among these, 50 were sophisticated spear-phishing attempts targeting our C-level executives, which could have led to significant data breaches or financial loss.”

For incident response, you might measure the average time to contain an incident. Instead of simply noting, “Our average time to contain incidents was 2 hours,” explain its importance: “Our average time to contain incidents was reduced to 2 hours, down from 5 hours last quarter. This improvement in response time minimized potential damage and recovery costs, enhancing our overall security posture.”

Improving Through KPIs

KPIs should lead to actionable insights and improvements and can be project-based. I once worked with a company struggling with vulnerability management. Their weekly reports only showed how many vulnerabilities were remediated, without context on the total number of vulnerabilities, their criticality, or service level agreement (SLA) compliance. By refining their KPIs to include these other indicators, we significantly improved the security of the environment.

And that’s the ultimate point of KPIs. It’s not to measure, it’s to improve.

Another company reported the percentage of their environment covered by a new security tool. Due to the complex nature of the environment and tool, it took almost a year to implement. The metric was useful as we rolled out the product, but once they reached 97% coverage, it lost its value. 

It’s essential to reassess and adapt KPIs to ensure they remain useful.

Effective security KPIs ensure you can:

• Align with business goals 
• Provide meaningful insights 
• Drive improvements 
• Are dynamic, contextual, and relevant
• Articulate the value and effectiveness of your security efforts to stakeholders 

By focusing on what truly matters and addressing the complexities of measuring security your KPIs will become powerful tools in enhancing your organization’s overall cybersecurity posture.

Tailor Your KPIs

It is crucial to know who the KPIs are for. Part of the purpose of KPIs is to ensure that stakeholders can quantify the value of the security program to evaluate its benefits. Tailoring your KPIs to your audience ensures they understand and appreciate the security measures you bring to the organization. What KPIs are you using in your security program, and more importantly, why?