Measuring Awareness Training Success: 7 Metrics that Matter

Here are seven metrics to help to show the benefits of security awareness and privacy awareness training and prove out your hard work.

How to Show Your Awareness Initiative Is Working

Proving your worth is difficult.

“How can we show that security and privacy awareness training is worth the money?” is a common question for those tasked with managing awareness programs.

Whether you’re just starting your journey toward establishing an awareness initiative or looking to upgrade an existing program, setting measurable goals for behavioral improvements is crucial.

Either way, the stakes are high.

Lack of ROI for awareness training can lead to reduction or a complete cut of funds for training when budget season rolls around.

Fortunately, straightforward ways exist to track effectiveness and set the stage for a successful awareness training initiative.

Here are seven metrics that help to show the benefits of training and prove out your hard work.

Tracking Risky Employee Behavior (with IT’s Help)

Chances are that your IT department has systems in place to track employee behavior in the form of network event and data loss prevention logs. Here are three software types that might be running behind the scenes of your corporate network:

  • Security Information and Event Management (SIEM) systems collect network event logs, such as unsecure login attempts, virus scans, and other security-related documentation for analysis.
  • Data Loss Prevention (DLP) software monitors the transmission of sensitive information to make sure an employee doesn’t send it to unauthorized destinations.
  • User and Entity Behavioral Analytics (UEBA, or UBA) tools are a way to parse information collected by SIEM and DLP systems and provide IT professionals prioritized trend information.

The information these systems collect can serve as a gauge for determining which employee behaviors are putting your organization at risk.

What to Do

Work with your IT team to set a baseline by logging risky events prior to your training event. A few months after your initiative starts, check these numbers to see if logged events have decreased.

How Often Incidents Are Reported

Real live employees are vital to your company’s information security posture, no matter the strength of your IT team’s electronic eyes and ears. No network monitoring system can spot confidential information left by the printer in public view, or un-badged visitors finding their way into a secure area.

Some segment of your company, be it IT or even HR, should have procedures in place for employees to report suspicious incidents.

What to Do

Review the frequency of reported incidents before training begins. Check if these reports increase as training progresses and in the months following. More reported incidents means your employees have developed sharper eyes for suspicious activity (not necessarily that more incidents are actually happening!).

Consider combining this information with SIEM or DLP data to identify decreases in how long it takes for a security incident to be detected (called “time to detection”). Also, look for increases in the number or percentage of breaches detected and resolved before any harm occurs.

Reported Phishing Email Percentage

Spotting the dreaded phishing email and knowing who to tell about it is a specific type of incident reporting. But given how frequent these attacks are, this metric is best pulled out and recorded on its own.

There are two ways to consider phishing from a metrics perspective. The first involves real phishing emails.

What to Do

If your company has incident reporting procedures in place, how to report a suspected phishing email is likely one of them. Collect numbers on frequency of reported phishing emails vs. phishing emails not reported to develop a reported percentage. Clicked phishing emails should also be included in this initial data gathering to help set a baseline of your employee’s proficiency for recognizing and correctly addressing this threat.

After your primary training has run its course, review these numbers to see how they changed. The goal is an increase in the percentage of phishing emails reported, and a decrease in clicked phishing emails (ideally to zero!).

If you’re deploying a simulated phishing tool as part of your awareness efforts, the metrics above still apply. Such a tool should provide a low-lift way of setting a baseline for phishing susceptibility before training by means of a simulated phishing campaign launched before training begins. Then run subsequent campaigns after training has been delivered to see how employee behavior around phishing emails has improved.

Want to dive deeper into simulated phishing campaigns, like the security awareness nerd you are? We’ve got more information on best practices for running simulated phishing campaigns here.

How Much Incident Remediation Costs

No cybersecurity measure is 100% effective.

Chances are your organization has had a run-in with a data breach, malware infection, or another kind of cyber incident. Such an event may even be the reason you’re in the market for security and privacy awareness training in the first place.

Typically, recovering from such an incident isn’t cheap.

What to Do

If these costs have struck your organization, set this number as a baseline before you launch your training initiative. Keep these figures in your back pocket in case another incident occurs and determine if training reduced overall incident remediation costs.

Fortunately, independent research suggests an awareness training program will do just that.

A commissioned study analyzing the ROI of MediaPRO’s approach to awareness training found that organizations experienced fewer malware incidents when training employees with MediaPRO, leading to a $58,968 reduction in incident remediation costs. A separate PWC study found that companies that trained their employees spent 76% less on security incidents than companies that offered no training.

Direct Assessment of Employee Knowledge

Assessing employee knowledge is a direct way to measure what they know about security and privacy best practices.

The design of such a survey can take many forms and can make use of the variety of free survey platforms out there.

What to Do

Create questions that address your organization’s most pressing security and privacy risks. Check out our most recent State of Privacy and Security Awareness Report for some ideas based on eight risk areas we see as important for almost any organization.

A good understanding of your organization’s goals and priorities will make sure you’re asking the right questions. After all, why ask employees if they know how to connect to networks via VPN if they’re not taking their computers out of the office?

Connect with your HR department for help deploying such a survey. HR may already use a survey tool for sending out surveys on employee benefits or other company-wide topics. Stick to no more than 15 to 20 questions, and a 10-minute completion time for most employees.

Deploy the survey at least twice: once before your initial training event, and once after. The responses will tell if your training stuck with your employees, or if supporting training materials (in the form of short videos, posters, or articles) are needed.

How Many Employees Complete Training

Training completion numbers become vital when compliance requirements come into play.

We’re vehemently against the “check-the-box” approach to awareness training. But if you’re in a specifically regulated industry (healthcare) or under the boot of regulations like the GDPR or CCPA (which require training), showing that your employees have taken training is vital.

What to Do

Any learning management system (LMS) worth its salt will be able to tell you how many employees have completed a given course in a specific timeframe. Such numbers are typically not sexy and won’t show progress outside of the primary training event. Still, completion rates are a necessary statistic to show that the bare minimum goal of a training initiative is being met.

Noting Employee Talk Around the Water Cooler

Beyond these numbers-based metrics, consider assessing your employees from a “softer” point of view. This means doing the leg work to see if the seeds a cultural shift to a risk-aware culture have been planted.

What to Do

Keep an ear open to discussions about your training topics before, during, and after your primary training deployment to see what conversations the content is generating.

You may notice, as you come into the office kitchen to get a cup of coffee, that your employees are talking about one of the videos you shared during training. Or maybe they’re discussing that particularly tricky phishing email that made its way to your marketing department.

When your employees happily joke about data classifications, when they brag about the difficulty of their passwords, and when they argue about the right answer on the latest quiz you sent out, you will know that you have started to make real progress in creating a risk-aware culture.

This is also an opportunity to gauge the quality of the training. Did that specific attempt at humor completely miss the mark? Was that physical security scenario too hokey? Keeping your ear tuned to discussions around the water cooler is a good way to find out.

Bringing It All Together

An important thing to remember is to look at these metrics comprehensively. No one metric should be taken as the sole measure of success.

Many security awareness training vendors take this bait when it comes to simulated phishing programs. Hyper-focus on phishing catch rates will only result in employees that are really good at one threat vector. But this will do little to support comprehensive of topics like identifying personal information or physical office security.

A comprehensive awareness program that touches multiple risks deserves a comprehensive approach to tracking success. Fortunately, a multi-topic awareness initiative gives you the opportunity to do just that. The more topics you address, the more data points you can collect and analyze to see the impact of your awareness initiative.

Another benefit is the continual data gathering that will reveal which risks your employees can improve on.

Since training should never be a one-and-done affair, keeping these assessment methods in mind when a training refresh or update is needed will allow you to alter your training program to address emerging risks.

One of the core elements of all of MediaPRO’s TrainingPacks is the ability to monitor training effectiveness and prove compliance requirements. Learn more about TrainingPacks by connecting with one of our experts. 

Share this Post

Learn More Today
;