Since 1988's Morris Worm, which infected 10% of the estimated 60,000 computers connected to the internet, cybersecurity has grown into an industry expected to exceed $1 trillion in global spending between 2017 and 2021. Cybercrime will cost the global business market an estimated average of $6 trillion annually through the same time frame! So how do we spend just enough money on cybersecurity to be resilient and achieve our business objectives despite disruptive events like cyber-attacks?

This admonition from management guru Peter Drucker leads us to another problem: benchmark studies show that we are failing to measure the effectiveness of our cybersecurity investments. Even when an organization can measure some of its cybersecurity capabilities, such as defect remediation window (DRW) or the total number of assets protected by antimalware software, this rarely translates to an overall summary of the organization's cybersecurity performance.

The challenge of measuring performance was addressed in project management using Earned Value Management (EVM), which can calculate actual performance against planned performance across a project's scope, schedule, budget, and expenses. EVM actually calculates two performance indices: schedule performance (Is the project on time?) and cost performance (Is the project within budget?). The calculation has three dimensions:

• Planned Value (PV): amount of work (in monetary value) expected to be completed
• Earned Value (EV): monetary value of work completed to date
• Actual Cost (AC): amount of money spent to date

For example, the diagram below shows a project that is behind schedule (PV is higher than EV) but below budget (EV is higher than AC). The performance indices are calculated as EV/PV, which would be less than 1, and EV/AC, which would be greater than 1.

Cybersecurity isn't a discrete project with defined beginning and end dates, nor is it a one-, two-, or even three-dimensional problem. However, we need to measure organizational cybersecurity performance, and applying some EVM concepts is a good place to start.

As in EVM, we need a few dimensions to calculate cybersecurity performance over a given time frame (e.g., quarter or year):

1. Cybersecurity expenditures. We need to know how much money was invested, and where, in cybersecurity, including the technical (e.g., hardware, software) and non-technical (e.g., personnel, policy development) expenses.
2. Actual events and activities that occurred. These may be planned or unplanned events.
3. Planned scope of cybersecurity events and activities that expenditures were intended to address. The planned scope covers a wide range: events, such as malicious attacks from outsiders or unintentional actions of employees, and activities, such as policy development and execution or monitoring of third-party performance (such as a cloud service provider).
4. Successful handling of events or activities.

We can now develop indicators of performance, based on the four dimensions of cybersecurity expenditures:

Here's an example of how to apply these indicators. Let's say your organization spent $5,000 in CYQ2 for site-wide antivirus protection to detect and quarantine 10 malware signatures (paid as $500/signature) and an additional $30,000 on a human analyst. At the end of the quarter, you examine your logs and other incident tickets to determine what actually occurred. Let's look at the indicators:

• Intended performance: The antivirus detected and quarantined 8 of the 10 planned-for malware signatures.
• Failed expenditure: The antivirus detected, but did not quarantine, 1 of the 10 planned-for malware signatures, resulting in infected devices on your network.
• Realized accepted risk: The organization consciously planned not to invest in two-factor authentication (2FA) due to resource constraints and was the victim of a password attack resulting in unauthorized access to personally identifiable information (PII).
• Unexpected impact: The organization was a victim of a Distributed Denial of Service (DDoS) attack as part of a larger botnet attack. The organization had not identified this as part of the potential or residual risks.
• Averted residual risk: The analyst detected and quarantined a new malware signature that the organization had been warned about, but the organization had not spent additional money on that signature in antivirus due to resource constraints.
• Unexpected impact averted: The antivirus detected and quarantined 3 malware signatures that were not part of the 10 planned.
• Planned, but event did not occur: The analyst spent two weeks reading through logs to determine if signature 9 had gone unnoticed by the antivirus, but this malware had never attempted to gain access to the organization's systems.
• Unnecessary expenditure: Though the organization paid $500 for malware signature 10 as part of the antivirus protection, this particular vulnerability had already been patched as part of the latest operating system upgrade.

The simplicity of this example hides the significant effort it takes to collect and analyze the inputs into the four dimensions of cybersecurity performance--let alone the output indicators, any one of which might have multiple instances. However, the example suggests that Cybersecurity Performance Indicators (CsPIs) are possible. CsPIs would enable succinct communication of the status of a cybersecurity program, as we do for other disciplines, such as finance or customer service.

My intention for this blog post is to start a discussion about creating CsPIs. Do you think these are the correct elements and dimensions? What would you add or remove to best define and describe the performance of your cybersecurity program? What measures and metrics are you currently using to achieve the objective of the CsPIs? Send your ideas to info@sei.cmu.edu.