CERT PODCAST SERIES: SECURITY FOR BUSINESS LEADERS: SHOW NOTES

A Workshop on Measuring What Matters

Key Message: To make better-informed decisions, take action, and change behaviors, leaders and staff need to demonstrate that metrics for managing cybersecurity risks derive from business objectives.

Executive Summary

This podcast summarizes the inaugural Measuring What Matters Workshop conducted in November 2014, and the team’s experiences in planning and executing the workshop and identifying improvements for future offerings. The Measuring What Matters Workshop introduces the Goal-Question-Indicator-Metric (GQIM) approach that enables users to derive meaningful metrics for managing cybersecurity risks from strategic and business objectives. This approach helps ensure that organizational leaders have better information to make decisions, take action, and change behaviors. [1]

Katie Stewart, Michelle Valdez, Lisa Young, and Julia Allen, the developers and facilitators of this workshop, are all members of CERT’s Cyber Resilience Management team. Further details about this workshop can be found in our workshop report [1].

PART 1: PURPOSE; GQIM PROCESS; WORKSHOP STRUCTURE

The first offering of the Measuring What Matters Workshop was conducted at the ISACA Information Security Risk Management Conference in Las Vegas, NV in November 2014 with 40 participants.

Purpose

Why do we measure?

• to make better-informed decisions
• to determine the appropriate actions to take
• to change behaviors and the culture within our organizations

But how do organizations, senior leaders, and managers figure out what are the right things to measure?

The purpose of this workshop is to describe a measurement approach that is driven by strategic and business objectives. Given this, participants learn how to derive operational metrics that help ensure that the organization is managing cybersecurity risk in the appropriate manner.

GQIM Process

The Goal-Question-Indicator-Metric (GQIM) Process used in the workshop is derived from CERT’s resilience measurement research that has been ongoing since 2010. It is based on earlier work performed by Vic Basili and Dieter Rombach [2], applied to software engineering.

The SEI’s Process Program added the “I” to Basili’s and Rombach’s GQM method in support of their Capability Maturity Model Integration (CMMI) model development and implementation.

CERT has been applying the GQIM process to operational resilience based on the CERT Resilience Management Model (CERT-RMM).

The key question when using the GQIM process is not “What metrics should I use?” A much more productive question is “What do I want to know or learn?”

Starting with a strategic or business objective, participants perform the following:

• Goal(G): Develop one or more goals for each objective, the meeting of which allows an objective to be met.
• Question(Q): Develop one or more questions that, when answered, help determine the extent to which the goal is met. Having the right questions are key to the effective use of this process.
• Indicator (I): Identify one or more pieces of information that are required to answer each question, to demonstrate the extent to which the goal is achieved.
• Metric (M): Identify one or more metrics that use selected indicators (data, information) to answer the question. Visualization and representation of the data can be performed during either the indicator or metric step in the process.

Workshop Structure

The workshop took place over one full day, consisting of

• lectures
• scenarios used to illustrate the GQIM process
• demonstrations of each step of the process based on the scenarios
• small group work

Based on pre-work and workshop discussions, participants selected a specific objective to work on in their small groups, with the intent of completing the GQIM process for that objective and having results to take back to their organizations.

PART 2: SCENARIOS; BUSINESS OBJECTIVES

Scenarios

We used the following scenarios during the workshop to illustrate and demonstrate the GQIM process:

• Ensure your children’s teeth are healthy by helping them learn how to brush their teeth (easily understood to introduce the process)
• Forbes cybersecurity incident: an spear phishing attack that occurred in February 2014, affecting Forbes.com and the Forbes publishing platform
• Incident management: mitigating the risks of business disruption and loss resulting from cybersecurity incidents

We plan to add additional scenarios based on recent cybersecurity incidents to this body of work.

Business Objectives

About 50% of the workshop participants provided strategic and/or business objectives in advance of the workshop. The CERT team used these to help select candidate table topics for small group work.

Developing a well-formed objective to use for the GQIM process is one of the more challenging steps.

The objectives that were used in the workshop are as follows:

• Protect customer information
• Keep software assets up-to-date
• User awareness of cybersecurity threats (training)
• Reliance on external parties
• Mitigate the risks of disruptive events/incidents
• Forbes scenario (social engineering/phishing)

PART 3: WORKSHOP INSIGHTS; FUTURE PLANS

We made the following observations and had the following insights during the conduct of the workshop:

• Participants didn’t always have a business objective or know what their business objective was. We spent time helping them discover this and its relevance to their risk management and cybersecurity programs.
• Starting with a current metric that they were measuring, we helped participants try to determine what an applicable business objective for that metric might be. Some discovered that a current metric might not be very important.
• If you can’t tie a current metric to something that is meaningful to the business – a critical service, a product, a mission objective – it’s not surprising that it’s hard to get management attention and funding for the measurement program.
• As a community, we are attempting to get cybersecurity measurement to have the same rigor and impact as financial measurement.
• The majority of participants were very engaged for the full workshop, particularly during the small group work.
• Participants started out asking “What are some good measures I can use in my organization?” By the end of the workshop, they were asking “How do I relate what I’m measuring now to a business objective?” In addition, they were also asking:
  • “Do I still even need to be measuring it?” and “Should I be throwing it out?”
• We armed those with technical, operational responsibilities with tools for more effectively communicating with their upper management.

We are in the process of turning this workshop into a 2-day CERT public course. The participants identified a number of key improvements that we are in the process of implementing. Some of these include:

• spending more time on formulating good business objectives
• giving participants more time to share their experiences and insights throughout the course
• providing more background on CERT-RMM and our resilience measurement research

Course announcements will be published on the SEI and CERT websites in the coming months.

PART 4: CUSTOMER EXPERIENCES

We have had the opportunity to use the GQIM process with several customers, for the following purposes and with the following insights:

• Building a better risk management program (both operational and enterprise risk), which includes an ability to quantify risk. Risk quantification is one form of measurement.
• Discovering that the GQIM process is broadly applicable for any type of measurement program: security planning and management, IT operations, business continuity/disaster recovery, etc.
• Given that all organizations operate in a resource-constricted environment, identifying the right data to help senior leaders make the right investment decisions is critical. Arming operational staff with a process for doing so, and a means to communicate in business terms, has great value.

Resources

[1] Stewart, Katie; Allen, Julia; Valdez, Michelle; Young, Lisa. Measuring What Matters Workshop Report (CMU/SEI-2015-TN-002). Software Engineering Institute, Carnegie Mellon University, January 2015.

[2] Allen, Julia & Davis, Noopur.Measuring Operational Resilience Using the CERT Resilience Management Model (CMU/SEI-2010-TN-030). Software Engineering Institute, Carnegie Mellon University, September 2010.

Allen, Julia & Curtis, Pamela. Measures for Managing Operational Resilience (CMU/SEI-2011-TR-019). Software Engineering Institute, Carnegie Mellon University, June 2011.

Allen, Julia; Curtis, Pamela; Gates, Linda. Using Defined Processes as a Context for Resilience Measures (CMU/SEI-2011-TN-029). Software Engineering Institute, Carnegie Mellon University, October 2011.

CERT Podcast: Measuring Operational Resilience, October 2011.

SEI Training Course: Measuring What Matters: Security Metrics Workshop