CERT PODCAST SERIES: SECURITY FOR BUSINESS LEADERS: SHOW NOTES

Comparing IT Risk Assessment and Analysis Methods

Key Message: Choose the IT risk assessment and analysis methods that are the best cultural fit for your organization.

Executive Summary

"Technical professionals are often called on to research, recommend, implement, and execute IT risk assessment and analysis processes. These processes provide important data used by management to responsibly grow and protect the business through good decision making for mitigating, accepting, transferring, or avoiding risk. These decisions must account for IT risks caused by emerging threats to the enterprise and vulnerabilities in the people, processes and technologies required for digital business." [1]

"Which method you choose for IT risk assessment and risk analysis is far less important than ensuring that the selected methodology is operationalized and a good fit for the corporate culture. The selected approach must be able to produce output that is meaningful to management, and supporting processes must account for assumptions, documentation, and potential gaming of the system. Tools should be leveraged, where possible, to ease method adoption." [1]

In this podcast, Ben Tomhave and Erik Heidt, research directors with Gartner Technical Professionals, discuss methods for IT risk assessment and analysis and comparison factors for selecting the methods that are the best fit for your organization.

PART 1: RISK ASSESSMENT METHODS AND COMPARISON FACTORS

Key Questions and Considerations

IT and information security professionals regularly ask:

• I’ve been told I need to do risk assessment. Where do I start?
• What methods should we be considering?
• What factors go into making a choice that is the best fit for our organization?

Much of the interest in answering these questions is being driven by compliance, promises from marketing, and public relations hype.

Professionals often start with idealistic criteria for selecting the right process. It is more useful to make this selection based on the method’s ability to inform better business decisions.

An effective risk method should help determine priorities and help make the right choices. Some examples are:

• improvements in controls
• determining the annual information security budget
• whether or not to add a new business feature given the organization’s risk tolerance
• a new regulatory or compliance requirement and whether current controls are adequate to meet this requirement

Risk Assessment and Analysis Methods

Some of the risk assessment and analysis methods that the Gartner team analyzed include the following:

• ISO 31000: Risk management – Principles and guidelines, establishes a number of principles that need to be satisfied to make risk management effective.
• ISACA COBIT 5: COBIT (formerly Control Objectives for Information and Related Technology) version 5 is a product of ISACA (formerly Information Systems Audit and Control Association); describes IT governance objectives and practices as well as a comprehensive risk management program.
• MAGERIT: Methodology for Information Systems Risk Analysis and Management, a Spanish standard mandated for use by all government agencies.
• NIST SP 800-30: U.S. National Institute of Standards and Technology (NIST) Special Publication (SP) 800-30 Managing Information Security Risk: Organization, Mission, and Information System View; documents a comprehensive approach to risk assessment.
• OCTAVE Allegro: developed by Carnegie Mellon University Software Engineering Institute’s CERT Division; provides a comprehensive risk assessment framework with details about performing risk analysis and includes the ability to incorporate other risk analysis methods.
• Factor Analysis of Information Risk (FAIR): a quantitative risk analysis method originally created by Jack Jones (CXOWARE), now a standard from The Open Group.

The hierarchy of risk management terms is risk management program or process, a subset of which is risk assessment. Risk analysis is a subset of risk assessment.

Comparison Factors

The factors that the Gartner team used to compare methods include the following:

1. Method type: risk assessment framework/process or risk analysis method
1. Framework: typically defines an overall risk management process including risk assessment and analysis. COBIT 5 is one example.
2. Method: typically focused on performing a specific set of analysis functions. FAIR is one example.
2. Analysis type: quantitative, qualitative, or both. MAGERIT is one example that supports both types of analysis.
3. Cost
4. Tools: available or not. OCTAVE provides spreadsheets to support its process. FAIR provides a range of automated tools.
5. Special skills required: by risk analysts
1. The more prescriptive methods are designed for a general audience. The more flexible methods typically require customization and therefore depend on more experienced risk analysts and training.
6. Method flexibility: flexible/customizable vs. prescriptive
7. Ramp-up time: time to complete the first analysis
8. Cycle time: time required for each subsequent analysis

Observations from the Field

One of the most important factors to consider is the fit between the method and the way in which decisions are made in the organization. Most organizations do not have a one-size-fits-all risk assessment approach. In most cases, organizations use a two-tiered approach:

• a baseline procedure for general risk assessment and identification, for filtering and vetting of risks
• other methods for performing deeper analyses of risks that fall outside the baseline, have high impact, or are high residual risks that need to be actively managed

A two-tiered approach is important because you can’t run every decision through an in-depth, quantitative risk analysis. This may take days or week to gather data and perform.

PART 2: TRADEOFFS – METHODS VS. FACTORS

Specialized Skills and Method Flexibility

These 2 factors are linked to the extent to which a given method is more or less prescriptive. Those that are more prescriptive tend to be relatively straightforward to understand and implement.

A counterpoint to this is that the separate risk documents that are derived from ISACA COBIT 5 provide extensive depth of coverage in terms of specific processes. However, COBIT 5 and its supporting documents are all intended to be customized, which does require a higher level of skill.

Additional examples include:

• FAIR: very prescriptive but does not provide for any customization. Training is required to understand how to use the quantitative analysis tools.
• OCTAVE: a much broader, more general framework that does support customization so it isn’t as prescriptive. As a result, analysts need to be reasonably skilled.

Investing in customization may be worthwhile if the method will be used for some time and has broad applicability across the organization. It is important to take the time to tie the method into the way the organization does business.

Additional Tradeoffs and Considerations – Part 1

Another consideration is being able to use assessment results to categorize and prioritize areas for improvement.

If your market sector is highly regulated or regularly has high impact risks to mitigate, investing the time and energy in a method such as FAIR may be the right thing to do. On the other hand, if such compliance requirements call for hundreds, thousands, or tens of thousands of risk assessments in a short period of time, some compromises regarding method selection are necessary.

With many assessments to perform, outside formal training may not be an option. In addition, the use of automated tools for capturing and scoring risk questionnaires may be required to collect high volumes of data. Short cycle times are essential.

It is important to select a method that fits within the time and energy of available resources.

Ramp-up Time vs. Cycle Time

Qualitative methods tend to have shorter ramp-up times while quantitative methods tend to have longer times. Because they are more qualitative, NIST 800-30 and OCTAVE likely have shorter ramp-up times.

Of the methods reviewed, COBIT 5 has the longest ramp-up time due to the degree of customization required. NIST 800-30, OCTAVE, and MAGERIT have mid-range ramp-up times.

OCTAVE has a medium to long cycle time due to the amount of analysis that is required for each assessment. FAIR has a shorter cycle time once you get through the somewhat longer ramp-up time. This is due to the use of automated tools.

Some methods are more iterative. For example, FAIR provides the ability to determine the size and impact of hazards and threats early in the process, which can be fine-tuned as you go. The more questionnaire-based methods are generally “one and done.”

High impact and high residual risks are almost always going to require additional, more in-depth analysis.

Annual vs. Continuous Risk Assessment

While there may be situations where an annual risk assessment still makes sense, effective risk management is conducted as an ongoing process. Cycle time needs to be considered when you are assessing key systems and key applications on a continuous basis.

Some risk assessment tools and methods are now being integrated earlier into the system and software development process, for example, as part of applications security testing.

PART 3: IMPORTANT CHARACTERISTICS FOR SELECTING THE “BEST FIT”

Differentiators

Qualitatively, there is not a great deal of difference in terms of how all of these methods function.

The most important factor is cultural fit. For example:

• OCTAVE fits well with an engineering and analytical mindset.
• COBIT 5 fits well with an audit, compliance, and governance mindset such as that found in the financial services sector.
• FAIR fits well with analysts who are numbers-oriented.
• MAGERIT is a good fit for organizations that want to use multiple methods, given its ISO 31000 structure.

Additional Tradeoffs and Considerations – Part 2

The following factors are also essential to consider:

• Scalability: allows an organization to select a centralized or decentralized approach for risk analysis and tool use. This ties back into specialized skills and flexibility, described above.
• Ability to produce meaningful output for key users and stakeholders: Qualitative methods tend to produce results such as “high, medium, and low” but when you have ten “high” findings, how do you determine which one is more important?

It is important to differentiate between risk management activities and compliance checks. These tend to get conflated by business professionals.

Given it is expensive to collect, consider how you are going to use assessment results to fulfill the immediate need as well as how you might use this data in the future. Think about what you can add to an assessment today to add value and leverage for the future.

Resources

[1] Tomhave, Ben; Heidt, Erik; Robins, Anne. “Comparing Methodologies for IT Risk Assessment and Analysis.” Gartner, 30 January 2014.

Refer to the links to publicly available methods listed in Part 1 above, under Risk Assessment and Analysis Methods.