Tuning detective controls is a key component of implementing and operating an insider threat program, and one we have seen many organizations struggle with. Our work helping organizations with their insider threat programs has revealed common challenges with any tool that generates alerts of potential insider risk, such as user activity monitoring (UAM), security information event management (SIEM), or user and entity behavioral analytics (UEBA) tools. In this blog post, we will discuss some of the challenges and best practices for tuning detective controls.

The most common challenge organizations experience with detective controls is the noisiness of the generated alerts. As discussed in "Effective Insider Threat Programs: Understanding and Avoiding Potential Pitfalls," faulty detective controls that generate alerts that overestimate actual insider risk can erode trust in and support of the insider threat program. We have identified two primary causes for noisy alerts:

• The insider risk indicator associated with the alert is flawed. The behaviors or activity that correspond to the alert are not actually associated with an increased risk to an organization's critical assets. This inappropriate association can happen when organizations adopt potential risk indicators without carefully considering their applicability to the organizations' industry sector, culture, risk appetite, critical assets, and policies and procedures. A potential risk indicator can also be flawed if it applies so broadly across an organization that it loses its power to differentiate between a potential risk and the baseline condition. For instance, see Kate Randal's example of using narcissism as a potential indicator among FBI special agents.
• The data and/or logic used to generate the alert is flawed. The insider risk indicator associated with the alert is adequate, but the measurement being used to identify the indicator needs to be refined. Common logical flaws include using fixed thresholds for indicators such as excessive volume or frequency of activity. Flawed data sources may be incomplete or ill-suited for a particular use case (for example, using chat logs to identify absenteeism).

Over-reliance on tools is another common mistake organizations make with their detective controls. Some UAM, SIEM, and UEBA tools come with pre-configured alerts for common potential risk indicators. While these pre-configured alerts may be a helpful starting point, they often lack organization-specific context. Without modification, they can generate high rates of false positives.

We encourage organizations to consider the pre-configured alerts of their detective controls as reference implementations: examples of how the features and functionality of the tools can be used to develop organization-specific alerts. Tools are designed to automate some portions of an insider threat data collection and analysis process but are not the process themselves. We recommend organizations consider adopting our high-level Framework for Effectively Developing Insider Threat Controls. When it comes to tuning detective controls, the software development lifecycle (SDLC) can help.

Think of the process of developing and refining detective controls in the context of the SDLC phases:

1. Requirements - Clearly specify which potential risk indicator or indicators you are developing detective controls for.
2. Design - Identify the inputs, algorithms, and outputs for the control.
3. Implementation - Create and configure the control in your UAM, SIEM, UEBA, or other tool.
4. Verification - Develop a test plan that validates that the control generates the expected responses.
5. Maintenance - Measure effectiveness of implementation, refining implementation and verification procedures as needed.

This process captures a repeatable, tool-agnostic method for developing and maintaining robust potential risk indicators. Taking this a step further, organizations should express the design and implementations of their detective controls in a tool-agnostic form. Doing so can avoid vendor lock-in and duplication of work--if the requirements, design, and implementation of your detective controls are only expressed within a certain tool, what happens if that tool goes away?

The design of UAM triggers can be conceptualized in a tool-agnostic way as a combination of the four following pieces of information:

• the data source
• the specific fields within that data source
• the analytic techniques that are applied to the fields
• the response options, or the actions taken when the trigger is fired

We recommend organizations develop and maintain a repository for their detective controls that documents the following attributes for each control:

• detailed descriptions for the control
• the data source, fields, analytic techniques, and response options associated with the control
• associated threat scenarios and potential risk indicators
• revision history of the trigger
• measures of effectiveness

With respect to testing detective controls, the SDLC provides another useful guidepost. IEEE Standard 829 provides a standard methodology for software test documentation that pairs nicely with the need to validate detective controls. Following this standard, the following information should be documented for each control:

• the prerequisite activities needed for the test
• the test procedure, or the specific actions to be taken
• verification points, or steps taken to confirm expected results, which should align with the prerequisites, test procedure steps, and the control's response options

A major part of the maintenance of insider threat detective controls involves measuring control effectiveness. Different conditions, such as the following, can indicate a properly working detective control:

• an alert that was triggered by an active insider anomaly
• an alert that led to an inquiry being opened
• an alert that led to a high-risk individual or activity being identified
• an alert that HR, legal, investigators, other insider threat program stakeholders found to be useful in supporting their decision making

With so many potential confirmatory conditions, it is important that organizations know up front which measures will be used for a given control. It is equally important to establish a process that allows for alerts that do not satisfy those confirmatory conditions. Dismissed alerts can help establish base rates of occurrence of potential risk indicators, which can allow organizations to determine if the risk indicator is effective in their organizational context, as well as provide valuable feedback into the refinement process for the data sources and analytics used by the detective control.

We encourage organizations to consider adopting this tool-agnostic, SDLC-inspired process for developing and refining their insider threat detective controls. De-coupling your detective control development process from your tools can help engage other insider threat program stakeholders who may have the context needed to most effectively refine your controls. De-coupling can also drive requirements for new tools or data sources.

For more from the SEI on how to implement these recommendations in your organization's insider threat program, check out our three-day Insider Threat Analyst training course. As always, send any feedback to insider-threat-feedback@cert.org.