Posts By :

Lisa Young

tips for developing improving metrics

Tips for Developing or Improving Metrics

Tips for Developing or Improving Metrics 1200 628 Lisa Young
Tips for Developing or Improving Metrics

Tips for Developing or Improving Metrics

by Lisa Young, VP of Cyber Risk Engineering

February 6, 2018

Reposted Content from ISACA Newsletter @ISACA Volume 1

Everywhere we turn, vast amounts of facts, figures, numbers, records and files are being processed, interpreted, organized, structured and presented in a way that turns those data bits and bytes into meaningful information. Putting the raw data into context is what makes information useful for business decisions and underlies many dashboards being developed across the enterprise. Data and information are important components for measurement and, if put into a suitable context, may also become meaningful metrics.

Let us begin with a few definitions and examples:

  • Data—Raw, unorganized facts, records, numbers, etc. An example is the number 2 or the letters “e, g, s.” By themselves, it is hard to know what exactly is meant by their use.
  • Information—Data that are structured, organized or presented in context to make them useful. An example is “I had 2 eggs for breakfast.”
  • Measure (or measurement)—Is the value of a specific characteristic of data. An example is “the number of staff that completed information security awareness training.” Without more context, it is hard to know what value is derived from the statement.
  • Metric—The aggregation of one or more measures to create a piece of business intelligence, in context. An example is “percentage of staff trained vs. expected (planned vs. actual numbers)” or “percentage of new users (internal and external) who have satisfactorily completed information security awareness training before being granted network access.” These statements give context for whether or not the information provided is meeting the intended objective. If I have 10 staff members and 9 of them have completed the relevant training, then my percentage of satisfactory completion is 90%. If I have 10,000 staff members and only 900 of them have completed the relevant training, then I know I still have more work to do, especially if the untrained staff have been granted access to the network.

Consistent, timely and accurate metrics are an important feedback mechanism for managing any activity. When seeking to develop or improve metrics, here are some considerations to keep in mind:

  • Establish objectives—What questions are intended to be answered with the metric? Who is the audience for the metric? Which information needs will be satisfied with the metric? Who collects the measurement data? What techniques for analysis and reporting will be used?
  • Prioritize objectives—Data collection and analysis are costly and time consuming. It is important to consider the purpose and intended use of the metrics. What actions or decisions would the metric inform? If no action, decision or behavior change occurs as a result of the metric, then why are you spending resources to collect and analyze the data?
  • Identify candidate metrics—Candidate metrics should be based on documented measurement objectives. Identify existing metrics that may already address the objective. Metrics may already exist to satisfy 1 purpose and may also be used for additional purposes or to answer additional questions.
  • Specify data collection and storage procedures—Procedures should be based on the objective to be satisfied and the capability of the organization for collecting, storing, managing and disposing of data. Remember, data by themselves may not be sensitive or personally identifiable, but when aggregated, there may need to be explicit procedures for protecting and sustaining the information and subsequently developed metrics. Being explicit about data collection and storage may also help with overall data management, maintaining data integrity and governance. Other considerations in this category are frequency of collection and where the source data are created, stored, used, transported, etc. Data flow diagrams are useful for better understanding the data’s unique characteristics and attributes.
  • Update objectives as needed—Do not be afraid to retire a metric if it is not driving decisions, behavior or actions. The most important consideration here is to ask yourself, “What is the value of this metric in comparison to another metric?” If the metric is not meeting the intended objective, then it is no longer useful to collect and maintain. You may need to iterate several times before getting to a small set of meaningful metrics that drive better decisions, actions and behaviors. Often, the best metrics are conveyed by reporting trends over time versus a single point-in-time metric.

Make sure your questions are the ones most important to your target audience (management, operations, strategic) and your assumptions are stated. If there are estimates used in the metric calculations (because you do not have a piece of data or have just started collecting and have no trends in the data), make sure to state that somewhere in your visualization. Good metrics are those that are used often, answer important business questions, cost little to collect in relation to their value, are easily collected and do not require extensive manual intervention or manipulation. There is a difference between metrics and metrics that matter. Lisa Young, CISA, CISM, is the past president of the ISACA West Florida (Tampa, Florida, USA) Chapter and a frequent speaker at information security conferences worldwide.

tips for understanding the role of rcsa in risk management

Tips for Understanding the Role of RCSA in Risk Management

Tips for Understanding the Role of RCSA in Risk Management 1200 628 Lisa Young

ISACA: Tips for Moving From a Controls-Based Approach to a Risk-Based Approach

by Lisa Young, VP of Cyber Risk Engineering

January 1, 2017

Reposted Content from ISACA Newsletter @ISACA Volume 1

Organizations exist to produce a product or deliver a service and generally have a strategy or a set of goals. Risk management is an organizational discipline that, when combined with strategic planning, ensures that the risk with the greatest potential negative impact on the ability to achieve the organization’s stated goals is identified, analyzed and responded to in an appropriate way (given the risk tolerances of that organization).

In September 1992, the Committee of Sponsoring Organizations of the Treadway Commission (COSO) released a 4-volume report titled Internal Control—Integrated Framework. This report presented a common definition of internal control, providing a framework against which internal control systems could be assessed and improved. Around the same time, the Turnbull Report was published and set out internal control best practices for UK-listed companies. After a few years of focus on internal control systems and corresponding internal controls, the management of risk was added to both the COSO and Turnbull reports. This was the genesis of risk and control self-assessment (RCSA) as we now know it.

An RCSA is one tool for surveying or interviewing the business and frontline personnel to understand their view of the risk factors that might impede their progress toward objectives. For the areas of concern identified as a potential risk, a set of corresponding controls that would assist in mitigating the risk or reducing its impact is determined. When an RCSA is used as the only source for risk identification, the organization’s capability to perform risk management is not fully developed, and important risk may go unnoticed. Here are some tips for thinking about how your organization identifies risk that may lead you to a more complete picture of the risk that your organization faces:

  • Do I begin with business goals and objectives and then identify IT-related risk to those business objectives? Many RCSAs are focused on known risk rather than new areas of concern or factors that have not materialized as realized risk yet.
  • Is my organization engaged in actively building skills in risk management? Do we have a common language for risk terms? Risk and controls are complementary, but they are not the same.
  • Do senior leaders in my organization seek out risk management insights to improve performance (not just manage the risk of noncompliance)?
  • Is robust and realistic scenario analysis a primary technique in my risk identification approach? If you are not using the COBIT 5 risk scenarios, consider looking at them and trying to incorporate them into your risk identification process.
  • Do business cases for all strategic initiatives (and major projects) include a detailed and specific description of risk in design, implementation and operations, along with steps to proactively manage them?
  • When conducting an RCSA, is the interviewee or survey participant asked about their concerns (that might not be part of the RCSA)?
  • Do I align strategic goals and objectives to a set of control objectives rather than prescribe a set of controls to use? Having a set of control objectives provides the ability to actively manage risk by changing the process or procedures, avoiding the activity that contributes to risk, or detecting a risky activity sooner. Controls are not the only way to manage risk.
  • Do I actively refine control objectives and the associated controls to make them simpler to save time and cost in design, implementation, use and monitoring?

Risk management is an ongoing organizational capability that can be improved over time. The goal is to keep the business operating with minimum impact from a realized risk or incident. Risk and control self-assessments are but one tool in the risk management tool kit. Make sure your RCSAs are robust enough to add value to the risk management process.

Lisa Young, CISA, CISM, is the past president of the ISACA West Florida (Tampa, Florida, USA) Chapter and a frequent speaker at information security conferences worldwide.

Summary

Organizations exist to produce a product or deliver a service and generally have a strategy or a set of goals. Risk management is an organizational discipline that, when combined with strategic planning, ensures that the risk with the greatest potential negative impact on the ability to achieve the organization’s stated goals is identified, analyzed and responded to in an appropriate way (given the risk tolerances of that organization).

Axio

Company

Support

Copyright 2018 Axio Global, Inc.

Axio360 NIST CSF

The time has come for you to take control of your cyber risk.