Tips for Developing or Improving Metrics

Reposted Content from ISACA Newsletter @ISACA Volume 1

Everywhere we turn, vast amounts of facts, figures, numbers, records and files are being processed, interpreted, organized, structured and presented in a way that turns those data bits and bytes into meaningful information. Putting the raw data into context is what makes information useful for business decisions and underlies many dashboards being developed across the enterprise. Data and information are important components for measurement and, if put into a suitable context, may also become meaningful metrics.

Let Us Begin With a Few Definitions and Examples:

• Data—Raw, unorganized facts, records, numbers, etc. An example is the number 2 or the letters “e, g, s.” By themselves, it is hard to know what exactly is meant by their use.
• Information—Data that are structured, organized or presented in context to make them useful. An example is “I had 2 eggs for breakfast.”
• Measure (or measurement)—Is the value of a specific characteristic of data. An example is “the number of staff that completed information security awareness training.” Without more context, it is hard to know what value is derived from the statement.
• Metric—The aggregation of one or more measures to create a piece of business intelligence, in context. An example is “percentage of staff trained vs. expected (planned vs. actual numbers)” or “percentage of new users (internal and external) who have satisfactorily completed information security awareness training before being granted network access.” These statements give context for whether or not the information provided is meeting the intended objective. If I have 10 staff members and 9 of them have completed the relevant training, then my percentage of satisfactory completion is 90%. If I have 10,000 staff members and only 900 of them have completed the relevant training, then I know I still have more work to do, especially if the untrained staff have been granted access to the network.

Consistent, timely and accurate metrics are an important feedback mechanism for managing any activity.

Considerations to Keep in Mind When Seeking to Develop or Improve Metrics

1. Establish objectives
   • What questions are intended to be answered with the metric?
   • Who is the audience for the metric?
   • Which information needs will be satisfied with the metric?
   • Who collects the measurement data?
   • What techniques for analysis and reporting will be used?
2. Prioritize objectives
   • Data collection and data analysis are costly and time consuming. It is important to consider the purpose and intended use of the metrics.
     • What actions or decisions would the metric inform?
     • If no action, decision or behavior change occurs as a result of the metric, then why are you spending resources to collect and analyze the data?
3. Identify candidate metrics
   • Candidate metrics should be based on documented measurement objectives. Identify existing metrics that may already address the objective. Metrics may already exist to satisfy 1 purpose and may also be used for additional purposes or to answer additional questions.
4. Specify data collection and storage procedures
   • Procedures should be based on the objective to be satisfied and the capability of the organization for collecting, storing, managing and disposing of data. Remember, data by themselves may not be sensitive or personally identifiable, but when aggregated, there may need to be explicit procedures for protecting and sustaining the information and subsequently developed metrics.
   • Being explicit about data collection and storage may also help with overall data management, maintaining data integrity and governance. Other considerations in this category are frequency of collection and where the source data are created, stored, used, transported, etc. Data flow diagrams are useful for better understanding the data’s unique characteristics and attributes.
5. Update objectives as needed
   • Do not be afraid to retire a metric if it is not driving decisions, behavior or actions. The most important consideration here is to ask yourself, “What is the value of this metric in comparison to another metric?” If the metric is not meeting the intended objective, then it is no longer useful to collect and maintain. You may need to iterate several times before getting to a small set of meaningful metrics that drive better decisions, actions and behaviors. Often, the best metrics are conveyed by reporting trends over time versus a single point-in-time metric.

Make sure your questions are the ones most important to your target audience (management, operations, strategic) and your assumptions are stated. If there are estimates used in the metric calculations (because you do not have a piece of data or have just started collecting and have no trends in the data), make sure to state that somewhere in your visualization. Good metrics are those that are used often, answer important business questions, cost little to collect in relation to their value, are easily collected and do not require extensive manual intervention or manipulation. There is a difference between metrics and metrics that matter. Lisa Young, CISA, CISM, is the past president of the ISACA West Florida (Tampa, Florida, USA) Chapter and a frequent speaker at information security conferences worldwide.