ISMS metrics: How do you measure effectiveness rather than tasks?

“90 per cent of measures completed” can be a useful progress indicator. It does not answer whether your most important information is better protected. The remaining ten per cent might concern the very process your operations depend on.

A useful ISMS metric starts with a decision it should support. You then define the measurement, data source and evaluation. This guide uses a recovery test to demonstrate the approach, with a metric profile you can adapt to other topics.

Define the measurement question before the metric

Do not start with whatever data your dashboard happens to provide. Start with a security objective and the question the responsible people need to answer.

Example: An important business process must be recoverable under agreed conditions. A suitable question is: “For which planned recovery scenarios do we have a valid test result that meets the specified criteria?”

The number of backup jobs created answers a different question. It can support operations, but it does not demonstrate successful recovery.

NIST SP 800-55 Volume 1 distinguishes measurement perspectives including implementation, effectiveness and efficiency. It also addresses data quality, uncertainty and documented measurement definitions. The following examples and threshold decisions are our working aid, not NIST target values.

Separate activity, result and effectiveness

One metric does not need to do everything. Two clearly defined measures are often more useful than a combined score whose components nobody can explain.

Define the numerator and denominator so others can reproduce the calculation

Hypothetical calculation: Tests are due for eight defined scenarios. Six meet the criteria, one fails and one has not been performed. The proportion of successfully tested scenarios out of all due scenarios is therefore 6/8, or 75 per cent.

If the denominator included only tests actually performed, the result would be 6/7, approximately 86 per cent. Both calculations are mathematically possible. They answer different questions. Without a definition, skipping a test could even improve the dashboard.

These figures are neither a recommendation nor an industry benchmark. What matters is the question you want the ratio to answer. Show absolute figures and particularly important unresolved cases too. One failure in a critical process can matter more than a good overall percentage.

Use a metric profile

Define in advance when a result counts as valid. A significant change to the process or technical environment can make an old test less meaningful. An unchanged green field would conceal that.

Track objectives and KPIs continuously in Kopexa

Kopexa lets you manage security objectives and their associated KPIs. Measurement values can be supplied dynamically through the API, so you do not have to gather them manually for every evaluation.

For our recovery example, the workflow is:

1. Set the objective: The defined business process must be recoverable under the agreed conditions.
2. Define the KPI: The proportion of successfully tested scenarios out of all due scenarios. Choose the target and evaluation period to suit your process.
3. Transfer the value: In this example, your connected data source calculates six successful scenarios out of eight due. Kopexa receives the resulting KPI value of 75 per cent through the API.
4. Address deviations: The responsible person assesses the failed and missing tests and derives measures. Test reports provide evidence for those measures.

The API keeps values current; the measurement definition determines their meaning. If your source supplies only completed tests, the omitted test remains invisible even with automatic transfer. Define how you handle the numerator, denominator and missing data, and how you detect a failed data import.

Interpret changes in context

A declining ratio can mean safeguards have deteriorated. It can also occur because you include additional systems or previously overlooked cases for the first time. Document changes to the scope and measurement method alongside the trend.

Missing data is information too. Mark it as “not assessable” or as a separate gap, rather than silently removing it from the evaluation. Sample the source data before using an apparent improvement as the basis for a management decision.

What decision follows from the figure?

End each evaluation with an action: repeat a test, address a critical fault, correct the scope or continue monitoring with a documented reason. A metric without an owner and a next decision remains a reporting figure.

One measurement question is enough to start. If your team can reproduce the calculation, explain its limitations and derive a specific measure, you have a sound basis. You can then add further security objectives.