CC BY
29
COMMUNICATION OF COST-EFFECTIVENESS OF SAFETY MEASURES BY USE OF
A NEW VISUALIZING TOOL
E.B. Abrahamsen & T. Aven
University of Stavanger, Stavanger, Norway
e-mail: eirik.b.abraliamsen@uis.no, teije.aven@uis.no •
W. Roed
Proactima AS, Stavanger, Norway e-mail: wr@proactima.com
ABSTRACT
A cost-effectiveness analysis (CEA) is often used as basis for comparisons between competing safety measures. In a CEA indices such as the expected cost per expected number of lives saved are calculated. These indices are presented to the decision-makers, and seen in relation to reference values, they form the basis for assessment of the effectiveness of the safety measures.
The appropriateness of using cost-effectiveness indices based on expected values have been thoroughly discussed in the literature. It is argued that uncertainty is not properly taken into account by the CEA, and extended frameworks for CEA are required. This paper represents a contribution to this end, by presenting a diagram that visualizes uncertainty in addition to the expected values as in the traditional CEA. The diagram is meant to be a presentation tool for semi-quantitative cost-effectiveness analyses used as a part of a screening process to identify safety measures to be assessed in a more detailed analysis. In the paper we discuss the use of the diagram in communication between analysts and other stakeholders, in particular the decision-makers. An example is presented to illustrate the applicability of the tool.
1 INTRODUCTION
Tremendous resources are spent on safety measures and the need for tools for supporting the decision-making is large. Cost-effectiveness analysis is such a tool, and it has shown to give useful support for comparisons between competing safety measures.
Different cost-effectiveness measures are used reflecting that there are many ways of expressing cost-effectiveness. We may think of a safety measure as cost-effective if it is (Petitti 2000):
• Less costly and at least as effective
• More effective and more costly, with the added benefit worth the added cost
• Less effective and less costly, with the added benefit of the alternative not worth the added cost
• Cost saving with an equal or better outcome
Quantitatively, and more precise, the cost-effectiveness can be expressed as a cost-effectiveness ratio, the ratio of change in expected costs to the change in expected effects. This type of ratio
(index) usually forms the basis for communication of cost-effectiveness between analysts and other stakeholders.
A cost-effectiveness ratio produced through a cost-effectiveness analysis provides valuable insights. However, many analysts and researchers have pointed out that cost-effectiveness indices based on expected values are not appropriate for evaluation and communication of cost-effectiveness - a picture of cost-effectiveness needs to include a broader reflection of uncertainties. The main problems are that the expected values are conditional on specific background knowledge, and the expected values could produce poor predictions. Surprises may occur, and by just addressing expected values such surprises may be overlooked (Aven 2008). Taleb makes a similar conclusion using the black swan logic (Taleb 2007). The inability to predict outliers (black swans) implies the inability to predict the course of history. An outlier lies outside the realm of regular expectations, because nothing in the past can convincingly point at its occurrence. We find also similar ideas underpinning approaches such as the risk governance framework (Renn 2008) and the risk framework used by the UK Cabinet Office (Cabinet Office 2002).
To improve the communication of the cost-effectiveness of safety measures between analysts and other stakeholders, a cost-effectiveness-uncertainty-diagram is presented in this paper. The diagram visualizes uncertainty in addition to the expected values as in the traditional cost-effectiveness analysis.
The diagram should not be looked at as a tool for visualising results from detailed cost-effectiveness analyses. The diagram is meant to be a presentation tool for semi-quantitative cost-effectiveness analyses used as a part of a screening process to identify safety measures to be assessed in a more detailed analysis.
The paper is organized as follows. In Section 2 we review and discuss the use of cost-effectiveness analysis in evaluation of safety measures. In Section 3 the visualisation tool for cost-effectiveness is presented. Then in Section 4 an example is used to illustrate the applicability of the tool. Finally, in Section 5 we draw some conclusions.
2 REVIEW AND DISCUSSION OF THE COST-EFFECTIVENESS ANALYSES
In evaluation of safety measures a cost-effectiveness analysis is often adopted. The decision on whether a safety measure should be implemented or not is by using such an analysis to large extent based on the calculated cost-effectiveness ratio. The ratios can be expressed either as a cost-effectiveness ratio, or as an effectiveness-cost ratio (Boardman et.al 2006). The review and discussion of the cost-effectiveness analysis that follows, focuses on the cost-effectiveness ratio which is by far the more commonly used ratio.
The method will be illustrated by an example of two competing safety measures; safety measure 1 and safety measure 2. The following notation is used in the example:
• Q; the investment cost associated with safety measure i (to simplify we assume that there is no annual cost associated with the safety measure)
• Z[, the total effect related to loss of lives if safety measure i is implemented (to simplify we assume that this is the only effect of interest)
• R; the reference value. The value clarifies how much money the decision-maker is willing to pay to obtain one unit of effectiveness.
In order to compare the cost-effectiveness between the two measures, the cost-effectiveness ratio for both measures is calculated. The cost-effectiveness ratio for safety measure 1 and safety measure 2 is equal to Q /Z, and C2 /Z2, respectively. Safety measure 1 is more cost-effective than safety measure 2 if Q !ZX < C2 /Z2. To see whether safety measure 1 is preferred to status quo or not, the cost-effectiveness ratio has to be compared with the reference value, R. Implementation of the safety measure is preferred to status quo if the decision-maker is willing to pay more to obtain one unit of effectiveness than the cost-effectiveness index expresses, which means that safety measure 1 is preferred to status quo if R>(C1/Zx).
In practical situations we cannot determine the cost and the effects with certainty. There is often large uncertainty about C and Z. As a result predictions are required, and the natural choice is to use expected values.
For example, let us look at a simplified case of a safety investment. The decision-problem is to decide whether or not a safety measure should be implemented. We assume that the expected investment cost is £0.8 million, and that the expected number of fatalities is reduced from 2.7 to 1.9 if the safety measure is implemented.
The calculated cost-effectiveness index for the safety measure is (in million pounds):
EC _ 0.8 EA ~ 2.7-1.9 ~
This value is often referred to as the implied value of a statistical life or the Implied Cost of Averting a Fatality (ICAF). We see that a cost-effectiveness analysis does not explicitly set a value to the benefit, e.g. value of a statistical life, as is required in a cost-benefit analysis, ref R2P (2001).
In many cost-effectiveness analyses we see that the decision is strongly based on the calculated cost-effectiveness index, which for this decision problem means that the decision-maker will prefer to invest in the safety measure if the decision maker's valuation of a statistical life (R) is higher than £1 million, while an investment in the safety measure will not be preferred if the valuation of a statistical life (R) is less than £1 million.
Valuable insight is provided through cost-effectiveness indices, but there is a need for a broader reflection of uncertainties, as discussed in Abrahamsen et.al 2004 and Aven (2008). The main argumentation is as mentioned in the introduction, that the expected values are conditional on specific background knowledge, and the expected values could produce poor predictions. To see this more clearly we can write the expected values in mathematical terms like E[X|K], where X is an observable quantity such as cost and K is the background knowledge. The background knowledge covers historical system performance data, system performance characteristics and knowledge about the phenomena in question. Assumptions and presuppositions are an important part of this knowledge. A result is that a true objective expectation value does not exist. Different analysts could come up with different values dependent on the assumptions and presuppositions made.
3 THE COST-EFFECTIVENESS-UNCERTAINTY-DIAGRAM
To improve the communication of the cost-effectiveness of safety measures between analysts and other stakeholders, we suggest to use a cost-effectiveness-uncertainty-diagram. This diagram better reflects the uncertainties than the cost-effectiveness indices.
The diagram reflects information about cost-effectiveness through three dimensions: 1) uncertainty, 2) expected cost and 3) the expected lives saved. The cost-effectiveness-uncertainty-diagram reflects the three dimensions by showing the expected cost on the x-axis, the expected saved lives on the y-axis and the uncertainty through different bubble sizes, see Figure 1. The cost-effectiveness of the safety measures is evaluated based on these three dimensions, and is represented by a colour (red, yellow and green). The red, yellow and green colour is in the diagram presented as black, dark grey and light grey, respectively. In the diagram below, four safety measures A, B, C and D are presented.
In the diagram attention is given to the expected number of saved lives as the expected effect, but could easily be adjusted to cover other dimensions of losses, for example related to the environment.
Figure 1. Graphical presentation of four safety measures in the cost-effectiveness-uncertainty-diagram.
The classification of safety measures into the cost-effectiveness-uncertainty-diagram is carried out on the basis of an understanding of the different dimensions as described in the following:
Expected Cost (EC):
The expected implementation cost of the safety measure. The expected implementation cost is considered as the centre of gravity of the probability distribution of the implementation cost.
Expected number of lives saved (EX):
The expected number of lives saved if the safety measure is implemented. The expected number of lives saved is considered as the centre of gravity of the probability distribution of the number of lives saved.
Uncertainty:
Uncertainty reflects the expected values' predictability of the real outcomes. High uncertainty in the cost-effectiveness-uncertainty-diagram may for example express that the assigned expected cost (EC) can give a poor prediction of the future cost.
In the cost-effectiveness-uncertainty-diagram four categories are used for both the cost and effectiveness dimensions, while three categories are used for the uncertainty dimension. Of course, the method may easily be adapted to more or less categories.
The categorisation process should be based on some guidelines or criteria to ensure consistency. In the following one example is given for all dimensions. The category classifications will be case-specific and subject to judgement by the analyst, but the descriptions could serve as guideline.
Expected cost:
• Very low: EC <£10.000
• Low: £10.000 < EC < £100.000
• Medium: £100.000 < EC < £1.000.000
• High: EC > £1.000.000
Expected number of lives saved:
• Very low: EX <0.01
• Low: 0.01 < EX <0.05
• Medium: 0.05 < EX <0.1
• High: EX >0.1
Alternatively, qualitative (non-quantified) categories may be used. This is in particular relevant in cases where a qualitative risk analysis is carried out.
Uncertainty:
Low uncertainty
All of the following conditions are met:
• The phenomena involved are well understood; the models used are known to give predictions with accuracy
• The assumptions made are seen as very reasonable
• Much reliable data are available
• There is broad agreement among experts
• Low variation in populations (low stochastic uncertainty)
High uncertainty:
One or more of the following conditions are met:
• The phenomena involved are not well understood; models are non-existent or known/believed to give poor predictions
• The assumptions made represent strong simplifications
• Data are not available, or are unreliable
• There is lack of agreement/consensus among experts
• High variation in populations (high stochastic uncertainty)
Medium uncertainty:
Conditions between those characterising high and low uncertainty, e.g.:
• The phenomena involved are well understood, but the models used are considered simple/crude
• Some reliable data are available
Note, that the degree of uncertainty must be seen in relation to the effect/influence the uncertainty has on the predicted values. For example, a high degree of uncertainty combined with high effect/influence on the predicted values will lead to that the conclusion that the uncertainty factor is high. However, if the degree of uncertainty is high but the predicted values are relatively insensitive to changes in the uncertain quantities, then the uncertainty classified in the diagram could be low or medium.
Table 1. Cost-effectiveness categories.
Category Description of cost-effectiveness category Expected cost (EC) Expected number of lives saved (EX) Uncertainty
High cost-effectiveness Measures associated with very low or low expected costs and with medium or high expected number of lives saved. (Independent of the uncertainty) Very low Medium Low, medium or high
Low Medium
Very low High
Low High
Low cost-effectiveness Measures associated with medium or high expected costs, with very low or low expected number of lives saved and with low uncertainty. Medium Very low The cost-effectiveness category will be considered one category up (medium) if the uncertainty is considered medium or high
High Very low
Medium Low
High Low
Medium cost- effectiveness Measures included in categories between those characterising high and low cost-effectiveness. High High The cost-effectiveness category will be considered one category up (high) if the uncertainty is considered medium or high
High Medium
Medium High
Medium Medium
Low Low
Low Very low
Very low Low
Very low Very low
The cost-effectiveness of the safety measures have to be decided through an evaluation of the three dimensions mentioned above. The categorisation of the cost-effectiveness should again be based on some guidelines to ensure consistency. One possible way for categorisation of the cost-effectiveness of the safety measures is given in Table 1.
Incorporation of the uncertainty dimension can lead to a reclassification of the cost-effectiveness for a safety measures seen in relation to a traditional cost-effectiveness analysis. We may start the cost-effectiveness classification by first rank the safety measures according to the two standard dimensions expected cost and expected (effectiveness) number of lives saved. Then we may adjust these up or down in case the uncertainties are considered high or low. In the example discussed in the next section, the uncertainties are considered high and hence the cost-effectiveness for the safety measures should be considered reclassified.
4 AN EXAMPLE
A risk analysis has been carried out for an existing road tunnel consisting of one tube with one lane in each direction. This is a low-traffic tunnel located on the countryside. The number of cars driving through the tunnel is, in average, two cars in each direction per minute. There are however large differences in traffic density during one day. For example in the morning and in the afternoon the traffic density is considerably higher than average, while in night hours there is hardly any traffic at all.
The tunnel is located in a district where, according to the geologists, the risk of rock fall is considerably higher than in most other tunnels. One potential risk reducing measure is to install rock protection bolts. This is however an expensive safety measure. Alternative measures have been considered, related to ordinary traffic accidents. For example, it has been discovered that the illumination of the lighting is not sufficient, increasing the risk of traffic accidents particularly when
entering and leaving the tunnel. Now, as a simplification for the purpose of the example, suppose two risk reducing measures only are considered:
• A: Install rock protection bolts in order to reduce the risk of rock fall
• B: Install more light fixtures and then increase the illumination to prevent traffic accidents
The alternative measures, and the associated properties in terms of expected number of lives saved, uncertainty and expected cost is presented below.
Risk reducing measure (A)
Install rock protection bolts.
Expected cost (EC): High: > £ 1 million
Expected number of lives saved (EX):
Category medium: Even though the probability of rock fall is by geologists considered high, the probability of a car being hit is low due to the low traffic density. Based on calculations, the expected number of lives saved is in category medium (between 0.05 and 0.1)
Uncertainty:
Category high: The typical situation is that one by one car drives through the tunnel every now and then. This means that most likely zero or one car will be hit by a rock fall even though the fall zone may be substantial. However, occasionally, and in particular during ruch hours, the cars tend to pass lines of typically 5 or 10. It is possible, though not likely, that an entire line of 10 cars will be hit by one single rock fall. This means that the number of lives saved by the rock protection bolts could be in the range from zero to 20 or more persons. Based on this, the uncertainty of the effect of the bolts is considered high.
Risk reducing measure (B)
Install new light fixtures
Expected cost (EC):
Medium: Between £ 100.000 and £1 million
Expected number of lives saved (EX):
Category medium: Based on experience from other tunnels high illumination is important to prevent traffic accidents. In this tunnel, the illumination is lower than what is considered best practice. The expected number of lives saved by increasing the illumination to best practice level is in the category medium.
Uncertainty:
Category low: The traffic accidents being prevented by installing new light fixtures involves 1 or 2 cars. Then the real number of lives saved by the risk reducing measure is 0-4, depending on the number of persons in each car. The phenomena involved are well-understood, and there is broad agreement among the experts on what the result of such accidents may be. Based on this uncertainty category low is applied.
Now, which of the two safety measures is the most cost-effective one? If only the expected risk reducing effect and the expected cost were taken into consideration, the natural candidate would be
measure B: This is the less expensive safety measure, and since the expected number of lives saved is equal for A and B it may be argued that B is the most cost-effective one.
Taking the uncertainty dimension into consideration, and applying the method presented in Table 1, it may be argued that safety measure A is the most cost effective one. The rationale behind this is that considering the expected number of lives saved and the expected cost only is not sufficient to evaluate cost-effectiveness: We have to take the uncertainties into consideration. In our case, it is considerable uncertainty about the number of lives saved of safety measure A; the rock protection bolts, in particular. The expected effect is low, since it is expected that the persons in zero or one car only will be saved. However, the actual number of lives saved could be much higher as described above. According to Table 1 we may change the cost-effectiveness by one category due to such considerable uncertainty.
The risk reducing measures are plotted in the cost-effectiveness-uncertainty-diagram in Figure 2 below.
Expected saved lives
High
Medium
Low
Very low
HO •
Very low Low Medium H i eli
Expected cost
Uncertainty
O Hisii
O Medium O Low
Cost-effectiveness H High
I Medium Low
Figure 2. Graphical presentation of the risk reducing measures.
5 CONCLUSION
Communication between analysts and other stakeholders of safety measures' cost-effectiveness is usually based on cost-effectiveness indices. These indices are based on expected values. In the literature it is argued that such indices are not appropriate for evaluation and communication of cost-effectiveness. A broader reflection of uncertainties is needed.
This paper presents a cost-effectiveness-uncertainty-diagram. By extending the cost-effectiveness description to also cover uncertainties beyond the expected values, we believe that the cost-effectiveness-uncertainty-diagram would be better able to provide a broad, informative and balanced picture of cost-effectiveness.
ACKNOWLEDGEMENT
The authors are grateful to two anonymous reviewers for valuable comments and suggestions to an earlier version of this paper.
REFERENCES
1. Abrahamsen EB, Aven T, Vinnem JE, Wiencke HS. 2004. Safety management and the use of expected values. Risk, decision and policy 9: 347-357.
2. Aven T. 2008. Risk analysis - Assessing uncertainties beyond expected values and probabilities. Wiley. N.J.
3. Boardman EA, Greenberg DH, Vining AR, Weimer DL. 2006. Cost-benefit analysis -Concepts and practice. New Jersey: Prentice Hall.
4. Cabinet Office. 2002. Risk: improving government's capability to handle risk and uncertainty. Strategy unit report. UK.
5. Petitti DB. 2000. Meta-analysis decision analysis, and cost-effectiveness analysis. New York: Oxford University Press.
6. Renn O. 2008. Risk governance: coping with uncertainty in complex world. London: Earthscan.
7. R2P2. 2001. Health and Safety Executive, Reducing Risks, Protecting People: HSE's decision-making process, HSE Books, Sudbury.
8. Taleb NN. 2007. The black swan: the impact of the highly improbable. London: Penguin.
