CC BY
273
EXPRESSING AND COMMUNICATING UNCERTAINTY IN RELATION TO
QUANTITATIVE RISK ANALYSIS
Roger Flage & Terje Aven
University of Stavanger, Stavanger, Norway e-mail: [email protected], [email protected]
ABSTRACT
A quantitative risk analysis (QRA) should provide a broad, informative and balanced picture of risk, in order to support decisions. To achieve this, a proper treatment of uncertainty is a prerequisite. Most approaches to treatment of uncertainty in QRA seem to be based on the thinking that uncertainty relates to the calculated probabilities and expected values. This causes difficulties when it comes to communicating what the analysis results mean, and could easily lead to weakened conclusions if large uncertainties are involved. An alternative approach is to hold uncertainty, not probability, as a main component of risk, and regard probabilities purely as epistemic-based expressions of uncertainty. In the paper the latter view is taken, and we describe what should be the main components of a risk description when following this approach. We also indicate how this approach relates to decision-making. An important issue addressed is how to communicate the shortcomings and limitations of probabilities and expected values. Sensitivity analysis plays a key role in this regard. Examples are included to illustrate ideas and findings.
1 INTRODUCTION
In a quantitative risk analysis/assessment (QRA), risk is typically described using probabilities and
expected values. For example, commonly used personnel risk indices in offshore QRA include individual risk (IR), potential loss of life (PLL) and fatal accident rate (FAR) (Vinnem, 2007). Individual risk and potential loss of life is defined as the probability of death of a randomly selected person and the expected number of fatalities, respectively, during a specified period of time
(typically a year). The fatal accident rate is defined as the expected number of fatalities per 108
exposed hours. For a comprehensive overview of risk measures (indices) for loss of life and economic damage, see Jonkman et al. (2003).
A QRA should provide a broad, informative and balanced picture of risk, with the purpose to
support decisions; for example about the implementation of risk reducing measures according to the ALARP principle. To achieve this, a proper treatment of uncertainty is a prerequisite. In this respect there are two main lines of thinking.
Most approaches to treatment of uncertainty in QRA seem to be based on the thinking that uncertainty relates to the calculated probabilities and expected values. This way of thinking leads to formulations like "uncertainties exist in most elements of the risk analysis: in top event frequencies, in branch probability models, and in consequence models/assessments" and "the absolute accuracy of the assessment is limited". Probabilities and expected values are interpreted as limiting relative frequencies, meaning that the probability of an event A is defined as the fraction of times the event A would occur if the situation considered were repeated (in real life or hypothetically) an infinite number of times. This means that an extended hypothetical population must be introduced. Variation in the population causes the event A to occur some, but not all of the times the situation is
repeated. This behaviour generates a "true" (objective) but unknown probability, P(A), which describes the uncertainty about the occurrence of the event A. The uncertainty described by P(A) is sometimes referred to as aleatory uncertainty, and lack of knowledge about the true value of P(A) as epistemic uncertainty; see e.g. Helton & Burmaster (1996a) - in particular Helton & Burmaster (1996b). Whereas epistemic uncertainty can be reduced, the aleatory uncertainty cannot and is sometimes called irreducible uncertainty (Helton & Burmaster, 1996b). Hence, if the true value of the probability were known, there would only be aleatory uncertainty. However, the probability P(A) is unknown (uncertain) and can only be estimated. The role of reliability and risk analysis is to provide an estimate, P*(A), of the underlying probability. Thus there are two levels of uncertainty involved: aleatory and epistemic.
The relative frequency interpretation is the one most often used for offshore QRA (Schofield, 1998).
We would argue that a conceptualisation like the one described above should be avoided. It causes difficulties communicating what the results of a risk analysis mean and easily leads to weakened conclusions in case of large uncertainties. We refer to Aven (2003).
An alternative approach is to hold uncertainty, not probability, as a main component of risk (Aven, 2008b), and regard probabilities purely as epistemic-based expressions of uncertainty. This means that probabilities are regarded as expressions of uncertainty as seen by the assessor(s), based on his/her (their) background knowledge (the Bayesian perspective). Then the probabilities are not themselves uncertain, but rather "probabilities in the light of current knowledge" (Lindley, 2006 p. 43). A probability always needs to be seen in relation to its basis.
The last point is emphasised in this paper. By pointing out that the background knowledge can hide uncertainties, we argue that a description of risk in QRA should cover more than the traditional probabilities and expected values. As uncertainty is the main component of risk, and probability is only a tool used to express uncertainty, possible uncertainty factors in the background knowledge need to be highlighted and their effect on risk and vulnerability assessed. This is a main message of the paper.
A similar thesis is developed by Schofield (1998), who refers to "confounding factors" that are "hidden from view". The confounding factors, it is said, may relate to inadequate information, difficulty in evaluating the appropriateness of assumptions, and difficulties in modelling complex phenomena.
The remainder of the paper is organised as follows. In Section 2 we describe what should be the main components of a risk description when uncertainty is a main component of risk. Section 3 addresses uncertainty factors, and presents a practical, semi-quantitative method for studying these. Section 4 looks at the total risk picture, and relates this picture to decisions concerning risk reduction and acceptance/tolerability. Section 5 concludes.
2 MAIN COMPONENTS OF THE RISK DESCRIPTION IN A QRA
To describe risk we need to take into account both the potential consequences (or the severity of the consequences) related to the activity or system being considered, and the associated uncertainties. The risk description presented here takes as a starting point the following general definition of risk (Aven, 2008b):
By risk we understand the two-dimensional combination of (i) events A and the consequences
C of these events, and (ii) the associated uncertainties U (about what will be the outcome), i.e.
(C,U). For simplicity we only write C, instead of A and C.
The main components of a risk description in line with this definition are reviewed in the following.
Events (A): An event is defined by ISO (2002) as the occurrence of a particular set of circumstances. In a QRA some main categories of events can be identified:
Initiating events (I): These are the events found at the centre of the well-known bow tie diagram. When the consequence spectrum of an event is clearly negative, which is usually the case in a QRA, the event is often referred to as an undesirable, unwanted or accident event.
Barrier failures (B): Barriers are introduced to reduce the likelihood of undesirable events (causation barriers (Vinnem, 2007)) and to limit the consequences of such events if they should occur (consequence/mitigation barriers (Vinnem, 2007)).
Consequences or outcomes (C): A consequence is defined by ISO (2002) as the outcome of an event. It is sometimes useful to distinguish between two levels of consequences or outcomes in a QRA:
Physical quantities (Z): The behaviour of the physical phenomena involved in the activity or system being studied forms an intermediate level of consequences or outcomes. Examples of relevant physical phenomena in an offshore QRA are fires (heat load, smoke dispersion, structural response) and explosions (overpressure, structural response).
Losses (L): The consequences or outcomes of primary concern are those that affect what humans value. Typical dimensions of loss studied in a QRA are human lives and health, the environment and material assets.
Predictions (C*): By prediction is meant a forecast of which value C will take. A prediction may be on the form of a single value or an interval.
Uncertainty (U): Uncertainty is understood as lack of knowledge about unknown quantities, i.e. about A and C. There is uncertainty about the occurrence of events (A) and what will be the consequences or outcomes (C) if an activity is carried out or a system is put into operation.
Probability (P): Probability is a tool used to express uncertainty about events, consequences and outcomes. Probabilistic expressions include probabilities, expected values, probability distributions and prediction intervals.
Background knowledge (K): The description of risk (i.e. the events and consequences considered and the description of uncertainty about these) depends on the background knowledge of the analyst. For an event A or consequence C the dependence of probabilities and expected values on the background knowledge K can be written as P(A|K) and E[C|K]. The background knowledge covers inter alia assumptions and presuppositions, historical system performance data and knowledge about the phenomena involved.
Sensitivity (S): The sensitivity of probabilistic risk indices can be investigated by altering the input parameters, or more generally the background knowledge. As pointed out by Bedford & Cooke (2001), sensitivity analysis is not the same as uncertainty analysis. Sensitivity analysis simply shows the effect on overall results of altered input parameters/values. By doing so it is possible to say something about the importance of assumptions and suppositions. A so-called backwards approach can be used to investigate how large changes in input parameters/values are needed to change conclusions.
The components listed above are the main elements of a QRA, i.e. the risk description covers (A,C,C*,U,P,K,S).
Vulnerability is an aspect of risk, and hence the vulnerability picture is a part of the risk picture. By vulnerability we understand the two-dimensional combination of consequences and associated uncertainties, given that an initiating event has occurred. Analogously with the structuring of the components of risk we write (C,U|I) for vulnerability, where I is the initiating event. The vulnerability picture includes the same elements as the risk picture, but is conditional on the occurrence of an initiating event. A description of vulnerability thus covers the components (A,C,C*,U,P,K,S|I).
The above structure represents an adjustment of the risk characterisation presented by Aven (2008b), in that it explicitly includes the sensitivity component (S) and identifies subcomponents of events (initiating events I, barrier failures B) and consequences (physical phenomena Z, losses L).
The risk description is illustrated in Figure 1. In the risk analysis a set of events and consequence categories are identified and studied. There is uncertainty about the events and their associated consequences. Specifically, there is uncertainty about the performance of barriers (B), the occurrence of initiating events (I), the outcome of physical quantities (Z) and the losses incurred (L). The uncertainty is relative to the background knowledge, i.e. altered background knowledge could cause decreased or increased uncertainty. Probability is used as a tool to express uncertainty, and risk and vulnerability is described by means of probabilities (e.g. P(I|K)), expected values (e.g. E[Z|I,K]), probability distributions (e.g. P(Z < z|I,K)) and prediction intervals (e.g. [zl, zu] such that P(zl < Z < zu|I,K) = p, where zl and zu are constants and 0 < p < 1). Underlying the entire assessment is the background knowledge of the analyst. By altering the background knowledge (K'), the sensitivity of the risk indices can be studied.
Events and consequences
B. D
Uncertainty
U(B1|K) U(I|K) U(B2|I,K) U(Z|K) U(L|K)
Probabilities and expected values
P(BJK) P(I|K) P(B2|I,K) E[Z|I,K] E[L|K], E[L|I,K]
P(Z < z|I,K) P(L < l|K), P(L < l|I,K)
Pfes Z < zu|I,K] P(.i< L < lu|K), P(ll < L < lu|I,K)
Sensitivity and risk reducing measures
P(BJK') P(I|K') P(B2|A,K') E[Z|I,K'] E[L|K'], E[L|I,K']
P(Z < z|I,K') P(L < l|K'), P(L < l|I,K')
P[zl< Z < zu|I,K'] P(l. < L < lu|K'), P(ll < L < lu|I,K')
Background knowledge
Analyst's knowledge Assumptions
Models, data and experts used
Figure 1. Risk description in a QRA according to the approach recommended in the paper 3 UNCERTAINTY FACTORS
The risk description in a QRA depends on a number of assumptions. For example, the assumptions in QRAs of offshore oil and gas installations commonly relate to:
- Time to detection of abnormal situation.
- Number of persons in the area (module) at the time of an accident.
- Number of immediate fatalities.
- Impact energy that a structure will be able to withstand.
- Etc.
The assumptions made form part of the background knowledge, and the risk description must be seen in light of this basis. Then it should be acknowledged that uncertainties could be hidden in the background knowledge. The assumptions may to a greater or lesser degree constitute uncertainty factors in the sense that the assumptions might not be valid. If assumptions turn out to be wrong, the result could be that the actual outcome of a predicted quantity is surprising relative to the assigned probabilities.
As an example, consider the analysis of explosion risk in a compressor module at an offshore oil and gas installation. Suppose that when determining the probability of ignition (event A) given a gas leak (initiating event I), P(A|I,K), a model is chosen which determines the gas dispersion pattern from the point of the gas leak in a very crude and simple way. The analyst has a rather low confidence in the predictive ability of this sub-model, but owing to various constraints more detailed and accurate methods are not available. In a professional analysis the models used should be considered sufficiently accurate for their purpose, but a high degree of accuracy is not always achievable. As the model used forms part of the background knowledge for the probability of ignition, the probability in itself is not uncertain. However, gas dispersion is an uncertainty factor in the background knowledge, and it should be highlighted when the probability is reported.
Furthermore, suppose that an assumption is made about the number of persons in the process area at the offshore installation at the time of an accident. In the calculation of individual risk it is assumed that the crew spends a certain fraction of their working time in the process area, which summarises to an expected number of persons in the area at the time of an accident. By doing so it is not properly reflected that the actual number of persons in the area at any time shows large variations, causing considerable uncertainty about the actual number of fatalities in the event of an accident.
To reflect uncertainty factors like the ones described above we suggest a semi-quantitative method as presented by Aven (2008a); adjusted to include consideration of both risk and vulnerability. In Table 1 uncertainty factors are analysed with respect to effect on risk and vulnerability. The effect on risk and vulnerability depends on two dimensions:
- Degree of uncertainty.
- Sensitivity of the relevant risk and/or vulnerability indices to changes in the uncertain quantities.
For example, a high degree of uncertainty combined with high sensitivity could lead to the conclusion that the uncertainty factor has a significant effect on risk. However, if the degree of uncertainty is high but the risk and/or vulnerability indices are relatively insensitive to changes in the uncertain quantities, then the effect on risk could be minor or moderate. The category classifications (minor, moderate, significant) will be case-specific and subject to judgement by the analyst, but the following descriptions could serve as a guideline: Significant uncertainty
One or more of the following conditions are met:
- The phenomena involved are not well understood; models are non-existent or known/believed to give poor predictions.
- The assumptions made represent strong simplifications.
- Data are not available, or are unreliable.
- There is lack of agreement/consensus among experts.
Minor uncertainty
All of the following conditions are met:
- The phenomena involved are well understood; the models used are known to give predictions with the required accuracy.
- The assumptions made are seen as very reasonable.
- Much reliable data are available.
- There is broad agreement among experts.
Moderate uncertainty
Conditions between those characterising significant and minor uncertainty, e.g.:
- The phenomena involved are well understood, but the models used are considered simple/crude.
- Some reliable data are available.
Significant sensitivity
Relatively small changes in base case values result in altered conclusions (e.g. exceeded risk acceptance criterion).
Moderate sensitivity
Relatively large changes in base case values needed to bring about altered conclusions. Minor sensitivity
Unrealistically large changes in base case values needed to bring about altered conclusions.
A semi-quantitative approach offers practicality and the suggested method may serve as a screening of uncertainty factors. If an uncertainty factor is found to have a significant effect on risk and/or vulnerability, it could be selected for a more thorough treatment in the analysis. This may not always be possible though. In practice, a QRA will always have to be based on a certain background knowledge (including a number of assumptions and suppositions). The probabilistic analysis will not be able to reflect all uncertainties.
Table 1. Uncertainty factors
Effect on risk Effect on vulnerability
Uncertainty factors Minor Moderate Significant Minor Moderate Significant
Gas dispersion/concentration x x
Number of persons in the area at the time of the accident x x
Note that the highlighted uncertainty factors in Table 1 are related to (potentially) observable quantities and events. It is not the probability of ignition that is uncertain. What is uncertain is the gas dispersion/ concentration, and consequently whether or not an ignition will take place.
Vulnerability is an aspect of risk, and so highlighting vulnerability can give an informative additional description of risk. Analysing the effect on vulnerability of the uncertainty factors involved can serve as an indication of the performance of the consequence/mitigation barriers. If the probability (frequency) of an initiating event is small, the effect of an uncertainty factor on risk is not necessarily very large, but the effect on vulnerability could be large.
4 THE TOTAL RISK PICTURE
When communicating the risk description to the decision-maker the following issues should be highlighted:
- Risk level and risk contributors
- Uncertainty factors
- Risk reducing measures
The main question concerning the risk level is whether the risk is high, low or typical for the system or activity being studied. This is illustrated by presenting the calculated risk together with reference values, e.g. values representing well established levels of intolerable/unacceptable and negligible risk. The calculated risk should however not be presented as the final answer. Other assumptions, databases or models would have resulted in different values, and this is important to reveal. Sensitivity is also a part of the risk picture. If the risk is relatively insensitive to changes in input quantities a strong message will be sent by the risk description.
Recall the explosion risk example above. The explosion risk analysis is part of the QRA of an entire offshore installation. The purpose of the QRA is to support a decision about risk acceptance and implementation of risk reducing measures. One way to present a broad risk picture is to report several risk and vulnerability indices. Let the unknown quantity X0(t) take the value 1 if a randomly selected person is killed during a period of t years and 0 otherwise. Furthermore, let N(t) denote the number of fatalities during a period of t years, and let T be the period of operation. Some examples of relevant risk and vulnerability indices which as a whole represent a broad risk picture are listed below.
P(X0(1) = 1|K) Probability of death of a randomly selected person during one year
E[N(1)|K] Expected number of fatalities during one year
P(N(T) > 1|K) Probability of one or more fatalities during the period of operation
E[N|I,K] Expected number of fatalities (during a specific period of time) given an
initiating event
P(N < n|I,K) Probability distribution of the number of fatalities (during a specific period of time) given an initiating event
E[N(1) > 5|K] Fraction of expected loss of lives owing to accidents with 5 or more fatalities E[N(1)|K] (sometimes referred to as major accidents)
Company acceptance criterion
Suggested boundary unacceptable/just tolerable risk for any substantial category of workers for any large part of a working life (Health and Safety Executive, 2001)
- About the observed individual risk on the Norwegian continental shelf (1996-2005) (Vinnem, 2007)
- Suggested boundary unacceptable/just tolerable risk for members of the public who have a risk imposed on them "in the wider interest of society" (Health and Safety Executive, 2001)
IR = 10-3
H Sensitivity ™ Calculated IR
IR = 10-4
IR = 10-5
Very low level of risk. Guideline boundary between tolerable/ broadly acceptable risk for both workers and the public |-|- ir = 10-6
(Health and Safety Executive, 2001)
Figure 2. Risk level, reference values and sensitivity in QRA example
The first two indices are commonly referred to as individual risk (IR) and potential loss of life (PLL), respectively. Suppose that the individual risk on the installation is calculated to be 2-10"4, and that this number proves to be relatively insensitive to changes in the assumptions made, databases used, etc. Presented together with reference values, as shown in Figure 2, the relatively robust message sent to the decision-maker is that the calculated individual risk is in the area of or slightly above the industry average.
The risk picture described above covers the probability and sensitivity components. Valuable insights are provided by these numbers, but considering the arguments in Section 3, a total risk picture needs to include a broader reflection of uncertainties.
Suppose that the risk description involves a prediction of the number of gas leaks Y(1) during one year, and that two sources of information regarding gas leaks are available. Use of the two sources results in markedly different values of E[Y(1)]. The difficulty in predicting Y(1) indicates that there are important aspects of the underlying phenomena that are not well understood. The number of gas leaks clearly has an effect on risk. This assertion can easily be supported by considering the typical modelling of gas leaks in offshore QRAs. A gas leak is a natural choice as initiating event in an event tree model, in which case the value of a risk index derived from the event tree will be proportional or approximately proportional to the number of gas leaks. The number of gas leaks then is recognised as an uncertainty factor with a significant effect on risk.
The total risk picture in a QRA today consists of probabilistic risk indices (P), sensitivity analyses (S) and a list of assumptions (part of K). What we suggest is an adjustment of this practice, where the description of risk is extended from probabilistic indices and sensitivity analyses, relying heavily on the validity of the background knowledge, to including the identification and classification of uncertainty factors in the background knowledge; see Figure 3.
(P,S,U) High confidence in avoiding fatal accidents?
Risk reducing measure M
(P,S,U|M) Changed confidence in avoiding fatal accidents?
Figure 4. Evaluation of risk reducing measures
Consider the assessment of a risk reducing measure M. The starting point is the situation without the measure implemented, i.e. the risk description covering (P,S,U). The question is whether the risk description (P,S,U|M) changes (increases) the confidence of the decision-maker in achieving the goals set, e.g. to avoid fatal accidents; see Figure 4.
Cost-benefit and cost-effectiveness indices such as the expected net present value (E[NPV]) and the implied cost of averting a fatality (ICAF) are commonly used to evaluate risk reducing measures. The expected net present value is defined as the expected discounted cash flow, and the ICAF is defined as the expected cost per expected number of averted fatalities (Aven, 2008b). Both indices belong to the probability component (P) of risk, and as such only constitute part of the total risk picture. In addition, sensitivity analysis plays a key role in the investigation of whether the conclusions drawn from the E[NPV] and the ICAF calculations are easily altered by changes to the assumptions made, data used, etc. The probability and sensitivity components (P,S) do not, however, take into account that the uncertainties are not properly captured by the calculated risk. An analysis of uncertainty factors with and without the risk reducing measure implemented is a way of achieving better insights into the effect of the risk reducing measures on these uncertainties, and thus cover the total risk picture (P,S,U).
5 CONCLUSIONS
It was pointed out in the introduction that the purpose of a QRA is to support decisions. The decision support role of QRA is emphasised by Falck et al. (2000) and discussed by Apostolakis (2004). We have argued that in order to provide a broad, informative and balanced picture of risk, the risk description needs to cover the components (A,C,C*,U,P,S,K). In the paper special attention has been given to the components (P,S,U), as we are concerned with the expression and communication of uncertainty.
Uncertainty is seen as a main component of risk and the risk description. The reason is that probability is only a tool for expressing uncertainty. Surprises relative to the assigned probabilities could occur if the background knowledge on which the probabilities are conditioned turns out to be wrong. Note that this is not to say that the probabilities themselves are uncertain. Probabilities are expressions of uncertainty based on a particular background knowledge. If the background knowledge changes, then the probability assignment might also change. However, for a given background knowledge the probability is not uncertain.
We propose a simple, practical method to classify uncertainty factors and thus characterise uncertainties that are not properly captured by probabilities and probabilistic risk indices. If left untreated these uncertainties could be lost and reduce the confidence in the analysis.
Extending the risk description to go beyond probabilities and expected values means that to evaluate risk acceptance and risk reducing measures (possibly with reference to the ALARP principle), the perspective of the decision-maker must be extended beyond pre-defined risk acceptance criteria and cost-benefit/effectiveness indices. Such criteria and indices as a rule cover only the probability component of risk. They do not take into account that a measure which reduces uncertainty could represent a risk reduction, even if a particular risk index remains more or less
unaffected by the measure. Presented with a risk description covering the components (P,S,U), the decision-maker must make an overall review and judgement.
ACKNOWLEDGEMENT
The work on the paper has been funded by The Research Council of Norway through the PETROMAKS research programme. The financial support is gratefully acknowledged.
REFERENCES
1. Apostolakis, G.E. 2004. How useful is quantitative risk assessment? Risk Analysis 24(3): 515-520.
2. Aven, T. 2008a. A semi-quantitative approach to risk analysis, as an alternative to QRAs. Reliability Engineering and System Safety 93(6): 790-797.
3. Aven, T. 2008b. Risk Analysis - Assessing Uncertainties beyond Expected Values and Probabilities. New York: Wiley.
4. Aven, T. 2003. Foundations of Risk Analysis - A Knowledge and Decision-oriented Perspective. Chichester: Wiley.
5. Bedford, T. & Cooke, R. 2001. Probabilistic Risk Analysis - Foundations and Methods. Cambridge: Cambridge University Press.
6. Falck, A., Skramstad, E. & Berg, M. 2000. Use of QRA for decision support in the design of an offshore oil production installation. Journal of Hazardous Materials 71: 179-192.
7. Helton, J.C. & Burmaster, D.E. (eds). 1996a. Reliability Engineering and System Safety 54(2-3): 91-262. Special issue on treatment of aleatory and epistemic uncertainty.
8. Helton, J.C. & Burmaster, D.E. 1996b. Guest editorial: treatment of aleatory and epistemic uncertainty in performance assessments for complex systems. Reliability Engineering and System Safety 54: 91-94.
9. Health and Safety Executive. 2001. Reducing risks, protecting people, HSE's decisionmaking process (R2P2). London: HSE books. http://www.hse.gov.uk/dst/r2p2.pdf
10. ISO. 2002. Risk management vocabulary. ISO/IEC Guide 73.
11. Jonkman, S.N., van Gelder, P.H.A.J.M. & Vrijling, J.K. 2003. An overview of quantitative risk measures for loss of life and economic damage. Journal of Hazardous Materials 99(1): 1-30.
12. Lindley, D.V. 2006. Understanding Uncertainty. Hoboken, NJ: Wiley.
13. Schofield, S. 1998. Offshore QRA and the ALARP principle. Reliability Engineering and System Safety 61: 31-37.
14. Vinnem, J.E. 2007. Offshore Risk Assessment: Principles, Modelling and Applications of QRA Studies. 2nd ed. London: Springer.
