CC BY
6
RISK ANALYSIS ON THE BASIS OF JUDGMENTS SUPPLIED BY
UNKNOWN EXPERTS
Lev V. Utkin, Sergey P. Shaburov •
"Tehtrans" JSC Fontanka River Quay, 117, St.Petersburg, 190031, Russia e-mail: lvu@techrw.spb.su, spsh@techrw.spb.su
The development of a system requires fulfilling the available standards of reliability and safety. Due to possible complexity of the system, its parameters often are determined by experts whose judgements are usually imprecise and unreliable due to the limited precision of human assessments. Therefore, an approach for computing probabilities of expert judgments and for analysing the risk of decision about satisfying the parameters to standards of reliability and safety is proposed in the paper. A numerical example considering a microprocessor system of central train control illustrates the proposed approach.
Keywords: expert judgments, imprecise probabilities, multinomial model, Dirichlet distribution, risk analysis, reliability and safety.
INTRODUCTION
The development of a system requires fulfilling the available standards of reliability and safety. Due to possible complexity of a system, it is difficult to precisely assess the system parameters characterizing its reliability and safety. Therefore, very often these parameters are determined by experts. Judgments elicited from human experts may be a very important part of information about systems on which limited experimental observations are possible. At the same time, they are usually imprecise and unreliable due to the limited precision of human assessments. When several experts supply judgments or assessments about a system, their responses are pooled so as to derive a single measure of the system behaviour. Judgments of reliable experts should be more important than those of unreliable ones. Various methods of the pooling of assessments, taking into account the quality of experts, are available in the literature [1]. These methods use the concept of precise probabilities for modelling the uncertainty and the quality of experts is modelled by means of weights assigned to every expert in accordance to some rules. It should be noted that most of these rules use some available information about correctness of previous expert opinions. This way might meet several difficulties. First, the behaviour of experts is unstable, i.e., exact judgments related to a system elicited from an expert do not mean that this expert will provide results of the same quality for new systems. Second, when experts provide imprecise values of an evaluated quantity, the weighted rules can lead to controversial results. For instance, if an expert with a small weight, say 0.1, provides a very large interval, say [0,10], for a quantity (covering its sample space), it is obvious that this expert is too cautious and the interval he supplies is non-informative, although this interval covers a true value of the quantity. On the other hand, if an expert with a large weight, say 0.9, supplies a very narrow interval, say [5,5.01], the probability that true value of the quantity lies in this interval is rather small. We can see that the values of weights contradict with the probabilities of provided intervals. It should be noted that sometimes we do not know anything about quality of experts or assignment of weights meets some ethical difficulties. This implies that weights of experts as measures of their quality can not be measures of the quality of provided opinions.
How in this case to compare the assessed system parameter with the available standards of reliability and safety? How to compute the risk of decision making after this comparison?
The main aim of the paper is to develop an approach for computing probabilities of expert judgments and to provide a tool for risk analysis taking into account these probabilities. At that these probabilities are not regarded as a result of the previous expert experience, but as a result of recent judgments provided by unknown experts. The experts are unknown in the sense that we have no prior information about their quality.
What are conditions for probabilities of judgements? First, they have to take into account the incompleteness of the available information and even total ignorance. Second, the probabilities have to take into account the overcautiousness of experts when they supply too large and non-informative intervals. Third, the probabilities have to take into account the overconfidence of experts when they supply intervals that are too narrow (or point-values) [3]. Fourth, the probabilities have to be simply updated after obtaining new judgments. Fifth, the probabilities are assigned not to experts, but to intervals provided by the experts. The first, second, and third conditions can be satisfied if to use imprecise or interval-valued probabilities [4,5,7]. The fourth condition is fulfilled if to assume that probabilities of intervals are governed by the Dirichlet distribution.
STATEMENT OF THE PROBLEM AND THE BASIC IDEA FOR ITS SOLUTION
Suppose that N experts assess a parameter u of a system defined on U = {up..., uL}. They supply a set of intervals {A1,A2,..., AN} of u such that every interval A. c U contains elements from Uwith indices J., i.e., A = {u : j e J }. At that, the number of elements in J. is L . Let u0 be a value of the standard safety. Our aim is to find probability that u is smaller than u0, i.e., Pr{u < u0}.
Suppose that the set {Ap...,AN} has identical elements such that there are cl, c2,...,cn identical intervals. Here n is a number of different intervals. Then N = ^n=i ci. Let us calculate possible numbers of occurrences of every element of U. Associate the set A. with an oblong box of size l. with one open side and the set U with L small empty boxes of size 1. The i-th oblong box contains c. balls which can move inside the box and we do not know location of balls in the i-th box because its open side is behind. Then we cover small boxes by the i-th oblong box and c. balls enter in l. small boxes with numbers from a set J t. We do
not know exact location of balls, but we know that they are in boxes with numbers from Ji . The same procedure is repeated n times. What can we say about possible numbers of balls in the small boxes now? It is obvious that there exist different combinations of numbers of balls except the case when l. = 1 for i = 1,..., n,
i.e., all sets A. consist of one element. Suppose that the number of the possible combinations is M. Denote the k-th possible vector of balls by n(k) = (n[*),...,n[t}), k = 1,...,M. If to assume that the sets A. occurred independently and a ball in the i-th small box has some unknown probability n., then every combination of
balls in small boxes produces the standard multinomial model. M possible combinations of balls produce M equivalent standard multinomial models. The models are equivalent in the sense that we can not choose one of them as a more preferable case.
For every model, the probability of an arbitrary event A c U depends on n(k), that is, we can find P(A | n(k)). So far as all the models are equivalent, even by precise probabilities of all categories only lower and upper probabilities of A can be computed
P(A) = min P(A | n(k)), P(A) = max P(A | n(k)).
k=1,...,M k=1,...,M
In particular, if all sets A. consist of single elements, that is, all oblong boxes are of size 1, then M=1 and
P(A) = P(A | n(k)), P(A) = P(A | n(k)). The following problem is to define n(k) and P(A | n(k)). In the case of multinomial samples, the Dirichlet distribution is the traditional choice.
Remark 1. It is worth noticing that the Dirichlet distribution should be regarded as one of the possible multinomial models that can be applied here.
Remark 2. Even if experts provide only characteristics of separate components of the system, their use leads to calculation of system parameters which also can be regarded as expert judgements (functions of expert judgements).
Remark 3. If U is some interval of real numbers, then we can always transform this universal set to a set with finite numbers of elements. Suppose that we have to find probabilities of an event A. Denote An+1 = A .
Let {i} = {(/p...,in,in+i)} be a set of all binary vectors consisting of n+1 components such that i. e {0,1}. For
every vector i, we determine the interval Bk (k = 1,...,2n ) as follows:
f \ f \
J
n+1 >
B =
I AI I A
j
VJ : 'j =0 J
'.
J
V J : i,=1 y
As a result, we obtain a set of non-intersecting intervals Bk such that B1 <u... <u B2n+1 = U . Moreover, all intervals A. can be represented as the union of a finite number of intervals Bk. This implies that every interval Bk corresponds to an element uk of the transformed universal set U* with the finite number (2n+1) of elements.
IMPRECISE DIRICHLET MODEL
The Dirichlet (s,a)prior distribution for n, where a = (a1,...,aL), has probability density function
L
p(n) = C(s,a) • nn-1, s > 0 , a e S(1, L),
j=1
where S(1,L) denotes the interior of the unit simplex, the proportionality constant C is determined by the fact that the integral of p(n) over the simplex of possible values of n is 1, a, is the mean of %. under the
Dirichlet prior and s determines the influence of the prior distribution on posterior probabilities.
Walley [6] pointed out several reasons for using a set of Dirichlet distributions to model prior ignorance about probabilities n:
1) Dirichlet prior distributions are mathematically tractable because they generate Dirichlet posterior distributions;
2) sets of Dirichlet distributions are very rich, because they produce the same inferences as their convex hulls and any prior distribution can be approximated by a finite mixture of Dirichlet distributions;
3) the most common Bayesian models for prior ignorance about probabilities n are Dirichlet distributions.
The imprecise Dirichlet model is defined by Walley [6] as the set of all Dirichlet (s,a) distributions such that ae S(1, L). For this model, the hyperparameter s determines how quickly upper and lower probabilities of events converge as statistical data accumulate. Walley [6] defined s as a number of observations needed to reduce the imprecision (difference between upper and lower probabilities) to half its initial value. Smaller values of s produce faster convergence and stronger conclusions, whereas large values of s produce more cautious inferences. At the same time, the value of s must not depend on L or a number of observations. The detailed discussion concerning the parameter s and the imprecise Dirichlet model can be found in [2,6].
By returning to the multinomial models considered in the example with boxes and balls and assuming that probabilities of balls are governed by the Dirichlet distribution, we can write the lower P(A, s) and
upper P(A, s) probabilities of an event A, whose elements have indices from a set J, as follows:
P( A ) . . f nkA) + sa(A) P( A ) n(k>(A) + sa(A) P(A, s) = min inf -, P(A, s) = max sup -,
k=1,...,M aeS(1,L) N + s k=1,-,M aeS(1,L) N + s
where a(A) = £ aJ, n(k)(A) = £
( k )
n .
J
J^J J^J
ANALYSIS OF EXPERT JUDGMENTS
Now we have to find n(k)(A) and a(A). The lower and upper probabilities P(A, s) and P(A, s) can be rewritten as
t)/ A X mink=1,...,M n(k)(A) + s • inf aES(1,L) a(A) -s. , maxk=1,...,M n(k)(A) + s • supaES(1,L) a(A)
P( A, s) =-—-, P (A, s) =-—-.
N + s N + s
Note that infaeS } a(A) is achieved at a(A) = 0 and supaeS(1L} a(A) is achieved at a(A) = 1 except a case when A=U. If A=U, then a( A) = 1 for the minimum and maximum.
In order to find the minimum and maximum of n(k)(A) we consider three intervals A1, A2, A3 such that A1 c A, A n A = 0, A3 n A ^ 0 and A3 & A . Numbers of their occurrences are c1, c2, c3, respectively, and c1 + c2 + c3 = N . It is obvious that all balls (c1) corresponding to the set A1 belong to the set A and n(k-1 (A) can not be less than c1. On the other hand, all balls (c2) corresponding to the set A2 do not belong to A. This implies that n(k)(A) can not be greater than N - c2. A part of balls corresponding to A may belong to A, but it is not necessary. Therefore, minn(k}(A) = c1 and maxn(k}(A) = N - c2. By extending this reasoning on an arbitrary set of A., we get the minimal, denoted L1 (A), and maximal, denoted L2( A), values of n(k >( A):
A(A) = min n<k)(A) = V c., L2(A) = max n(k)(A) = N - V c. = V c..
k=1,...,M ^^ k=1,...,M ^^ ^^
i : A.i c A i : A. nA=0 i : A. nA^0
Then there hold
P(A,s) = j&L. P(A,s) = LN++±.
N + s N + s
SPECIAL CASES
Let us consider some important special cases.
1) Coinciding judgments. Suppose that there are N coinciding judgments, i.e., A1 = ... = AN = [a, a]. This means that all experts have the same opinions about unknown value of u. Then we can write P(A.) = N /(N + s), P(A.) = 1. If N , then there holds P(At) = P(At) = 1. This property supports the intuitive sense. Indeed, if we have many identical judgments, we begin to think that these judgments are true even if we do not know anything about each expert. If N = 1, then there hold P(Ai) = 1/(1 + s), P(Ai) = 1. The result corresponding to the case N = 1 shows that the precise Dirichlet model (s = 0) gives lower and upper probabilities of events 1. This is the incorrect conclusion. How can we totally believe one judgment? This contradiction can be avoided by using the imprecise Dirichlet model (s > 0).
2) Conflicting judgments. Suppose that there are two conflicting judgments A1 = [a1, a1], A = [a2,a2], a <a2, i.e., A nA2 =0 . Then there hold P(A.) = 1/(2 + s), P(A.) = (1 + s)/(2 + s). If there are N conflicting judgments, then P(A.) = 1/(N + s), P(A.) = (1 + s)/(N + s). If N , then there holds P( A) = P( A) = 0 .
3) Noninformative judgments (overcautiousness of experts). Suppose that there is one judgment
A1 = [inf U,supU]. Then P(A1) = P(A1) = 1. This implies that the overcautiousness of experts can be
properly modelled. Indeed, even if we do not believe to an expert, but he (she) provides very overcautious judgments, then probabilities of these judgments have to be large though the expert is unreliable.
RISK ANALYSIS
If there is the standard value u0 of reliability or safety, then the parameter u can be compared with u0 by means of computing the probability distribution function at point u0. Suppose that U is the real line restricted by some values infU and supU. Then we can define lower and upper cumulative probability distribution functions of a random parameter u, about which we have data in the form of intervals Ai , i = 1,...,n . By using the above results and taking into account the fact that a(A) = 1 by A=U, we get
1
F(u0,s) = P({u e U : u < u0},s) = <
N + s A : sup A. <u
V c, u0 < supU
;up A. <u0
1, u0 = sup U
F(u0,s) = P({u e U : u < u0},s) = <
—s--1--1— V c , u0 > inf U
N + s N + s A :f <u0
0, u0 = inf U
The above expressions are obtained by considering lower and upper probabilities of the interval A = [inf U, u0]. If it is necessary to find probabilities of the complementary event {u > u0}, then the following equalities can be used: P({u e U : u > u0},s) = 1 -P({u e U : u < u0},s) ,
P({u e U : u > u0},s) = 1 -P({u e U : u < u0},s).
Example. We consider a microprocessor system of central train control "Tract" developed by "Tehtrans" JSC. According to specialized standard related to safety of such systems, the probability of a hazardous failure during 10 years must be less than 2.6*10-6, i.e., u0=2.6*10-6. In order to prove the safety of the developed microprocessor system, 10 experts supply indirectly intervals for probabilities of hazardous failures of the system (see Table 1). The intervals Ai are given in the second column. Numbers of identical
intervals c. are given in the third column. Values of L1(i) and L2(i) are given in the fourth and fifth
columns. If we take s = 1, then P( A. ,1) and P( A. ,1) are given in the sixth and seventh columns. By using
the expressions for lower and upper distribution function, we find lower P({u < 2.6 x 10-6},1) and upper
P({u < 2.6 x 10-6},1) probabilities that the system parameter (probability of a hazardous failure) less than 2.6x10-6:
P({u < 2.6 x 10-6},1) = 9/11 = 0.82 , P({u < 2.6 x 10-6},1) = 1.
It is obvious that risk of decision should be calculated on the basis of the lower probability. Therefore, we get the value of risk 1-0.82=0.18. This is rather large value and, therefore, it is necessary to carry out some additional expert elicitation or to improve the system, for instance, by using additional redundancy of main components that has been done.
Table 1. Expert intervals and their probabilities
i A x10-6 c A(i) L2(j) P( A ,1) P( A ,1)
1 [0.6, 1.6] 3 5 8 0.45 0.82
2 [0.1, 1.6] 1 8 10 0.73 1
3 [0.0, 3.1] 1 10 10 0.91 1
4 [0.6, 2.6] 1 6 8 0.55 0.82
5 [0.6, 1.1] 2 2 8 0.18 0.82
6 [0.1, 0.6] 2 2 4 0.18 0.45
CONCLUSION
The method for analyzing the expert judgments has been considered in the paper. On one hand, it is very simple from computational point of view. On the other hand, it does not use information about experts and takes into account imprecision of expert information and deficiency in statistical data. The application of imprecise Dirichlet model allows us to avoid possible errors of traditional statistical analysis under condition when the number of available judgments is rather small. The resulting assessments can be simply updated after obtaining new expert judgments. It is worth noticing that the method can be simply extended on a case of heterogeneous judgments when experts provide information different in kind. Moreover, the method also can be extended on a case of experts with known parameters of their quality.
REFERENCES
1. Cook, R.M. Experts in Uncertainty. Opinion and Subjective Probability in Science. Oxford University Press, New York, 1991.
2. Coolen F.P.A. An imprecise Dirichlet model for Bayesian analysis of failure data including right-censored observations. Reliability Engineering and System Safety, 56, 61-68, 1997.
3. D. Dubois and H. Kalfsbeek. Elicitation, assessment and pooling of expert judgement using possibility theory. In C.N. Manikopoulos, editor, Proc. of the 8th Inter. Congress of Cybernetics and Systems, pages 360-367, Newark, NJ, 1990. New Jersey Institute of Technology Press.
4. V.P. Kuznetsov. Interval Statistical Models. Radio and Communication, Moscow, 1991. in Russian.
5. P. Walley. Statistical Reasoning with Imprecise Probabilities. Chapman and Hall, London, 1991.
6. P. Walley. Inferences from multinomial data: learning about a bag of marbles. Journal of the Royal Statistical Society, Series B, 58:3-57, 1996.
7. K. Weichselberger. Elementare Grundbegriffe einer allgemeineren Wahrscheinlichkeitsrechnung, vol. I Intervallwahrscheinlichkeit als umfassendes Konzept. Physika, Heidelberg, 2001.
