CC BY
39
EXPERIMENTAL COMPARISON OF TEST DERIVATION METHODS FOR AN EMBEDDED FSM
KOUFAREVA I., TRENKAEV V., YEVTUSHENKON.
Tomsk State University Tomsk, Russia
irene@molsoft.home.tomsk.net,
{vad, yevtushenko}@elefot. tsu.ru
Abstract. The paper presents experimental results on comparison of test generation techniques for a system of interacting FSMs with at most one faulty component. We consider a test suite derived by enumeration of all faulty systems and compare the obtained test suite with those derived by techniques based on an embedded equivalent and a mutation machine. We also estimate fault coverage of a test suite traversing each transition in the component of our interest.
1. Introduction
A complex system is often specified as a collection of interacting components. If a behavior of each component of the collection is described by an FSM then a behavior of the overall system can be described by a composed FSM [1]. In this case, a complete test suite for a system of interacting FSMs can be derived using diverse FSM-based techniques started by Hennie [Hennie64] and its modifications. The methods deal with a huge fault domain of all machines with a restricted number of states, and return high quality tests. However, in practical situations the obtained tests become too long to be of a practical use. By these reasons, another more practical fault model for communicating FSMs [see for example, Yevt91] is considered in a number of papers. According to that model, a system structure is preserved in all implementations and at most one component can be faulty. Under these assumptions it is possible to develop test derivation methods for a system w.r.t. single-component faults; such methods are also called test derivation methods for a component FSM. Given a complete test suite for each component FSM w.r.t. an appropriate fault domain, we merge them into the unique test suite that is complete w.r.t. the considered fault model. The considered fault model is good enough since a probability that at most one component FSM is faulty is much more than that of two and more machines. Moreover, it is well known tests complete w.r.t. single faults usually capture a lot of other faults. On the other hand, the fault model enables to deal with a component FSM that usually is much less than the overall system and derive test suite w.r.t. different fault domains for different component F SMs. Given a system of interacting FSMs and a fault domain of a component FSM, there are a number of methods for test derivation for a component FSM.
In this paper, we present the results of computer experiments of test suite derivation using these methods. We calculate fault coverage w.r.t. stuck-at-faults of a test suite traversing each transition of a component FSM and estimate test suites which are derived based on a reduced form of a so-called e -
machine [1, 2] (an embedded equivalent [Petr97]) and on a mutation machine [PoRe97, Kouf99].
The rest of the paper has the following structure. Section 2 comprises preliminary definitions. Methods for test derivation for a component FSM are shortly sketched in Section 3. Section 4 presents the results of performed experiments.
2. Finite State Machines
Finite state machine (FSM) A, usually simply called a machine throughout this paper, is a quintuple < S, I, O, h, s0 >, where S is a finite nonempty set of states with the initial state s0, I and O are input and output alphabets, and h c S x I x O x S is a behavior relation. There is a transition from state s to state s' in FSM A under Input/Output pair I/O if (s, i, o, s) e h. If for each pair (s,i) e S x I there is at most single pair (s',o) e Sx O such that (s,i,o,s') e hthen FSM a is said to be deterministic. Ifforeach pair(s,i) e S x I there exists (s',o) e S x O such that (s,i, o, s') e h then FSM A is said to be complete; otherwise, A is called a partial FSM. FSM B =< S',I,O,g,s0 > , S c S, is a submachine of A if g c h. Similarto [Kouf99], given a complete FSM A, we further denote Sub(A) the set of all complete deterministic sub-machines of FSM A.
As usual, relation h can be extended to the sets I and O , that is h c S x I x O x S. An Input/Output sequence a / P such that (s0, a, P,s) e h for some state s e S is called a
trace of A.
Given two machines B and A, FSM B is a reduction of A if the set oftraces ofFSM B is a subset ofthat of A, written B < A . FSMs B and A are equivalent, written B = A, if their sets of traces coincide.
3. Testing the head component FSM of a series composition
In this section, we sketch methods for test suite derivation for a component FSM. To compare the techniques we consider a series composition of two FSMs where only the head FSM can be faulty and assume that only output faults can occur in the component implementation.
3.1. Output faults
Test suite derivation methods based on output faults are widely used. Moreover, in [David93] it is theoretically shown that for an appropriate FSM class, a test suite complete w.r.t. output faults detects “almost all” faults.
There are a number of papers where fault coverage of a transition tour that detects all output faults, is estimated w.r.t. other fault domains [Heer2000]. In this paper, given a digital circuit, we estimate fault coverage of a transition tour of an FSM that describes a behavior of the digital circuit, w.r.t. stuck-at-faults and input variable permutation. A test suite that captures more than 95% of single stuck-at-faults is known to be good enough for detecting functional faults in a digital circuit. The results of performed experiments (Table
1) show that usually a transition tour detects about 96% of single stuck-at-faults and 99 % of input variable permutations.
64
R&I, 2003, Ns 3
T o formally describe a set of output faults of a deterministic machine A =< S, I, O, h, sg > , we introduce a so-called mutation machine [Kouf99] Ma =< S, I, O, H,sg > where for each transition (s,i, o,s') e h and each output o'e O the transition (s, i, o, s') is in the relation H. Any implementation ofA with output faults is a deterministic submachine of M a, ie. the setSub(M a) is the fault domain for the machine A .
3.2. Series composition of FSMs
Consider a system in Figure 1 that is the series composition of two deterministic component FSMs. Let A =< S, I, U, h,sg > and B =< T, U, O,g, tg > be the head and tail component machines of the system.
I
A
Q
Fig. 1. Series composition of FSMs.
A behavior ofthe overall system is described with a composed machine A • B =< S x T, I, O,f, sgtg > where for each two pairs st,s't'e S x T, each input i'e I and each output o e O it holds that
(st,i,o,s't') £ f » 3u £ U[(s,i,u,s') e h & (t,u,o, t') e g].
Moreover, the tail component B is supposed to be faulty-free while a head component ’ s implementation is a submachine of a nondeterministic FSM M a . We are required to construct a test suite w.r.t. output faults of the head component FSM,
i.e. a test suite w.r.t. a fault domain Sub(Ma ) • B , where Sub(Ma)• B denotes the set of all machines A'»B, A'e Sub(M a) . Test suite is said to be complete w.r.t. fault domain Sub(Ma)• B if for each FSM A'• B, A'e Sub(Ma) , that is not equivalent to A• B there is a test pattern that detects the FSM. Since number of faulty systems is finite a complete test suite can be derived by explicit enumeration of all faulty systems.
Another approach is based on a transition tour of the component FSM A that is known to be a complete test suite w.r.t. output faults for a component in isolation. The results of our experiments clearly show that if no access to internal outputs is granted, the coverage of a test suite derived in this manner can be very low, about 70% of output faults of the component FSM (see Table 2). More rigorous analysis must be performed to study properties of interacting FSMs for which methods based on traversing transitions of each component FSM return tests with satisfactory fault coverage.
The experiments also have been performed for test suite derivation based on a so-called embedded equivalent and a mutation machine. These methods do not require explicit enumeration of faulty systems as for test generation w.r.t. stuck-at-faults, and are based on the fault domain of the head component machine and are known to return complete test suites. We briefly present them in the next section.
R&I, 2003, N 3
3.3. Test derivation based on the fault domain ofthe head component machine
Consider composition ofFSMs A and B in Figure 1. It is well known not each faulty implementation of the head component FSM can be detected if no access is granted to the internal output U of the head component FSM. The set of FSMs which can replace the head component FSM without changing a behavior of the overall system, can be described as the set of all reductions of an appropriate nondeterministic FSM called here the embedded equivalent. The technique for derivation of the embedded equivalent of the head component FSM is presented in [1-3].
Let A =<S,I,U,h,sg > and B =<T,U,O,g,tg > be
deterministic head and tail component FSMs while the composed FSM is the machine
C = A • B =< S x T,I,O,f,sgtg >,Q c S x T,qg = sgtg The embedded equivalent Ag is the machine < Q X C hE,qgtg > where for each two pairs qt,q't'e Q x T , each input i e I and each output u e U it holds that
(qt, i, u, q't') e hE » 3o e O[(q, i,o, q') e f & (t, u,o, t') e g].
A complete test suite derived fromnondeterministic embedded equivalent Ag w.r.t. the fault domain is known [Kouf.Evt97] to be complete w.r.t. output faults of the head component FSM. However, generally the embedded equivalent is a partial FSM and the latter make s difficult te st generation. The author of [Trenthesis] suggests a technique for a complete test suite generation based on the largest complete submachine of Ag [1, 2] that is called the reduced embedded equivalent and denoted Agg .
Here we notice it s possible to escape derivation of the composed FSM if an appropriate approximation of the embedded equivalent is enough for test derivation. For example, a so-called k-approximation of an embedded equivalent can be derived by direct simulation of a reference system under input sequences of length k. However, in this case, if the system under test passes a test suite we can guarantee the correctness of a component implementation only w.r.t. an input sequence of length up to k. Additional research is needed on how to construct an embedded equivalent if the composed FSM is unavailable due its huge dimension.
Another approach to test suite derivation also based on the fault domain of the head component FSM, was presented in [Petr96b]. The idea behind the approach is to describe a set of all faulty composed machines A' • B , A'e Sub(Ma), using the set of deterministic sub-machines of the machine Ma • B. Since the set Sub(Ma)• B is a subset of Sub(Ma • B), a test suite for the composed machine C = A • B complete w.r.t. the fault domain Sub(Ma • B) is also a complete test suite w.r.t. Sub(Ma ) • B .
In other words, a complete test suite for a component FSM can be derived as a complete test suite w.r.t. the fault domain
Sub(M) where M is a nondeterministic FSM such that the set of all deterministic sub-machines of M comprises each
65
Table 1. Experimental estimation of a transition tour w.r.t. stuck-at and input permutation faults.
Name 5 i o Single stuck-at- faults (enum.) Permutation faults (enum.) Output faults (transition tour)
t l t l t l SAF PF
1 2 3 4 5 6 7 8 9 10 11 12
bbara 10 4 2 7 139 1518 1876 0 716 95.6 100
bbtas 6 2 2 0 124 2 60 0 88 74.2 100
beecount 7 3 4 3 105 16 26 0 182 100 100
dk14 7 3 5 1 97 14 12 0 162 100 100
dk15 4 3 5 1 89 1 10 0 89 100 100
dk17 8 2 3 1 99 1 8 0 96 97.5 100
dk27 7 1 2 1 39 0 9 0 36 90 100
dk512 15 1 3 4 105 6 13 0 83 89.5 100
donfile 24 2 1 2 119 56 522 0 334 100 97.2
ex2 19 2 2 21 19 342 11 0 35 99.2 100
ex3 10 2 2 9 20 26 10 0 28 98.5 100
ex5 9 2 2 4 33 12 6 0 57 93.6 100
ex6 8 5 8 2757 84 - - 0 863 100 -
ex7 10 2 2 3 22 10 9 0 31 96.5 100
lion 4 2 1 0 38 0 7 0 42 100 100
lion1 4 2 1 0 37 1 7 0 47 100 100
lion9 9 2 1 1 107 11 13 0 113 95.3 100
mc 4 3 5 0 103 1 16 0 103 100 100
modulo12 12 1 1 0 19 0 76 0 103 100 85
faulty implementation. Methods for derivation of such a test suite are proposed in [Kouf99, PoRe97].
4. Experimental results
In this section, we present the results of computer experiments comparing the above techniques for test derivation. They were performed using Pentium II machine and software tools designed with Borland C++ and Microsoft Visual C++ 6.0 compilers. For the experiments we use random generated FSMs as well as benchmarks [Yang91].
4.1. Estimation of a transition tour of an FSM
As mentioned above, given an isolated FSM, a transition tour is known to be a complete test suite w.r.t. output faults ofthe FSM. In Table 1, we present experimental results of a transition tour estimation w.r.t. stuck-at and input permutation faults in combinational logic of digital circuits. As we discussed above, test suites detecting such faults capture a lot of functional faults in digital circuits and are widely used in practice.
Given a digital circuit, three different test suites are considered. One of them is a transition tour of the digital circuit while two other are derived by explicit enumeration of single stuck-at-faults and input variable permutations respectively. We compare test suite length and time period necessary for its derivation.
We consider 19 benchmarks in .kiss-format [Yang91]. We denote s,i and o numbers of states and input/output Boolean variables of the benchmark respectively (columns 2-4 of Table 1). Columns 5-6 ofthe table contain derivation time (in seconds) and total length of a complete test suite w.r.t. single stuck-at-faults in the combinational logic of the circuit obtained by enumeration of such faults. Columns 7-8 contain derivation time and total length of a complete test suite w.r.t. permutation faults in the combinational logic that also is obtained by fault enumeration. Columns 9-12 comprise information about a transition tour that is known to be complete w.r.t. output faults. SAF labels column 11 containing fault coverage of the transition tour w.r.t stuck-at-faults, while PF (column 12) is the coverage w.r.t input permutation faults. For all considered benchmarks, test derivation based on a transition tour takes less than 1 sec. By direct inspection one can assure a transition tour has good fault coverage w.r.t. traditional fault domains, its length is comparable with that of tests obtained by enumeration, while its derivation is much easier.
4.2. Testing the head component of a series composition
Fault coverage of a transition tour of the head component machine. We estimate fault coverage of a transition tour of the head component FSM A w.r.t. its output faults. A transition tour is known to detect all such faults in the isolated FSM A. However, as mentioned above, when A is embedded in the series composition some output faults of A may cause transfer faults in the composed FSM as well as increase its number of states. By these reasons, output faults of the head component FSM can become latent and undetected by a transition tour. The results of performed experiments are represented in Table 2. We denote s, i and o number ofstates, inputs and outputs of component machines, respectively (columns 1-5). The results shown in Table 2 represent average
66
value of fault coverage over 100 series composition of corresponding dimension. The results clearly show sometimes a transition tour detects less than 70% of single output faults ofthe head component FSM. The reason is the tail component machine masks the output response of the head component machine to a test case.
Test suite derivation based on embedded equivalent and mutation FSM. We consider the series composition of two randomly derived FSMs with various numbers of states, inputs and outputs. In Table 3, numbers of states of the head and tail component FSMs are shown in columns 1 and 3 respectively, while columns 2 and 4 comprise number of their inputs.
We first consider a test suite derivation based on explicit enumeration of all faulty composed machines A'»B, A' e Sub(Ma ). Average time of test derivation and length of such tests are shown in columns 5-6 of Table 3. Apparently, the approach can be applied only for small component machines and usually is out of a practical use. However, a derived test suite is very close to optimal and can be used for estimation of tests derived by other approaches.
As we already mentioned, a test suite derived from a composed machine by W-method is also hardly of a practical use because of its length (columns 11-12).
Table 2. Fault coverage of a transition tour of a
head component
Head Tail Length Coverage
s i o s o isolated in context
1 2 3 4 5 6 7 8
5 4 4 5 4 51 61 79
10 4 4 10 4 115 137 79
10 7 5 8 3 210 252 70
12 5 6 12 7 177 212 85
10 10 10 10 10 270 324 83
R&I, 2003, N 3
Methods based on a mutation machine (columns 7-8) and an embedded equivalent (columns 9-10) return test suites that can be shortly derived and have total length comparable with that of a test suite derived by explicit enumeration of mutant FSMs. Here we notice that in experiments we do not watch intersection of test suites for sub-machines of a mutation FSM (Step 2 of the method for test derivation in Section 3.4). If this inspection is added to software tools total length of test suites (columns 7-10) will be much lower.
Table 3. Test derivation for a series composition w.r.t. output faults in a head component: random components
Head Tail Enum. Mutation machine Embedded equivalent W-method
s i s i t l t l t l t l
1 2 3 4 5 6 7 8 9 10 11 12
2 4 2 2 0 1 9 0 37 0 19 0 390
2 4 4 2 0 1 8 0 85 0 33 0 734
2 4 8 2 1 2 0 0 213 0 57 7 12799
2 8 2 2 81 4 1 0 77 0 41 0 472
2 8 4 2 133 3 8 0 196 0 75 0 1178
2 8 8 2 268 4 1 0 526 0 143 1 5192
2 4 2 4 57 2 7 0 51 0 39 0 249
2 4 4 4 94 2 4 0 122 0 124 0 646
2 4 8 4 183 2 5 0 308 0 229 150 59323
2 8 2 4 - - 0 98 0 90 0 329
2 8 4 4 - - 0 336 0 247 4 4513
2 8 8 4 - - 6 678 0 772 12 15861
4 4 2 2 94 3 8 0 79 0 45 295 55552
4 4 4 2 168 4 2 0 182 0 122 8 14775
4 4 8 2 440 4 3 0 610 0 167 2882 314896
4 8 2 2 - - 0 160 0 97 0 1335
4 8 4 2 - - 0 527 0 152 214 42330
4 8 8 2 - - 1 1304 0 267 12 18079
4 4 2 4 - - 0 107 0 94 0 1523
4 4 4 4 - - 0 313 0 345 13 16353
4 4 8 4 - - 0 685 0 605 85 45980
4 8 2 4 - - 0 211 0 175 0 2050
4 8 4 4 - - 0 629 0 468 387 51712
4 8 8 4 - - 1 1534 0 1070 73 28676
8 4 2 2 - - 0 200 0 156 90 34691
8 4 4 2 - - 0 652 0 239 3011 334943
8 4 8 2 - - 2 1114 0 675 - -
8 8 2 2 - - 3 379 0 241 12 11720
8 8 4 2 - - 1 873 0 432 260 61917
8 8 8 2 - - 2 2379 0 693 - -
8 4 2 4 - - 0 224 0 191 47 32216
8 4 4 4 - - 1 622 0 694 1234 151197
8 4 8 4 - - 5 1372 1 1842 - -
8 8 2 4 - - 0 456 0 436 1 5009
8 8 4 4 - - 0 1206 0 1485 13 16603
8 8 8 4 - - 8 2981 0 3616 - -
Table 4 represents experimental results for the same test suite derivation methods for a number of benchmarks.
Table 4. Test derivation for a series composition w.r.t. output faults in a head component: benchmarks
Head component Tail component Mutation machine Embedded equivalent
Name s i Name s i o t l t l
1 2 3 4 5 6 7 8 9 10 11
shirtreg 8 1 dk27 7 1 2 30 483 0 67
bbtas 6 2 dk17 8 2 3 7 597 0 299
bbara 10 4 dk17 8 2 3 20 2059 0 1382
dk27 7 1 dk17 8 2 3 5 280 0 98
donfile 24 2 shiftreg 8 1 1 0 86 0 20
dk16 27 2 dk15 4 3 5 30 1847 0 511
ex2 19 2 ex3 10 2 2 - - 10 1589
ex3 10 2 ex5 9 2 2 - - 0 64
ex5 9 2 ex7 10 2 2 - - 0 523
ex2 19 2 ex7 10 2 2 - - 0 585
lion 4 2 dk27 7 1 2 - - 0 99
5. Conclusion
The paper presents experimental comparison of test suites derived for a component FSM by diverse test derivation techniques. The experiments have been performed for the head component FSM of a series composition w.r.t. output faults. The obtained results clearly show that a test engineer should be careful with a test suite that traverses each transition of a component FSM, since sometimes such a test suite detects only 70% of single output faults of the component. More rigorous analysis is necessary to clarify properties of a system of communicating FSMs where the approach can be effectively used. We also show huge redundancy of a test suite derived from the composed FSM by W -method. Methods based on the embedded equivalent and mutation FSM return tests that are longer than an optimal test suite derived by explicit enumeration of all possible mutants of a component FSM. However, the divergence is much less than that when a test suite for a component FSM is derived by W-method from the composed machine. Moreover, in the case of the embedded equivalent and mutation machine, a long test suite is a corollary of some drawbacks of the test derivation methods; thus, a test suite can be shortened by improvement of methods of test suite derivation w.r.t. the fault model < A, <, Sub(M) > .
References:
[Aziz95] Aziz A., Balarin F., Brayton R.K., Di Benedetto M.D., SaldanhaA., Sangiovanni-VincentelliA.L. Supervisory control of finite state machines // Proceedings of International Conference on Computer-Aided Verification, 1995. P. 279-292.
[Cava2000] Cavalli A., Prokopenko S., Yevtushenko N. Fault detection power of a widely used test suite for a system of communication FMSs // Testing of communications systems: tools and techniques (TestCom2000), Ottawa, Canada, 2000. P.35-59.
[Heer2000] Heerink L., Feenstra J., Tretmans J. Formal test automation: the conference protocol PHACT // Testing of communications systems: tools and techniques (TestCom2000), Ottawa, Canada, 2000. P.211-221.
[Cava99] Cavalli A., Lee D., Rinderknecht C., Zaidi F. Hit-and-Jump: an algorithm for embedded testing with applications to in services // Proceedings of the IFIP International Conference F ORTE/ PSTV99, Kluwer Academic Publishers, 1999. P. 41-56.
R&I, 2003, N 3
67
[Chow78] Chow T.S. Test software design modeled by finite state machines // IEEE Transactions, SE-4, N° 3, 1978. P. 178-187.
[David93] DavidR., Brzozowski J.A., Jurgensen H. Random test length for bounded faults in RAMs // Proceedings of the 3rd European Test Conference, The Netherlands, 1993. P. 149-158.
[Fuji91] Fujiwara S., Bochmann G.V., KhendekF., Amalou M., Ghedamsi A. Test selection based on finite state models. IEEE Trans. SE-17, N°6. 1991. P. 591-603.
[Kam97] Kam T., Villa T., BraytonR., Sangiovanni-VincentelliA. Synthesis of finite state machines: functional optimisation. Kluwer Academic Publishers, 1997. 283 p.
[Kouf99] Koufareva I., Petrenko A., Yevtushenko N. Test generation driven by user-defined fault models // Testing of Communicating Systems: Methods and Applications. Kluwer Academic Publishers, 1999. P. 215-233.
[Kouf2000] Koufareva I. Nondeterministic FSMs application to test generation for logic control circuits. PhD thesis, Tomsk, 2000. 176 p.
[Lee96] LeeD., SabnaniK.K., KristolD.M., PaulS. Conformance testing of protocols specified as communicating finite state machines - a guided random walk based approach // IEEE Trans. on Communication, 1996. Vol. 44, N°5.
[Lima97] Lima Jr. L.P., Cavalli A.R. A pragmatic approach to generating test sequences for embedded systems // Proceedings of the 7th IWTCS, Chapman & Hall, 1997. P. 288-307.
[Petr94] Petrenko A., Yevtushenko N., and Dssouli R. Testing strategies for communicating FSMs // Proceedings of the 7th IWTCS, 1994. P. 193-208.
[Petr96a] Petrenko A., Yevtushenko N., Bochmann G.V. and Dssouli R. Testing in context: framework and test derivation // Computer communications, 1996. Vol. 19. P. 1236-1249.
[Petr96b] Petrenko A., Yevtushenko N., Bochmann G.V. Fault models for testing in context // Proceedings of the IFIP 1 st Joint International Conference FORTE /PSTV, Chapman & Hall, 1996. P. 163-178.
[Petr97] Petrenko A., Yevtushenko N. Fault detection in embedded components. Proceedings of the 7th IWTCS, Chapman & Hall, 1997. P. 272-287.
[Star72] Starke P.H. Abstract Automata, North-Holland / American Elsevier, 1972. 419 p.
[Vasi73] Vasilevsky M.P. Failure diagnosis of automata, Cybernetics, Plenum Publishing Corporation, NY, 1973. N° 4. P. 653-665.
[Vuong89] VuongS.T., Chan W. W.L., ItoM.R. The UIO-method for protocol test sequence generation. Proceedings of the 2nd IFIP International Workshop onProtocol Test Systems, 1989. P. 161-175.
[Yang91] Yang S. Logic synthesis and optimisation benchmarks user guide // Technical report 1991-IWLS-UG-Saeyang, MCNC, Research triangle park, NC, January 1991.
[Yevt90] Yevtushenko N., Petrenko A. A method of constructing a test experiment for an arbitrary deterministic automaton. Automatic Control and Computer Science, Allerton Press Inc., New York,
1990. Vol. 24, N° 5. P.65-68.
[Yevt91] Yevtushenko N., Matrosova A. On one approach to automata network checking sequence construction // Automatic Control and Computer Science, Allerton Press Inc., New York,
1991. Vol 25, N° 2. P. 3-7.
68
R&I, 2003, N 3
