CC BY
37Conformance Testing With State Mapping
Vadim Mutilin
Institute for System Programming of Russian Academy of Sciences 25, B. Kommunisticheskaya, Moscow, Russia
mutilin@ispras.ru
Abstract—This work considers conformance testing (functional testing). The main distinction from the other works is the availability of mapping from implementation states to specification ones. This information allows us to simplify test development and to reduce test execution time. We introduce a novel implementation relation called conff and composite test suites. The conff relation minimizes the size of generated test suite. Composite test suite is a compact representation of traditional test suite. Furthermore, it allows to reduce test execution time.
I. Introduction
Testing is an essential part of modern software engineering. With the development of software engineering methods, the complexity of created systems increases. As result, testing becomes more complex. By estimations of experts, testing takes more than a half of all the time of creation of modern software systems, that enables to speak about testing as about one of the key activities in modern software engineering.
Testing is an execution of implementation with the subsequent checking of results on conformity to the requirements. Depending on the requirements, several kinds of testing are distinguished: conformance testing (functional testing), performance testing, robustness testing, stress testing, reliability testing and availability testing.
There are black box testing and a white box testing. In a black box testing only the information on interaction of system with an environment is accessible. Whereas in a white box testing the information on structure of system is accessible. There are a plenty of gray box testing methods in which the part of the information on system is accessible.
In this work we concentrate on gray box, conformance testing. That distinguishes the work from traditional methods of conformance testing, in which testing is performed by black box methods.
In conformance testing, functional specification and implementation exhibiting some behaviour are given. The purpose of testing is to check the conformance of the given implementation to the specification. Let SPECS be a set of specifications, IMPS be set of implementations, then conformance could be represented as a relation conforms-to C IMPS x SPECS. Expression IUT conforms-to M means that IUT is a correct implementation of specification M.
In this work we consider specifications given in the formal notations. However, unlike specifications, implementations are real physical objects, such as pieces of hardware or software. They are considered as objects interacting with the
environment, but not amenable to formal reasoning. This makes it difficult to give a formal definition of conforms-to. Therefore in order to reason formally about implementations, we assume that any implementation is modelled by formal object SysIUT € MODS, where MODS is a set of models. This assumption is called the test hypothesis [2], [19]. Note that the test hypothesis assumes only that the model exists, but not that it is known a priory.
Thus, the test hypothesis allows to define conformance as a relation between formal models of implementation and specification. This relation is called implementation relation imp C MODS x SPECS. Implementation IUT € IMPS is said to be correct with respect to M € SPECS, IUT conforms-to M, if and only if the model of implementation SysIUT € MODS is imp-related to M: SysiUT imp M.
The behaviour of implementation is investigated by performing experiments with it and observing reactions that the implementation produces to these experiments. A specification of such experiments is called a test case, and the process of applying a test to an implementation - test execution. Formally we define test execution on the model of implementation, however, according to the test hypothesis, the same behaviour will be observed at test execution on the real implementation.
During test execution, a number of observations are made, e.g. occurring events, stimuli, reactions are recorded. At the end of the test execution we like to interpret the observations in terms of being right or wrong, so we return verdict. If the test confirms the correctness of implementation, then verdict is pass, otherwise - fail. We say that the test passes, if it ends with a verdict pass, and otherwise we say that it fails .
A set of tests is called a test suite. Implementation passes a test suite, if it passes all the tests in it. Ideally, we would like to have such test suite that implementation passes it if and only if it conforms to the specification. A test suite with this property is called complete. It can distinguish exactly between conforming and non-conforming implementations. Unfortunately, in practice this requirement is very strong: complete tests are usually infinite and consequently not executable. Hence, a weaker requirement is usually posed: the test suite should be sound, which means that any correct implementation passes this suite. Probably, that some incorrect implementations also pass the suite, but if implementation does not pass, it indeed does not conform to the specification.
In the conformance testing process there are two main phases: test generation/writing and test execution. The impor-
tant characteristics of these phases are size of input data for obtaining a test suite and time of test execution.
In automatic test generation, the only input is specification from which tests are generated automatically. As it was noted, in practice we need additional data for test selection to get a test suite practically suitable for execution. For some reasons tests can be created manually. Imperfection of test generation and selection methods concern to such reasons, for example, if selection methods do not allow to get a good-enough test suite, executable in practice. Besides, representation of specification can not allow to perform automatic generation, for example, if the specification is given in the form of preconditions and postconditions representing predicates on states, input and output data.
On test execution phase it can be used both fully prepared test suite and partially prepared test suite from which tests are constructed during execution. Process, in which tests are completely generated at execution, is called on-the-fly generation. Time of execution is interpreted as quantity of the executed interactions with implementation. The more interactions the longer is test execution. Thus, we can recognize two important characteristics of testing process:
1) Size of input data;
2) Time of test execution.
In this work we consider implementations, which provide information about their state. This information can be obtained in different ways: by reading of accessible fields, global variables; by function calls, with the established correctness; by reading data from a database. Besides, it is required to provide the mapping from this information to specification states. This mapping is developed by test developers. In practice it is required to write a function, which first reads the information of an implementation state, and secondly, establishes connection between the information and specification state.
Formally, we consider, that there is a set of mappings MAPPINGS : SPECS x MODS ^ FUNC, where for each specification M and implementation Sys the function f = MAPPINGS(M,Sys) is given, which maps implementation states to specification ones. Note that f can be partially defined, i.e. the corresponding specification state can be defined not for all implementation states.
The purpose of the work is to use the given state mapping from implementation states to specification ones for reduction of size of input data and time of test execution.
This work is organized as follows. Section II introduces labelled transition systems used as formalism for specifications and implementations. Section III gives basic testing concepts for labelled transition systems, such as a test case, a test suite, a test run, passing a test case, passing a test suite and the notions of completeness, soundness and exhaustive of a test suite. Also the implementation relation conf is described in this section. The main results of this work are presented in section IV. A novel implementation relation conff which uses state mappings given in MAPPINGS is introduced in it. The algorithm for complete test suite generation that is more effective than algorithm in [18] for conf is developed. Then, composite test
suites are introduced, which are the compact representations of regular test suites. Test cases in these suites are intended for execution in different specification states, not only in the initial state as it is in traditional test suites. Besides reduction of input data size composite test suites allow to reduce test execution time as compared with test suites derived from them. In section V some concluding remarks are given, among which a relation to practice and a future work.
II. Labelled Transition Systems
We use labelled transition systems (LTS) [18] for formalization of the behaviour of systems.
Definition 1: Labelled transition system is a 4-tuple {S, L, T, s0), where S is a set (possibly infinite) of states, L is a set of labels, T is the transition relation T C S x (L U {t}) x S, s0 is the initial state.
The labels in L represent the observable interactions of a system. The special label t e L represents unobservable, internal action.
We denote the class of all labelled transition systems over L as LTS(L). We restrict LTS(L) to labelled transition systems that do not have cycles composed of internal actions.
The set of all finite sequences in alphabet L is denoted by L*, with e denoting the empty sequence. If o1,o2 e L*, then o1 ■ o2 is the concatenation of o1 and o2. With |o| we denote the length of the sequence. Some additional notations are introduced in definitions 2 and 3.
Definition 2: Let p = {S, L, T, s0) be a labelled transition system, s, s' e S, ¡-i(i) e L U {t}, v(i) e L and o e L*.
Ml-
Ml-
Ml-
s
• ■ Mn ■ ' Mn
• ' Mn
-def
-def
-def
-def
-def
def
def
def
def
(ss>) e T 3so,..., sn : s 3s'
s0
Ml
Ml - ■■■• Mn
not 3s' : s Ml! T • ■
s = s' or s -
3si, s2 : s ==
3so,.. - ,sn
s'
^ si — s = so
s2
..
s
vn
=n sn
s' :
not 3s' : s =^ s'
Definition 3: Let p = {S, L, T, s0) be a labelled transition system and s e S, o e L*.
1) traces(s) =def {o e L* I s =^}
2) init(s) =def {v e L I s =>}
3) s after o =def {s' e S I s =^ s'}
4) der(p) =def {s l3o e L* : so =^ s}
5) p has finite behaviour if 3n e Nat, such that Vo e traces(so) : lol < n
6) p is a finite state if der(p) is finite.
7) s is deterministic if for all sequences o e L*, s after
o has at most one element. If o e traces(p), then we
overload s after o to denote this element.
M
Definition 4: The state is finite if V^ e L U{t} holds s -/^.
T
Definition 5: State is stable if s ^.
The set of all stable states is denoted by Ss. Later on, we consider that so e Ss.
s
s
s
s
s
n
s
s
M
s
s
s
s
s
III. Conformance Testing
Starting point for conformance testing is a specification given in some formal notation and implementation, that is an object interacting with it’s environment. We make an assumption that each implementation has a formal model. This assumption is referred to as the test hypothesis.
In this paper labelled transition systems, or any language with underlying semantics of labelled transition systems, are considered as the formal notation for specifications. A model of implementation also considered as labelled transition system.
A. Implementation Relation
Conformance is defined by means of an implementation relation between the models of implementation and the specifications, in our case it is relation imp C LTS(L) x LTS(L): implementation Sys e LTS(L) conforms to specification M e LTS(L) if and only if Sys imp M.
Many possibilities for implementation relations have been studied, for example, observation equivalence [12], strong and weak bisimulation equivalence [13], [16], failure equivalence and preorder [7], testing equivalence and preorder [15], failure trace equivalence and preorder [1], generalized failure equivalence and preorder [10], and many others [20], [6].
In this paper we chose the implementation relation conf [5]. This relation was extensively studied in the literature, see for example [18].
Definition 6: A deadlock of p in a state s is a trace a e L*, after which no more observable actions are possible:
after a deadlocks
Definition 7: The sets of observations, obs and obs'
%
=def 3s' : s ^ and init(s') = 0
' respectively, that an observer u e LTS(L) of system p e LTS(L) can make, are given by deadlocks, respectively the traces of the synchronized parallel composition of u and p:
obs(u,p) =def {a e L* | (u\\p) after a deadlocks}
obs'(u,p) =def {a e L* I (u\\p) =^}
Definition 8: Let p e LTS(L), s e S, a e L* and A C L, then
V
s after a refuses A =def 3s' : s ^ s' and Vv e A : s' ^ Definition 9: Let M = {SM, L, TM, sM) be a specification and Sys = {S, L, T, s0) be an implementation.
Sys conf M =def Vu e LTS(L) : (obs(u, Sys) fi traces(sM)) C obs(u,M) and (obs'(u, Sys) f
traces(sM)) C obs'(u,M)
Statement 1: Sys conf M if and only if (Va e
traces(M), VA C L: Sys after a refuses A implies
M after a refuses A).
The proofs are given in [14], [17].
Figure 1 gives examples of labelled transition systems. p2 conf p1 does not hold, since p2 after a refuses {c} not
implies pi after a refuses {c}. It is easy to check that
p3 conf p1, p5 conf p4, but p4 conf p5 does not hold.
Fig. 1. Examples of labelled transition systems
B. Test Suite
Test cases are derived from the specification, and applied to the implementation. Test cases are constructed in such a way that from the results of applying them it can be concluded whether the implementation conforms to specification.
A test case is a specification of the behaviour of a tester in an experiment to be carried out with an implementation under test. Such behaviour, like other behaviours, can be specified by a labelled transition system. An experiment should last for finite time, so the test case should have finite behaviour. Moreover, a tester executing a test case would like to have as much control as possible, so nondeterminism in a test case is undesirable. To be able to decide about the success of a test a verdict is attached to each state of the test case.
Definition 10: A test case t is a 5-tuple {Sl,L,Tt,v,st0), such that {St, L, LtT, Tt, s0) is a deterministic labelled transition system with finite behaviour, and v : St ^ {pass, fail} is a verdict function.
The class of test cases over action in L is denoted by LTSt (L). Definitions for LTS(L) is extended to LTSt(L) by applying them to underlying labelled transition system.
Definition 11: A test suite T is a set of test cases, i.e. a subset of LTSt(L).
A test case run is modelled by synchronous parallel composition of the test case with an implementation, which continues until the deadlock is reached. This deadlock may occur when test case reaches a final state, or when the composition reaches a state where the actions proposed by the test case cannot be accepted be the implementation. An implementation passes a test run if and only if the verdict of the test case in a deadlock state is pass. Since an implementation can behave nondeterministically different test runs of the same test case can lead to different deadlock states, and hence to different verdicts. An implementation passes a test case if and only if all possible test runs lead to verdict pass.
Definition 12:
1) A test run of a test case t e LTSt(L) with an implementation Sys e LTS(L) is a trace of the synchronous
pass pass
pass
b j
U pass O' pass Fig. 2. Examples of test cases
m ro
pass
IV. Using State Mapping
In this section we consider specifications and implementations for which the information about correlation between their states is given. We assume that for each specification and implementation there is a mapping from implementation state to specification state. In practice this mapping is accomplished by the procedure which reads state data from the implementation and then constructs the corresponding specification state.
By the semantics, t action in LTS can be executed at arbitrary time. Hence, reading of state data in the states with this action (unstable states) is complicated. In practice we cannot guarantee that the data will be read in a certain unstable state. In this connection we assume that the mapping exists only for stable implementation states.
Formally, we assume that for each pair (specification, implementation) there is a function f : Sg ^ SM, where Sg is a set of stable implementation states and SM is a set of stable specification states. By MAPPINGS we refer to a mapping, which for each pair (specification, implementation) contains a corresponding mapping f. Implementation relation depending on MAPPINGS is referred to as imp(f). We
fi: f2:
io^o ¡o—o
¡1^1 ¡1—1
i2—>2 ¡2—2
¡3^3 ¡3—3
¡4—2 ¡4—3
Fig. 3. Specification and implementation with state mappings
parallel composition of t and Sys leading to a deadlock: a is a test run of t and Sys =def (t||Sys) after a deadlocks
2) An implementation Sys passes test case t, if all test runs lead to a deadlock states, where verdict is pass:
Sys passes t =def (t||Sys) after a deadlocks implies v(t after a) = pass
3) An implementation Sys passes a test suite T, if it passes all test cases t eT :
Sys passes T =def Vt eT : Sys passes t
Figure 2 shows examples of test cases. p4 not passes ti, since t1|p4 after a deadlocks and v(t1 after a) = fail.
p1 passes {t1,t2}, p2 not passes t1, p3 passes {t1,t2},
p4 passes {t3,t4}, p5 passes {t3,t4}, p6 passes {t5,t6}.
Definition 13: Let M be a specification, imp be an implementation relation and T be a test suite, then
T is complete if VSys : Sys imp M ^ Sys passes T
T is sound if VSys : Sys imp M ^ Sys passes T
T is exhaustive if VSys : Sys imp M ^ Sys passes T
Fig. 4. Implementation supplemented with state mappings
write S'ys imp(f) M, if S'ys imp M with state mapping f = MAPPINGS(M, Sys).
Existence of f allows to supplement system Sys with transitions representing the current specification state. Hence, we define
Sys =def (S, L U SM,T U T,so),
M =def (SM, L U SM, TM U TM, sM),
where
T
T = {(s,f(s),s) | seS},
M = {(sM,sM,sM) I sM eSM}.
Let’s consider an example of specification and implementation with the given state mapping. Figure 3 shows specification m1, implementation i1, and two possible state mappings f1 and f2. Figure 4 shows implementations i2 and i3, which are the result of supplementation of implementation ii with state mappings fi , f2 respectively.
A. Implementation Relation conff
Consider implementation relation conff, which depends on state mappings given in MAPPINGS. We assume that for each specification M = {SM,L,TM,sM) and implementa-
tion Sys = {S, L, T, so) mapping f : Ss MAPPINGS(M, Sys)) is given.
Definition 14: Let M = {SM ,L,TM
S M
(f =
M
) be a specifica-
tion, Sys = {S, L,T, s0) be an implementation, and f be a state mapping.
Sys conff M =def
1) S'ys conf M, and
2) ya e traces(M): s0 s in Sys, where s G Sg
implies f (s0) f (s) in M.
Statement 2: Sys conff M implies Sys conf M.
In example shown in figure 3 i1 conff m1 holds, if mapping is f1, and does not hold, if mapping is f2.
a
a
b
—»
t2 t3
\ °a
b
pass 2 3
pass pass
Fig. 5. Test suite example for conff
Consider another way of conff definition. For implementations supplemented with state mappings the following statement holds.
Statement 3: Sys conff M if and only if Sys conf M.
From this follows that test suite generation for implementation relation conff can be done by test generation algorithm for conf described in [18] by applying it to M.
B. Test Generation for conff
The notions of test case, test suite, test run and passing a test case are the same, with the only difference that as implementation we consider system Sys, supplemented with state mapping. Furthermore, test cases together with labels from L include labels corresponding to specification states S M.
In contrast to conf, test generation for conff becomes simpler because it is stronger: besides inclusion of observable traces and deadlocks it requires coincidence of the corresponding specification states.
Test suite generation algorithm consists in applying construction 1 to all specification traces traces(M).
Construction 1: Let a = j1 •... • jn G traces(M) is given. Construct a labelled transition system x = {Sx, L, Tx, sX) that consists of the only trace traces(x) = {a}, deterministic for
all sequences, i.e. x consists of transitions ; s'n. From x construct test case t = (Sx U Sn Tn,v,sX), where function v on states sx e Sx as follows.
Let a be a trace leading to sx, i.e. sx Consider two cases.
, L U SM ,T , Sn and Tn
sx after a.
(s
M
) and ( s
M
)=*}.
Theorem 1: Let T be a test suite obtained by applying construction 1 to all specification traces, then T is complete for implementation relation conff.
Proof: We devide proof into two parts. To show that T is complete it is sufficient to prove that
1) VT C T, T is sound;
2) T is exhaustive.
Prove 1. Suppose that one of the test cases t e T does not pass on Sys. Then test case deadlocks in a state st: v(st) = fail, a is it’s test run. By construction 1, st is not finite state. Hence, there exists n e L U SM: Sys after a refuses {^}. But by construction 1, a • n e traces(M). Hence, M after a refuses {^} does not hold. From statement 1 follows that S ys conf M does not hold.
Prove 2. It is required to show that if Vt eT Sys passes t, then Sys conff M, i.e. by definition 14 and statement 1 it is sufficient to prove that
1) Va e traces(M), VA C L: Sys after a refuses A implies M after a refuses A
2) Va e traces(M): s0 ==>• s in Sys, where s e Sg implies f (s0) f (s) in M.
At first we show 2. Let a e traces(M) and s0 ==> s in Sys. Let t be a test constructed on the base of a and Sys passes t. By construction 1, f (s) e sM after a; otherwise, test case finishes with the fail verdict. From this it follows that f (so) => f (s) in M.
Now we show 1. Let a e traces(M ) and Sys after a refuses A, i.e.
3s' e Sys after a : V^ e A : s'
(1)
We prove by contradiction. Let M after a refuses A does
not hold, i.e. s
M
M
sM 3^ e A
M
By
xU
are
1) If t\\M after a deadlocks, then
a) Assign v(sx) = fail
b) Add to Tn the following set of transitions
{(sx, s', new) | s' G S' fl SM, new - new state}, where S' - reachable specification state, in which deadlock is possible, i.e. S' = {sM | (sx, sM) ==$■
c) Enlarge Sn with new states added to Tn, for which we assign v(new) = pass.
2) Otherwise, assign v(sx) = fail.
The example of the test suite generated from all specification traces of m1 (fig. 3) with applying construction 1, is shown in figure 5.
proved condition 2, we have f (s0) =^ f (s'). Hence, 3j G
A : f (s')
Let t be a test case constructed on the base of a • j and sl = t after a. By condition, Sys passes t. By construction 1, the test case state sl does not have transitions with label f (s'), since there is no transition by j in specification M (f (s') =>), and by assumption (1), implementation
does not have transition by j in state s' (s' ^). Hence t\\Sys after a deadlocks. But by construction 1, v(sl) = fail. This contradicts the fact that implementation passes the test case. ■
Note 1: Test generation algorithm described in [18], generates a test suite such that total number of traces of test cases is no less than the number of specification traces. As distinct from it, algorithm proposed here is restricted to all specification traces.
In example shown in figure 6, proposed test generation algorithm for specification m2 generates a complete test suite consisting of three test cases t1, t2 and t3. Whereas algorithm in [18] generates a complete test suite consisting of infinite number of test cases. The cause of infinite number is self-loops representing transitions by specification states. However, even
0
x
x
x
^2
o.
a
b
pass pass
U
2 pass 3'
pass pass
Fig. 6. Comparison of generated test suites
f:
¡0—0 ¡1 — 1 ¡2—>2 ¡3—3 ¡4—4
Tî3
enabled: 0—>{tl, t2}
1—{ta}
2—{t4, ts}
3—{t6}
4—{}
t1
t3 t4 t5
3
1 b 3 3 C 4
pass 2j3 Pass Pass ^ pass
pass pass pass
Fig. 8. Example of composite test suite
implementation Sys e LTS(L) in a state s is a trace of the synchronous parallel composition of t and Sys leading to a deadlock:
a is a test run of t and Sys in s = (t||Sys') after a deadlocks,
-def
where Sys' - system Sys with the initial state s.
2) An implementation Sys passes test run a of a test case t in a state s, if the test run leads to a deadlock, where verdict is pass:
Sys passes test run a of t in s =def
v(t after a) = pass
3) An implementation Sys passes test case t in s, if it passes all test runs in this state:
Sys passes t in s
def
Fig. 7. Specification and implementation with state mapping
if we apply the algorithm to the original system without selfloops, generated test suite will additionally contain test case
t4.
C. Composite Test Suites
We consider test suite T containing test cases that can be executed not only in the initial state, but in arbitrary state as well. For each state sM with the help of the function enabled we define the possibility of execution of a test case t T in this state.
Definition 15: Composite test suite Te for specification M is a pair {T, enabled), where T is a test suite and enabled : SM - 2T is a function defining a set of test cases which can be executed for each state sM S M.
Figure 8 shows composite test suite for specification m3 in figure 7.
The notions of test run, passing a test case and a test suite with an implementation are the same with the only difference that definitions are made not only for initial state, but for arbitrary state as well.
Definition 16:
1) A test run of a test case t G LTSt(L) with an
ya : S'ys passes test run a of t in s To check the possibility of execution of test case t in implementation state s we need information about corresponding specification state. We assume that for each specification and implementation we have function f mapping stable implementation states into stable specification states.
Definition 17 introduces additional notations for test case execution in stable states.
Definition 17: Given an implementation Sys = {S, L,T, s0), a mapping f, a test suite Te = {T, enabled), s1, s2 G Sg, t G X and a G L*.
s1 —— s2 =def a is test run of t in s1, t G enabled(f (s1)), and s1 s2
s1 —- s2 =def 3a Definition 18:
1) A run of composite test suite Te with implementation Sys is a set Q, containing such chains as:
ti t2,&2 tn,an
so —- s1 ------ ... —- sn,
t, a
si ------------► s2
G T, si e S and s0 is the initial
t, a
of chains are called run
where ai G L*, t. state of Sys. Elements s items.
2) An implementation Sys passes run Q of composite test
suite Te, if for each run item s1 test run a of t in s1.
s2 in Q Sys passes
tl
3
3) An implementation Sys passes test suite Te, if it passes all test runs Q:
Sys passes Te =def yQ : Sys passes run Q Note, that for any composite test suite there exists a run including any other runs. Obviously, that a run constructed as a union of chains in all possible runs has such property.
Consider examples of runs Q of composite test suite in figure 8 with implementation i3 (fig. 7).
^2
ta
pass
Q1
Q2 =
i0 ^ il ^ i3 ^ i4 i0 t2ab2 i2
i0 —2 i2 ^ i3 -^ i4 i0 t2ab2 i2 ^ 4 i4
i0 t2ab3 i3
Definition 19: Let M be a specification, imp be an implementation relation and Te be a composite test suite, then
Te is complete if ySys : Sys imp M ^ Sys passes Te
Te is sound if ySys : Sys imp M ^ Sys passes Te
Te is exhaustive if ySys : Sys imp M ^ Sys passes Te
1) Derived test suites: Let Te be a composite test suite. Consider the notion of concatenation t of two test cases 11,12 G Te.
Definition 20: Let t1 = {S1, L U SM, T 1,v1, s1) and t2 = {S2,L U SM,T2, v2,s2). Test case t is the result of concatenation of two test cases t1, t2, if t = {S1 U S2, L U
pass
pass
Fig. 9. Example of derived test suite
respectively, and the beginning of transition is a state where test case ti finishes, i.e. U after a.. And vice versa, for each
failed test run a1
s
a2
s
M n— 1
a
of a test case
SM, T1 U T2 U Tt, vt, s1), where Tt = {(s1, sM, s^)}, at that t G T constructed by successive concatenation of t1... tn,
v1(s1) = pass, t2 G enabled(sM), vt(s) = v 1(s) for s G S1 there exists run Q where the test case tn fails in chain so —1
and vt(s) = v2(s) for s G S2.
Now we construct test suite T = derive(Te) from Te. Each test case t T in it is successive concatenation of test cases t0,..., tn, U G Te, where t0 G enabled(sM). Test suite T is called derived from composite test suite Te.
Figure 9 shows test suite T3 derived from composite test suite Te in figure 8.
Derived test suites have very important property: they pass with the same set of implementations as composite ones from which they are derived.
This enables us to talk about composite test suites as about compact representations of regular test suites. Indeed, derived test suites are always larger by both a number of test cases and a total number of transitions of all test cases in the test suite.
Statement 4: For any composite test suite Te, a test suite T = derive(Te) derived from it and an implementation Sys, the following property holds
Sys passes T ^ Sys passes Te
Proof: For proving this fact it is sufficient to note that
s1
t2,&2
tn
sn, and sM = f (si).
2) Time of Test Execution: For estimation of test execution time we count the number of interactions between a test case and an implementation. For the sake of simplicity we do not consider the possible difference in execution time between different interactions. Moreover we do not take into account execution times of internal actions. At first, this metrics becomes independent from labels L. At second, it does not depend on the number of internal actions which can vary for the same test run.
Definition 21: An execution time of test run a of a test case t with implementation Sys is p(a) =def |a|.
A total execution time of a test suite T can be calculated by the following formula
tET a is run of t
for each failed run Q of Te with failed chain s0 s1 ... —^ sn e Q, where k is the number of the first failed test case, there exists a test case t in suite T constructed by successive concatenation of t1.. .tk, where transitions added during concatenation of test cases ti, ti+1 contain labels f (si)
t2,a2
This formula is applicable only for finite test suite with finite number of test runs for each test case. Obviously, that for other test suites execution time is infinite.
T
3
5
Let’s compute execution time for derived test suite T3 on Note, that p(Q3) = 20, whereas p(T3) = 53. i3. For that consider all possible test runs of ti;... , t8 e T3. Statement 6: For all implementations Sys
ti a l
t2 a l b 3
t3 a l b 3 c 4
t4 a b 2, a b 3
t5 a b 2 a 3, a b 3
te a b 2 a 3 c 4, a 3 b
t7 a b 2 a c 4, a b 3
t8 a b 2, a b 3 c 4
Total sum of all test runs is p(T3) = 53.
An execution time of run of a composite test suite is a total sum of all run items.
Definition 22: An execution time of run Q is
p(Q) =
E
P(ai) + P(a2) + ■■■ + f->(an)
*1 É2
- «I
^ sne^
of runs Qi, Q2 on i3: p(Q1)
9,
An execution time
pQ) = 16.
Let’s estimate execution time of composite test suites. As it was noticed earlier for composite test suites there exists a run containing run items of any other run. Thus, for time estimation purpose we compare execution time of runs Q with all test runs of derived test suite T.
The following statement holds.
Statement 5: Let Te be a composite test suite, T = derive(Te). For any system Sys and any run of composite test suite Q such that Sys passes Q, if total execution times of T and Q are finite, then the following condition holds:
p(Q) < p(T)
Proof: For proving this fact it is sufficient to show that for any chain s0 si ... f'--— sn e Q, there exists a test case t in suite T constructed by successive concatenation of test cases ti.. .tn, transitions added during concatenation of test cases ti, ti+i contain labels f (si), and the beginning of transition is a state where test case ti finishes, i.e. ti after ai. An execution time of this test case p(ai • f (si) • a2 • ... • f (sn-i) • an) is not less than execution time of the selected chain of Q. ■
Note, that passing a run Q by implementation depends on passing test case in run items s s'. From that follows, that runs containing the same sets of run items pass or fail simultaneously. We distinguish maximal runs Qmax containing all possible run items for given implementation.
Definition 23: Maximal run Qmax is a run, such that for any Q all run items y.s s' are contained in chains of
Qmax.
For test suite T3e in figure 8 and implementation i3 one of the possible Qmax is Q3
i0 ^ il i3 —,4 i4
i0 t2^2 i2 ^ i3
•n t2 ? a -b-2 -rv t5 i0 —> i2
i0 t2ab3 i3
i4
Sys passes Qmax ŸQ '■ Sys passes Q
As well as for any run, statement 5 holds for Qmax. Hence, execution time of Qmax is less than execution time of derived test suite.
V. Conclusion
The work concerns questions of gray box conformance testing. Besides the possibility of interaction with implementation, the test system has a possibility of reading the states of implementation and mapping these states to specification ones.
in section iV, it is shown, how this possibility, given in the form of mappings MAPPINGS, can be used to improve the methods of conformance testing.
A novel implementation relation conff was defined and a test suite generation algorithm is simplified in comparison with other known generation algorithms for conf. Note, that there are other implementation relations and the possibility of simplifying generation for them needs further investigation. Among such relations it is deserved to mention widely spread relation ioco and it’s improvements.
Composite tests, presented in this work, allow to essentially simplify the description of test suite. Besides, the way of test definition allows to reduce the execution time in comparison with test suites derived from them with the same capability to recognize incorrect implementations.
The idea of composite tests is very close to test sequence generation methods in testing using automata or finite state machine (FSM) models [11]. These methods are improved for the more general case - labelled transition systems. State mapping can be considered as a more general case of status messages, because the mapping does not require implementation states to be equal to specification ones.
in practice composite test suites are successfully used in UniTESK technology [8], [9]. However, existing formalization of UniTESK approach is based on automata models. Conditions on specifications, implementations and tests at which automata equivalency is guaranteed [3], [4] are determined for these models. In UniTESK composite test runs are limited by the only chain, which corresponds to traversal of automata model of specification. obviously, it is possible only with constraints on a specification and tests, whereas herein the more general case is considered: runs can contain any number of chains.
Let’s note that in this work the question of effective complete composite test suite generation is not addressed. obviously, that complete test suite in a initial state can serve as such suite. However in this case the advantages of composite tests are neglected.
References
[1] J. C. M. Baeten and W. P. Weijland. Process algebra. Cambridge University Press, New York, NY, USA, 1990.
so
4
[2] Gilles Bernot. Testing against formal specifications: A theoretical view. In TAPSOFT ’91: Proceedings of the International Joint Conference on Theory and Practice of Software Development, Volume 2: Advances in Distributed Computing (ADC) and Colloquium on Combining Paradigms for Software Developmemnt (CCPSD), pages 99119, London, UK, 1991. Springer-Verlag.
[3] Igor B. Bourdonov, Alexander Kossatchev, and Victor V. Kuliamin. Irredundant traversal algorithms of directed graphs. deterministic case. Programming, 5:59-69, 2003.
[4] Igor B. Bourdonov, Alexander Kossatchev, and Victor V. Kuliamin. Irredundant traversal algorithms of directed graphs. nondeterministic case. Programming, 1:2-17, 2004.
[5] Ed Brinksma. A theory for the derivation of tests. pages 63-74, 1988.
[6] Ed Brinksma and Jan Tretmans. Testing transition systems: an annotated bibliography. pages 187-195, 2001.
[7] C. A. R. Hoare. Communicating sequential processes. Prentice-Hall, Inc., Upper Saddle River, NJ, USA, 1985.
[8] A. Khoroshilov. Specification and testing components with asynchronous interfaces. PhD thesis. ISP RAS, Moscow, 2006.
[9] V. Kuliamin, A. Petrenko, A. Kossatchev, and I. Bourdonov. Unitesk approach to test development. Programming, 29:25-43, 2003.
[10] Rom Langerak. A testing theory for lotos using deadlock detection. In Proceedings of the IFIP WG6.1 Ninth International Symposium on Protocol Specification, Testing and Verification IX, pages 87-98, Amsterdam, The Netherlands, The Netherlands, 1990. North-Holland Publishing Co.
[11] D. Lee and M. Yannakakis. Principles and methods of testing finite state machines - A survey. In Proceedings of the IEEE, volume 84, pages 1090-1126, 1996.
[12] Robin Milner. A Calculus of Communicating Systems. Springer-Verlag New York, Inc., Secaucus, NJ, USA, 1982.
[13] Robin Milner. Communication and concurrency. Prentice-Hall, Inc., Upper Saddle River, NJ, USA, 1989.
[14] R De Nicola. Extensional equivalence for transition systems. Acta Inf., 24(2):211-237, 1987.
[15] R. De Nicola and M. C. B. Hennessy. Testing equivalences for processes. Theoretical Computer Science, 34(1-2):83-133, November 1984.
[16] David Park. Concurrency and automata on infinite sequences. In Proceedings of the 5th GI-Conference on Theoretical Computer Science, pages 167-183, London, UK, 1981. Springer-Verlag.
[17] Jan Tretmans. A Formal Approach to Conformance Testing. PhD thesis. University of Twente, 1992.
[18] Jan Tretmans. Conformance testing with labelled transition systems: implementation relations and test generation. Comput. Netw. ISDN Syst., 29(1):49-79, 1996.
[19] Jan Tretmans. Testing concurrent systems: A formal approach. In
CONCUR ’99: Proceedings of the 10th International Conference on Concurrency Theory, pages 46-65, London, UK, 1999. Springer-Verlag.
[20] Rob J. van Glabbeek. The linear time - branching time spectrum ii. In CONCUR ’93: Proceedings of the 4th International Conference on Concurrency Theory, pages 66-81, London, UK, 1993. Springer-Verlag.
