CC BY
42
ВЕСТНИК ТОМСКОГО ГОСУДАРСТВЕННОГО УНИВЕРСИТЕТА
2009 Управление, вычислительная техника и информатика № 1(6)
UDK 519.7
Alexandre Petrenko and Nina Yevtushenko
REFINING SPECIFICATIONS IN ADAPTIVE TESTING OF NONDETERMINISTIC FINITE STATE MACHINES
This paper addresses testing of nondeterministic FSMs. An implementation FSM is allowed to be less nondeterministic than its specification, so the reduction relation between machines is the conformance relation. Execution of a given test needs to be adaptive to avoid application of unneeded input sequences, since the test can have different inputs depending on output produced by the implementation under test in response to previous input. To further reduce testing efforts during test execution, the paper suggests performing refinement of the specification FSM by removing traces absent in the implementation, which has already passed some subtest of a given complete (sound and exhaustive) test. The test execution process can be terminated as soon as the executed so far subtest is complete for the current refined version of the specification. The sufficient conditions formulated in the paper can be used for this purpose.
Keywords: Finite State Machine (FSM), adaptive testing, fault model, complete test suite w.r.t. the fault model.
The need for adaptive tests for nondeterministic machines has widely been discussed in various work, see, e.g., [1 - 10], [13, 14], [17 - 21]. When testing a nondeterministic implementation FSM each input sequence used in a given test needs to be repeatedly executed to make sure that all the implemented traces with this input sequence are observed. The number of repetitive applications of a sequence of input stimuli to satisfy the all-the-weather, aka complete testing assumption [11], is application dependent; however, it is even unknown when it is sufficient to terminate this process in a general case. To the best of our knowledge, the work of Hwang et al. [8] is the only attempt in determining the repetition numbers of test sequences in a probabilistic setting. However, there has been little work done on minimizing the number of input stimuli of a given test that need to be executed against a given possibly nondeterministic implementation FSM.
In this paper, we consider the offline testing of nondeterministic FSMs. As opposed to online testing, where test generation and execution are merged into a single process, see, e.g., [3], [14], in offline testing, tests are first generated from a given specification and then executed against a given implementation. This type of testing avoids repetitive test generation especially when several implementations (or its versions) have to be tested against a given specification. In this setting, the overall testing efforts can be reduced by minimizing a part of given tests that needs to be executed against a particular implementation.
If both, a specification and its implementations are allowed to be nondeterministic then the reduction relation between FSMs is a natural choice for a conformance relation, since an Implementation Under Test (IUT) is allowed to be less nondeterministic than its specification. Several publications in the area use the notion of a test from the case of deterministic specification FSMs; where tests are simply input sequences. This creates a number of problems in defining an adaptive test and formalizing its adaptive execution.
For this reason, an adaptive test case is presented as a tree in [2, 4 - 7, 9, 17], where in each node only a single input triggers outgoing transitions.
Hierons in [4, 5], derives an adaptive test assuming that implementation FSMs are deterministic. The idea is to use candidate machines which may model the behavior of an implementation FSM, in other words, to refine the fault domain which contains all possible implementation machines. As a result, the algorithm terminates when an implementation FSM fails the test or is recognized up to distinguishable states. Thus, this approach deals with the problem of identification of an implementation FSM, while the initial problem statement concerns just checking of the implementation which is usually less expensive than machine identification. Hierons and Ural [7] also address the problem of adaptive execution of test sequences against a deterministic implementation. The proposed approach consists in finding such an ordering of input sequences that minimizes the total number of input sequences that needs to be executed. In [6], Hierons extends the approach for the case when an implementation FSM can be non-deterministic. In this paper, in order to terminate traces of a test earlier than in [16] Hierons refines not the fault domain but the product of the specification and an implementation FSM when the latter already produced expected output responses to some input sequences. Termination rules and algorithms in [4 - 6] depend on the number of states of an implementation FSM, and should be completely reconsidered when extending the methods proposed there to another fault domain. In [17], we elaborated an approach to test generation using the notion of a test as a tree FSM. A complete test is defined as sound and exhaustive, i.e., each conforming IUT (a reduction of the specification FSM) has to pass the test, while each non-conforming implementation (not a reduction of the specification FSM) of the fault domain has to fail the test.
All the known methods for deriving tests with respect to the reduction relation keep the same original specification FSM during test generation, while in this paper, we propose to refine the specification FSM by removing traces absent in the implementation, which has already passed some subtest of a given complete (sound and exhaustive) test. A refined specification usually becomes more deterministic and thus, a shorter complete test may exist for the refined specification FSM. The test execution process can be terminated as soon as the executed so far subtest is complete for the current refined version of the specification. The sufficient conditions for the completeness of a test with respect to the reduction relation when the upper bound on the number of states of an IUT is known, formulated in the paper can be used for this purpose. This approach does not differentiate between deterministic and nondeterministic implementation FSMs and can be easily extended to other fault domains with respect to which sufficient conditions for the test completeness are known. A proposed approach is based on checking experiments with FSMs, which are usually known to be shorter that the identification experiments; however, additional research is needed to compare the effectiveness of the proposed approach and that of the approach in [4] when an implementation FSM is deterministic.
The paper is organized as follows. Section 2 contains basic definitions related to nondeterministic finite state machines including the reduction relation. Section 3 defines a test as a nondeterministic acyclic FSM. A method for adaptive test execution of a given complete test against a given possibly nondeterministic implementation FSM which is based on a consecutive refinement of a given specification is elaborated and illustrated on an example in Section 4. Section 5 concludes the paper.
1. Definitions
Definition 1. A Finite State Machine (FSM) A is a 5-tuple (S, s0,I, O, h), where
• S is a finite set of states with the initial state s0;
• I and O are finite non-empty disjoint sets of inputs and outputs, respectively;
• h is a behavior function h: S x I ^ 2SxO, where 2SxO is the powerset of S x O. The
function h has two projections h1 and h2: for each (s, a) e S x I, it holds that hl(s, a) = = {s'| 3o e O s.t. (s', o) e h(s, a)} and h2(s, a) = {o | 3s' e S s.t. (s', o) e h(s, a)}.
We slightly abuse the notation h by referring to the behavior function extended to input words.
An input a is a defined input at state s if h(s, a) ^ 0; otherwise, an input a is
undefined at state s.
A word a e (IxO)* of the automaton Ax in state s is a trace of A in state s; let Tr(s) denote the set of all traces of A in state s, while Tr(A) denote the set of traces of A in the initial state. Given a trace set T, we denote Max(T) the set that contains completed trace of the set T, i.e., those traces are not proper prefixes of the other traces of T. Given sequence a e (IxO)*, the input projection of a, denoted a;I, is a sequence obtained from a by erasing symbols in O. Input sequence p e I* is a defined input sequence in state s of A if there exists a e Tr(s) s.t. p = a;I. We use Q(s) to denote the set of all defined input sequences for state s and Q(A) for the state s0, i.e., for A. Q(A) = Tr(A);I = I* holds for any complete machine A, while for a partial FSM Q(A) c I*.
We use the following categorization of state machines.
Definition 2. FSM A = (S, s0, I, O, h) is
• completely specified (a complete FSM) if h(s, a) ^ 0 for all (s, a) e S x I;
• partially specified (a partial FSM) if h(s, a) = 0 for some (s, a) e S x I;
• trace-harmonized if for each sequence aa e Q(A) input a is defined at each state
h1(s0, a);
• trivial, if S = {s0} and h(s0, a) = 0 for all a e I;
• state deterministic if |h'(s, a)| < 1 for all (s, a) e S x I;
• output deterministic if |h2(s, a)| < 1 for all (s, a) e S x I;
• deterministic if |h(s, a)| < 1, i.e., if it is state and output deterministic;
• nondeterministic if |h(s, a)| > 1 for some (s, a) e S x I;
• observable if the automatonAx = (S, s0, I x O, 8), where 8(s, (a,o)) = s' iff (s', o) e h(s, a), is deterministic;
• acyclic if its transition diagram is acyclic;
• output-complete if for each pair (s, a), h(s, a) = 0 or h2(s, a) = O;
• regular if it is observable, output-complete, acyclic and has at each state at most one defined input; the set of its traces is called regular.
We define several relations between states of observable and traced-harmonized FSMs.
Definition 3. Given observable and traced-harmonize FSM A = (S, s0, I, O, h) and s,
t e S,
• s and t are (trace-) equivalent, s = t, if Tr(s) = Tr(t);
• t is quasi-equivalent to s, t 3 s, if Q(t) 3 Q(s) and {p | p e Tr(s) & p^ = a} =
= {p | p e Tr(t) & p^ = a} for each a e Q(s);
• t is trace-included into (is a reduction of) s, t < s, if Tr(t) c Tr(s);
• s is not a reduction of t, written s t, if Tr(s) £ Tr(t); if s is not a reduction of t
and there exists an input sequence a e Q(s) n Q(t) s.t. {p | p e Tr(s) & p^ = a} £
{p | p e Tr(t) & p;I = a} we use the notation s t in order to refer to the input sequence a that evidences that s is not a reduction of t; in this case, we say that a trace p e Tr(s) that is not a trace at state t, i.e., p g Tr(t), distinguishes s from t;
• s and t are r-distinguishable, s ¥ t, if there exists a regular set T of traces s.t. Max(T) n Tr(s) n Tr(t) = 0, called a set r-distinguishing s and t; then the set of traces T n Tr(s) is the r-distinguishing set of s w.r.t. t, denoted d(s, t), and T n Tr(t) is the
r-distinguishing set of t w.r.t. s, denoted d(t, s); we also use the notation s ¥ T t when we need to refer to the set T of traces r-distinguishing s and t.
The notion of r-distinguishability (or adaptive distinguishability) is used in several work, e.g., [15 - 17, 2, 6]; below we briefly sketch how states can be efficiently checked for the r-distinguishability and how corresponding r-distinguishing sets can be determined. As usual, for a trace a e Tr(s), s-after-a denotes the state reached by A when it executes the trace a from state s. If s is the initial state s0 then instead of s0-after-a we write A-after-a.
Definition 4. Given an FSM A = (S, s0, I, O, h), states s, t e S,
• s and t are r(1)-distinguishable if there exists input a s.t. h2(s, a) n h2(t, a) = 0;
• given k > 1, s and t are r(k)-distinguishable, if the states are r(k - ^-distinguishable or there exists an input a e I s.t. for each trace (a,b), b e h2(s, a) n h2(t, a), states s-after-(a,b) and t-after-(a,b) are r(k - 1)-distinguishable.
Proposition 1. States s and t are r-distinguishable if there exists k > 0 s.t. states s and t are r(k)-distinguishable.
If h2(s, a) n h2(t, a) = 0 for some input a then the set of traces {(a,b) | b e O} r-distinguishes states s and t. If the states s and t are r(k)-distinguishable, but not r(k - 1)-distinguishable and there exists an input a e I s.t. for each b e h2(s, a) n h2(t, a), states s-after-(a,b) and t-after-(a,b) are r(k - 1)-distinguishable, while d(s-after-(a,b), t-after-(a,b)) and d(t-after-(a,b), s-after-(a,b)) are their r-distinguishing sets, respectively, then the set of traces d(s, t) = {(a,b) | b e h2(s, a)} u {(a,b)y | b e (h2(s, a) n h2(t, a)) a y e d(s-after-(a,b), t-after-(a,b))} is a r-distinguishing set of s w.r.t. t and d(t, s) = {(a,b) | b e h2(t, a)} u {(a,b)Y | b e (h2(s, a) n h2(t, a)) a y e d(t-after-(a,b), s-after-(a,b))} is that of t w.r.t. s [16, 17].
The above relations could also be applied to states from different machines. Considering the disjoint union of these machines, we have B r A, if and only if t0 r s0, where r e {<, =£, ¥}.
There exists a simple means to characterize the maximal common behavior of two machines.
Definition 5. Let A = (S, s0, I, O, h) and B = (T, t0, I, O, g) be FSMs. The intersection of A and B is an FSM A n B = (Q q0, I, O, 9) with the state set 0 c S x T, the initial state (s0, t0) = q0, and the behavior function 9: Q x I ^ 2QxO, where Q is the smallest state set s.t. ((s', t'), o) e 9((s, t), a) iff (s', o) e h(s, a) and (t , o) e g(t, a).
This operation on machines will be used in refining a given specification FSM. We also define a complete FSM which contains a given prefix-closed set of traces. This construction is useful in characterizing the set of implementation machines which have passed some test and thus shown to contain a certain set of traces of the specification machine.
Given a finite set T of traces over the sets I and O, we derive the observable machine M(T) = <pref(T), e, I, I, XT>, where the state setpref(T) is the set of all prefixes of each trace of the set T with e as the initial state, and the transition relation XT = {(p, a, pa) | pa e prefT)}. By construction, the set Tr(M(T)) coincides with the prefix closure of the set T. We complete the FSM M(T) by adding a designated dnc state that has a self loop transition for each pair (a, o) e I x O. For each state p e pref(T) and each input a undefined at state p, we define Xt(p, a) = {(dnc, o) | o e O } and denote M(T)the
completed form of M(T). The set {C | C 3 M(T)} contains each FSM C s.t. Tr(C) 3 T and C is a reduction of M(T).
The following proposition indicates how refinement of an FSM can be performed by using a set of common traces and intersection of machines.
Proposition 2. Given a complete FSM A = (S, s0, I, O, h) and a subset T of traces of A, for each complete FSM B e {C | C 3 M(T)}, if B < A then B < A n M(T).
Proof. Consider FSM B = (Q, q0, I, O, f) of the set {C | C 3 M(T)}, i.e., B 3 M(T). Given a trace y e Tr(A), let a be the longest prefix of y that is in the prefix closure of T, i.e., y = ap . Then, by definition of M(T), ap e Tr( M(T). Thus, y e Tr(A n M(T)).
2. FSM Tests
In this paper, we assume that a specification FSM is a complete observable, but not necessarily deterministic, machine, while any implementation FSM is complete, but not necessarily deterministic and observable. One could use a standard procedure for automata determinization to convert a given FSM into an observable one.
Definition 6. Given initially connected FSM U = (T, t0, I, O, X), U is a test if it is an observable acyclic output-complete FSM with the designated pass and fail states, s.t. X(t, a) = 0 implies t e {pass, fail} and if (fail, o) e X(t, a) then t-after-(a,o')p = pass for some o' 4 o and (a,o')p e (IxO)*.
We use the notion of a trivial test to denote a trivial FSM with the single initial pass state where no input is defined. Given a test U, we further refer to traces which take U from the initial state to the fail state as to fail traces and denote Trfail(U) the set of all fail traces. Pass traces are defined as follows, Trpass(U) = Tr(U)\Trfail(U).
Note that inputs (outputs) of the specification machine are also inputs (outputs) of a test, so a tester, executing such a test, applies inputs of the test to an implementation under test, observes its outputs and produces a corresponding verdict after observing completed traces. The tester produces the verdict fail whenever the implementation FSM executes an input/output sequence that takes the test to the state fail.
The test may have transitions with different inputs from a same state. In several work, it is not allowed to ensure the controllability of test execution. We leave the choice of inputs to a tester; executing such a test it simply selects one among alternative inputs during a particular test run. To execute another input, the tester first uses a reliable reset operation assumed to be available in any implementation to reset the implementation to its initial state and then re-executes the preamble taking the test to this input. Test execution continues until no more executable input sequences in the test are left. A strategy to achieve this is considered as an implementation detail of the tester and is not discussed here. Note that a test, as defined above, corresponds to what is called a test suite in other work, where a test case has no choice between inputs. We
also assume that all possible output reactions of a non-deterministic implementation to any input sequence can be observed in testing by repetitive application of the sequence (the complete testing assumption [11]).
Given two tests U1 = (Ti, tJ0, I, O, Xi) and U2 = (T2, t20, I, O, X2), Ui is a subtest of U2 if Tr(U1) c Tr(U2) and Trpass(U1) n Trfail(U2) = 0. To combine tests, we define the union of tests.
Definition 7. Let U1 = (T1, t10, I, O, X1) and U2 = (T2, t20, I, O, X2) be two tests s.t. the sets Trpass(U1) and Trfail(U2) as well the sets Trpass(U2) and Trfail(U1) are disjoint and T1 n T2 = {pass, fail}. The union U1 u U2 of tests U1 and U2 is the initially connected observable form of an FSM (Q, p0, I, O, y), where Q = {p0} u T1 u T2, p0 g T1 u T2, and the behavior function y : Q x I ^ 2QxO is defined as follows: for each a e I it holds thaty(p0, a) = h(s0, a) u g(t0, a); for each s e T1, y(s, a) = h(s, a), and for each t e T2,
y(s, a) = g(t, a). A trace y of U1 u U2 that is not a proper prefix of another trace is a pass trace if y is a pass trace of U1 or U2; otherwise, y is a fail trace of U1 u U2.
Given a state s of FSM A and a finite prefix closed set T of traces at state s, we derive the test Us(T) called the test closure of the set T in state s. For each prefix |a(i,o) e (IxO)* of each trace of the set T, the test Us(T) has a fail trace |J.(i,o') for each o' = O s.t. M-(i,o') g T. Each trace y e T that is not a proper prefix of another trace is a pass trace of Us(T).
To characterize conformance in this paper, we use the reduction relation (Definition 3). Recall that reduction conformance relation requires that in response to each input sequence a conforming implementation FSM produces only output sequences of the specification FSM. Given a complete observable FSM A = (S, s0, I, O, h), let 3(4) be a set of complete (implementation) machines over the input alphabet I and the output alphabet O, called a fault domain. FSM B e 3(A) is a conforming implementation machine of A w.r.t. the reduction relation if B < A.
Definition 8. Given the specification FSM A, a test U = (T, t0, I, O, X), and an implementation FSM B e 3(A),
• B passes the test U, if the intersection B n U has no state where the FSM U is in the state fail. The test U is sound for FSM A in 3(A) w.r.t. the reduction relation, if any B e 3(A), which is a reduction of A, passes the test U.
• B fails U if the intersection B n U has a state where the FSM U is in the state fail. The test U is exhaustive for FSM A in 3(A) w.r.t. the reduction relation, if any B e 3(A), which is not a reduction of A, fails the test U.
• The test U is complete for FSM A in 3(A) w.r.t. the reduction relation, if it is sound and exhaustive in 3(A) w.r.t. the reduction relation.
The set 3(A) that contains all complete FSMs with at most m states is denoted 3m(A). A test is m-complete if it is complete in the fault domain 3m(A). Clearly, an m-complete test is also complete in any subset of 3m(A), i.e., an m-complete test is also k-complete for any k < m.
The following proposition states that a test complete for a refined specification machine is also complete for the original specification machine once all machines which do not possess traces used in the process of refinement are excluded from the fault domain.
Proposition 3. Given the complete specification FSM A, let T be a subset of traces of A. If a test W is complete for A n M(T) in 3(A) then W is complete for A in 3(A) n
Proof. Let W be a complete test for A n M(T). Consider FSM B e 3(A) n {C | C 3 M(T)} that is a reduction of A. Due to Proposition 2, B is a reduction of A n M(T) and thus, B passes the test W, i.e., W is a sound test for A in 3(A) n {C | C 3 M(T)}. Let now B e 3(A) n {C | C 3 M(T)} be not a reduction of A. In this case, B is not a reduction of A n M(T), i.e., B fails the test W, and thus, W is exhaustive for A in
3(A) n {C | C 3 M(T)}.
Since W is sound and exhaustive for A in 3(A) n {C | C 3 M(T)}, the test W is complete for A in 3(A) n {C | C 3 M(T)}.
3. Testing Nondeterministic Machines
3.1. Sufficient conditions of the test m-completeness
In this section, we formulate conditions stating when a test is m-complete for a given nondeterministic specification machine, mostly relying on a test generation method which we elaborated in [17], where, however, sufficient conditions are not stated explicitly.
Given a specification FSM A, states 5, p e S and trace ß e Tr(s), let pref,p(ß) be the set {ra | ra e pref(ß) a s-after-ra < p}. We define a relation <s,p on the set pref,p(ß), s.t. ra <s,p ra' for ra, ra' e pref,p(ß), if |ra| < |ra'| and s-after-ra < s-after-ra'.
Proposition 4 [17]. Given a specification FSM A, states s, p e S and trace ß e Tr(s), the relation <s,p is a partial order on the set pref ,p(ß).
Let <s,p be a partial order relation on the set pref ,p(ß). Given a chain Cs,p(ß) of the poset (prefs,p(ß),<sp), the length of Cs,p(ß) is denoted l(Cs,p(ß)). By definition, we assume that l(Cs,p(ß)) = 0 if the set pref ,p(ß) is empty.
A state s e S is deterministically reachable (d-reachable) in FSM A if there exists an input sequence % s.t. for each trace a of FSM A with the input projection % it holds that A-after-a = s; in this case, the set of all traces of FSM A with the input projection % is a d-transfer set for the state s. Given a set Sd of deterministically reachable states of FSM A, we determine a minimal subset (a set is minimal w.r.t. the inclusion) of the set Sd, called a d-core of the set Sd, that contains the initial state and, for each state s e Sd, a state that is a reduction of s. A set K of traces of A is a d-core cover of the set Sd if K has the empty sequence and for each state s in the d-core of the set Sd, the set K contains a d-transfer set for the state s. Given a subset R of states of A, we denote Rd = R n Sd and Kd(R) a d-core cover of the set Rd. For each state p e R n Sd, the set Kd(R) contains a d-transfer set for a state that is a reduction of state p.
A set R c S s.t. for all p1, p2 e R, p1 ^ p2 implies p1 ¥ p2 is called a set of pairwise
r-distinguishable states of A, we use RA to denote the set of all such sets of states of A.
Note that a trivial test is m-complete when there exists R e RA s.t. |R| > m. The reason is that in this case, the fault domain 3m(A) has no conforming implementation FSM. In the
following, we assume that m > |R| for all R e RA. We also assume that each singleton
R = {p} is an item of RA; in this case, by definition, the empty sequence is said to r-distinguish states of R.
Theorem 1. Given a complete specification FSM A, let U be a sound test for A in 3m(A). The test U is m-complete for A if each trace of U that is not a trace of A is a fail trace and there exists a set Sd of d-reachable states of A, a d-core cover Kd of the set Sd with the following properties:
• e e Kd;
• for each trace a e Kd, hl(s0, a) = {s}, and each sequence p of length mn that is a trace of A, it holds that there exists a pass trace in U with a prefix y of p s.t. for some set
R e Ra and for each p e R there exists a chain Csp(y) s.t. 'LpeRl(Cs,p(Y)) + |R n Sd| = m + 1, and for each pair of r-distinguishable states s1 and s2, the set Tr(U) contains the sets |ad(sb s2) and %d(s2, s1) whereA-after-^ = s1, A-after-% = s2, and |a, % e{v| v e Kd(R) u
UpeRaQp(Y)}.
Proof. According to Proposition 13 [17], given B = (T, t0, I, O, g), B e 3m(A), B ^ A,
that passes the test U, there exist a e Kd, hl(s0, a) = {s}, aY e U, s.t. for any set R e Ra
if EpeRl(Cs,p(Y)) + |R n Sd| = m + 1 then there exist (s', t) and (s", t), s' ¥ s", in the set
{(A n B)-after-ra | ra e (K(y) u UpeRaCs,p(Y))}, where K(y) c Kd and for each r e (R u
{A-after-ra | ra e UpeRaCs,p(Y)}) that has a reduction in the d-core of Sd there exists ct e K1(y) s.t. A-after-CT < r. Let sequences ra1 and ra2 be sequences which take A n B to (s', t) and (s'', t). According to Theorem 1, the set Tr(U) contains the sets ra1d(s', s'') and ra2d(s'', s'). By the definition of the sets d(s', s") and d(s", s'), the set of traces of FSM B at state t is not contained in d(s', s") or in d(s”, s'), and thus, B fails the test U.
For more detail and explanation on m-complete test generation for nondeterministic FSM, we refer the reader to our previous work [17].
According to Theorem 1, the length of an m-complete test does not significantly depend on the number of states of the specification FSM, it rather depends on the sets of d-reachable and r-distinguishable states and the degree of nondeterminism in the specification FSM. Thus, it is worth to check if given an FSM A = (S, s0, I, O, h) and a natural m, there exists a more deterministic reduction Am of the FSM A s.t. for each complete FSM C with at most m states, it holds that if C < A then C < Am. FSM Am is called an m-reduction of A. According to Theorem 1, we can expect that a shorter m-complete test can be derived for the specification FSM Am. A straightforward approach to derive an m-reduction of A is to explicitly enumerate all possible reductions of A with m states, derive the union of these reductions and intersect the union with the FSM A. However, such an approach is impractical, since the number of possible reductions of a given FSM with the given number of states can be exponential. On the other hand, it is difficult to know in advance whether an obtained m-reduction will be simpler than the given specification FSM, for example, whether an m-reduction has fewer states than the FSM A. Moreover, it is an open question what is the relationship between two m-reductions; i.e., whether any two m-reductions of a given FSM are equivalent. All the above issues need additional research before using m-reductions in test derivation.
Another way of making the specification FSM more deterministic is to use some knowledge about an implementation FSM which can be obtained through testing. For example, if an implementation FSM is more deterministic that the specification then after applying some inputs to the implementation FSM and observing the corresponding output sequences, we could refine the specification FSM. For some applied input
sequence, only fewer traces with this input projection should be left in the specification and if T is the set of such traces then the machine M(T) is the “loosest” machine that can represent each implementation FSM which has T as a subset of the trace set. According to Proposition 3, a complete test can now be derived for a more deterministic specification FSM A n M(T) and due to Theorem 1; this test is expected to be shorter than the one derived for the initial, more nondeterministic, specification FSM A. In the following section, we formally describe such a refinement of the specification FSM in adaptive test execution.
3.2. Adaptive Test Execution Method
A simple minded test execution method would try to repeatedly execute each and every sequence of input stimuli used in a given test against an implementation under test and terminate its application only when all the produced traces take the test into pass state or a trace occurs which takes the test into fail state. As an example, consider the specification FSM shown on the left-hand side of Figure 1 and the test in Figure 2. The trace (a,1)(a,0) is one of the shortest traces of the test, this trace uses a shortest sequence of input stimuli of length two, namely aa, not counting the reset input. At the same time, if a given implementation produces in response to the input sequence aa the output sequence 02 instead of 10, the input sequence aa needs be extended to a longest input sequence, e.g., abbaa. The test shown in Figure 2 contains 13 different input sequences of the total length of 65 inputs (including resets and excluding prefixes) which need to be executed in the worst case, when, for example, the implementation is equivalent to the specification machine. Moreover, in this case, some input sequences should be executed several times in order to allow an IUT to produce all possible output responses to each input sequence.
The problem we are trying to address in this section is as follows: can the results of executing a part of a given test be used to terminate the test execution process earlier?
The idea of the proposed solution is to refine the specification FSM, by removing traces absent in a given implementation under test, which has already passed some subtest of a given complete test. The test execution process can be terminated as soon as the executed so far subtest is complete for the current refined version of the specification. The sufficient conditions formulated in Section 3 can be used for this purpose. This idea leads to the following.
Adaptive test execution method for checking whether an implementation under test with at most m states is a reduction of the specification FSM
Input. A complete FSM A = (S, s0, I, O, h), an m-complete test U = (T, t0, I, O, X) for FSM A w.r.t. the reduction relation, and an implementation FSM B in the form of a procedure Procedure(V) which for a given test V returns fail if B fails V or if B passes V the non-empty set of completed traces Trpass( V) n Tr(B).
Output. The verdict fail if B is a non-conforming implementation and the verdict pass if B is a conforming implementation as well as a subtest G of the test U that is a
complete test for A in 3m(A) n {C | C 3 M(Trpass(G))} and a refined specification At s.t. for each complete FSM D e 3m(A) n {C | C 3 M(Trpass(G))}, it holds that if C < A if and only if C < Ai.
1. Initialize G as a trivial test, i:= 1, Ai:=A.
2. Choose an arbitrary subtest U of U s.t. U^ £ G;I and for each pass trace a e U, the subtest U contains all the pass traces of U with the input projection a;I and go to Step 3. If there is no such subtest then go to Step 7.
3. Call Procedure(U), where Procedure(U) = Trpass(U) n Tr(B) if it does not return fail, else stop and return the verdict fail.
4. Derive the test closure of Procedure(U) in the initial state, i.e., the test Ui = Us0(Procedure(U)); G:= G u Ui.
5. Determine and reduce the FSM M (Procedure(U)) by merging its equivalent states.
6. Increment i by one and derive Ai:= M (Procedure(U)) n Ai_1. If the test G is not complete for Ai in 3m(Ai) according to the sufficient conditions of Theorem 1, then delete from U each trace which contains a e Trfail(G) as its proper prefix and replace the pass state U-after-a by fail state and go to Step 2.
7. Return the complete test G for A in the set 3m(A) n {C | C 3 M(Trpass(G))} and produce the verdict pass.
Theorem 2. The above procedure terminates and produces the pass verdict if and only if B is a reduction of A.
Proof. The verdict fail can be produced in one of the intermediate steps if and only the IUT has a trace that is a fail trace of the test U and thus, the verdict fail can be produced if and only the IUT is not a reduction of A.
Let the test G satisfy sufficient conditions of Theorem 1 for Ai in 3m(Ai) and the verdict fail was not produced in one of the intermediate steps, an implementation at hand is a reduction of Ai and thus, is a reduction of A.
If in Step 2 no new subtest can be chosen then G = U0(T), where T = Procedure(U).
Three cases are possible for an FSM B e 3(A): 1) B is a reduction of M(T) n A; 2) B is a not a reduction of M(T) n A, but B is a reduction of A; 3) B is not a reduction of A.
Case 1: Consider a trace y e Us0(T) n Tr(B). If y e T then, by construction of Us0(T), Y is a pass trace in Us0(T). If y g T then y is a prefix of a pass trace in Us0(T), since U is sound for A in 3(A).
Case 2: There exists a trace y e Tr(B) s.t. y^ e T;I. Denote |a the shortest prefix of y that is not a trace of M(T)n A. By construction of Us0(T), |a is a fail trace in Us0(T).
Case 3: FSM B fails the test U, i.e., FSM B fails the test Us0(T), as the latter preserves all the fail traces of U. ■
We now comment on the complexity of the above algorithm. The length of a resulting test suite can only decrease; in the worst case the whole initial test suite will be applied to an IUT. The intersection of two FSMs has the polynomial complexity w.r.t. the number of their states. The union of two FSMs, generally, has the exponential complexity according to the determinization operator. However, in Step 4, the input projections of G and Ui do not intersect and thus, there is no need to use the determinization operator. We also can stop to check the sufficient conditions in Step 6 any time facing increasing complexity; though, in this case, a longer resulting test suite might be obtained. Thus, it is always possible to execute the algorithm in polynomial time.
The method is illustrated in the next section.
3.3. Example
Consider the specification and implementation machines, shown in Figure 1. The specification FSM A with three states is nondeterministic, while the implementation FSM B with two states is deterministic and chosen to be a reduction of A. We use the FSM B as an implementation of the procedure Procedure(V) which for a given test V returns the non-empty set Trpass(V) n Tr(B) of completed pass traces, since B is a conforming implementation in this example.
a/0
Fig. 1. The specification A and implementation B FSMs; initial states are depicted in bold
Assuming that a given implementation has at most two states (as the FSM B), i.e., m = 2, we generate an m-complete test U for FSM A w.r.t. the reduction relation using the method [17]. Note that any two states of the specification A are not r-distinguishable; states 1 and 3 are d-reachable, thus, Sd = {1, 3}, the d-core cover Kd = {e, (b,{0, 1})}, where (b,{0, 1}) = {(b,0), (b,1)}. The 2-complete test U is shown in Figure 2, where the nodes inherit the names of states of the FSM A, note that all the deadlock states are pass states. To reduce the clutter, all the fail traces are omitted, for example, two transitions from the initial state to the fail state labeled a/2 and b/2 are not shown in the figure.
Fig. 2. The pass traces of 2-complete test U for the FSM A w.r.t. the reduction relation
Executing the first step, we initialize G as a trivial test, i:= 1, Aj.= A. Next, we select the right-hand side subtree in Figure 2 (depicted with dotted transitions) for a subtest U with the following input projections {baa, bab, bba, bbb)}. The set of completed pass traces Trpass(U ) = {(b,{0, 1})(a,{0, 1})(a,{0, 1}); (b,{0, 1})(a,{0, 1})(b,{0, 1}); (b,{0, 1})(b,{0, 1})(a,{0, 1}); (b,{0, 1})(b,{0, 1})(b,{0, 1})} contains 32 traces. The procedure Procedure(U) returns the set of completed traces Trpass(U) n Tr(B) = {(b,0)(a,1)(a,1), (b,0)(a,1)(b,1), (b,0)(b,1)(a,0), (b,0)(b,1)(b,0)}.
In the forth step, we obtain the test closure of Procedure(U ) in the initial state, i.e., the test U = US0(Procedure(U' )) (Figure 3).
The fifth step returns the FSMM (Procedure(U')), which represents all FSMs which can pass the test Uu shown in Figure 4. The FSM has no equivalent states.
Next, we construct the first refined specification A2 = M (Procedure(U')) n A1 (Figure 5).
Fig. 5. The FSM A2 = M (Procedure(U)) n A1; where states are renamed as follows: x1 - 1; y3 - 2; p3 - 3; t3 - 4; z1 - 5; z2 - 6; z3 - 7
Fig. 3. The test U1 = USQ(Procedure(U)), where p denotes pass and f - fail states
Fig. 4. The FSM Mi (Procedure(U)) in the first iteration
The FSM A2 has five d-reachable states, 1, 2, 3, 4, and 7. There are four pairs of r-distinguishable states, (1, 2), (1, 3), (2, 4), (4, 5). The obtained machine refines the original specification FSM A, its traces constitute a proper subset of that of A; it is “more” deterministic than A, at the price of having more states. We delete from U each trace which contains a e Trfail(G) as its proper prefix. For example, all the traces with the prefix (6,0) will be deleted from the set of pass traces in Figure 2. The test G is not m-complete for A2 and m = 2, according to Theorem 1, as it violates its sufficient conditions. For example, it contains no trace which starts with the input a.
Step 2 has to be executed again. To satisfy the above mentioned sufficient condition, we select another subtest U of the 2-complete test U (dashed transitions in Figure 2), with input a followed by a and b. The set of completed pass traces Trpass(U) = = {(a,0)(a,{0, 2}), (a,0)(b,{0, 2}), (a,1)(a,{0, 1}), (a,1)(b,{0, 1})} contains eight traces. Clearly, U1/ <x Gh, where the test G = U1, see Figure 3. The procedure Procedure(U’) returns now the set of completed pass traces Trpass(U) n Tr(B) = {(a,0)(a0), (a,0)(b.0)}.
In the forth step, we obtain the test closure of Procedure(U ) in the initial state, i.e., the test U2 = US0(Procedure(U)) and G = U1 u U2 (Figure 6).
The fifth step returns now the FSM M (Procedure(U)), Figure 7, which represents all the FSMs as its reductions which can pass the test U2, shown in Figure 6. The FSM has no equivalent states.
The refined specification FSM becomes now A3 = M (Procedure(U)) n A2 (Figure 8). The reduction relation between states of A2: 1 < 6; 1 < 2 < 7; 5 < 8; 3 < 6; 3 < 4 < 8. Pairs of r-distinguishable states are (1, 3); (1, 4); (2, 3); (2, 4); (3, 5); (3, 7); (4, 5); (4, 7). The test G for FSM A3 satisfies the conditions of Theorem 1, i.e., the test G is
Fig. 6. The tests U2 = US0(Procedure(U)) and G = Ui u U2
Fig. 7. The FSM hi (Procedure(U)) in the second iteration
2-complete for A3. Due to Proposition 3, the test G is complete for A in 32(A) n { C | C 3
M(Trpass(G))}.
Fig. 8. The FSM A3 = M (Procedure(U)) n A2, where states are renamed as follows: x1 - 1; >6 - 2; z2 - 3; z3 - 4; z4 - 5; z5 - 6; z6 - 7; x7 - 8
The original test U (Figure 2) contains, as mentioned above, 13 input sequences of the total length of 65 inputs (including resets) and all these sequences should be applied to the given implementation FSM B (Figure 1) if the tester does not take into account output responses produced by the FSM B, i.e., it executes this test in a preset manner.
However, the tester executes this test adaptively, i.e., it only applies a suffix of an input sequence that is defined in the test after the output response of B to the prefix of this sequence. For example, the tester would not apply the input sequence abbab to the FSM B, since it should be applied only when an IUT has the output response 02 to the prefix ab, but the FSM B replies with 00. In such adaptive execution of this test against the given deterministic implementation FSM B (Figure 1), the tester would use ten input sequences of the total length of 54.
At the same time, the proposed method needs only six input sequences of the total length of 18. This indicates that refining the specification FSM can bring a significant saving in testing efforts, especially in dealing with nondeterministic implementation FSMs which require repeated test runs driven by a same input sequence to observe all the implemented traces. In fact, each dropped input sequence of the original test would have been repeatedly executed to make sure that all the implemented traces with this input projection are observed. The saving tends to increase when a given implementation FSM is less nondeterministic than its specification FSM.
Conclusion
We considered testing of nondeterministic FSMs, when tests are first generated from a given specification and then executed against a given implementation FSM. This type of testing avoids repetitive test generation when several implementations (or its versions) have to be tested against a given specification. In this setting, the overall testing efforts can be reduced by minimizing a part of given tests that needs to be executed against a particular implementation. The novelty of the proposed solution lies in the iterative refinement of the original specification during adaptive test execution. The test execution terminates when the executed test is found to be complete for the refined specification FSM.
Several possible avenues for future work can be indicated. First, we have already mentioned that characterization of all possible reductions of a given FSM which have
a given number of states (^-reductions) is an open interesting problem. Then, the proposed method does not provide any hint on which subtest of a predefined test has to be chosen at each step of the iteration. It looks like ideas similar to those proposed in [7] can be of help to address this issue. Finally, our current work is to use the idea of specification refinement in online testing, thus refining both, the specification and test.
REFERENCES
1. AboElFotoh H., Abou-Rabia O., and Ural H. A test generation algorithm for protocols modeled as non-deterministic FSMs // The Software Eng. J. 1993. V. 8(4). P. 184 - 188.
2. Alur R., Courcoubetis C., and Yannakakis M. Distinguishing tests for nondeterministic and probabilistic machines // Proc. of the 27th ACM Symp. on Theory of Comp. 1995. P. 363 -372.
3. Dorofeeva M., Petrenko A., Vetrova M., and Yevtushenko N. Adaptive test generation from a nondeterministic FSM // Radioelektronika i Informatika. 2004. No. 3. P. 91 - 95.
4. Hierons R.M. Adaptive testing of a deterministic implementation against a nondeterministic finite state machine // The Computer J. 1998. V. 41(5). P. 349 - 355.
5. Hierons R.M. Testing from a non-deterministic finite state machine using adaptive state counting // IEEE Trans. Computers. 2004. V. 53 (10). P. 1330 - 1342.
6. Hierons R.M. Using candidates to test a deterministic implementation against a nondeterministic finite state machine // The Computer J. 2003. V. 46 (3). P. 307 - 318.
7. Hierons R.M. and Ural H. Concerning the ordering of adaptive test sequences // Proc. of the 23rd IFIP International Conference on Formal Techniques for Networked and Distributed Systems (FORTE 2003). 2003. LNCS volume 2767. P. 289 - 302.
8. Hwang I., Kim T., Hong S., and Lee J. Test selection for a nondeterministic FSM // Computer Communications. 2001. V. 24/12 (7). P.1213 - 1223.
9. Kloosterman H. Test Derivation from Non-Deterministic Finite State Machines // Proc. of the IFIP TC6/WG6.1 Fifth International Workshop on Protocol Test Systems. 1992. P. 297 - 308.
10. Kufareva I., Yevtushenko N., and Petrenko A. Design of tests for nondeterministic machines with respect to reduction // Automatic Control and Computer Sciences. 1998. No. 3. P. 1 - 6.
11. Luo G.L., Bochmann G.v., and Petrenko A. Test selection based on communicating nondeter-ministic finite-state machines using a generalized Wp-method // IEEE Trans. Software Engineering. 1994. V. 20(2). P. 149 - 161.
12. Luo G., Petrenko A., and Bochmann G.v. Selecting test sequences for partially specified nondeterministic finite state machines // Proc. of the IFIP TC6/WG6.1 Seventh International Workshop on Protocol Test Systems. 1994. P. 95 - 118.
13. Miller R., Chen D., Lee D., Hao R. Coping with nondeterminism in network protocol testing. testing of communicating systems // Proc. of the IFIP TC6/WG6.1 19th International Conference on Protocol Test Systems. 2005. LNCS volume 3502. P. 129 - 145.
14. Nachmanson L., Veanes M., Schulte W., et al. Optimal strategies for testing nondeterministic systems // Proc. of ISSTA. 2004. Software Engineering Notes ACM. V. 29. P. 55 - 64.
15. Petrenko A., Yevtushenko N., Lebedev A., and Das A. Nondeterministic state machines in protocol conformance testing // Proc. of the IFIP TC6/WG6.1 Sixth International Workshop on Protocol Test Systems. 1993. P. 363 - 378.
16. Petrenko A., Yevtushenko N., and Bochmann G.v. Testing deterministic implementations from their nondeterministic specifications // Proc. of the IFIP TC6/WG6.1 Ninth International Workshop on Protocol Test Systems. 1996. P. 125 - 140.
17. Petrenko A. and Yevtushenko N. Conformance tests as checking experiments for partial nondeterministic FSM // Proc. of the 5th International Workshop on Formal Approaches to Testing of Software (Fates 2005). 2005. LNCS volume 3997. P. 118 - 133.
18. Tretmans J. Test generation with inputs, outputs and quiescence // Proc. of TAGAS’96. 1996. P. 127 - 146.
19. Tripathy P. and Naik K. Generation of adaptive test cases from nondeterministic finite state models. Protocol test systems H Proc. of the IFIP TC6/WG6.1 Fifth International Workshop on Protocol Test Systems. 1992. P. 309 - 320.
20. Zhang F. and Cheung T. Optimal transfer trees and distinguishing trees for testing observable nondeterministic finite-state machines H IEEE Trans. Software Engineering. 2003. V. 29(1). P. 1 - 14.
21. Yevtushenko N., Lebedev A., and Petrenko A. On checking experiments with nondeterministic automata !! Automatic Control and Computer Sciences. 1991. V. 6. P. Si - 85.
Статья представлена кафедрой программирования факультета прикладной математики и кибернетики Томского государственного университета. Поступила в редакцию 1З января 2009 г.
