CC BY
30Recognition and Explanation of Incorrect Behavior in Simulation-based Hardware Verification
Mikhail Chupilko*, Alexander Protsenko^
Institute for System Programming of the Russian Academy of Sciences (ISPRAS) t National Research University Higher School of Economics (NRU HSE) {chupilko,protsenko} @ispras.ru
Abstract—Simulation-based unit-level hardware verification is intended for dynamical checking of hardware designs against their specifications. There are different ways of the specification development and design correctness checking but it is still difficult to diagnose something more than incorrect data on some or other design outputs. The proposed approach is not only to find erroneous design behavior but also to make an explanation of incorrectness on the base of resulted reactions based on special mechanism using a list of explanatory rules.
I. Introduction
Taking up to 80% of the total verification efforts [1], verification of HDL designs remains being very important. We expect the verification labor costs to be decreased by means of more convenient and substantial diagnostic information. The most complicated problem that underlies in all the approaches to hardware verification is how to represent the specification in machine-readable form that can be both convenient for development and useful for verification purposes. Typically, the specifications can be represented by means of temporal assertions (like in SystemVerilog in general and in Unified verification methodology [2] in particular), or using implicit contracts in form of pre- and post-conditions applied for each operation and micro-operation [3], or by means of executable models. The way of assertion usage lacks of certain incompleteness as assertions covers some of other quality and their possible violation shows only the quality without any guesses why it has happened. To guess something in this case, we should have had a bit higher representation of specifications. The way of implicit specification by means of contracts allows showing which micro operation does not work, but it is still difficult to interpret such information as such interpretation requires lower specification representation.
The executable specification can be considered as being the most useful in the error explanation. To be the most appropriate, the specification should imitate the logic architecture of HDL designs, and the test system is to have mechanisms of explanations the results of simulation. Exactly such mechanisms based on executable specifications are implemented in the proposed approach to test system development as it will be shown later.
The rest of the paper is organized as follows. The following chapter introduces the method of specification and test system development. The third chapter tells about reaction checker work. The fourth chapter reveals the theory underlying the explanatory mechanism. The fifth chapter says a few words about implementation of the approach in C++ library named C++TESK Testing ToolKit [4].
Diagnostics information Fig. 1. Common architecture of test system
Then a few words about the approach application are given. The seventh chapter concludes the paper.
II. Specification and Test System Architecture
The typical test system for unit-level simulation-based hardware verification includes the following three parts: generator of stimuli, reaction checker, and design under verification (DUV) connected to the test system via special adapter. The proposed approach follows the same tradition but formulates properties of test system components more strictly. Let us shortly consider all the parts of the ordinary test system developed according to the approach (see Figure 1) and then review reference model development more thoroughly.
It should be noticed that fully colored elements in Figure 1 are derived from the supporting library (C++TESK), halfcolored elements are developed for each DUV manually on the base of the supporting library, and white-boxed elements are developed fully manually.
Test oracle is the test system core. In fact, the test oracle works as a typical reaction checker; it receives stimuli flow from stimuli generator, receives implementation reactions (DUV reactions) enveloped into messages, and compares them with model reactions produced by reference model. Each message consists of a number of fields carrying data. Only messages with the fields of the same data types are comparable. The test oracle includes a replacement for the reference model which is called reference model environment. The environment consists of a list of operations and functional dependencies between data on output and input interfaces. The operation
description is based on extension of external reference model with timing properties.
The other parts of the test oracle are reaction matcher, and diagnostics subsystem. The reaction sequence made by the reference model is processed by the reaction matcher. It consists of processes each of which processes reactions on one particular reference model interface. As each reference model reaction is bound to a particular output interface, so that all the reactions are subdivided into a set of model interfaces. A reaction arbiter is defined for each output model interface. This component orders model reaction as follows.
When the model reaction is received by the reaction matcher, special process waiting for correspondent implementation reaction is started. If the implementation reaction is found, the process asks the reaction arbiter of the interface whether it can catch the reaction. The reaction arbiter contains a list of model reactions, registered at the interface where the arbiter is defined and not yet matched to the implementation reactions. The match process asking the arbiter about possibility of catching, the arbiter checks the list and according with a strategy of reaction selection (i.e. FIFO, LIFO, data matching) permits or forbids the matching process to catch the implementation reaction.
It is the way of reaction arbitration on each output interface. If the catching is allowed, the model reaction is deleted from the arbiter‘s reaction list, and the couple of model and implementation reaction is sent to the diagnostics subsystem. If the catching is forbidden, the matching process returns to the state of looking for the next implementation reaction. If the waiting for implementation reaction timeout is reached (the timeout can be set up to each interface separately), the reaction is sent to the diagnostics subsystem alone without implementation reaction marked as missing reaction. Besides processes looking for implementation reactions launched by model reactions, special processes named listeners are launched by test system for each interface. Each listener is bound to a particular interface and works as follows.
It contains an infinite loop of receiving implementation reaction, shaping the message with the reaction data, checking whether the reaction is matched to the correspondent model reaction at the next cycle after the implementation is completely received by test system. If the matching has happened, the listener returns to its first state and starts looking for the next implementation reaction. If the listener finds out that the implementation reaction has not been taken by any model reactions, it has been waiting for a certain implementation reaction timeout, having placed the implementation reaction into special buffer, offering next model reaction to match with the given implementation reaction. If the implementation reaction timeout is reached, the reaction is sent to the diagnostics subsystem alone without model reaction marked as unexpected reaction.
III. Reaction Checker Algorithm
The reaction checker work can be described by means of an algorithm showing clearly its possibility of catching all visible DUV defects. To provide the algorithm, some introduction might be useful. There are two definitions, the algorithm and a theorem about the reaction checker work.
All the input and output signals of DUV (implementation) are subdivided into input and output interfaces. The set of input and output interfaces of the reference model (specification) matches the one of the implementation (In and Out). Alphabets of stimuli and reactions of the implementation and specification also match each other (X and Y). Set of implementation state (Simpl) and specification states (Sspec) speaking generally might differ but initial states of implementation and specification are marked out (simpIo G Simp; and
sspeco G Sspec).
Applied during testing to input interface in G In stimuli
are elements of the sequence Xi
<(*i,ti))!
where
xi G X is a single stimulus, tj G N0 is the time mark of its application (ti < ti+1,i = 1,n - 1). The set of stimuli sequences applied during testing to input interfaces will be denoted as X =< Xini,..., XTinn > and called stimuli sequence. Stimuli sequence admissibility is defined by definitional domain Dom C |J^=0(X x N0)fc.
Implementation answering the stimuli sequence X produces reactions Ymp(X) = <(v'i,ti))”=1 and sends them to the output interface out G Out, where yi G Y is a single reaction, ti G N0 is time of its sending (ti < t'i+1,i = 1, m - 1). Let the set of reaction sequences emitting by the implementation to all interfaces be denoted as Yimpl =< Ympl,..., Ympr > and called implementation reaction sequence.
Specification answering the stimuli sequence X produces reactions Ys°Ufc(X) = <(yi,ti))k=1 and sends them to the output interface out G Out, where yi G Y is a single reaction, ti G N0 is time of its sending (ti < ti+1, i = 1, k - 1). Let the set of reaction sequences emitting by the implementation to all interfaces be denoted as Yspec =< Ys°UCl,..., YfpJ* > and called specification reaction sequence.
Let each output interface out G Out to be equipped with reaction production timeout At°Mt g N0. Let also each finite stimuli sequence results in a finite reaction sequence. Let us denote single element of reaction sequence Y = <(yi, ti))k=1 as y[i] = (yi, ti). The operation of element removing from the reaction sequence Y\ (y, t) is defined as follows: if the element being removed is absent in the sequence, the result consists of the former sequence; if the element is in the sequence, its first entrance in the sequence will be removed. The sequence length is denoted as m = |Y |.
Definition 1: The implementation's said to correspond to the specification if Vout G Out and VX G Dom |Yimp;(X)| =
I Yspfc (X )| = m°Mt is satisfied and there is a rearrangement
4 of the set {1,,m°Mt} so that Vi G {1,...,m°Mt}
t,- < /< t,- + Atout f is satisfied, where j
((i).
Definition 2: The implementation behavior is said to have an observable failure if the implementation does not correspond to the specification or 3X g Dom and Bout G Out so that either |Ymp;(^)| = |Y>s'peC(-X)|, or for each rearrangement
7r°Mt of the set {1,..., m°Mt} Bi G {1,..., m°“4} for which
yi = yj t, > ti Lti > t, + AtoU
is satisfied, where j = n°Mt(i).
Action 1 reactionMatcher[Yjmp;, YSpec]
Guard: true Input: Y^mpl ? Xspec Y* -t= Y
Y spec ^ Y spec
for all i € |Yimpi | do tno^ ti
y now
Y spec
Y now ^
Y missing ^ if Ynow
Y missing
^def ect
end if
Y now ma if Ynow
matched
{(y,t)|3j ^ {(y,t) € = 0 then
^ ^defect
(y,t) = -s*pec[j] A t < tn
Yspew It + At<tnow}
pi ynow M Y missing
now
spec
w ^ J(y Y
tched ^ {(y, 6) € Ys
then
^defect ^ ^defect ^ (yi,ti)
end if
(ymatched?/matched) ^ argmin(y,t)G Y \ (ymatched, tmatched)
|y = yi A t < ti < t + At}
y * y *
Y spec Y spec
end for
if |YspecI =
Yd
defect
end if
return Ydefect
then
^def ect
n Y
spec
Lemma 1: If reaction sequence Yimpl and Yspec are finite, and |Yimpl| = | Y'^spec | then test oracle returns negative verdict.
Proof: Suppose the main cycle of the algorithm not to find a failure. In this case the number of elements in sequence Ys*pec (which at the first step was equal to the number of elements in Yspec) will be decreased to the number, which the sequence Yimpl contains. If |Yimpl| > |Yspec|, then there is no step of the test oracle algorithm to find reaction from sequence Yspec correspondent to current being worked under reaction from sequence Yimpl. In this case test oracle finishes its work with negative verdict. We had supposed that such a situation cant occur, so that |Yimpi| < |Yspec|. In this case |Yspec| = |Yspec|-|Yimpl | > 0 and the oracle finishes its work with negative verdict in due to condition if |Ys*pec| = 0 then return (false) after the main cycle having finished. ■
Theorem 1: Test oracle working according to the proposed algorithm allows constructing significant tests (it means that oracle is not mistaken having found certain defect).
Proof: The case when |Yimpl | = |Yspec| meaning that there are different numbers of implementation and specification reactions is considered to be erroneous according to the definition 2. It was considered in the lemma and shown that the test oracle in this case does return negative verdict.
Let us consider the case |Yimpl | = |Yspec| = 0. Here the main cycle of test oracle work is not executed, the condition if |Yspec| = 0 then return (false) is not satisfied too and the test oracle returns positive verdict (true). The case of empty sequences is understood as correct according to the definition 2. According to the induction rule of inference, let us suppose that for the case |Yimpl| = |Yspec| = n test oracle returns verdict correctly. Let us prove that the same situation takes place if the numbers of elements in sequences are equal to n +1. According to the definition 2, defect can be found if for each rearrangement n of set 1,..., n3i € 1,..., n, when
Type name Reaction pair Definition of type
NORMAL (r spec , rimpl ) dataimpl & if ace spec — ifaceimpl & timemin < time < timemax
INCORRECT (r spec , rimpl ) dataspec = dataimpl & if acespec — if aceimpl & timemin < time < timemax
MISSING (rspec, NULL) ^rimpl ^ Rimpl \ -r,normal,incorrect ■ £ Rimpl : ifacespec = ifaceimpl & timemin < time < timemax
UNEXPECTED (NULL, rimpi) |rspec ^ Rspec \ Rnp0ercmal,incorrect : ifaceimpl — ifacespec & timemin < time < timemax
= yj tj > ti ti > t,- + Atout
is satisfied, where j = n(i).
Let us remove last elements of sequences and make sequences where the numbers of elements are equal to n and to which test oracle works correctly. Let us consider the case of the following two removed reactions. Negative verdict can be returned only in two cases: the first one is if Ym°swing = 0 then return (false), the second one is if Ymowched = 0 then return (false). The first case occurs only when ti > t, + Atout according to the definition 2, and the second case takes
place only in when
y'i = y, t, > ti
according to the definition 2.
Therefore, test oracle returns negative verdict only when there is erroneous reaction in any finite reaction sequences. ■
IV. Diagnostics subsystem Let reaction checker use two sets of reactions:
spec
{rspec^ }i=0 and -^impi — {rimplj }, = 0.
R
Each specification reaction consists of four elements: rspec = (data, iface,timemin,timemax). Each
implementation reaction includes only three elements: rimpi = (data, iface, time). Notice that timemin and
timemax show an interval where specification reaction is valid, while time corresponds to a single timemark: generation of implementation reaction always has concrete time mark.
The reaction checker has already attempted to match each reaction from Rspec with a reaction from Rimpl, making a reaction pair. If there is no correspondent reaction for either specification or implementation ones, the reaction checker produces some pseudo reaction pair with the only one reaction. Each reaction pair is assigned with a certain type of situation from the list normal, missing, unexpected, incorrect.
For given reactions rspec € Rspec and rimpl € -^impl,
these types can be described as in Table I. Remember that each reaction can simultaneously be located only in one pair.
The diagnostics subsystem has its own interpretation of reaction pair types (see Table II). In fact, the subsystem translates original reaction pairs received from the reaction checker into new representation. This process can be described
as M ^ M*, where M = {(rspec,rimpi,type)i} is a
}
t
Type name Reaction pair Definition of type
NORMAL (r spec , rimpl ) dataspec —
INCORRECT (r spec , rimpl) dataspec —
MISSING (rspec, NULL) -/1 7-1 \ nnormai.incorrect ?rimpZ € PimpZ\Pimp
UNEXPECTED (NULL, fimpj ) pr /= p \ pnorma!,incorrect -P ' spec € pspec \ pspec
TABLE ii
DIAGNOSTICS SYSTEM REACTION PAIR TYPES
Rule 5: If there is a missing reaction pair and an incorrect reaction pair (aspec, null), (bspec, aimpl) :
set of reaction pairs marked with type from the list above. M* = {(rspec, rimpl, type*)*} is a similar set of reactions pairs but with different label system. It should be noticed that these might be different M * dependent on the algorithm of its creation (accounting for original order, strategy of reaction pair selection for recombination, etc). This question will be discussed after so called transformation rules are presented.
Having made reaction pair set, reaction matcher sends it to the diagnostics subsystem to process them providing verification engineers with explanation of problems having occurred in the verification process. The diagnostics subsystem is underlain with a special algorithm, consisting of consequent application of the set of so called rules, each of which transforms the reaction pairs. Some rules decrease the number of pairs, having found pairs with correspondent implementation and specification reactions, collapse them and write diagnostics information into log-file. Other rules make it possible to recombine reaction pairs for better application of rules from the first type. The third part of rules uses special technique to find similar reactions according to the distant function to recombine the reaction pairs for better readability but do. The distant function can be implemented in three possible ways. To begin with, it may account the number of equal data fields in two given messages. Second, Hamming distance may be used as one can compare not only the fields but the bits of data carried by the fields. The measure of closeness between two given reactions is denoted as C(rspec, rimpl).
Each rule consists of one or several pairs of reactions. In cases of missing of unexpected reactions, one of the pair elements is undefined and called null. Each pair of reaction is assigned with model interface. Left part of the rule shows initial state and right part (after the arrow) shows result of the rule application. If the rule is applied to several reaction pairs, they are separated with comma. Now, let us review all these twelve rules that we found.
Rule 1: If there is a pair of collapsed reactions, it should be removed from the list of reaction pairs. (null, null) ^ 0.
Rule 2: If there is a normal reaction pair
(aspec, aimpl ) : dataaspec dataaimpl, it should be
collapsed. (aspec, aimp;) ^ (null,null).
Rule 3: If there are two incorrect reaction
pairs (aspe^ bimpl), (bspe^ aimpl) : dataaspec
dataaimpl & databspec = databimpi, these reaction pairs
should be regrouped. {(aspeo (bspeo aimpi)}
{ (aspe^ aimpl ), (bspe^ bimpl ) }.
Rule 4: If there is a missing reaction pair and an unexpected reaction pair (aspec, null), (null, aimpl) : dataaspec = dataa.mpi, they should be united into one reaction pair.
{(aspec,null) (^'W11, aimpl)} ^ {( ^spe^ aimpl)}.
dataa , these reaction pairs should
^^spe^ null), (bspec, aimpl)} ^
null)}.
datan
aspec
be regrouped.
{(aspec, aimpl), (bspec
Rule 6: If there is an unexpected reaction pair and
an incorrect reaction pair (null, aimpl), (aspec, bimpl) :
datan = data„. ,, these reaction pairs should
aspec azmpi 7 *
be regrouped. { (nul^ aimpl), ^spe^ bimpl)} =^'
{ ^spe^ aimpl ), (null, bimpl ) }.
Rule 7: If there are
pairs (aspec, bimpl ), (cspec, aimpl )
dataaimpi, these reaction regrouped.
^^pe^ ^mpl^ (cspe^ aimpl)}
two incorrect reaction
: datan
aspec
pairs should be
=>
{(aspec, aimpl), (cspec, bimpl)}.
The rules 1-7 allow finding the closest reaction pairs. The algorithm of their implementation is shown in 2 and 4 algorithms.
Action 2 matcfe[(rispec ,rijmpi ^ (r2spec ,r2impj )]_____
Input: RP1 = ^1^ ,ri!mp, ),RP2 = (r2Spec ,r2impi )
for all rule_number G |NormalRules| do
if rules[rule_number].isApplicable(RPi, RP2) then
return rule_number end if end for return 0
Action 3 /uzz^match[(rispec ,r1impl ), (r2spec ,r2impl ),rule] Input: RPi = (rispfc ,riimpi ),RP2 = (r2spec ,r2impi )
proximity_metric ^ 0
for all rule_number G |FuzzyRules| do
if (metric* = rules [rule_number].metric(RPi, RP2)) > proximity_metric then
proximity_metric ^ metric* rule ^ rule_number end if end for
return proximity_metric
Action 4 apply_normal_rules[{(rspec,rimpi)i}]
Input: {(rspec , rimpl)i}
for all r G |{(rspec,r*mpl)i}| do
if !r.collapsed then
for aU p G ^^spe^ rimpl )i |} do
if !r.collapsed&!p.collapsed then
if rule_number = match(r,p) then
(rspeci+i , rimpli+1 ), (r speCi+2 , rimpii+2 ) ^
rules [rule_number].apply_rule(r, p) r.collapsed ^ true p.collapsed ^ true return end if end if end for end if end for
Action 5 apply_/uzzy_rules[{(rspec,rjmpi)i}]
Input: {(rspec , rimpl )i }
for аll r G |{(rspec,rimpOi}| do
if !r.collapsed then metric* ^ 0
for all p G |{(rspec,rimp;)i|} do
if !r.collapsed&!p.collapsed then
metric = /uzzy_match(r, p, rule_number) if metric > metric* then metric* ^ metric rule_number* ^ rule_number si ^ r
s2 ^ p end if end if end for
if metric* > 0 then
(rspeci+i, rimpli+i ^ (rspeCi+2 , rimpli+2 ) rules[rule_number*].apply_rule(si, s2) si.collapsed ^ true s2.collapsed ^ true return end if end if end for
When the rules from the list of normal rules have been applied, the sets Rspec and Rimpl does not contain any not yet collapsed reactions with identical data. In this part of diagnostics subsystem work the stage of fuzzy rules (See 3 and 5 algorithms) comes.
{(nul^ aimpl, null) (aspec, bimpl
)}
=>
Rule 8: If
there
are
two
reaction
c(aspec, ajmp()
pairs
<
{(aspec, bimpl ), (bspec,aimpl)} c(aspec, bimpl) & c(aspec, ajmp() < c(bspec,aimpi) or
c(bspec, bimpl) < c(aspeo bimpi) & c(bspec, bimpl) <
c(bspec, aimpl), where c is the selected distance function and the value of c is the best amoung other fuzzy rules, these reaction pairs should be regrouped. {(aspeo bimpl ), (bspec,aimpi)} ^
{(aspec, ajmpi ), (bspec, bimpl)}
Rule 9: If there are two reaction pairs {(aspec, null), (null, aimpl)} and the value of the selected distant function c = (aspec, aimpl) is the best amoung other fuzzy rules, these reaction pairs should be regrouped.
{(aspec, null), (hu11, aimp|)} ^ {(aspec, aimpi)}
Rule 10: If there
spec
ill), (b.
'spec, ajmpi
are two reaction pairs
l)} : c(aspec, ajmp() <
{(a
c(bspec, aimpl), where c is the selected distance
function and the value of c is the best amoung other fuzzy rules, these reaction pairs should
be regrouped. {(aspec, null), (bspec, aimpl)} ^
{(
spec, ajmp( ), (bspec
), (bs
ll)}
Rule 11: If there are
{(hu11, ^mpl^ (aspec, bimpl ) }
two reaction pairs
c(aspec, aimpl) <
c(aspec, bimpl), where c is the selected distance function and the value of c is the best amoung other fuzzy rules, these reaction pairs should be
regrouped.
{(аspec, ^mpl^ (hu11, bimpl ) }
When all the metrics of fuzzy rules have been measured and all the most suitable rules have been applied, the time of the last rule comes.
Rule 12: If there is a reaction pair (aspec, ai .) with both specification and implementation parts, it should be collapsed.
(aspe^ aimpl ) (hu11, null).
The last rule allows transforming all the incorrect reaction pairs to show the diagnostics for the whole list of reaction pairs. Typically, after the application of each rule, the history of transformation is traced and then it is possible to reconstruct the parents of the given reaction pairs and all the rules they are undergone. Such a reconstruction of the rule application trace we understand as the diagnostics information.
V. Implementation
The proposed approach to development of test systems, reference model construction, reaction correctness checking, and diagnostics subsystem has been implemented in the open source library C++TESK Testing ToolKit [4] developed by ISPRAS. The library is developed in C++ language to be convenient for verification engineers. It contains macros enabling the engineers to develop all the parts of the test systems which should be done by hands. Some parts, like diagnostics subsystem algorithm, are hidden inside of the tool.
Results of diagnostics work are shown each time after the verification is over. Now they look like tables with all found errors and results of rule application: new reaction pair sets and the way of their obtaining.
VI. Results
The C++TESK testing toolkit including diagnostics subsystem has been used in the number of projects of industrial microprocessor development in Russia. The aim of the all approach is unit-level verification and on this level it can be a competitor to widely used UVM mentioned in the introduction.
It might be shown by the following fact. Typically, we started verification by means of C++TESK starts when the whole system had been already verified by UVM-like approaches. In spite of power of UVM, it does not include means to direct test sequence generation, which C++TESK does, means of quick analysis of verification results as diagnostics subsystem etc.
Results of application of different approaches depend on the qualification of the engineers and their familiarity with the approach. And on this point, we should say that our toolkit was used by people now being close to its development kitchen and despite it, they exactly managed to find those bugs we have already mentioned.
VII. Conclusion
The proposed approach to simulation-based unit-level hardware verification solves in some sense the task of dynamical checking of hardware designs against their specifications. It includes both means of specification development and diagnostics subsystem producing an explanation of incorrectness
on the base of special mechanism using formally represented specifications and a list of explanatory rules.
The approach has been used in the number of projects and shown its possibility to find defects and help verification engineers to correct them by means of diagnostics information.
Our future research is connected with more convenient representation of diagnostics results by means of wave-diagrams, localization of found problems in source-code.
References
[1] J. Bergeron, Writing Testbenches: Functional Verification of HDL Models. Kluwer Academic Pub, 2003.
[2] Unified verification methodology. [Online]. Available: http://www.uvmworld.org
[3] M. Chupilko and A. Kamkin, “Specification-driven testbench development for synchronous parallel-pipeline designs,” in Proceedings of the 27th NORCHIP, nov. 2009, pp. 1-4.
[4] C++tesk homepage. [Online]. Available:
http://forge.ispras.ru/projects/cpptesk-toolkit/
