CC BY
35
UDC 519.7:616.7+616.8
APPLICATION OF ANALYTICAL LEARNING TO THE SYNTESIS OF NEURAL NETWORK FOR PROCESS CONTROL OF PHYSICAL
REHABILITATION
Alexander Trunov,
Petro Mohyla Black Sea State University, First Vice - Principal, Professor, Medical
Equipment and System dept.,
p. h. d., 10, 68 Desantnykiv Str. Mykolayiv, 54003, Ukraine, E-mail :
trunovalexandr@gmail.com
Abstract — The problem of analytical learning of artificial neural network (ANN) is consider. Solutions in the analytic form for synaptic weight coefficients (SWC) as recurrent sequence are obtained. Convergence of recurrent approximation for two scheme of approach by a linear and quadratic curve are proved and discussed
Keywords — neural network; analytical learning; solutions - recurrent sequence; convergence
I. INTRODUCTION
The development of physical rehabilitation and evidence-based medicine methods [1]-[5] has been determined by a large extent achievements associated with the success of creating databases and knowledge bases for each patient throughout the life. The transitions from data bases, which have been accumulated personal data over the years and quantitative data analysis at this moment to knowledge data bases [1], [4] generates knowledge and formulate a prediction of quantitative indicators of the patient state of health as a result of the proposed course of physiotherapy or treatment. This is especially actual, when transitioning characteristics of the database, which are built for certain types of treatment and conditions are determined by the data analysis to knowledge bases, what are formed during the rehabilitation of injured as a result of accidents and various wounded. The experience of the neural network application in general, defines its as one, that contains a list of promising prospects [1]-[5]. Success application of networks and neural networks in particular [2], to creation of knowledge databases is particularly attractive approach [4]-[5]. In addition, it should be notice, that use of neural network for building of regulators are constrained of small values speed of retraining directly during the regulatory process [3], [7]. However, existing methods for learning such networks is not those, what can to allow quickly determine of the SWC, so that they have to will be improved. The
methods widely known in the literature are based on definition of gradients, back propagation error, genetic selection are quite slow. They do not work properly, especially in conditions of un-monotonicity and necessity of choice at the global level of minimum. Advances modeling [6], [7]-[10] and design [8]-[10], which achieved recently in [14]-[17], show that the method recurrent approximation (MRA) [14]-[15] with intellectualization of process analysis of functions behavior [16] can produce fast algorithms for oscillating processes [16], [17].
The first samples of application to analytical studies of ANN provided by MRA is done in [17], but the main problem, which is not solved until the present time -proving of convergences and comprising to type of proposed algorithms to analytical learning of ANN.
Thus, the generalization of system support of decision making for control of physiotherapy process and formulation of analytical expressions for SWC for arbitrary standards as recurrent sequence, mathematically proof of its existence and convergence for ordinary forms of activation function is the main purposes of this article.
To achieve this goal, the following tasks will be pointed, such as:
to obtain the expressions for analytical calculation SWC, as a function of purpose for
ordinary standards of neurons;
to prove the existence and convergence recurrent sequence of analytical expressions for SWC of the control system of process;
to receive solution as a sequence and demonstrate indicator of rapidity of convergence on the examples of two approaches: zero error of output to the reference values and requirement of minimizing the sum of squares error in reference values.
II. PROBLEM OF ANALYTICAL LEANING FOR ANN
For presentation of the general problem of neural network training will be considered two cases: single-layer artificial neuron and multi-layered ANN.
a. Single - layer artificial neuron.
Let consider an artificial neuron, that contains n inputs and one output. Assume, that all inputs are given at inputs of neuron from the output sensors as n component vector X. After amplification signal of each component, i.e. multiplication by the coefficients synaptic weights m and shift the specified value ®0 is formed by predetermined activation function, which is denoted as h (X), output Y . Suppose, that
the output signal is a scalar quantity. Suppose, that after examining the phenomenon observed M experimental facts. For these experiments, information about the values of all components of the vector of inputs and output are collected. We pose the problem to find the value of synaptic factors that ensure full compliance with the
standards of exemplary or satisfy the requirement of a minimum sum of squared deviations from them.
We introduce n dimensional space, that appears in the input vector X and output Y, then according to the structure of the neuron activation function argument will be S -scalar, linear combination of inputs shifted relative to the origin and which also appears in this space:
N
S = 'ZaiXi + ao. b. * * * * * * * * * (1)
i=1
We introduce an activation function assuming that it’s continuous together with its derivatives and it’s differentiable function, then the problem is reduced to the analytical study the problem of finding the roots of a scalar equation in the form:
8m = 0; 7 = 1, N; m = 1, M, (2)
where is denoted sm - error of output to the reference value Dm, which is
& m = D m
h + ®o j
and x7m, h [ xm j - the value component of vector for standard input and an activation
function, respectively, which belong to the indicated subscript. Established system of equations is complete in conditions, performance requirements: M = N +1.
The second version of the task is minimization of the sum of squared errors for all
known examples of standard
2
; i = 1, N; m = 1, M. (3)
M f N
min Г Dm - h Г a 'Xim ^Oo
(Oi m=1 1 i=1 J
b. Multi-layered neural net-work
Consider an artificial neural network, which contains the n inputs and m outputs.
Assume that all inputs are given to neuron from the n sensors and dials initial conditions input n component vector X . The network contains q layers of Nq
neurons in the layer. Each neuron from q layer has inputs and outputs. Two neurons:
first i transmitting information over the course from q -1 layer is connected to the second example j from q layer characterized synaptic weights Of and value of shift a-q . Suppose, that for M experiments, information about the values of all
components of the inputs and output vectors are collected. We set the problem to find
the value of synaptic weights, what ensure full compliance with the standards of exemplary or satisfy the requirement of a minimum sum of squared deviations from
them. We introduce n dimensional space that appears in the input vector X and output Y. Consider two neurons: first i in direction of the transfer of information
from the q -1 layer and the second j series from q layer. Output i - th is the input to
neuron j - th, as a result of the way Sq - becomes to j neuron as input value and its
activation function hkjj} [Sqj is equal to output Y(kj):
N (q}
c<q — v ,„(q'KSq).
s kj Z x kj & kij
і=1
. (q) = ,/q-1'>Sq),^(q)-
& k0 j Z y ki & kij ' & k0 j;
і=1
v(q) = u (q)
Y k hkj
(sq); Djm
= Y (Q)
Y kjm '
where subscript k denoting affiliation to k - th components of n(q) dimensional input and output vector. Given these designations, the analytical task learning neural network is reduced to two tasks to ensure zero error (2) or the problem of minimizing the sum of squares of errors (3) or combined - minimizing the squared error and zero error for selected standards, whose solution is presented for the first time in the works [17].
Thus, for analytical studies neural network formulate three types of tasks. First - the problem is the exact configuration layer network last known configuration at the other layers
8{o=D kjm - hkqq (Sj) = 0; i = 1, Nf; m = 1, M, (4)
where indicated - k - component of the error vector output of i - th neuron for m - th standard, n^ - the number of neurons in the last layer Q. Second - the problem is the exact configuration of the last layer of the network and random selection method other customizable layers. Third - the problem of approximate configuration layers based on the requirement of minimizing the sum of squares error
M
min Z
^q m=1
& kij
D
kjm
1, N m = 1, M; S
q
k
i=1
(5)
III. ANALYTICAL SOLUTION OF TRAINING NEURON PROBLEM
Problem (4) and (5) are reduced to the problem of finding roots of nonlinear equations, since the requirement of continuity activation function is implemented only as a non-linear continuous function. This same problem is reduced as problem finding SWC, that a set as the problem of minimizing the sum of squares of errors (3) or (5). In [17] are demonstrated, what under differentiability of vector function (2), or the lefts parts of the system (4) is an effective MRA. Assume what all activation functions are continuous and N+1 time differentiated, written sequence in a number of conditions (2) - first approach according to the MRA:
- for the scheme of approach by a linear
h ’ (si)-V
1 Ni 8 Ni
)=a
+ A"
Ni 8 І si)
(i)
1 _ 8 i \ 8 Nq , .
+-Z—i) Z't-^ Za
6 j=i 8a к; j=i 8a j j=1
j= 8©k
8 hU’S)
(iL
1 Ni 8 Ni * (i) ^hj(S|) + -Zt^Z'-г^ '
2 j=i 8a к j= 8a к
+
„( i)= akj
8wki
(i
a# )=a_
+....+ = 0; a = j к = 1,Nf); j = 1,N(e); m = 1, M; (6)
for the scheme of approach by a quadratic curve
Ni ... 8hк)(S'j)
h к) ( Stj)-D
(t )
ak =a j=
+ ZAakjP+1
8akq)
'kj
1 Nq , x
+ - Z Aa( i)
ak)=a
Z a Jcjp+1 ~ (q) Z
2 j= 8a -1 j=i
8 Nq , ,
ZAaj+1 x
8h к >( Sij)
8ak ’
+ ■
a
38ak) j“
^(q)-„
akj =
". <q, 8h ki ’( Sij)
Z Aakjp
8ak ’
„(і)_„ a=
= 0; к = 1,N f; j = 1,NQ m = 1, M
(S). _ -
j
x
(7)
Solution of nonlinear system (6) shall be introduced in the form of recurrent sequences for determining p+1 -th approach values of approximation of SWC akil+1
with layer i for the p+1 - th by MRA cubic approximation :
- for the scheme of approach by a linear
„( i) - „( i) _ j
a ksp+1 ~ a ksp A
h к) (Si)-D
8h к) (sij)
" „ (i) Shi (si)
z i)= a kj
Z Aa(i)
j=1. j *s
kjp+1
8a'k1
a'k'=a
1 Ni
2 j=1. j *s
+1 Z Aa “+
8®a ’
Ni , x
x Z Aa a
j=1
8a к)
akj
(i)_
1 Ni , .
+ -. Z Aa+1
6 j=1. j*s
d
«к. (i) 8 .V. (i) (S«)
8ak >
Z^'g^x Za
j=1 8 a) j=1
8a kii
aj )=a
> X
-1-1
8hki >( Sij)
<4i ’
8 N\ (i) 8hki>(Sij)
„( i)_ a kj
2 8a kj j = 1
Z Aaj,-
8ak)
8 «у.(i) 8 (i)8hк>(si)
, -(i)_ a kj
6 8aki) j=1
Z Aai
8a^^) j=1
kjp
8ak)
, -(i)_ a kj
X
к = 1.N(S); j = 1.N(S); s = 1.N(S); j * s. (8)
It should be noted, that the summation should be considered a condition j * s that arises as a result of the transposition terms on the right side of the equation to the left. For the solution of the problem using cubic approximation for scheme of approach by a quadratic curve will create recurrent sequence to determine p +1 - th value approximation ratio synaptic weights akip+1
.Л?) - „(я).
® kjp+1 ® kjp
D( ? )
Pk/p 2 /f( ? )
2 Akjp
+
^ D(q) ^ pkjP
2 A(q)
V 2 A— у
(q)
F kjP
/Я
p
(9)
where are indicated
a (я) = A—
і a
2 a®(q)
' ks
B * k)(s-
B®(q)
J7( q) =
F kjp =
h k) (Si)-В 3Si)
' ks
kjm
a »qA (q) Bh-)(sj)
(q) N®)
®<k )=fl
за®к?) /=i
a®!?)
Wkj)=a
• D(q) =
; Я P =
Bh-) (s^)
a®!? >
^(q )=
® к
„ (q )=
®kj
Nq
N
j=1, j * s
+ N A®g+1
Bhk1 (S^)
В® >
3®k
(q)
a n?
a?)=
®kj =
за®к k
NA®-)
(q)3* -' (SB)
a®-1
,,(q)=
®j =
k = 1, N(Q);
®ik )=fl
s = 1,N(Q); j = 1,NlQ); j * s; m = 1,M.
1 Nq
T N
2 ;=1,; * s
+1 N A®^p)+1
a n?
;a = ®
л?)
a® ®-=w * s
N Aj x
(Q).
It should likewise be noted, that the summation should be considered a condition j * s, what arises as a result of the transposition terms on the right side of the equation to the left.
Solution of the problem of minimizing the sum of squared errors is also feasible using the method of recurrent approximation [14]-[16]. To implement of its firstly derivative condition (5) as a function for each of the SWC, and then substitute out into the left side series by the MRA are written based on the second approach -requirement of minimizing the sum of squares error:
- for the scheme of approach by a linear
n? I Bh к ' (sq)
N ,
NI >
1 N? , ,
+~N A®S+1
2 j=1
[ Dkjm - h -) (S?)]
(q)=
®kj
N?
■N'
j=1
+ NA®(q)
kjp+1'
a Nq^Bhjkjl(sk,-Г (?w ,-i;
nI „ [D„m-hd’(--)]:
a® к )
a®k)
(q)=
®kj
a Nj [ah—(sljV (?)/ q\-i]
3®/—A®- ® x S1 a®-> [D--hkk (Skk)]:
a n? ,,
—: N A®kq)-
(10)
(q)
®# =a
1 N? , ,
+ “N A®—+1^-q N
6 k=1 a®-) j=1
a n? ,, a n? ,,
NA®“-- NA®k x
a® к > k=1
a ^ [ah-)(sljV (?)/ q\-i
a®— S1 a®- [D--hk (SkЯ
= 0;
„( ?)= ®kj
k = 1, N f; j = 1, N(Q); m = 1, M;
- for the scheme of approach by a quadratic curve
N? j5h(k](Si)
kN11 a®Lq)
[ D kjm - h (k ) ( Sj-)]
(? )=
®kj
Nq
N'
j=1
+ N A®(q)
k/p+1'
В .vj [Bhkk'(SkjГ (q)/ >-i;
nI „ [---h->()]:
a® к) k=1
a®k)
(q)= ® kj
2
X
1 Nq , , Q Nq , ,
+ ~ Z^—^ Z^ '' (
2 j=1 Qak) j=i QaV j=
Q Nq<\Qh'k)(siV {q), чи
~H) Z | q (q) [Dijm-hkj (Si)]
1 Nq
Q Nq
Q Nq
+-Z WUjQq Z АС^ z a 4 (q) -
6j=1 Qa'kj- J=1 Qaiq) j=1 Qaiq) j=1
Qaj Q Nq
Zj
(11)
(q)
akj =a
Sh'q'(Sl)r , q), q,n
q (q) \_Dk,m hi (Si)]
Qa kj
= 0;
al)=a
k = 1N; J = 1,N Q); m = 1, M.
IV.
THE ASSESSMENT OF CONVERGENCE.
Denote for simplicity and compactness of presentation and recording of the left side of conditions (6) or (10) the general expression L(a) . An algorithm of solution considered equations is a sequence, that under the conditions of existence convergence in general will the both sided and quadratic [9],[12]-[16]. For this scheme of approach by a linear we can write:
a+1 = a -
L(an)
L a )V (an у where is noticed:
v a) =
1 + z 1 dLi(a) (A n)k 1
Z(k)! dak l ' (a) x=an v n/
Let denote a* the root to which tends solution of system (6) or (7) and let suppose, what a function L(a) is integrated within square in the range [a0,a* ], then norm can be introduced
llL HI
a
J [L(a)] da
ao
1 / 2
For this conditions value of error of solution can be estimated as:
a+1 -a =
L(an)
or
La
l a )v (a)
Minimum and maximum can be determined as forms:
L(a )V(a ) n n
GK+1 -an\\ <
La
L’(a )V(a )
nn
L ’ (a )V(a ) n n
L'(a ) ; L'(a )V(a ) n' j’ n n
N
L'(a ) +Z-
n
1 dkL (a)
(k)! dak x=a
(A n)k-1.
min
mm
max
max
Thus, maximum of the second derivative of L(an) at a point of the root determines the speed of convergence, and therefore a third derivative L(a*) at a point on the root
determines the speed of convergence and the convergence is quadratic and both sided.
Now consider scheme of approach by a quadratic curve of (7) or (11) [13]-[16]. The proof of quadratic convergence would be made in two ways. The first - is to find the boundary, and then study its behavior. The second calculation is made of the function of recurrent transformations.
So let us assume that all activation functions and as result L(a) are continuous and N+1 time differentiated under B(a) ^ 0 the borders of ratios:
L'(a) with L(a) 2L(a)B(a)
lim
lim
B(a)^o B(a) ’ ’ b(®)^o B(a)
B(a) = 2^
8kL(a)
k =2
SA k
A'
a)
k-2 n-1
exist and lim
B( a )^0
[ L'(a)Y
< l, where is indicated
A = 0 k!
Under these conditions for scheme of approach by a quadratic curve solution can be simplified and written:
a
-a
L'jco) 1T L 2L{co)B{co) B{co) [L'(co)f ’
and under additional conditions for the assumption of convexity features the latest will have simplified border
lim (an+i -an) = -
B(a)^0
L(a) L '(a)
lim (an+i-an)
B(®)^0
= lim
B(®)^0
an+1 -an|| =
L(a)
V (a)
Thus, for the border we have result equal to a linear approximation scheme with linear approximation, rate of convergence of which as it’s shown in [9],[ 12]-[17] is both sided and quadratic.
Let we prove by another approach, that quadratic approximation schemes generally occurs both sided and quadratic convergence. We introduce the notation:
L '(an)
a n+1 a n
■ +
B(an) \
L Xan) B(an).
2L(an)
B(an)
Apply directly definition of norm to deference of two serial solutions for roots of system and compute its value at the point of root we obtain:
an+l a nn
L' (an) + Г L' (an) 2 2L(an) < L' (an) + Г L' (a„) 1 ( L(a)B(a)
B(an) І _ B(an) _ B(an) B(an) _ B(a„) \ {[ L’a )
L(a)
La)
(12)
This result shows, that only for sine minus in the case of quadratic approximation scheme is converge and it’s both sided and quadratic, and its speed substantially depends from the properties of the activation function and the type of task and less -from schemes of approximation. Thus, for considered problems of analytical learning to the synthesis of a neural network for process control physical rehabilitation, value of error is predetermined by the maximum value of the third and lowest value of
fourth derivative of activation function. In other case, for the problem of minimizing the sum of squares error, speed, of convergence is predetermined by maximum value of fourth derivative and lowest minimal value of fifth derivative. This specific results are formulated requirements for units of control system of physical rehabilitation. The consideration of two schemes of approach by a linear and quadratic curve is demonstrated formally equal results at the point a * of solution under existence and boundedness of L) and L'(o) functions. Practically speed of convergence of the sequence of solutions generated is determined by the evaluation of maximum derivatives of functions activation. That’s why estimation of maximum of L(a„)|
and \l'(o)\ in the range of definition of components input vector will be indicate
rapidity of convergence for different schemes of approximations and form of activation functions.
CONCLUSIONS
1 The task of analytical learning of neural network is reduced to analytical recurrent sequence as solutions a system of nonlinear algebraic equations.
2 The speed of convergence of solutions as the sequence is determined by the evaluation of maximum derivatives of functions activation and can be estimated as inequality relations.
REFERENCES
[1] Willett P. K., Thiagalingam Kirubarajan, Component and Systems Diagnostics, Prognosis, and Health Management, Том 4389 - 2001
[2] Artificial Neural Networks in Medicine
http://www.emsl.pnl.gov:2080/docs/cie/techbrief/NN.techbrief.ht
[3] Croall I.F., Mason J.P., Industrial Applications of Neural Networks (research reports Esprit,
I.F.Croall, J.P.Mason)
[4] A Novel Approach to Modelling and Diagnosing the Cardiovascular System http://www.emsl.pnl.gov: 2080/docs/cie/neural/papers2/keller.wcnn95.abs.html
[5] Pattern Recognition of Pathology Images
http ://kopernik-eth.npac.syr.edu: 1200/T ask4/pattern.html
[6] Learning internal representations by error propagation by Rumelhart, Hinton and Williams (1986).
[7] Alkon, D.L, Memory Storage and Neural Systems, Scientific American, July, 1989, p. 42-50
[8] Minsky and Papert, Perceptrons, An introduction to computational geometry, MIT press, expanded edition, 1969
[9] Trunov A. “Robust Fault Diagnosis of State and Sensor Faults in Nonlinear Multivariable Systems, / Trunov A., Polycarpou M.//” Proceedings of the 1999 American Control Conference, pp. 608-612, June 1999.
[10] Trunov A., “Robust Nonlinear Fault Diagnosis: Application to Robotic Systems,/ Trunov A., Polycarpou M.//” Proceedings of the 1999 IEEE Conference on Control Applications, pp. 1424-1429, August 1999.
[11] Bellman R. E. Quasilinearization and nonlinear boundary - value problems/ Bellman R. E, Kalaba R. E: New York, 1965.-183 p.
[12] Bellman R. E. Dynamic programming and modern theory of control / Bellman R. E, Kalaba R. - M.: Sience, 1969. -458 p.
[13] E. Halley, A new, exact, and easy method of finding the roots of any equations generally, and that without any previous reduction, Philos. Trans. Roy. Soc. London, 18 (1694), pp. 136-145.
[14] Trunov A.N. Application of methods recurrent approximation for solution of non-linear problems /Trunov A. // - Methods and algorithms of parametric analysis linear and non-linear models of transfer. Scientific Transaction Vol. 16.- М., MSOPU, 1998, p.142-156.
[15] Trunov A.N. Recurrence approximation in problems of modeling and design // Monografy - Mykolayiv: Petro Mohyla BSSU - 2011, 272 p.
[16] Trunov A.N. Intellectualization of the models’ transformation process to the recurrent sequence, European Applied Sciences, Ort Publishing, # 9 - 2013, vol. 1, p.123-130.
[17] Trunov A.N. Application of the recurrent approximation method to synthesis of neuron net for determination the hydrodynamic characteristics of underwater vehicles/Trunov A.//- Problem of Information Technology, Jornal, Vol. #02(016), Dec. 2014, p. 39-47
