CC BY
74
Vestn. Samar. Gos. Tekhn. Univ., Ser. Fiz.-Mat. Nauki
[J. Samara State Tech. Univ., Ser. Phys. Math. Sci.], 2018, vol. 22, no. 3, pp. 549-556 ISSN: 2310-7081 (online), 1991-8615 (print) d http://doi.org/10.14498/vsgtu1592
MSC: 65F10, 65F22
Implicit iterative schemes based on singular decomposition and regularizing algorithms
A. I. Zhdanov
Samara State Technical University,
244, Molodogvardeyskaya st., Samara, 443100, Russian Federation.
Abstract
A new version of the simple iterations implicit method based on the singular value decomposition is proposed. It is shown that this variant of the simple iterations implicit method can significantly improve the computational stability of the algorithm and at the same time provides a high rate of its convergence. The application of the simple iterations implicit method based on the singular value decomposition for the development of iterative regularization algorithms is considered. The proposed algorithms can be effectively used to solve a wide class of ill-posed and ill-conditioned computational problems.
Keywords: implicit iterative schemes, singular value decomposition, computational stability of iterative schemes, convergence rate, iterative regular-ization methods.
Received: 20th December, 2017 / Revised: 22nd August, 2018 / Accepted: 3rd September, 2018 / First online: 9th November, 2018
Introduction. Iterative methods of regularization play an exceptionally important role in solving a large class of ill-posed and discrete ill-conditioned problems [1-6]. They can be used to solve problems encountered in dynamics and kinetics, mathematical economics, geophysics, potential theory, antenna synthesis, acoustics, automatic processing of physical experiment results, determination of the shape of a radio pulse emitted by a source, plasma diagnostics, in ground or air geological prospecting (mathematical processing of measurements), in solving the inverse kinematic task of seismic, space research (spectroscopy), medicine (computed tomography), and processing image task.
Research Article
Q ©® The content is published under the terms of the Creative Commons Attribution 4.0 International License (http://creativecommons.org/licenses/by/4.0/) Please cite this article in press as:
Zhdanov A.I. Implicit iterative schemes based on singular decomposition and regularizing algorithms, Vestn. Samar. Gos. Tekhn. Univ., Ser. Fiz.-Mat. Nauki [J. Samara State Tech. Univ., Ser. Phys. Math. Sci.], 2018, vol. 22, no. 3, pp. 549-556. doi: 10.14498/vsgtu1592. Author's Details:
Alexander I. Zhdanov A http://orcid.org/0000-0001-6082-9097
Dr. Phys. & Math. Sci., Professor; Head of Department; Dept. of Higher Mathematics & Applied Computer Science; e-mail: zhdanovaleksan@yandex.ru
Iterative regularizing algorithms are obtained from classical iterative schemes using special stopping criteria that provide them with regularizing properties. The regularization effect in these algorithms is achieved due to matching the number of iterations with initial errors—as a regularization parameter is the number of iterations of the iterative algorithm (iterative residual principle of Tikhonov). Such a choice of the regularization parameter is easily realized, especially in iterative methods, and in most practical problems leads to results on accuracy close to optimal.
Most of the practically used iterative regularizing algorithms apply explicit or implicit iterative schemes (methods) of simple iterations [1-3]. In terms of the convergence rate, implicit iterative schemes have significant advantage over explicit schemes. In this connection, implicit iterative schemes play a more important role for the methods of iterative regularization.
The computational stability of implicit iterative schemes is important. This stability strongly depends on the method of solving systems of linear algebraic equations (SLAE) at each iteration. It is known that the methods of solving SLAE on the basis of singular decomposition (SVD-methods) have the largest computational stability [7,8]. These methods allow to solve effectively even SLAE with ill-determined rank [9,10]. A disadvantage (when solving problems large dimension) is the fact that the SVD method is poorly implemented on parallel high-performance platforms. However, in recent years some researches eliminating this lack have recently appeared [11-15].
In this paper, we propose a variant of the implicit method of simple iterations based on the use of singular decomposition (SVD-method). The speed and the numeric stability of this iteration algorithm are studied. The application of the proposed method for constructing iterative regularization algorithms is considered here.
1. Problem formulation and implicit method of simple iterations. We
consider the problem of solving systems of linear equations of the form:
Ax = b,
where A e Rmxra, m ^ n.
The pseudo-solving of the initial SLAE is defined as follows
x* = A+b,
where A+ is a pseudo-inverse Moore-Penrose matrix. If rank A = n, then = (ATA)-lAT.
Applying the Gauss symmetrization, the original SLAE lead to:
ATAx=b
or
ATAx + (xx = ATb + (xx, a > 0.
For this system, the classical implicit method of simple iterations has the following form [1,2,16]:
(aEn + ATA)xk+i = axk + ATb, (1)
where k = 0,1,....
This algorithm is widely used to obtain iterative regularizing algorithms [1-3]. 2. An implicit iteration method based on SVD. Using the singular value decomposition [7-9], an arbitrary matrix A can be represented as follows:
A = U £VT, (2)
where U = [u1u2 ■ ■ ■ un] e Rmxn and V = [v1 v2 ■ ■ ■ vn] e Rnxn are orthogonal matrixes, £ = diag(ai,..., an), a1 ^ a2 ^ ... ^ an, ai are singular numbers, and Ui and Vj are respectively left and right singular vectors of matrix A. The iterative process (1) can be written as:
xk+i = (aEn + ATA)-1(axk + AT b)
or
Xk+1 = &aXk + 9a, (3)
where $a is the transition matrix,
$a = a(ATA + aEn)-1 and §a = (AT A + aEn )-1AT b.
Using the SVD decomposition (2), let us perform the following transformations:
,T4 1 ~^n)-1 =a[V (£2 + aEn)VT]—1 = aV (£2 + aE.n)-ur T
$a = a(ATA + aEn)-i = a[V (£2 + aEn)V 1 ]-i = aV (£2 + aEn)-iV 1 =
n
a
n t
VjVl . -, a2 + a '
ga = ( AT A + aEn)-1ATb = [V (£2 + aE,n)VT ]-1V £U Tb =
= V (£2 + aEn)-1V TV £U Tb = V (£2 + aE,n)-1£U Tb =
n
Ev- 2 -—uT b.
■ i a-2 + a -
Therefore, the iterative process (3) (implicit method of simple iterations) can be described on the basis of the singular decomposition in the form of:
n -p
xk+i = aV 2 k v- + ga, (4)
f a 22 + a
where
n avTb
^ = Z Ofta^ (5)
k = 0,1,... ; a > 0, xo is an arbitrary initial condition.
If x0 = 0, then x\ = ga and, respectively,
Xk+1 = a
£
i= 1
vjxk_ e"2 + a
Vi + ga,
(6)
where k = 1,2,....
3. Algorithm convergence and conditionality. The necessary and sufficient condition for convergence of the simple iterations implicit method (3) is determined by the spectral radius of the transition matrix
p($a) = Amax (En + a-1ATA)-1
1
1
Xmin(En + a-1ATA) 1 + a-1a2n (A)
Since rank A = n, then an > 0 and, therefore, the convergence condition of the simple iterations implicit method can be written as follows
P($a) =
1
1+ 7"
< 1,
(7)
where 7 =
It follows directly from (7) that the smaller a is, the smaller the spectral radius of the transition matrix of the iterative process (3) is and, accordingly, the higher the rate of its convergence is.
However, the parameter a has the opposite effect on the computational conditionality of the simple iterations implicit method (3)—the smaller the parameter a is, the greater the number of the algorithm is.
We show that for the same parameter a of the condition number of the classical simple iterations implicit method (2) and simple iterations implicit method (4)-(6) on the basis of SVD are essentially different.
In general, the simple iterations implicit method can be written as follows:
Xk+1 = argmin
A \ f b \
Jx ^ Jaxk )
VâE,
(8
Different variants of implementing the simple iterations implicit method differ only in the method of solving the least squares problem at each iteration:
min
A VaEn
>-(
b
V®Xk
(9)
Let
A
> "(
b
Vaxk
= \\A«x - b^\\l
then (8) can be presented in the following way:
xk+1 = A+b^\ where A+ is a Moore-Penrose pseudoinverse matrix.
2
2
2
(
)
2
Since rankA« = n, then A+ can be calculated by the formula:
A+ = (AhAa) .
In this case, the expression (9) will correspond to the classical form of the simple iterations implicit method, i.e.
xk+l = {ATaAa)-lATab^ = (ATA + aEn)-1 (AT.^En) ( J^ ) =
= (ATA + aEn)-\ATb + axk ).
Thus, the spectral condition number of the problem at each k iteration for the classical simple iterations implicit method (1) is equal to
cond2(ATA + aEn) = ||(ATA + aEn)||2||(ATA + aEn)-1||2 =
= Amax(ATA + aEn) = o2 + a Amin(ATA + aEn ) a2 + a'
The implicit SVD-based simple iteration method is applied directly to the problem (9) and hence the condition number for this algorithm is determined by the matrix condition number
Aa =
( JAEn )
Thus, the spectral condition number for the algorithm (4) is equal to
/2 + \ 1/2
eond2(Aa) = \\AJ\2 -\\A + \\2 = ^^ .
V + ^ J
This means that the SVD-based algorithm (4) has a significant advantage over the classical algorithm (1) in solving ill-conditioned problems.
4. Regularizing iterative algorithms. Let us take approximate values (A, b) instead of the exact source data (A, b), where \\b — b\\ ^ S.
Then the regularized variant of the simple iterations implicit method based on SVD (4), (5) will have the following form
n t
EVi Xk / \
—Vi + ga, (10)
• , at + a
i=i i
where
n T£
EaiuT b
. + a '
i=i 1
k = 0,1,... ,ks; a > 0, x0 is arbitrary initial condition, and
ks = min{k G N : ||Axk - 6|| ^r5}, (11)
t ^ 1 is some parameter. Often chosen in practice t = 1 + emach, where emach is machine Epsilon.
Here, the stop number of plays the role of the regularization parameter, and the parameter a is the convergence rate controller (10). Expression (11) is the iteration discrepancy Tikhonov's principle.
If x0 = 0, then x\ = ga and, respectively,
where k = 0,1,... ,ks.
Conclusion. In this report, a new variant of the simple iterations implicit method based on the singular value decomposition (SVD method) is proposed. The iterative regularization algorithms obtained on the basis of the proposed implicit iteration schemes are considered. The main advantage of the proposed implicit singular value iteration schemes is to increase the computational stability of the implicit simple iteration method. This fact is especially important in solving poorly conditioned computational problems.
Competing interests. I declare that I have no competing interests.
Author's Responsibilities. I take full responsibility for submitting the final manuscript
in print. I approved the final version of the manuscript.
Funding. This research received no specific grant from any funding agency in the public, commercial, or not-for-profit sectors.
References
1. Vainikko G. M., Veretennikov A. Yu. Iteration Procedures in Ill-Posed Problems. Moscow, Nauka, 1986, 186 pp. (In Russian)
2. Bakushinsky A. B., Goncharsky A. V. Iterative methods for solving ill-posed problems. Moscow, Nauka, 1986, 186 pp. (In Russian)
3. Matysik O. V. Explicit and implicit iteration procedures of solving ill-posed problems. Brest, Brest State Univ., 1986, 186 pp. (In Russian)
4. Donatelli M. On nondecreasing sequences of regularization parameters for nonstation-ary iterated Tikhonov, Numer. Algor., 2012, vol.60, no. 4, pp. 651-668. doi: 10.1007/ s11075-012-9593-7.
5. Landi G., Loli Picolomini E., Tomba I. A stopping criterion for iterative regularization methods, Appl. Numer. Math., 2016, vol.106, pp. 53-68. doi: 10.1016/j.apnum.2016.03. 006.
6. Buccini A., Donatelli M., Reichel L. Iterated Tikhonov regularization with a general penalty term, Numer Linear Algebra Appl., 2017, vol.24, no. 4, e2089. doi: 10.1002/nla.2089.
7. Golub G. H., Van Loan C. F. Matrix computations, Johns Hopkins Series in the Mathematical Sciences, vol. 3. Baltimore, etc., The Johns Hopkins University Press., 1989, xix+642 pp.
8. Bjork A. Numerical Methods in Matrix Computations, Texts in Applied Mathematics, vol.59. Cham, Springer, 2015, xvi+800 pp. doi: 10.1007/978-3-319-05089-8.
9. Malyshev A. N. Introduction to numerical linear algebra (with an application of algorithms on FORTRAN). Novosibirsk, Nauka, 1991, 229 pp. (In Russian)
10. Hansen P. C. Rank-deficient and discrete ill-posed problems. Numerical aspects of linear inversion, SIAM Monographs on Mathematical Modeling and Computation, vol. 4. Philadelphia, PA, SIAM, Society for Industrial and Applied Mathematics, 1997, 247 pp.
11. Yamazaki I., Tomov S., Dongarra J. Sampling Algorithms to Update Truncated SVD, In: IEEE International Conference on Big Data (Big Data), 2017. doi: 10.1109/BigData. 2017.8257997.
12. Gates M., Tomov S., Dongarra J. Accelerating the SVD Two StageBidiagonal Reduction and Divide and Conquer Using GPUs, Parallel Computing, 2018, vol. 74, pp. 3-18. doi: 10. 1016/j.parco.2017.10.004.
13. Kabir K., Haidar A., Tomov S., Bouteiller A., Dongarra J. A Framework for Out of Memory SVD Algorithms, In: High Performance Computing, ISC 2017. Lecture Notes in Computer Science, vol. 10266; eds. J. Kunkel, R. Yokota, P. Balaji, D. Keyes. Cham, Springer, pp. 158178. doi: 10.1007/978-3-319-58667-0_9.
14. Dong T., Haidar A., Tomov S., Dongarra J. Optimizing the SVD Bidiagonalization Process for a Batch of Small Matrices, Procedia Computer Science, 2017, vol. 108, pp. 1008-1018. doi:10.1016/j.procs.2017.05.237.
15. Haidar A., Kabir K., Fayad D., Tomov S., Dongarra J. Out Of Memory SVD Solver for Big Data, In: 2017 IEEE High Performance Extreme Computing Conference (HPEC), 2017. doi: 10.1109/HPEC.2017.8091029.
16. Verzhbitskij V. M. Numerical Methods (Linear Algebra and Nonlinear Equations). Moscow, Oniks 21 Century Publishing House, 2005, 432 pp. (In Russian)
Вестн. Сам. гос. техн. ун-та. Сер. Физ.-мат. науки. 2018. Т. 22, № 3. С. 549-556
ISSN: 2310-7081 (online), 1991-8615 (print)
d http://doi.org/10.14498/vsgtu1592
УДК 519.612
Неявные итерационные схемы на основе сингулярного разложения и регуляризирующие алгоритмы
А. И. Жданов
Самарский государственный технический университет,
Россия, 443100, Самара, ул. Молодогвардейская, 244.
Аннотация
Предложен новый вариант неявного метода простых итераций на основе сингулярного разложения. Показано, что данный вариант неявного метода простых итераций позволяет существенно повысить вычислительную устойчивость алгоритма и при этом обеспечивает высокую скорость его сходимости. Рассмотрено применение неявного метода простых итераций на основе сингулярного разложения для разработки итерационных регуляризирующих алгоритмов. Предлагаемые алгоритмы могут быть эффективно использованы для решения широкого класса некорректных и плохо обусловленных вычислительных задач.
Ключевые слова: неявные итерационные схемы, сингулярное разложение, вычислительная устойчивость итерационных схем, скорость сходимости, итерационные методы регуляризации.
Получение: 20 декабря 2017 г. / Исправление: 22 августа 2018 г. / Принятие: 3 сентября 2018 г. / Публикация онлайн: 9 ноября 2018 г.
Конкурирующие интересы. Конкурирующих интересов не имею. Авторский вклад и ответственность. Я несу полную ответственность за предоставление окончательной версии рукописи в печать. Окончательная версия рукописи мною одобрена.
Финансирование. Исследование выполнялось без финансирования.
Научная статья
<В ©® Контент публикуется на условиях лицензии Creative Commons Attribution 4.0 International (https://creativecommons.org/licenses/by/4.0/deed.ru) Образец для цитирования
Zhdanov A. I. Implicit iterative schemes based on singular decomposition and regularizing algorithms, Vestn. Samar. Gos. Tekhn. Univ., Ser. Fiz.-Mat. Nauki [J. Samara State Tech. Univ., Ser. Phys. Math. Sci.], 2018, vol. 22, no. 3, pp. 549-556. doi: 10.14498/vsgtu1592.
Сведения об авторе
Александр Иванович Жданов А http://orcid.org/0000-0001-6082-9097 доктор физико-математических наук, профессор; заведующий кафедрой; каф. высшей математики и прикладной информатики; e-mail: zhdanovaleksan@yandex.ru
