CC BY
40
1.2. REVISED SOLUTION OF ILL-POSED ALGEBRAIC SYSTEMS
FOR NOISE DATA
Ternovski Vladimir V., associate professor, VMK Dep. Lomonosov Moscow State University, Russia. E-mail: [email protected]
Khapaev Mikhail M., professor, VMK Dep. Lomonosov Moscow State University, Russia
Abstract: The problem of numerical solution of linear algebraic equations is known to be ill-posed in the sense that small perturbation in the right hand side may lead to large errors in the numerical solution. It is important to verify the accuracy of approximate solution by taking all possible errors of elements of a matrix, a vector of the right hand side, and roundoff errors into account. There are computational difficulties with ill-posed systems as well. If to apply standard methods, for example, a method of Gauss elimination, for such systems it isn't possible to catch the correct solution though discrepancy can be less accuracy of data-in and roundoff errors. The small discrepancy doesn't guarantee proximity to the correct solution. Actually there is no need for preliminary study of assessing whether a given system of linear algebraic equations is inherently ill-conditioned or well-conditioned. The new approach to the solution of algebraic systems based on statistical effect in matrixes of a big order is considered. The conditionality of the systems of equation changes with a high probability at a matrix distorted by random noise. After standard methods to be applied the received "chaotic" solution is used as a source of a priori information in more general variational problem
Index terms: ill-posed problems, condition numbers, random matrix
ВЕРОЯТНОСТНЫЙ ПОДХОД В АЛГОРИТМАХ НЕКОРРЕКТНЫХ
ЗАДАЧ
Терновский Владимир Владимирович, доцент факультета ВМК. МГУ им. М.В. Ломоносова. Email: vladimir@chatroulette. com
Хапаев Михаил Михайлович, профессор факультета ВМК. МГУ им. М.В. Ломоносова
Аннотация: Множество некорректных задач сводится к решению плохо обусловленных систем алгебраических уравнений. В свою очередь, известны вариационные методы решения плохо обусловленных систем, позволяющие выделить искомое решение. На точность численного решения влияют обусловленность системы, погрешности задания элементов матрицы, вектора правой части, а также ошибки округления. В действительности, возможно отказаться от предварительного исследования системы на обусловленность. В работе развивается новый подход к решению некорректных задач, основанный на статистическом эффекте в матрицах большого порядка. Обусловленность систем улучшается с большой вероятностью при зашумлении матрицы. Изучается вопрос, какую задачу можно считать плохо или хорошо обусловленной и как ее решать. Для решения систем применяются стандартные методы линейной алгебры, причем полученное классическое «хаотичное» решение используется как источник априорной информации в более общей задаче условной минимизации уточнения решения. Тем самым устанавливается соответствие между классическими методами линейной алгебры и алгоритмами некорректных задач.
Ключевые слова: некорректные задачи, плохообусловленные системы, методы регуляризации
1. INTRODUCTION
Many direct and iterative traditional methods for numerical solution of systems of well-conditioned linear algebraic equations are known. As a result, the classical methods for solving ill-conditioned systems tend to be unstable. It is less known that the commonly used methods of regularization of ill-conditioned problems and classical methods can be connected.
Let A be a square matrix that is ill-conditioned Ax = b. (1)
The solution of such algebraic systems can be a difficult task [1]. When is a matrix A ill-conditioned? Or maybe a question should be better rephrased as: are there robust condition number estimators? The number of elements of a matrix can be so big that it is difficult to investigate system (1) on conditionality because of loss of accuracy of calculations and computer
time consuming. Moreover, it is possible to change value of spectral condition number as much as we can multiplying any equation by any number
cond ( A)
i -i
(2)
It is wrong to use condition number as the criterion of quality of ill-conditioned system because it is difficult to calculate the criterion itself. Complexity of calculation of (2) is connected with an assessment of norm of the inverse matrix A_1 . Such calculations can be incorrect.
Therefore, the definition of condition number is exposed to criticism in literature. Other options of condition number are discussed in [2] and other papers. It should be noted that it makes sense to use the number (2) only if it can be really found with the fixed accuracy.
At the current stage of development of computers it is convenient to find the solution of systems of the linear equations via commercial software like MATLAB, MATHEMATICA, MAPLE, etc. Popularity of such software is caused by opportunity to carry out calculations both on the personal computer, and on a supercomputer with parallelization of calculations and controlled accuracy.
There are systems of linear equations that can not be solved with the help of commercial software. Standard software in this case gives an irregular, "chaotic" solution. The question is, whether it is possible to consider system of the algebraic equations (1) as ill-conditioned problem if it can not be solved by standard methods of linear algebra. Or whether, in other words, it is possible to consider system as ill-conditioned, without calculating the condition number? It is interesting, whether it is possible to avoid using the condition number at all.
Authors consider the new approach in research of such systems that doesn't demand calculation of conditionality of a system that is based on statistical effect of improvement of spectral properties of perturbed matrix.
If the system (1) is not well-conditioned, the concept of the solution has to be revised, it is necessary to consider errors of b and A. That is
Ax = b , but ||X-X = f>>0, where isb -the perturbed value of right hand side vector b and A - perturbed matrix , and X- perturbed solution, x - the required solution, f is the residual. When classical approach is used, without additional tricks, the perturbed solution can be useless. The inverse matrix of perturbed matrix can have correctly calculated because perturbed matrix is well-conditioned as it will be shown below. There is general statement: if the matrix (operator) of A"1 does not exist, the inverse perturbed matrix exists with a high probability. A. N. Tikhonov and [1, 3] developed the theory of the ill-posed problems. The focus of Tikhonov method is to choose the compact class and the solution with minimal norm taking into account errors of input data.
It should be noted that the relevance of the task of solving ill-conditioned systems does not raise doubts: as such systems occur in numerous engineering applications: recovery of images, spectral analysis, digital signal processing, etc. Implementation of variational algorithms takes much more time, than standard methods of Gauss elimination. However, the solution received by programs of computer mathematics can be chaotic. Again the question is whether the obtained solution useless or not.
Consider a methodical example of calculation of the algebraic system with Hilbert's matrix 1
H (i, j)
i + j-1
, i = 1..N, j = 1..N.
We define a vector of the right hand parts for function -|X+1 on interval from -1 to +1, multiplying a matrix H through by a vector 1 — |— 1+2(i-1)/(N-1)|,i = 1..N. At N=400 calculation for the LinearSolve program of the Mathe-matica package shows development of instability in the solution. For recovery of the stable solution it was required to keep at least 560 digits.
From here it is possible to draw a conclusion on reliability of standard programs in a case when elements of a matrix and a vector of the right part are known precisely or when they are integers, or when they can be found with a controlled accuracy.
Observe that using of LeastSquares method gives us the stable solution of a problem with Hilbert matrix up to N=2000 and more that tells us about efficiency of the LeastSquares method for lack of errors of the right hand part. If we want to perturb right hand side, accepting a hypothesis of additivity of errors, all methods of computer mathematics are unsuitable and the solution is unstable. LeastSquares method and the pseudo-inverse also lead to the irregular solution. In that case, all classical methods for solving ill conditioned systems are usually unstable.
There are topical issues which it is necessary to answer in this paper:
1. How to reduce condition number?
2. How to get the correct solution of ill-conditioned via standard programs of computer mathematics?
3. How to increase the accuracy of the classical solution of system of the algebraic equations?
The idea of the present article is that the approximate solution from classical method could be useful for variation problem. The assumption is based on the fact that classical solutions, even chaotic, nevertheless are a source of a priori information and are in the set of possible solutions.
2. CONDITIONALLY OF SYSTEM WITH THE NOISY MATRIX
Consider the general problem of solving ill-conditioned system of the linear equations when the right hand side and a matrix are subjected to a perturbation. Observe that spectral properties of a matrix change to the best (!) as the condition number of ill-conditioned system decreases with growth of noise amplitude [4]. Thus the condition number of a real-valued random matrix slowly grows as Ln[N] + 1.537 , where N - dimension of a matrix [5]. In Fig. 1a the calculated condition numbers of a random matrix are displayed, from where follows that these numbers with a high probability do not reach critical values.
Fig.la. The condition number of pseudorandom matrices with NxN dimension.
According to the results of [4] the norm of the inverse contaminated matrix of ill-posed problem and condition number are easily calculated. In Fig. 1b the value of Hilbert matrix condition number depends on amplitude of noise in an interval (l0"6,l0"5 ).
ocnd 1.4*108r
1.2*10'
1.0*10'
8.0 *u0
6.0 * 10N 4.0 * 10 7
4. *10"6 6. *10"6 8. *10"6
0.00001
Tolerance
Fig. 1b. Condition number of Hilbert Matrix 1000x1000 is under small perturbation.
From calculation is shown that with a high probability ill-conditioned system will become well-conditioned under perturbation. Expression "with a high probability" means that with small probability it is possible to specify such distribution of random numbers when the value of condition number has emissions of great values. Noise is always present at measurements therefore it is represented natural to add noise to an exact matrix and to improve conditionality of system of the linear equations. Under perturbation of the right hand side for Hilbert matrix of the previous task with an amplitude of noise of 10 7, the LinearSolve method gives the irregular solution. Other methods yield the same result which indicates chaotic,
7
irregular properties of the solution. Nevertheless, the residual of such solution f=2.7-10"5 and all calculations are made with control of accuracy. Thus, it is possible to claim that the solution turned out "chaotic", but it also is the exact classical solution of the perturbed system. Such solution cannot satisfy us though, according to results [4], the system of the equations has to be well-conditioned with a high probability. We will notice that methods of smoothing do not lead to recognition of the hidden solution if such methods don't use a priori information about errors. It is necessary to correct the obtained solution within the errors. We will note again that the contamination of a matrix was made forcibly for the purpose of improvement of conditionality.
There are also other effective ways to reduce condition number. For example, it makes sense to consider the extremal problem of condition
number of a matrix A = A + aD on a set of diagonal matrices Dwith fixed a and constraint ||d|| < 1.
3. THE VARIATIONAL ALGORITHM
Let the perturbed system Ax = b , where X = x+ Sx, b = b+Sb, A = A+8A. Add to original system Ax = b on the right and at the left a vector SAx. In consequence, subtract the perturbed system. This gives us an inequality
A(x - x) <|Sfr|| + ||SA||x|| <& + h max||xll, (3)
on Gauss elimination or other known methods is used.
Consider other problem, when there is a matrix A (perhaps already perturbed, but errors are unknown), and the right hand side is known only approximately.
In other words, let be the equation Ax = b ,
b - b <a, where A is known, b is unknown.
Hence using the solution from the equation Ax = b with forcibly distorted matrix A = A + SA the extremal problem can be formulated as
\\X || = inf{ ||x||: ||A(x - x)|| < a + h||~|} . (5)
From the computing point of view, the problem (5) is simpler, than (4), as the right hand side of constraint a + h||~|| is a precise calculation. On noting that the perturbation SA is known in this case it is possible to set the extremal problem:
1*1 = F
Ax - A ~
(6)
where x - is the solution of a classical method like Gauss elimination, C _ is the amplitude of
an error of the right part, h - is an absolute error of a matrix SA.
Finally, suppose a variation problem of conditional minimization for solution X :
IXII = inf{ IIx|| : ||A(x - x)|| < c + h max ||x||}, (4)
where norm || || is a finite-dimensional, for example, the Frobenius norm.
We now claim that problem (4) is not the same as Tikhonov one [1] because instead of
b we use the solution x, that is correctly calculated from well-conditioned system Ax = b
The formulated variation problems are solved by the standard function of NMinimize of the Wolfram company.
4. IMPLEMENTATION
We define the ill-conditioned matrix A on that main diagonal there are (+1), lower diagonals are zero and is above as (-1). In the Math-ematica language 10.1 such matrix NXN is that :
SparseArray [{{i i _} ^ 1, {i _, j _}/; j > i 1},{ N, N },0].
In spite of the determinant Det [ A ] = Det [ A= 1, to calculate the inverse matrix for N=512 is not possible because of cond(A) > 2510. If we perturb A adding random matrix SA with an amplitude a = 0.01, the inverse matrix is calculated and the solution from LinearSolve is submitted in Fig. 2a.
x
Artificial blow-out
Fig.2a. The wrong classical solution is from intentionally perturbed system.
Though the solution coincides with analytical model(square-wave pulse) in the majority of points, but contains an essential mistake on the left side. If such classical solution doesn't satisfy the researcher, it can be used in a task (5) or (6). The final variation solution in Fig. 2b. is close to model in all points.
Fig.2b. Restored variational wave is from classical
solution on Fig.2a. Note: In this example it is possible to compare the solutions, but it maybe limited to the classical solution for huge matrices as the variation demands expensive resources of the computer. 5. WELL-CONDITIONED SYSTEMS The question of well-founded estimates of the solution of well-conditioned systems in the presence of noise in coefficients is insufficiently studied though there is a literature on such systems. The statement about reduction of condition number for ill-conditioned systems under perturbation is wrong for well-conditioned systems of a big order. On the contrary, with a high probability the condition number of a noisy matrix will grow with growth of noise amplitude and dimension of system as the
norm of a random matrix grows. If the matrix of A is non-degenerate and
fi<— ■ <7> \\A\\ cond (A)
hence A + SA is also non-degenerate (the theorem 2.3.1 [6]). For such systems the relative error of the solution at exact right right hand side b is calculated according to a known inequality [6]
W
INI
cond ( A)
<
1 - cond ( A)
SA
(8)
For an example we investigate Toeplitz matrix of dimension N=1000x1000 with cond = 696745. Matrix A perturbed with noise amplitude by 10-2. It is easy to check a condition of the above-mentioned theorem and an inequality (8). Value of the right part of an inequality (8) shows slow growth ~ 1/5ON depending on the dimension of system.
If to consider influence of noise on condition number at the fixed number N=1000, the system with well-conditioned Toeplitz matrix becomes ill-conditioned. Observe an exponential growth of condition number of a matrix A + SA though value of condition number cond = 3028 of a random matrix SA is small in comparison with a noisy matrix. One can notice a violation of the estimate (8) if a noise amplitude more than h= 0.048 and inequality (7) is not valid. Then the problem of solving of the system is interpreted as incorrect and can be also solved by a variational method. Modeling on a grid N=500 showed reduction of an error of the classical solution and the subsequent improvement by a variation method by 10 times.
6. CONCLUSION
In this paper we have analyzed the ill-posed problems from classical and regularization point of view. Many numerical ill-posed problems are reduced to systems of the linear equations. For the solution of ill-posed systems there are still no methods that could be considered as the best and final one. The purpose of current work was to create a method of the
x
solution of systems that does not take into account the study on conditionality.
Two steps method of the solution of systems of the linear equations offered in the present paper allows getting the numerical solution. If the algebraic system is well-posed, there is no sense to apply the time consuming variation method demanding expensive computing resources. Influence of approximate coefficients of a matrix on the solution error in this case is minimal.
With ill-posed systems the situation is opposite. It is considered that the classical numerical methods are not applicable for such systems. If the classical solution is distorted and irregular, but the residual is small, it can be used as a source of a priori information to get the varia-tional solution.
Forced contamination of matrices opens a way of association of different classical algorithms and variation methods of ill-posed problems (Fig. 3).
5. A. Edelman. Eigenvalues and condition numbers of random matrices. SIAM j. Matrix Anal. Appl., Vol.9, No. 4, October, 1988, Pages 543- 560.
6. David S. Watkins. Fundamentals of Matrix Computations, Third Edition John Wiley and Sons, July 2010, 644 pp.
Fig. 3. Connection between classical and regulari-zation methods
Список литературы:
1. A. Tikhonov and V. Arsenin, Solutions of ill-posed problems. Winston, Washington, DC(1977).
2. N. N. Kalitkin, L. F. Yuhno, L. V. Kuz'mina, Quantitative criterion of conditioning for systems of linear algebraic equations, Mathematical Models and Computer Simulations October 2011, Volume 3, Issue 5, pp 541-556
3. A. Bakushinsky and A. Goncharsky, Ill-posed problems: theory and applications. Springer Netherlands (1994).
4. Terence Tao and Van Vu. Smooth analysis of the condition number and the least singular value. Mathematics of computation ,Volume 79, Number 272, October 2010, Pages 2333-2352
