CC BY
8
MSC 46N10, 65K10, 90C06 DOI: 10.14529/mmp220309
A MODIFICATION OF DAI-YUAN'S CONJUGATE GRADIENT ALGORITHM FOR SOLVING UNCONSTRAINED OPTIMIZATION
Y. Najm Huda,1, I. Ahmed Huda2
University of Duhok, Duhok, Kurdistan Region, Iraq
2University of Mosul, Mosul, Iraq
E-mail: [email protected], [email protected]
The spectral conjugate gradient method is an essential generalization of the conjugate gradient method, and it is also one of the effective numerical methods to solve large scale unconstrained optimization problems. We propose a new spectral Dai-Yuan (SDY) conjugate gradient method to solve nonlinear unconstrained optimization problems. The proposed method's global convergence was achieved under appropriate conditions, performing numerical testing on 65 benchmark tests to determine the effectiveness of the proposed method in comparison to other methods like the AMDYN algorithm and some other existing ones like Dai-Yuan method.
Keywords: unconstrained optimization; conjugate gradient method; spectral conjugate gradient; sufficient descent; global convergence.
Introduction
The conjugate gradient method (CGM) is one of the most widely used approaches for solving nonlinear optimization problems. We consider the following unconstrained optimization problem:
min f (x), (1)
where f (x) : Rn ^ R is continuously differentiable and bounded below when solving equation (1), the iteration method is employed and is written as:
xr+1 = xr + ar dr, (2)
where ar is a line search-based optimal move [1]. One of the most inexact line searches is the Wolfe conditions to achieve convergence in the conjugate gradient methods, ar is guaranteed to be present as:
f (xr+i) - f (xr) < 8arVf (xr)Tdr, (3a)
|gr+iTdrI <-^grTdr, (3b)
where 0 < 5 < a < 1. More information may be found in [2]. The directions dr are calculated using the following rule:
d = / -9r+i ; for r =0 1 (4)
dr+1 = \ -gr+i + вгdr ; for r > 1 j ' (4)
where er £ R is a conjugacy coefficient. There are many Conjugate Gradient techniques based on different er process choices, for example [3-8]. The most significant distinction
between the spectral gradient approach and the gradient conjugate method is calculating the search path. The spectral gradient method's search path is as follows:
dr+1 — — 9r+i gr+1 + sr
(5)
where sr = ardr and 0r+1 is the spectral parameter. The spectral conjugate gradient is a powerful tool for solving large-scale unconstrained optimization problems. The spectral conjugate gradient algorithms are presented in many papers, for more details see [9-20].
1. Derivation of New Parameter
An essential idea is to equate the vector of the conjugate gradient method to the vector of quasi-Newton method such that dQ+1 = d^+i, then
—Pr Hr+igr+i — -0r+igr+i + Pr)Y Sr >
(6)
where the matrix Hr+1 is positive definite and symmetric, and we use the scale from [21] defined as follows:
Pr
VTrVr
sjyr'
(7)
We use the DY conjugate gradient algorithm because it achieves convergence according to the Wolfe criterion and has the same property as FR [4] because it is similar in the numerator and the same property as HS [3], which is similar in the denominator. Therefore,
VÏVr u
K Vr
-@r+igr+i +
llg-
r + i l
sJ Vr
(8)
T
yr yr T TT
——yr nr+igr+i
s r Vr
Û T I I|gr+i! T
-Vr+lVr 9r+1 + —Tf—yr Sr,
ST yr
since Hr+iyr — sr ^ vTHr+i
T
T
Vr Vr T T
-Tf,— sr gr+1 — —fcV+iyr 9r+1 + \\9r+1 sT Vr
Then the new spectral is known in the following form:
12
nSDY Ur+i
lg
r+i I
vVT gr+i
+
VÏVr
T
Sr 9r+1
vVT gr+i
(9)
then the proposed search direction is defined as
r+i
lg
r+i I
vJ gr+i
+
VÏVr
sjvr
T
sr
VÏ9r+1
gr+i +
lg
r+i l
sï Vr
(10)
Lemma 1. Suppose that the sequences generate (2) and (10) computed by line search (3a) and (3b), then the sufficient descent condition holds.
Proof. Define the spectral gradient direction as follows:
SDY DY
dr+i — —"r+i gr+i + pr sr,
2
s
r
r
2
2
2
s
r
where 0f+1y is defined in (9)
\\9
r+11
yT 9r+i
+
T \ T yr yr \ sr gr+1
sTyr / yTgr+1
gr+1 +
\9
r+1 \
sT yr
(11)
Multiply equation (11) by (,,flr+1„2). We get
dr+i 9r+1 к
r+1 \
\9
r+1 \
yT gr+1
+
£ Уг
T
Sr 9r+1
sTyr / yTgr + 1
\9
r+1 \
\9
+
\9r+1\ r 9r+1
r+1 \
sT yr
\9
r+1 \
Since gr+1 < ||yrII ||gr+i|| and s^gr+i < s^yr, then
dr+1 gr+1 ^ \ 9r+1 \ 9r+1
12 —
< —-
\9
+
r+1 \
T
sr
sT yr
From the strong Wolfe condition, s^gr+1<-a s^gr, and since we get s^gr<
■ -sr Уг
3r a+1
^r+1 gr+1
и
12 —
< -
\9
r+1 \
r+1 \
\9
-—a-
T
~Sr Уг CT+1
r+1 \
sT Уг
<-
a
\9
r+1 \
a+1
Since yr=gr+1-gr we get: dT
'r+1 gr+1 < _ llgr+l-grll O-
\\9r+1\\2 \\gr+1|
f_^ llfr+lll I a
a+1 \\gr+1\ a+1
Let ф = (l — then we get:
dT+1 gr+1 < -^\\9r+1\\2-
2. Convergence Analysis of New Method
□
Assumption I. The following are some fundamental assumptions in this paper. (a) The level set ^ is bounded. (b) The function is convex and continuously differentiable, its gradient g (x) is Lipchitz continuous, i.e., there exists a constant L > 0 such that x) ~ 9(y)\ < L\\x ~~ VII f°r x,y E Q.
2
2
s
r
2
2
2
2
r
r
r
r
Lemma 2. Suppose that Assumption I holds then for any CG-method, the direction dr+1 is a descent, and the step size ar is achieved by (3a, 3b) if
£
Then
>1 IK+1W2
1
00.
lim (inf ||gr|| =0) . (12)
r—y^o
Lemma 3. Suppose that Assumption I holds, where dr+1 is defined by (10), then the method satisfies equation (12).
Proof. Since /3 is positive [7], then 0 < / < 1,
PDY\ < > 0,
(13)
\ftSDY I
|er+1 |
\\9
r+11
y'r 9r+1
+
iSDY \
<
'r+1 | —
\\9:
r+11
\9
r+1 \
\ T \ \9r+19k\
+
from Powell restart |9T+19r| > 0, 2 \\gr+1\2 ,
iSDY \
<
\9
r+1 \
0, 8 \\g.
+
r+1 \
T
2 0 [ sr Vr
sjyr 0,8 \\gr+1\
ef+f\ < ^2, ^2 > 0.
(14)
From equation (5) and using (13) and (14)
\\dr+1\\2 < ^2\\9r+1 \ \2 + \\Sr\\ = ,
<x
E-
k>1
1 1 ^ -IT > — 7 1
dr+1\\
œ.
r>1
(15) □
3. Results and Discussion
In this section, the new algorithm specified by equation (10) is compared to the classical DY algorithm [7] as well as AMDYN algorithm [11], which define Qr+\ as follows:
e.
r+1
yï 9r+1
il || 2 \\gr+1\2(sT gr+1) T
\\9r+i\\--r--\-sr9r+1
Vr Sr
We chosen 65 unconstrained optimizations in the range [n = 1000, 2000,..., 10, 000], broadly and based on generalized [22]. All algorithms used Wolfe condition a = 0, = 0, 0001. The codes are adopted with double precision and using the Fortran language. All of these codes are authored by Andrei [11,23]. We applied the performance profile of Dolan and More [24] to demonstrate the algorithm's efficiency, where the upward curve indicates that the new algorithm (SDY) is better than the classic DY. Also, we compare the results with AMDYN based on the number of iterations (NOI), number of functions and gradient ratings (NOF & g) and CPU time (time). Representation of practical results is presented in Figure.
2
2
2
2
2
1
Conclusion
The aim of this work is to propose new computing schemes for the spectral parameter OfDi . This technique is such that the search direction always possesses a sufficient descent property. As a result, the presented new spectral conjugate gradient possesses global convergence under a robust Wolfe search. Using a collection of 65 optimization tests to solve problems, and numerically comparing the conduct of this new algorithm to that of some conjugated gradient algorithms DY and AMDYN, we show that the proposed algorithm outperforms previous conjugate gradient algorithms due to computational performance.
(a) NOI (b) NOF & g
Performance profile: 1.332000e-01
(c) Time of CPU Performance profile
References
1. Alhawarat A., Salleh Z. Modification of Nonlinear Conjugate Gradient Method with Weak Wolfe-Powell Line Search. Abstract and Applied Analysis, 2017, vol. 2017, no. 2, pp. 1-6. DOI: 10.1155/2017/7238134
2. Nocedal J., Yuan Ya-Xiang. Analysis of a Self-Scaling Quasi-Newton Method. Mathematical Programming, 1993, vol. 61, no. 1, pp. 19-37. DOI: 10.1007/BF01582136
3. Hestenes M.R., Stiefel E. Methods of Conjugate Gradients for Solving Linear Systems. Journal of Research of the National Bureau of Standards, vol. 49, no. 6, pp. 409-435. DOI: 10.6028/jres.049.044
4. Fletcher R., Reeve C.M., Function Minimization by Conjugate Gradients. The Computer Journal, 1964, vol. 7, no. 2, pp. 149-154. DOI: 10.1093/comjnl/7.2.149
5. Polak E., Ribiere G. Note sur la convergence de methodes de directions conjuguees. Mathematical Modelling and Numerical Analysis-Modélisation Mathématique et Analyse Numérique, 1969, vol. 3, no. 1, pp. 35-43. (in French)
6. Polyak B.T. The Conjugate Gradient Method in Extremal Problems. Computational Mathematics and Mathematical Physics, 1969, vol. 9, no. 4, pp. 94-112. DOI: 10.1016/0041-5553(69)90035-4
7. Dai Yu-Hong, Yuan Yaxiang. A Nonlinear Conjugate Gradient Method with a Strong Global Convergence Property. SIAM Journal on Optimization, 1999, vol. 10, no. 1, pp. 177-182.
8. Dai Yu-Hong, Li-Zhi Liao. New Conjugacy Conditions and Related Nonlinear Conjugate Gradient Methods. Applied Mathematics and Optimization, 2001, vol. 43, no. 1, pp. 87-101. DOI: 10.1007/ s002450010019
9. Hassan B., Abdullah Z., Jabbar H. A Descent Extension of the Dai-Yuan Conjugate Gradient Technique. Indonesian Journal of Electrical Engineering and Computer Science, 2019, vol. 16, no. 2, pp. 661-668. DOI: 10.11591/ijeecs.v16.i2.pp661-668
10. Jie Guo, Zhong Wan. A Modified Spectral PRP Conjugate Gradient Projection Method for Solving Large-Scale Monotone Equations and its Application in Compressed Sensing. Mathematical Problems in Engineering, 2019, vol. 2019, pp. 23-27. DOI: 10.1155/2019/5261830
11. Neculai A. New Accelerated Conjugate Gradient Algorithms as a Modification of Dai-Yuan's Computational Scheme for Unconstrained Optimization. Journal of Computational and Applied Mathematics, 2010, vol. 234, no. 12, pp. 3397-3410. DOI: 10.1016/j.cam.2010.05.002
12. Eman H., Rana A.Z., Abbas A.Y. New Investigation for the Liu-Story Scaled Conjugate Gradient Method for Nonlinear Optimization. Journal of Mathematics, 2020, vol. 2020, article ID: 3615208, 10 p. DOI: 10.1155/2020/3615208
13. Gaohang Yu, Lutai Guan, Wufan Chen. Spectral Conjugate Gradient Methods with Sufficient Descent Property for Large-Scale Unconstrained Optimization. Optimization Methods and Software, 2008, vol. 23, no. 2, pp. 275-293. DOI: 10.1080/10556780701661344
14. Ibrahim Sulaiman Mohammed, Yakubu Usman Abbas, Mamat Mustafa. Application of Spectral Conjugate Gradient Methods for Solving Unconstrained Optimization Problems. An International Journal of Optimization and Control: Theories and Applications, 2020, vol. 10, no. 2, pp. 198-205.
15. Wang Li, Cao Mingyuan, Xing Funa, Yang Yueting. The New Spectral Conjugate Gradient Method for Large-Scale Unconstrained Optimisation. Journal of Inequalities and Applications, 2020, vol. 2020, no. 1, pp. 1-11. DOI: 10.1186/s13660-020-02375-z
16. Jian Jinbao, Yang Lin, Jiang Xianzhen, Liu Pengjie, Liu Meixing. A Spectral Conjugate Gradient Method with Descent Property. Mathematics, 2020, vol. 8, no. 2, article ID: 280, 13 p. DOI: 10.3390/math8020280
17. Danhausa A.A., Odekunle R.M., Onanaye A.S. A Modified Spectral Conjugate Gradient Method for Solving Unconstrained Minimization Problems. Journal of the Nigerian Mathematical Society, 2020, vol. 39, no. 2, pp. 223-237.
18. Al-Arbo A., Rana Al-Kawaz. A Fast Spectral Conjugate Gradient Method for Solving Nonlinear Optimization Problems. Indonesian Journal of Electrical Engineering and Computer Science, 2021, vol. 21, no. 1, pp. 429-439. DOI: 10.11591/ijeecs.v21.i1.pp429-439
19. Hassan Basim, Jabbar Hawraz. A New Spectral on the Gradient Methods for Unconstrained Optimization Minimization Problem. Journal of Zankoy Sulaimani, 2020, vol. 22, no. 2, pp. 217-224. DOI: 10.17656/jzs.10822
20. Liu J.K., Feng Y.M., Zou L.M. A Spectral Conjugate Gradient Method for Solving Large-Scale Unconstrained Optimization. Computers and Mathematics with Applications, 2019, vol. 77, no. 3, pp. 731-739.
21. Al-Bayati A.Y. A New Family of Self-Scaling Variable Metric Algorithms for Unconstrained Optimization. Journal of Education and Science, 1991, vol. 12, pp. 25-54.
22. Neculai A. An Unconstrained Optimization Test Functions Collection. Advanced Modeling and Optimization, 2008, vol. 10, no. 1, pp. 147-161.
23. Neculai A. New Accelerated Conjugate Gradient Algorithms for Unconstrained Optimization. Technical Report, 2008, vol. 34, pp. 319-330.
24. Dolan E.D., More J.J. Benchmarking Optimization Software with Performance Profiles. Mathematical Programming, 2002, vol. 91, no. 2, pp. 201-213. DOI: 10.1007/s101070100263
Received November 25, 2021
УДК 519.6+517.972
DOI: 10.14529/ mmp220309
МОДИФИКАЦИЯ АЛГОРИТМА СОПРЯЖЕННЫХ ГРАДИЕНТОВ ДАЙ-ЮАНЯ ДЛЯ РЕШЕНИЯ БЕЗУСЛОВНОЙ ОПТИМИЗАЦИИ
Ю. Наджм Худа,1, И. Ахмед Худа2
1 Университет Духока, г. Духок, Ирак 2Университет Мосула, г. Мосул, Ирак
Метод спектральных сопряженных градиентов является существенным обобщением метода сопряженных градиентов, а также одним из эффективных численных методов для решения крупномасштабных задач безусловной оптимизации. Мы предложили новый спектральный метод сопряженных градиентов Дай-Юаня для решения нелинейных задач безусловной оптимизации. Глобальная сходимость предложенного метода была достигнута при соответствующих условиях, проведены численные эксперименты на 65 эталонных тестах, показывающие эффективность предложенного метода по сравнению с другими методами, такими как алгоритм ЛМБУМ и некоторыми другими существующими методами, такими как метод Дай-Юаня.
Ключевые слова: неограниченная оптимизация; метод сопряженных градиентов; спектральный сопряженный градиент; достаточный спуск; глобальная конвергенция.
Наджм Ю. Худа, кафедра математики, колледж наук, университет Духока (г. Духок, Ирак), [email protected].
Ахмед И. Худа, PhD, кафедра операционных исследований и интеллектуальных техник, колледж компьютерных наук и математики, Университет Мосула (г. Мосул, Ирак), [email protected].
Поступила в редакцию 25 ноября 2021 г.
