A Quadratic Model based Conjugate Gradient Optimization Method
ISAM H. HALIL1, ISSAM A.R. MOGHRABI2, AHMED A. FAWZE3, BASIM A. HASSAN3,
HISHAM M. KHUDHUR3
1Department of Mathematics, College of Science,
Kirkuk University, Kirkuk,
IRAQ
2Department of Computer Science, College of Arts and Sciences,
University Central Asia, Naryn,
KYRGYZ REPUBLIC
3Department of Mathematics, College of Computer Science and Mathematics,
University of Mosul, Mosul,
IRAQ
Abstract: - In this paper, we introduce a nonlinear scaled conjugate gradient method, operating on the premise
of a descent and conjugacy relationship. The proposed algorithm employs a conjugacy parameter that is
determined to ensure that the method generates conjugate directions. It also utilizes a parameter that scales the
gradient to enhance the convergence behavior of the method. The derived method not only exhibits the crucial
feature of global convergence but also maintains the generation of descent directions. The efficiency of the
method is established through numerical tests conducted on a variety of high-dimensional nonlinear test
functions. The obtained results attest to the improved behavior of the derived algorithm and support the
presented theory.
Key-Words: - unconstrained optimization, conjugate gradient methods, line search methods, global
convergence, quadratic modelling, non-linear programming.
Received: March 13, 2023. Revised: October 29, 2023. Accepted: November 13, 2023. Published: December 6, 2023.
1 Introduction
Our concern in this paper is problems of the form:
𝑚𝑖𝑛 (𝑥), where 𝑥∈𝑅 , (1)
for which 𝑓 is a differentiable convex function. For
large dimension n, conjugate gradient (CG) methods
are among the most sophisticated and
straightforward approaches proposed to solve (1),
due to storage considerations. Starting with a guess
𝑥∈𝑅, the CG method produces the sequence
󰇝𝑥󰇞 using the recurrence:
𝑥 𝑥𝛼𝑑, for 𝑘0,1, (2)
where 𝛼 is a positive scalar representing the step
length along the search direction 𝑑, which is
calculated using some line search method. The
search vectors are derived using:
𝑑 𝑔 if k0
𝑔𝛽𝑑 if k0.
󰇛3󰇜
The parameter 𝛽 in (3) is referred to as the
conjugacy parameter and 𝑔 is the gradient vector
evaluated at 𝑥. The primary factor in CG
strategies is that they generate search directions 𝑑,
which are downhill. The particular choice of the
scalar parameter 𝛽 defines different and new
conjugate gradient algorithms. For convergence
purposes and for aiding in ensuring that the
generated search directions are downhill, the step
length 𝛼 in (2) is computed subject to satisfying the
strict conditions, [1,2], given by:
𝑓󰇛𝑥𝛼𝑑󰇜𝑓󰇛𝑥󰇜𝛿𝛼𝑔
𝑑 󰇛4󰇜
𝑑
𝑔󰇛𝑥𝛼𝑑󰇜𝜎𝑑
𝑔 , 󰇛5󰇜
where 0𝛿𝜎 to guarantee that 𝑑 is a
downward direction.
For a quadratic convex problem with positive
definite A of the form:
𝑓󰇛𝑥󰇜
𝑥𝐴𝑥𝑏𝑥𝑐 (6)
WSEAS TRANSACTIONS on MATHEMATICS
DOI: 10.37394/23206.2023.22.101
Isam H. Halil, Issam A.R. Moghrabi,
Ahmed A. Fawze, Basim A. Hassan, Hisham M. Khudhur
E-ISSN: 2224-2880
925
Volume 22, 2023
and for accurate line searches used to solve:
𝛼𝑎𝑟𝑔𝑚𝑖𝑛

𝑓
󰇛𝑥𝛼𝑑󰇜 , 󰇛7󰇜
the following conjugacy condition holds:
𝑑
𝐴
𝑑0, 󰇛8󰇜
for any 𝑖𝑗. The CG techniques, typically used
to solve systems of linear equations, are derived
with satisfying conditions (8), as is the case with the
original algorithm in, [6]. Almost all CG algorithms,
such as those in, [3], [4], [5], [6], [7], [8], [9], satisfy
(8) for quadratic objective functions and positive
definite Hessian matrix.
In this paper, we provide new variants of the
CG relations in (2) and (3) that are similar
conceptually to those in, [10], [11], [12], that
always satisfy: 𝑔
𝑑 0. 󰇛9󰇜
By the mean value theorem, for a nonlinear function
𝑓 there is a value 𝜉 such that:
𝑑
𝑦𝛼𝑑
𝑓󰇛𝑥𝜉𝛼𝑑󰇜𝑑. (10)
As a result of this, the following conjugacy criterion
appears to be an appropriate substitute for (8):
𝑑
𝑦0. 󰇛11󰇜
Using 󰇛3󰇜 and (11) leads to the relation (upon pre-
multiplying both sides by 𝛼):
𝛽𝑣
𝑦𝑔
𝑦0, 󰇛12󰇜
for 𝑣𝑥𝑥 or, equivalently, the conjugacy
parameter, [13]:
𝛽
 
. (13)
This paper has the following outline. We
provide a novel scaled conjugate gradient approach
in subsection 2. In Section 3, the proof that the
suggested algorithm generates a search direction
that complies with the descent requirements at each
iteration is provided. The new CG-methods' global
convergence property is proven in Section 4. The
effectiveness of the suggested CG method is
established with some numerical findings in Section
5. The final Section offers some conclusions and
future research suggestions.
2 New Scaled CG Method
We focus attention in this paper on solving
unconstrained minimization problems using the
recurrence: 𝑥 𝑥𝛼𝑑 , 󰇛14󰇜
where 𝛼0 is acquired to satisfy the conditions
for line search in (4) and (5), and 𝑑 is some
computed search direction using:
𝑑 𝜙𝑔𝛽𝑣, 󰇛15󰇜
for some scalars 𝜙 and 𝛽, whose derivation
specifies the fingerprints of the CG method.
The scalar parameters 𝜙 and 𝛽 in Equation
(15) are obtained in our approach for all 0k
based on the descent condition:
𝑔
𝑑 𝜙𝑔
𝑔
𝛽𝑔
𝑣0. 󰇛16󰇜
From 󰇛15󰇜 and (16), we get:
𝑔
𝑑 𝜙𝑔
𝑔𝛽𝑔
𝑣𝛿0,
which gives:
𝑔
𝑑 𝜙𝑔
𝑔
𝛽𝑔
𝑣𝛿. 󰇛17󰇜
Now, using the conjugacy condition 󰇛11󰇜 also
gives:
𝑦
𝑑 𝜙𝑦
𝑔𝛽𝑦
𝑣0. (18)
Let us define:
𝜔𝑔𝑦
𝑣. (19)
Assuming that 𝜔0, if using (17) and (18)
leads to:
𝜙𝛿𝑦
𝑣
𝜔
and 𝛽

. (20)
Therefore, from 󰇛15󰇜 and 󰇛20󰇜 we obtain:
𝑑
 
𝑔󰇛
󰇜
󰇛
󰇜𝑣,
or, equivalently,
WSEAS TRANSACTIONS on MATHEMATICS
DOI: 10.37394/23206.2023.22.101
Isam H. Halil, Issam A.R. Moghrabi,
Ahmed A. Fawze, Basim A. Hassan, Hisham M. Khudhur
E-ISSN: 2224-2880
926
Volume 22, 2023
𝑑
 𝛿
𝑔󰇩𝑔𝑦
𝑔
󰇛𝑦
𝑣󰇜𝑣󰇪
𝛾𝑔

󰇛
󰇜𝑣. (21)
3 The Descent Feature of the
Algorithm
The following is a presentation of the scaled
conjugate gradient (CG) approach.
Step one. Choose 𝑥∈𝑅 and the line search
parameters 𝛿 and 𝛿 such that 0𝛿𝛿1.
Compute 𝑓󰇛𝑥󰇜 and 𝑔. Take 𝑑𝑔 and 𝛼
1/𝑔.
Step two. Check if the looping can be continued. If
𝑔10, we will cease.
Step three. Update the variables 𝑥 𝑥𝛼𝑑
and compute 𝛼 0 that satisfies the
line search criteria (4) and (5).
Step four. Compute the parameter 𝛽 using
equation (13).
Step Five. Compute 𝑑 𝛾󰇛𝑔𝛽𝑑󰇜. If
the restart strategy in, [5], namely 𝑔
𝑔
0.2𝑔, is met, then use 𝑑 𝛾𝑔.
Else, define 𝑑 using (21). Compute 𝛼

. Set 𝑘𝑘1, and continue to step 2.
The following next theorem establishes that the
search vectors generated by the derived scaled CG
technique are guaranteed to meet the descent
property, for Wolfe conditions-satisfied line
searches.
Theorem 1
Assuming 𝛼 meets conditions (3) and (4), hence,
𝑑 given by󰇛21󰇜 is a downhill direction.
Proof: Since 𝑑𝑔, we have 𝑔
𝑑
𝑔0. Multiplying 󰇛21󰇜 by 𝑔
, we have
𝑔
𝑑

𝑔
𝑔


𝑔
𝑣


𝑔
𝑔𝑦
𝑣
𝑦
𝑔𝑦
𝑣𝑔
𝑣.
󰇛22󰇜
Applying ( 𝑢𝑣
󰇛𝑢𝑣󰇜 to (22) with
𝑢󰇛𝑦
𝑣󰇜𝑔 and 𝑣󰇛𝑔
𝑣󰇜𝑦, we get:
𝑔
𝑑
𝛿
|𝑔|󰇛𝑦
𝑣󰇜󰇩|𝑔|𝑦
𝑣
1
2󰇣|𝑔|𝑦
𝑣𝑔
𝑣󰇛|𝑦|󰇜󰇤󰇪
or
󰇛23󰇜
𝑔
𝑑


󰇣
𝑔|𝑦
𝑣
𝑔
𝑣󰇛∣𝑦󰇜󰇤.
󰇛24󰇜
The last term in equation (24) tends to zero
closer to the minimum, so 󰇛24󰇜 can be expressed as:
𝑔
𝑑
∥|
󰇣
|𝑔|𝑦
𝑣󰇤

.
As a result, the direction 𝑑 meets the criteria for
a descent: 𝑔
𝑑 0.
4 Convergence Analysis
Using similar approaches to those in, [5], [14], [15],
we assume that the objective function 𝑓 satisfies the
Lipschitz continuity criterion on the level set L and
is also substantially convex. So,
𝐿󰇝𝑥∈R:𝑓󰇛𝑥󰇜𝑓󰇛𝑥󰇜󰇞.
Assumption A: There are values of 𝜇0 and 𝐿
such that, [11]:
󰇛∇𝑓󰇛𝑥󰇜∇𝑓󰇛𝑦󰇜󰇜󰇛𝑥𝑦󰇜𝜇𝑥𝑦|
and 󰇛∇𝑓󰇛𝑥󰇜∇𝑓󰇛𝑦󰇜󰇜𝐿𝑥𝑦
from 𝐿, for all 𝑥 and 𝑦.
Lemma 1:
Let us assume that Assumption A is true. Then
consider the general conjugate gradient approach in
(2) and (3), where 𝑑 is a downhill direction and 𝛼
is computed to satisfy the line search conditions (4)
and (5). If


∞.
Then it follows that
𝑙𝑖𝑚
→ 𝑖𝑛𝑓𝑔∣0. (31)
Similar details can be found in [16], [17].
Theorem 2:
Assume that Assumption A holds, then 󰇝𝑥󰇞
is computed by the Algorithm (3.1) with 𝛾0.
Then 𝑙𝑖𝑚
→𝑖𝑛𝑓𝑔0. (32)
Proof:
WSEAS TRANSACTIONS on MATHEMATICS
DOI: 10.37394/23206.2023.22.101
Isam H. Halil, Issam A.R. Moghrabi,
Ahmed A. Fawze, Basim A. Hassan, Hisham M. Khudhur
E-ISSN: 2224-2880
927
Volume 22, 2023
We assume the conclusion is false because of the
seeming conflict. Suppose that 𝑔0 for all 𝑘. By
then 󰇛28󰇜, we get
𝑦
𝑑󰇛𝑔𝑔󰇜𝑑𝜇𝛼𝑑. (33)
In contrast, 𝑦𝑔𝑔𝐿𝑣.
Hence,
𝑔
𝑦𝑔‖‖𝑦𝐿𝑔‖‖𝑣. (34)
Therefore, from 󰇛21󰇜, we have
𝑑
∥𝑔󰇩𝛾𝛾𝐿
𝑔
𝑣
𝜇𝛼
𝑑
𝑣
𝑔
𝛾𝛾𝐿∣𝑔
𝜇∣𝑑
𝑣
𝑔
𝛾𝛾𝐿𝜀𝛼
𝜇
𝜔∥𝑔
......󰇛35󰇜
where 𝜔𝛾𝛾
. This relation implies
that



 1∞. (36)
Therefore, 𝑙𝑖𝑚
→ 𝑖𝑛𝑓
𝑔
0.
4 Numerical Outcomes
This section presents the outcomes of the
computational experiments. A comparison has been
made between the standard HS algorithm and the
new scaled conjugate gradient method. Both
methods were tested using a Fortran
implementation. The test problems can be found in
references, [1], [16], [17], [18], [19]. In the
numerical experiments, the dimensions evaluated
were n=1000 and 10000 for each test function. We
selected fifteen large-scale, generalized, and
unconstrained problems. To determine the
termination of the process, we utilized the inequality
𝑔10 as the criterion. Table 1 provides a
view of the results derived from testing both
methods using the line parameters σ=0.001 and
𝛿0.9 in equations (4) and (5). The columns in
Table 1 indicate the following items
Problem: function name;
Dim: the dimension of the problem;
TNOI: total iterations number;
TIRS: total restarts number.
Table 1. Numerical comparison between the new
algorithm and HS Algorithm Comparison of
algorithms for 𝑛1000
Test Problems
New HS
k
NOI
IRS
NOI
IRS
Freudenstein and
Roth
137 125 838 810
Perturbed
Quadratic
328 94 392 116
Diagonal 2 199 56 200 64
Diagonal 3 1745 1591 F* F*
Extended Three
Expo Terms
28 21 25 18
Extended Powell 75 20 75 20
Extended
Maratos
64 29 85 44
Extended Cliff 11 9 13 10
Extended Wood 28 11 28 11
Quadratic QF2 395 122 403 121
EG2(CUTE) 113 48 F* F*
Tridiagonal
Perturbed Quad.
348 101 356 108
Drench (CUTE) 48 32 84 69
Diagonal 6 20 12 20 12
Sinquad (CUTE) 167 75 172 76
1848 707 2693 1479
Table 2. Numerical comparison between the
new algorithm and HS Algorithm for 𝑛
10000
Test Problems
New HS
k
NOI
IRS
NOI
IRS
Freudenstein and
Roth
15 8 F* F*
Perturbed
Quadratic
1223 326 1243 332
Diagonal 2 684 207 686 203
Diagonal 3 F* F* F* F*
Extended Three
Expo Terms
65 57 215 207
Extended Powell 82 26 97 29
Extended Maratos 64 29 64 28
Extended Cliff 16 11 24 22
Extended Wood 27 9 29 9
Quadratic QF2 1235 361 1285 368
EG2(CUTE) F* F* F* F*
Tridiagonal
Perturbed Quad.
1157 331 1297 376
Drench (CUTE) 51 40 102 89
Diagonal 6 18 9 22 11
Sinquad (CUTE) 437 29 1005 358
5059 1435 6069 2032
WSEAS TRANSACTIONS on MATHEMATICS
DOI: 10.37394/23206.2023.22.101
Isam H. Halil, Issam A.R. Moghrabi,
Ahmed A. Fawze, Basim A. Hassan, Hisham M. Khudhur
E-ISSN: 2224-2880
928
Volume 22, 2023
Table 2 above presents a numerical comparison
between the new algorithm and the HS Algorithm
for n=10000.
5 Conclusion
In this study, a novel method of scaled nonlinear
conjugate gradient is developed that, under certain
assumptions, boosts its global convergence property
for uniformly convex functions. Additionally, it
successfully fulfills the essential descent condition,
which is commonly observed in standard gradient
algorithms. The utility of the suggested new scaled
types was shown in the reported numerical
experiments. The obtained results reveal, for at least
the chosen set of test problems, that the developed
algorithm reduces by an overall 61% NOI and
78.82% IRS against the standard Hestenes-Stiefel
(HS) algorithm. The relative utility of the new
approach 󰇛𝑛1000,10000󰇜 is presented in Table
3.
Table 3. Relative utility of the new approach 󰇛𝑛
1000,10000󰇜
Tools NOI IRS
HS-Algorithm 100 % 100%
New-Algorithm 39% 21.18 %
The new method is promising and deserves
further exploration of a wider spectrum of problems
by extending the method to constrained
optimization, exploring parallel and distributed
implementations for scalability, [8], [20], [21],
developing adaptive parameter tuning schemes, and
analyzing sensitivity to initial conditions. It is also
worth integrating the method with machine learning
models, [22], assessing robustness to noisy
objectives, comparing it with state-of-the-art
methods, exploring real-world applications, and
developing user-friendly interfaces for wider
accessibility. These directions aim to enhance the
algorithm's versatility, efficiency, and practical
applicability across diverse optimization scenarios.
References:
[1] E. D. Dolan and J. J. Moré,Benchmarking
optimization software with performance
profiles,” Math. Program. Ser. B, vol. 91, no.
2, pp. 201–213, 2002, doi:
10.1007/s101070100263.
[2] B. A. Hassan and H. A. Alashoor, “On image
restoration problems using new conjugate
gradient methods,” Indonesian Journal of
Electrical Engineering and Computer
Science, vol. 29, no. 3, p. 1438, Mar. 2023,
doi: 10.11591/ijeecs.v29.i3.pp1438-1445.
[3] H. M. Khudhur and A. A. M. Fawze, "An
improved conjugate gradient method for
solving unconstrained optimization and
image restoration problems," Int. J.
Mathematical Modeling Numerical
Optimization, vol. 13, no. 3, pp. 313–325,
2023, doi: 10.1504/IJMMNO.2023.132286.
[4] B. A. Hassan and H. Sadiq,Efficient New
Conjugate Gradient Methods for Removing
Impulse Noise Images,” Eur. J. Pure Appl.
Math., vol. 15, no. 4, pp. 2011–2021, Oct.
2022, doi: 10.29020/nybg.ejpam.v15i4.4568.
[5] S. Aji, A. B. Abubakar, A. I. Kiri, and A.
Ishaku, “A Spectral Conjugate Gradient-like
Method for Convex Constrained Nonlinear
Monotone Equations and Signal Recovery,”
Nonlinear Convex Anal. Optim., vol. 1, no. 1,
pp. 1–23, 2022.
[6] M. R. Hestenes and E. Stiefel, "Methods of
conjugate gradients for solving linear
systems, "Journal of Research National
Bureau Standard (1934), vol. 49, no. 6, pp.
409–436, Dec. 1952, doi:
10.6028/jres.049.044.
[7] N. Andrei, “An accelerated conjugate
gradient algorithm with guaranteed descent
and conjugacy conditions for unconstrained
optimization,” Optimization Methods and
Software, vol. 27, no. 4–5, 2012, doi:
10.1080/10556788.2010.501379.
[8] M. M. Abed, U. Öztürk, and H. M. Khudhur,
“Spectral CG Algorithm for Solving Fuzzy
Nonlinear Equations,” Iraqi J. Computer
Science and Math., vol. 3, no. 1, pp. 1–10,
Jan. 2022, doi:
10.52866/ijcsm.2022.01.01.001.
[9] Y. A. Laylani, B. A. Hassan, and H. M.
Khudhur, “Enhanced spectral conjugate
gradient methods for unconstrained
optimization,” Computer Science, vol. 18,
no. 2, pp. 163–172, 2023.
[10] N. Andrei, “Numerical comparison of
conjugate gradient algorithms for
unconstrained optimization,” Stud.
Informatics Control, vol. 16, pp. 333–352,
2007.
[11] H. N. Jabbar and B. A. Hassan, “Two-
versions of descent conjugate gradient
methods for large-scale unconstrained
optimization,” Indonesian Journal of
Electrical Engineering and Computer
Science, vol. 22, no. 3, p. 1643, Jun. 2021,
WSEAS TRANSACTIONS on MATHEMATICS
DOI: 10.37394/23206.2023.22.101
Isam H. Halil, Issam A.R. Moghrabi,
Ahmed A. Fawze, Basim A. Hassan, Hisham M. Khudhur
E-ISSN: 2224-2880
929
Volume 22, 2023
doi: 10.11591/ijeecs.v22.i3.pp1643-1649.
[12] Y. Ismail Ibrahim and H. Mohammed
Khudhur, “Modified three-term conjugate
gradient algorithm and its applications in
image restoration,” Indonesian Journal of
Electrical Engineering and Computer
Science, vol. 28, no. 3, pp. 1510–1517, Dec.
2022, doi: 10.11591/ijeecs.v28.i3.pp1510-
1517.
[13] I. A. R. Moghrabi, "A New Formulation for
Quasi-Newton Methods," WSEAS
Transactions on Mathematics, vol. 18, pp.
79-84, 2019, DOI:
[14] E. G. Birgin and J. M. Martínez, “A Spectral
Conjugate Gradient Method for
Unconstrained Optimization,” Appl. Math.
Optim., vol. 43, no. 2, pp. 117–128, 2001,
doi: 10.1007/s00245-001-0003-0.
[15] Y. Laylani, B. A. Hassan, and H. M.
Khudhur, “A New Class of Optimization
Methods Based on Coefficient Conjugate
Gradient,” Eur. J. Pure Appl. Math., vol. 15,
no. 4, pp. 1908–1916, 2022.
[16] Y. H. Dai, “New Conjugacy Conditions and
Related Nonlinear Conjugate Gradient
Methods,Appl. Math. Optim., vol. 43, no. 1,
pp. 87–101, Jan. 2001, doi:
10.1007/s002450010019.
[17] Z.-F. Dai, “Two modified HS type conjugate
gradient methods for unconstrained
optimization problems,” Nonlinear Anal.
Theory, Methods Appl., vol. 74, no. 3, pp.
927–936, 2011.
[18] N. Andrei, “An Unconstrained Optimization
Test Functions Collection,” Adv. Model.
Optim., vol. 10, no. 1, pp. 147–161, 2008.
[19] Z. Aminifard and S. Babaie-Kafaki, “Dai–
Liao extensions of a descent hybrid nonlinear
conjugate gradient method with application
in signal processing,” Numer. Algorithms,
vol. 89, no. 3, pp. 1369–1387, Mar. 2022,
doi: 10.1007/s11075-021-01157-y.
[20] A. Olawale, I. Osinuga, R. Raji, "A Globally
Convergent Hybrid FR-PRP Conjugate
Gradient Method for Unconstrained
Optimization Problems", WSEAS
Transactions on Mathematics, vol. 20, pp.
736-744, 2021.
[21] S. Singh, S. Gutta, A. Hadaegh, "Stock
Prediction Using Machine Learning",
WSEAS Transactions on Computer
Research, vol. 9, pp. 152-158, 2021.
[22] M. Malik, I. Mohammed Sulaiman, A. Bala
Abubakar, G. Ardaneswari, S. Sukono. A
new family of hybrid three-term conjugate
gradient method for unconstrained
optimization with application to image
restoration and portfolio selection. AIMS
Mathematics, 2023, 8(1):1-28.
doi: 10.3934/math.2023001.
Contribution of Individual Authors to the
Creation of a Scientific Article (Ghostwriting
Policy)
- Isam H. Halil contributed to the mathematical
derivations.
- Issam Moghrabi drafted the document and
contributed to the derivation.
- Ahmed Fawze carried out the numerical tests on
the new method.
- Basim Hassan did the coding necessary for
carrying out the tests.
- Hisham Khudhur summarized the results and did
proofreading.
Sources of Funding for Research Presented in a
Scientific Article or Scientific Article Itself
No funding was received for conducting this study.
Conflict of Interest
The authors have no conflicts of interest to declare.
Creative Commons Attribution License 4.0
(Attribution 4.0 International, CC BY 4.0)
This article is published under the terms of the
Creative Commons Attribution License 4.0
https://creativecommons.org/licenses/by/4.0/deed.en
_US
WSEAS TRANSACTIONS on MATHEMATICS
DOI: 10.37394/23206.2023.22.101
Isam H. Halil, Issam A.R. Moghrabi,
Ahmed A. Fawze, Basim A. Hassan, Hisham M. Khudhur
E-ISSN: 2224-2880
930
Volume 22, 2023