1. Presented byPritam BhadraPranamesh ChakrabortyIndian Institute of Technology, Kanpur11 May 2013Comparative study of algorithms ofNonlinear Optimization

2. Methods of Nonlinear OptimizationThe methods for nonlinear optimization are:1. Conjugate gradient methodsa) Fletcher-Reevesb) Polak- Ribiere2. Powells conjugate direction method3. Quasi-Newton methodsa) Davidon-Fletcher-Powell (DFP) methodb) Broyden-Fletcher-Goldfarb-Shanno (BFGS) method 3. Functions used for comparison of algorithms1. Booth function2. Himmelblau function3. Beale function4. Ackley function5. Goldstein function6. Cross-in-tray function7. Bukin function8. Rosenbrock function 4. For each function(two-dimensional), local minima is obtainedfrom 6 initial points:a) (0,0)b) (1,1)c) (3,3)d) (5,5)e) (-2,-2)f) (-4,-4)g) (-6,-6)for 3 ranges of (step-size) (0,10) (-5,20) (-50,50) 5. Booth function2 2( , ) ( 2 7) (2 5)f x y x y x yQuadratic function of 2 variables3d plot of Booth function 6. Booth functionContour plot of Booth function 7. Booth function Exact same results for F-R and P-R since the function is quadraticInitialPointFletcher-Reeves Polak-Ribiere (0,10) (-5,20) (-50,50) (0,10) (-5,20) (-50,50)FinalPoint# offunctionevaluations# ofiterationsFinalPoint# offunctionevaluations# ofiterationsFinalPoint# offunctionevaluations# ofiterationsFinalPoint# offunctionevaluations# ofiterationsFinalPoint# offunctionevaluations# ofiterationsFinalPoint# offunctionevaluations# ofiterations(0,0) (1,3) 20 2 (1,3) 15 2 (1,3) 15 2 (1,3) 20 2 (1,3) 15 2 (1,3) 15 2(1,1) (1,3) 20 2 (1,3) 15 2 (1,3) 15 2 (1,3) 20 2 (1,3) 15 2 (1,3) 15 2(3,3) (1,3) 20 2 (1,3) 15 2 (1,3) 15 2 (1,3) 20 2 (1,3) 15 2 (1,3) 15 2(5,5) (1,3) 20 2 (1,3) 15 2 (1,3) 15 2 (1,3) 20 2 (1,3) 15 2 (1,3) 15 2(-2,-2) (1,3) 20 2 (1,3) 15 2 (1,3) 15 2 (1,3) 20 2 (1,3) 15 2 (1,3) 15 2(-4,-4) (1,3) 20 2 (1,3) 15 2 (1,3) 15 2 (1,3) 20 2 (1,3) 15 2 (1,3) 15 2(-6,-6) (1,3) 20 2 (1,3) 15 2 (1,3) 15 2 (1,3) 20 2 (1,3) 15 2 (1,3) 15 2 The algorithm converges in 2 steps as it is a 2-d quadratic problem 8. Booth functionInitialPointDavidon-Fletcher-Powell (DFP)methodBroyden-Fletcher-Goldfarb-Shanno(BFGS) method Powells conjugate direction method (0,10) (-5,20) (-50,50) (0,10) (-5,20) (-50,50) (0,10) (-5,20) (-50,50)FinalPoint# ofiterationsFinalPoint# ofiterationsFinalPoint# ofiterationsFinalPoint# ofiterationsFinalPoint# ofiterationsFinalPoint# ofiterationsFinalPoint# ofiterationsFinalPoint# ofiterationsFinalPoint# ofiterations(0,0)(1,3) 2,36 (1,3) 2,29 (1,3) 2,23 (1,3) 2,40 (1,3) 2,31 (1,3) 2,26(3.4,1.08)(1,3) 1 (1,3) 1(1,1)(1,3) 2,43 (1,3) 2,29 (1,3) 2,38 (1,3) 2,42 (1,3) 2,28 (1,3) 2,32(2.6,1.72)(1,3) 1 (1,3) 1(3,3)(1,3) 2,35 (1,3) 2,29 (1,3) 2,32 (1,3) 2,44 (1,3) 2,29 (1,3) 2,31 (1,3) 1 (1,3) 1(5,5)(1,3) 2,32 (1,3) 2,27 (1,3) 2,22 (1,3) 2,37 (1,3) 2,28 (1,3) 2,24NotWorkingreaching toan(1,3) 2 (1,3) 2(-2,-2)(1,3) 2,44 (1,3) 2,28 (1,3) 2,39 (1,3) 2,35 (1,3) 2,28 (1,3) 2,31arbitrarypointandoscillates(1,3) 1 (1,3) 1(-4,-4)(1,3) 2,36 (1,3) 2,27 (1,3) 2,26 (1,3) 2,38 (1,3) 2,26 (1,3) 2,27around thepoint.(1,3) 1 (1,3) 1(-6,-6)(1,3) (2,38) (1,3) 2,31 (1,3) 2,36 (1,3) 2,36 (1,3) 2,25 (1,3) 2,31 (1,3) 1 (1,3) 1DFP and BFGS works very well for Booth functionPowells method works nice for proper range ofalpha. A bit large range of alpha can be taken asthere is only a few local minima of the function.For range of alpha between 0 and 10 it did not workbecause in some steps for ensuring the step to bedescent the value of alpha is coming out to benegative. 9. Himmelblau function2 2 2 2( , ) ( 11) ( 7)f x y x y x y3d plot of Himmelblau function 10. Himmelblau functionContour plot of Himmelblau function 11. Himmelblau functionInitialPointFletcher-Reeves Polak-Ribiere (0,10) (-5,20) (-50,50) (0,10) (-5,20) (-50,50)Final Point# ofiterationsFinalPoint# ofiterationsFinal Point# ofiterationsFinal Point# ofiterationsFinal Point# ofiterationsFinal Point# ofiterations(0,0) (3,2) 8 (3,2) 8 (3,2) 8 (3,2) 6 (3,2) 6 (3,2) 6(1,1)(-3.7793,-3.2832)28 (3,2) 13 (3,2) 13 (3,2) 7(-3.7793,-3.2832)7(-3.7793,-3.2832)7(3,3) (3,2) 12 (3,2) 12 (3,2) 12 (3,2) 7 (3,2) 6 (3,2) 6(5,5)(-3.7793,-3.2832)12 (3,2) 11 (3,2) 11(-3.7793,-3.2832)8 (3,2) 6 (3,2) 7(-2,-2)(-2.8051,3.1313)12 (3,2) 18 (3,2) 8(-3.7793,-3.2832)12(-2.8051,3.1313)8 (3,2) 7(-4,-4)(-2.8051,3.1313)11 (3,2) 17 (3,2) 50(-2.8051,3.1313)12(-2.8051,3.1313)7(3.5844,-1.8481)7(-6,-6)(-3.7793,-3.2832)29 (3,2) 36(-3.7793,-3.2832)5(-2.8051,3.1313)18(-3.7793,-3.2832)9(-3.7793,-3.2832)5 P-R works better than F-R (as per # of iterations). has no significant effect on # of iterations. 12. Himmelblau functionInitialPointDavidon-Fletcher-Powell (DFP) methodBroyden-Fletcher-Goldfarb-Shanno (BFGS)method Powells conjugate direction method (0,10) (-5,20) (-50,50) (0,10) (-5,20) (-50,50) (0,10) (-5,20) (-50,50)FinalPoint# ofiterations Final Point# ofiterations Final Point# ofiterations Final Point# ofiterationsFinalPoint# ofiterationsFinalPoint# ofiterationsFinalPoint# ofiterationsFinalPoint# ofiterationsFinalPoint# ofiterations(0,0)(3,2) 9 (3,2) 9 (3,2) 9 (3,2) 9 (3,2) 9 (3,2) 9 (3.02,1.996)1(1,1)(3,2) 7(-3.7793, -3.2832)6 (3,2) 13 (3,2) 7(-3.7793,-3.2832)6 (3,2) 13 (3.00,1.99)1(3,3)(3,2) 6 (3,2) 6 (3,2) 6 (3,2) 6 (3,2) 6 (3,2)6notworking(3.01,2.00)1(5,5)(-3.7793,-3.2832)7 (3,2) 7 (3,2) 7(-3.7793, -3.2832)7 (3,2) 7 (3,2) 7 (3.00,2.00)2(-2,-2)(-3.7793,-3.2832)6(-3.7793, -3.2832)6 (3,2) 8(-3.7793,-3.2832)6 (3,2) 11(-2.8051,3.1313)12 (3.5844, -1.8481)2(-4,-4)(-2.8051,3.1313)7(-2.8051,3.1313)7(3.5844,-1.8481)7(-2.8051,3.1313)7(-2.8051,3.1313)7(3.5844,-1.8481)16(-3.7789, -3.2832)2(-6,-6)(3,2) 8 (3,2) 8(-3.7793,-3.2832)5 (3,2) 8 (3,2) 8(-3.7793,-3.2832)5(-3.7789, -3.2832)2For range of alpha -50 to 50 the algorithm may somehow help reach near local minimstill does not converge, rather it gets distracted to another arbitrary point. For range ofdoes not provide descent direction at each step. 13. For alpha -50,50 starting from(3,3) ..though there is local minima in the vicinistill it oscillates and does not convergeiterationsx 14. Beale function2 2 2 3 2( , ) (1.5 ) (2.25 ) (2.625 )f x y x xy x xy x xy3d plot of Beale function 15. Beale functionContour plot of Beale function 16. Beale functionInitialPointFletcher-Reeves Polak-Ribiere (0,10) (-5,20) (-50,50) (0,10) (-5,20) (-50,50)FinalPoint# ofiterationsFinalPoint# ofiterationsFinalPoint# ofiterationsFinalPoint# ofiterationsFinalPoint# ofiterationsFinalPoint# ofiterations(0,0) (3,0.5) 15(3.0001,0.5)17(2.9999,0.5)15 (3,0.5) 9 (3,0.5) 9 (3,0.5) 9(1,1) (3,0.5) 14 (3,0.5) 14 (3,0.5) 11 (3,0.5) 17(3.0001,0.5)16 (3,0.5) 11(3,3) Infinite(2.9999,0.5)11 Does not converge Does not converge (3,0.5) 10 Does not converge(5,5) Infinite Infinite Does not converge Does not converge Infinite (3,0.5) 12(-2,-2) Does not converge (3,0.5) 10 (3,0.5) 10 Does not converge(3.0001,0.5)8(3.0001,0.5)8(-4,-4) Infinite Does not converge Does not converge (3,0.5) 10 (3,0.5) 10 (3,0.5) 22(-6,-6) Infinite (3,0.5) 20 (3,0.5) 21 Infinite(3.0001,0.5)38 Does not converge 17. Beale functionx1x2Beale function for (3,3) initial point and =(0,10) for P-R method 18. Beale functionx1x2Beale function for (3,3) initial point and =(-50,50) for F-R method 19. Beale functionInitialPointDavidon-Fletcher-Powell (DFP) method Broyden-Fletcher-Goldfarb-Shanno (BFGS) method Powells conjugate direction method (0,10) (-5,20) (-50,50) (0,10) (-5,20) (-50,50) (0,10) (-5,20) (-50,50)FinalPoint# ofiterationsFinalPoint# ofiterationsFinalPoint# ofiterationsFinalPoint# ofiterationsFinalPoint# ofiterationsFinalPoint# ofiterationsFinalPoint# ofiterationsFinalPoint# ofiterationsFinalPoint# ofiterations(0,0)(3,0.5) 8 (3,0.5) 8 (3,0.5) 8 (3,0.5) 8 (3,0.5) 8 (3,0.5) 8 (3,0.5)4(3,0.5)4(3,0.5)3(1,1)(3,0.5) 10 (3,0.5) 9 (3,0.5) 9 (3,0.5) 9 (3,0.5) 13 (3,0.5) 15(11,1) 2(-4.04,1.21)7Convergingveryslowly(3,3)NaN 100 (3,0.5) 10 NaN 100 NaN 100 (3,0.5) 10 varying 100notvarying(3,3)__(-0.1198,3.0003) 3(5,5)(3,0.5) 118 (3,0.5) 12 (3,0.5) 947.93,.8650 (3,0.5) 12 (3,0.5) 40notvarying(5,5)may begetstuckto asaddlepoint(0,5) 3(-2,-2)NaN 100 (3,0.5) 10 (3,0.5) 8 varying 100 (3,0.5) 7 (3,0.5) 7 (3,0.5)4(3,0.5)4(-4,-4)(3,0.5) 117 (3,0.5) 10 (3,0.5) 15 (3,0.5) 10 (3,0.5) 10 (3,0.5) 12(4.7703,0.7383) 1(4.7188,0.7384) 4(-6,-6)(20.66,0.95)50 (3,0.5) 21 NaN 100(-113.86,1.0074)8 (3,0.5) 21 varying 100(6.842,.832)(6.79,0.83)3For range of alpha between -50 to 50 alpha as well as x oscillates between large range probably because it gets distractedtoo much in some iteration steps and converges very slowly sometimes. 20. Ackley function2 20.2 0.5( ) 0.5(cos(2 ) cos(2 ))( , ) 20 20x y x yf x y e e e3d plot of Ackley function 21. Ackley functionContour plot of Ackley function 22. Ackley functionInitialPointFletcher-Reeves Polak-Ribiere (0,10) (-5,20) (-50,50) (0,10) (-5,20) (-50,50)FinalPoint# ofiterationsFinalPoint# ofiterationsFinalPoint# ofiterationsFinalPoint# ofiterationsFinalPoint# ofiterationsFinalPoint# ofiterations(0,0)Infinite Infinite Infinite Infinite Infinite Infinite(1,1)(0.9685,0.9685)4(-0.9685,-0.9685)3 (-57,-57) 2(0.9685,0.9685)4(-0.9685,-0.9685)3 (-57,-57) 2(3,3)(0.9685,0.9685)4(-1.9745,-1.97453 (-82,-82) 4(0.9685,0.9685)4(-1.9745,-1.97453 (-82,-82) 4(5,5)(-0.9685,-0.9685)4(-1.9745,-1.97454 (83,83) 4(-0.9685,-0.9685)4(-1.9745,-1.97454 (83,83) 4(-2,-2)(-0.9685,-0.9685)4(-1.9745,-1.97453 (83,83) 4(-0.9685,-0.9685)4(-1.9745,-1.97453 (83,83) 4(-4,-4)(-0.9685,-0.9685)4(1.9745,1.9745)4 (-83,-83) 4(-0.9685,-0.9685)4(1.9745,1.9745)4 (-83,-83) 4(-6,-6)(-0.9685,-0.9685)2(-1.9745,-1.97454 (-83,-83) 4(-0.9685,-0.9685)2(-1.9745,-1.97454 (-83,-83) 4At(0,0) gradient of f(x,y) in not defined and hence the result shows Infinite=(-50,50) gives local minima which are far away from the initial point.P-R and F-R gives exactly same results for all and all initial points. 23. Ackley functionInitialPointDavidon-Fletcher-Powell (DFP) methodBroyden-Fletcher-Goldfarb-Shanno (BFGS)method Powells conjugate direction method (0,10) (-5,20) (-50,50) (0,10) (-5,20) (-50,50) (0,10) (-5,20) (-50,50)FinalPoint# ofiterationsFinalPoint# ofiterationsFinalPoint# ofiterationsFinalPoint# ofiterationsFinalPoint# ofiterationsFinalPoint# ofiterationsFinalPoint# ofiterationsFinalPoint# ofiterationsFinalPoint# ofiterations(0,0)NaN 100 NaN 100 NaN 100 NaN 100 NaN 100 NaN 100oscillatingoscillatingoscillatingbut it hasbeenidentified that(1,1)(.96885,.9685)4(-.96885,-.9685)3(-57,-57)2(.96885,.9685)4(-.96885,-.9685)3(-57,-57)2oscillatingoscillatingoscillatingit hasfunction(3,3)(.96885,.9685)4(5.9887,5.9887)4(.96885,.9685)4(.96885,.9685)4(5.9887,5.9887)4(.96885,.9685)4oscillatingoscillatingoscillatingvaluezero(5,5)(49,49) 4 (0,0) 6(-295,-295)4 (49,49) 4 (0,0) 6(-274,-274)4oscillatingoscillatingoscillatingor nearlyzero(-2,-2)(-.96885,-.9685)4 (0,0) 6 (0,0) 5(-.96885,-.9685)3 (0,0) 6 (0,0) 5oscillatingoscillatingoscillatingatseveralpoints(-4,-4)(.96885,.9685)4(.96885,.9685)4(.96885,.9685)4(-17,-17)4(.96885,.9685)4(.96885,.9685)4oscillatingoscillatingoscillating(-6,-6)(-.96885,-.9685)3(-.96885,-.9685)4 (0,0) 6(-.96885,-.9685)2(-.96885,-.9685)4 (0,0) 5oscillatingvarying varying 24. It can be perceived from Powells method that Ackley fn has local minima atseveral pointsBut as the algorithm reaches that point algorithm does not stop, it just oscillatesaround it and one can guess that it happens possibly due to large range of alpha(the function has several local minima within a very short distance) and thisperception was validated when the algorithm reached to local or globalminimum points when we ran the algorithm for small range of alpha (-2,3)starting from several starting points!For alpha (-5,20) the value sometimes almost reaches the minima butthen bounce back to another point not close to the previous pointAckley function 25. In Powell method for Ackley function for range of alpha (-2,3) it happily reachedA minima at (0,0) within 2 iterations though powells method did not work wellfor large range of alphaxyAckley function 26. Goldstein-Price function2 2 2 2 2 2( , ) (1 ( 1) (19 14 3 14 6 3 ))(30 (2 3 ) (18 32 12 48 36 27 ))f x y x y x x y xy y x y x x y xy y3d plot of Goldstein-Price function 27. Goldstein-Price functionContour plot of Goldstein-Price function 28. Goldstein-Price functionInitialPointFletcher-Reeves Polak-Ribiere (0,10) (-5,20) (-50,50) (0,10) (-5,20) (-50,50)FinalPoint# ofiterationsFinalPoint# ofiterationsFinalPoint# ofiterationsFinalPoint# ofiterationsFinalPoint# ofiterationsFinalPoint# ofiterations(0,0)Infinite (0,-1) 11 (0,-1) 25 Infinite (0,-1) 8 (0,-1) 9(1,1)Infinite (0,-1) 15 (0,-1) 19 Infinite (0,-1) 10 (0,-1) 26(3,3)Infinite Infinite (0,-1) 18 Infinite Infinite (0,-1) 34(5,5)Infinite Infinite (0,-1) 39 Infinite Infinite (0,-1) 22(-2,-2)Infinite Infinite (0,-1) 33 Infinite Infinite (0,-1) 10(-4,-4)Infinite Infinite (0,-1) 49 Infinite Infinite (0,-1) 11(-6,-6)Infinite InfiniteDoes notconvergeInfinite InfiniteDoes notconverge =(-50,50) works better compared to other ranges.Changing to(-100,100) results in convergence for (-6,-6) initial point also. 29. Goldstein-Price functionDFP and BFGS gives better result (less # of iterations) compared to F-R and P-R .InitialPointDavidon-Fletcher-Powell (DFP) methodBroyden-Fletcher-Goldfarb-Shanno (BFGS)methodPowells conjugate direction method (0,10) (-5,20) (-50,50) (0,10) (-5,20) (-50,50) (0,10) (-5,20) (-50,50)FinalPoint# ofiterationsFinalPoint# ofiterationsFinalPoint# ofiterationsFinalPoint# ofiterationsFinalPoint# ofiterationsFinalPoint# ofiterationsFinalPoint# ofiterationsFinalPoint# ofiterationsFinalPoint# ofiterations(0,0) (0,-1) 12 (0,-1) 6 (0,-1) 5 (0,-1) 12 (0,-1) 6 (0,-1) 5oscillating(0,-1) 1(0,-.999)1(1,1)(1.8,0.2)12 (1.8,2) 6 (0,-1) 7(1.8,0.2)12 (1.8,2) 6 (0,-1) 7 (2.986,1) 2 (0,-1) 4(0.003,-.99)1(3,3) NaN 100 NaN 100 (0,-1) 11 NaN 100 (0,-1) 17 (0,-1) 11oscillating(0,-1) 2 (0,-1) 5(5,5) NaN 100 NaN 100 (0,-1) 11 NaN 100 (0,-1) 17 (0,-1) 12 (0,-1) 3 (0,-1) 3(-2,-2)(0,-1) 16 (0,-1) 32(-.6,-.4)7 (0,-1) 16 (0,-1) 12(-.6,-.4)7(2.497,.675)4(-.6,-.4)3 (0,-1) 4(-4,-4)NaN 100(-.6,-.4)22 (0,-1) 16 NaN 100 (0,-1) 14 (0,-1) 11(2.99330.9896)3 (0,-1) 3oscillating(-6,-6)NaN 100 NaN 100 (0,-1) 10 NaN 100 (0,-1) 17 (0,-1) 10(1.84340.2297)3 (0,-1) 3*3 (0,-1) 3 30. Cross-in-tray function2 20.1100( , ) 0.0001 sin( )sin( ) 1x yf x y x y e3d plot of Cross-in -Tray function 31. Cross-in-tray functionContour plot of Cross-in -Tray function 32. Cross-in-tray functionInitialPointFletcher-Reeves Polak-Ribiere (0,10) (-5,20) (-50,50) (0,10) (-5,20) (-50,50)FinalPoint# ofiterationsFinalPoint# ofiterationsFinalPoint# ofiterationsFinalPoint# ofiterationsFinalPoint# ofiterationsFinalPoint# ofiterations(0,0) Infinite Infinite Infinite Infinite Infinite Infinite(1,1)(1.3494,1.3494)2(1.3494,1.3494)2(1.3494,1.3494)2(1.3494,1.3494)2(1.3494,1.3494)2(1.3494,1.3494)2(3,3)(1.3494,1.3494)2(-1.3494,-1.3494)2(-1.3494,-1.3494)2(1.3494,1.3494)2(-1.3494,-1.3494)2(-1.3494,-1.3494)2(5,5)(4.4910,4.4910)2(4.4910,4.4910)2(4.4910,4.4910)2(4.4910,4.4910)2(4.4910,4.4910)2(4.4910,4.4910)2(-2,-2)(-1.3494,-1.3494)2(-1.3494,-1.3494)2(1.3494,1.3494)2(-1.3494,-1.3494)2(-1.3494,-1.3494)2(1.3494,1.3494)2(-4,-4)(-4.4910,-4.4910)2(-4.4910,-4.4910)2(-4.4910,-4.4910)2(-4.4910,-4.4910)2(-4.4910,-4.4910)2(-4.4910,-4.4910)2(-6,-6)(-4.4910,-4.4910)2(-1.3494,-1.3494)2(-1.3494,-1.3494)2(-4.4910,-4.4910)2(-1.3494,-1.3494)2(-1.3494,-1.3494)2At(0,0) gradient of f(x,y) in not defined and hence the result shows InfiniteFor =(0,10), positive initial point gives positive local minima only whichis not the case for other . 33. Cross-in-tray functionAll methods give local minima closest to initial point.Davidon-Fletcher-Powell (DFP) method Broyden-Fletcher-Goldfarb-Shanno (BFGS) method Powells conjugate direction method (0,10) (-5,20) (-50,50) (0,10) (-5,20) (-50,50) (0,10) (-5,20) (-50,50)FinalPoint# ofiterationsFinalPoint# ofiterationsFinalPoint# ofiterationsFinalPoint# ofiterationsFinalPoint# ofiterationsFinalPoint# ofiterationsFinalPoint# ofiterationsFinalPoint# ofiterationsFinalPoint# ofiterationsNaN NaN NaN NaNNaN NaNoscillating(0,-1) 1 oscillating(1.3494,1.3494)2(1.3494,1.3494)2(1.3494,1.3494)2(1.3494,1.3494)2(1.3494,1.3494)2(1.3494,1.3494)2 oscillating(0,-1) 4 oscillating(1.3494,1.3494)2(-1.3494,-1.3494)2(-1.3494,-1.3494)2(1.3494,1.3494)2(-1.3494,-1.3494)2(-1.3494,-1.3494)2oscillating(0,-1) 2oscillating(4.491,4.491)2(4.491,4.491)2(4.491,4.491)2(4.491,4.491)2(4.491,4.491)2(4.491,4.491)2 oscillating(0,-1) 3 oscillating(-1.3494,-1.3494)2(-1.3494,-1.3494)2(1.3494,1.3494)2(-1.3494,-1.3494)2(-1.3494,-1.3494)2(1.3494,1.3494)2oscillating(-.6,-.4) 3oscillating(-4.491,-4.491)2(-4.491,-4.491)2(-4.491,-4.491)2(-4.491,-4.491)2(-4.491,-4.491)2(-4.491,-4.491)2 oscillating(0,-1) 3 oscillating(-4.491,-4.491)2(-1.3494,-1.3494)2(-1.3494,-1.3494)2(-4.491,-4.491)2(-1.3494,-1.3494)2(-1.3494,-1.3494)2oscillating(0,-1) 3*3oscillating 34. Bukin function2( , ) 100 | 0.01 | 0.01| 10|f x y y x x3d plot of Bukin function 35. Bukin function3d plot of Bukin function 36. Bukin functionInitialPointFletcher-Reeves Polak-Ribiere (0,10) (-5,20) (-50,50) (0,10) (-5,20) (-50,50)FinalPoint# ofiterationsFinalPoint# ofiterationsFinalPoint# ofiterationsFinalPoint# ofiterationsFinalPoint# ofiterationsFinalPoint# ofiterations(0,0)Infinite Infinite Infinite Infinite Infinite Infinite(1,1)Does not converge Does not converge Does not converge Does not converge Does not converge Does not converge(3,3)Does not converge Does not converge Does not converge Does not converge Does not converge Does not converge(5,5)Does not converge Does not converge Does not converge Does not converge Does not converge Does not converge(-2,-2)Does not converge Does not converge Does not converge Does not converge Does not converge Does not converge(-4,-4)Does not converge Does not converge Does not converge Does not converge Does not converge Does not converge(-6,-6)Does not converge Does not converge Does not converge Does not converge Does not converge Does not converge 37. Bukin functionx1x2Bukins function for (-2,-2) initial point and =(-50,50) for F-R method 38. Bukin functionInitialPointDavidon-Fletcher-Powell (DFP) method Broyden-Fletcher-Goldfarb-Shanno (BFGS) method Powells conjugate direction method (0,10) (-5,20) (-50,50) (0,10) (-5,20) (-50,50) (0,10) (-5,20) (-50,50)FinalPoint# ofiterationsFinalPoint# ofiterationsFinalPoint# ofiterationsFinalPoint# ofiterationsFinalPoint# ofiterationsFinalPoint# ofiterationsFinal Point# ofiterationsFinal Point# ofiterationsFinal Point# ofiterations(0,0)(-0.01620.0000) 1(-0.01620.0000) 2(-0.01620.0000) 1(1,1)(10,1) notconvergingproperly 1(10,1) notconvergingproperly 1varying oroscillating(3,3)NOTWORKINGvarying oroscillatingvaryingvarying oroscillating(5,5)varying oroscillatingoscillatingvarying oroscillating(-2,-2)varying oroscillating(0,0)1varying oroscillating(-4,-4)varying oroscillatingconvergesto(14.32482.0520)but rherewere somepointshaving lessfunctionvalue 6varying oroscillating(-6,-6)varying oroscillating(-0.01620.0000)1varying oroscillating 39. For Powell method alpha (-50,50) starting from (3,3) the ultimate point just oscillates fro(-17.02,3) to (17.02,3)xy 40. Rosenbrock function12 2 211( ) {100( ) ( 1) }ni i iif X x x x 41. Initial PointFletcher-Reeves Polak-Ribiere (0,10) (-5,20) (-50,50) (0,10) (-5,20) (-50,50)FinalPoint# ofiterationsFinalPoint# ofiterationsFinalPoint# ofiterationsFinalPoint# ofiterationsFinalPoint# ofiterationsFinalPoint# ofiterations(0,0,0,0,0,0,0,0)Does not converge Does not converge Does not converge Does not converge Does not converge Does not converge(1,1,1,1,1,1,1,1)(1,1,1,1,1,1,1,1)1(1,1,1,1,1,1,1,1)1(1,1,1,1,1,1,1,1)1(1,1,1,1,1,1,1,1)1(1,1,1,1,1,1,1,1)1(1,1,1,1,1,1,1,1)1(3,3,3,3,3,3,3,3)Does not converge Does not converge Does not converge Does not converge Does not converge Does not converge(5,5,5,5,5,5,5,5)Does not converge Does not converge Does not converge Does not converge Does not converge Does not converge(-2,-2,-2,-2,-2,-2,-2,-2)Does not converge Does not converge Does not converge Does not converge Does not converge Does not converge(-4,-4,-4,-4,-4,-4,-4,-4)Does not converge Does not converge Does not converge Does not converge Does not converge Does not converge(-6,-6,-6,-6,-6,-6,-6,-6)Does not converge Does not converge Does not converge Does not converge Does not converge Does not convergeThe point (1,1,1,1,1,1,1,1) is itself a minimum point and hence itconverges for that initial point onlyRosenbrock (8 variable) function 42. Rosenbrock (8 variable) functionF-R very slowly converges 10 the minimum point.F-R method for =(0,10) and (-2, -2, -2, -2, -2, -2, -2, -2) initial point 43. InitialPointDavidon-Fletcher-Powell (DFP) method Broyden-Fletcher-Goldfarb-Shanno (BFGS) method (0,10) (-5,20) (-50,50) (0,10) (-5,20) (-50,50)Final Point# ofiterationsand # offunctionevaluations Final Point# ofiterationsand # offunctionevaluations Final Point# ofiterationsand # offunctionevaluations Final Point# ofiterationsand # offunctionevaluations Final Point# ofiterationsand # offunctionevaluations Final Point# ofiterationsand # offunctionevaluations(0,0,0,0,0,0,0,0)(1,1,1,1,1,1,1,1)59(1,1,1,1,1,1,1,1)45(1,1,1,1,1,1,1,1)45(1,1,1,1,1,1,1,1)46(1,1,1,1,1,1,1,1)45(1,1,1,1,1,1,1,1)45(1,1,1,1,1,1,1,1)(1,1,1,1,1,1,1,1)1(1,1,1,1,1,1,1,1)1(1,1,1,1,1,1,1,1)1(1,1,1,1,1,1,1,1)1(1,1,1,1,1,1,1,1)1(1,1,1,1,1,1,1,1)1(3,3,3,3,3,3,3,3)slowlyconverging(1,1,1,1,1,1,1,1)44(1,1,1,1,1,1,1,1)39(1,1,1,1,1,1,1,1)31(1,1,1,1,1,1,1,1)34(1,1,1,1,1,1,1,1)37(5,5,5,5,5,5,5,5)notconvergingor veryslowlyconverging(1,1,1,1,1,1,1,1)44(1,1,1,1,1,1,1,1)271(1,1,1,1,1,1,1,1)75(1,1,1,1,1,1,1,1)48(1,1,1,1,1,1,1,1)72(-2,-2,-2,-2,-2,-2,-2,-2)slowlyconverging/oscillating(1,1,1,1,1,1,1,1)69(1,1,1,1,1,1,1,1)65(1,1,1,1,1,1,1,1)52(1,1,1,1,1,1,1,1)54(1,1,1,1,1,1,1,1)51(-4,-4,-4,-4,-4,-4,-4,-4)slowlyconverging/oscillating(1,1,1,1,1,1,1,1)slowlyconverging/oscillating(1,1,1,1,1,1,1,1)64(1,1,1,1,1,1,1,1)56(1,1,1,1,1,1,1,1)41(1,1,1,1,1,1,1,1)56(-6,-6,-6,-6,-6,-6,-6,-6)(1,1,1,1,1,1,1,1)30(1,1,1,1,1,1,1,1)126(1,1,1,1,1,1,1,1)182(1,1,1,1,1,1,1,1)23(1,1,1,1,1,1,1,1)37(1,1,1,1,1,1,1,1)62Rosenbrock(8 variable) function 44. Conclusions For quadratic problems, all methods gives satisfactory results.Powells method is working satisfactorily for small (i.e.lower-dimensional)problems [particularly for =(-5,20)]DFP-BFGS works very good for bad functions (e.g Goldstein Pricefunction) where F-R and P-R does not work well.Suitability of DFP, BFGS for higher dimensional problems need to bestudied more.Range of step size if chosen small and searches on both side of the givenpoint [=(-5,20)], it works well for all problems.