Project on ? Newton-RaphsonMethod

Nguyen Quan Ba Hong∗

Doan Tran Nguyen Tung†

Students at Faculty of Math and Computer Science,

Ho Chi Minh University of Science, Vietnam

email. dtrngtung@live.com

email. nguyenquanbahong@gmail.com

blog. http://hongnguyenquanba.wordpress.com ‡

June 29, 2016

Abstract

This paper contains my team’s notes about the Newton-Raphson method.

∗Student ID: 1411103.†Student ID: 1411352.‡Copyright c© 2016 by Nguyen Quan Ba Hong, Student at Ho Chi Minh University of

Science, Vietnam. This document may be copied freely for the purposes of education andnon-commercial research. Visit my site http://hongnguyenquanba.wordpress.comto get more.

1

Contents

1 Introduction 31.1 Historical notes . . . . . . . . . . . . . . . . . . . . . . . . . . . . 31.2 Later studies. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 31.3 A glance at Newton-Raphson method . . . . . . . . . . . . . . . 4

2 The Newton-Raphson Method 42.1 Geometric view point . . . . . . . . . . . . . . . . . . . . . . . . 52.2 Analytical view point . . . . . . . . . . . . . . . . . . . . . . . . . 6

3 Selected examples 7

4 The Newton-Raphson method has fallen! 74.1 The Newton method can go bad . . . . . . . . . . . . . . . . . . 74.2 Drawbacks . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 74.3 What is wrong with Newton-Raphson . . . . . . . . . . . . . . . 84.4 Newton-Raphson method is not always applicable . . . . . . . . . 9

5 Analysis and explain 10

6 Convergence’s conditions 11

7 Improvements 117.1 Cubic iteration . . . . . . . . . . . . . . . . . . . . . . . . . . . . 117.2 Householder’s iteration . . . . . . . . . . . . . . . . . . . . . . . . 12

8 High order iteration 138.1 Householder’s methods . . . . . . . . . . . . . . . . . . . . . . . . 138.2 Modified methods . . . . . . . . . . . . . . . . . . . . . . . . . . 13

9 Generalization 14

10 Convergence of Newton-Raphson method 16

11 Newton’s method for several variables 17

12 Extension to systems of equations 19

2

1 Introduction

1.1 Historical notes

Since it is not possible to solve all equations of the form f(x) = 0 exactly, anefficient method of approximating solutions is useful. The algorithm discussedin this paper was discovered by Sir Issac Newton, who formulated the result in1669. Later improved by Joseph Raphson in 1690, the algorithm is presentlyknown as the Newton-Raphson method, or more commonly Newton’s method.

Newton’s method involves choosing an initial guess x0, and then, through aniterative process, finding a sequence of numbers x0, x1, x2, x3, ... that convergeto a solution. Some functions may have several roots. Later we see that theroot which Newton’s method converges to depends on the initial guess x0. Thebehavior of Newton’s method, or the pattern of which initial guesses lead towhich zeros, can be interesting even for polynomials. When generalized to thecomplex plane, Newton’s method leads to beautiful pictures.

A method for finding the roots of an arbitrary function that uses the derivativewas first circulated by Isaac Newton in 1669. John Wallis published Newton’smethod in 1685, and in 1690 Joseph Raphson (16481715) published an improvedversion, essentially the form in which we use it today.

Newton’s work was done in 1669 but published much later. Numerical meth-ods related to the Newton Method were used by alKashi, Viete, Briggs, andOughtred, all many years before Newton.

Raphson, some 20 years after Newton, got close to the equation, but only forpolynomials P (y) of degree 3, 4, 5, . . . , 10. Given an estimate g for a root,Raphson computes an improved estimate g+x. He sets P (g+x) = 0, expands,discards terms in xk with k ≥ 2, and solves for x. For polynomials, Raphson’sprocedure is equivalent to linear approximation.

Raphson, like Newton, seems unaware of the connection between his methodand the derivative. The connection was made about 50 years later (Simpson,Euler), and the Newton Method finally moved beyond polynomial equations.The familiar geometric interpretation of the Newton Method may have beenfirst used by Mourraille (1768). Analysis of the convergence of the NewtonMethod had to wait until Fourier and Cauchy in the 1820s.

1.2 Later studies.

The method was then studied and generalized by other mathematicians likeSimpson (1710-1761), Mourraille (1720-1808), Cauchy (1789-1857), Kantorovich(1912-1986), ... The important question of the choice of the starting point wasfirst approached by Mourraille in 1768 and the difficulty to make this choice is

3

the main drawback of the algorithm.

• Although the bisection method will always converge on the root, the rateof convergence is very slow.

• A faster method for converging on a single root of a function is the Newton-Raphson method.

• Perhaps it is the most widely used method of all locating formulas.

1.3 A glance at Newton-Raphson method

This section is concerned with the problem of root location; i.e. finding thosevalues of x which satisfy an equation of the form f(x) = 0 for a given functionf(x). An initial estimate of the root is found by drawing a graph of the func-tion in the neighborhood of the root. This estimate is then improved by using atechnique known as the Newton-Raphson method. The method is based upona knowledge of the tangent to the curve near the root. It is an iterative methodin that it can be used repeatedly to continually improve the accuracy of the root.

The Newton-Raphson method, or Newton Method, is a powerful technique forsolving equations numerically. Like so much of the differential calculus, it isbased on the simple idea of linear approximation. The Newton Method, prop-erly used, usually homes in on a root with devastating efficiency.

2 The Newton-Raphson Method

Newton-Raphson method. Newton-Raphson method is the most effectivemethod for finding roots by iteration f(x) = 0. The method consists of thefollowing steps:

1. Pick a point x0 close to a root. Find the corresponding point (x0, f (x0))on the curve.

2. Draw the tangent line to the curve at that point, and see where it crossesthe x-axis.

3. The crossing point, x1, is your next guess. Repeat the process startingfrom that point.

In fact there are many ways to improve this numerical search for the root. Inthis section we examine one of the best methods: the Newton-Raphson method.To obtain the method we examine the general characteristics of a curve in theneighborhood of a simple root.

4

Figure 1: Example for a curve and its root x∗.

Figure 2: Tangent of the curve at (x0, f(x0) cut Ox at (x1, 0).

2.1 Geometric view point

Consider the following diagram showing a function f(x) with a simple root atx = x∗ whose value is required. Initial analysis has indicated that the root isapproximately located at x = x0. The aim of any numerical procedure is toprovide a better estimate to the location of the root.

The basic premise of the Newton-Raphson method is the assumption thatthe curve in the close neighborhood of the simple root at x is approximately astraight line. Hence if we draw the tangent to the curve at x0, this tangent willintersect the xaxis at a point closer to x∗ than is x0: From the geometry of thisdiagram we see that

x1 = x0 − PQ

5

But from the right-angled triangle PQR we have

RQ

PQ= tan θ = f ′ (x0)

and so

PQ =RQ

f ′ (x0)=

f (x0)

f ′ (x0)

x1 = x0 −f (x0)

f ′ (x0)

If f(x) has a simple root near x0 then a closer estimate to the root is x1 where

x1 = x0 −f (x0)

f ′ (x0)

This formula can be used time and time again giving rise to the following:

The Newton-Raphson iterative formula.If f(x) has a simple root near xn then a closer estimate to the root is xn+1

where

xn+1 = xn −f (xn)

f ′ (xn)

This is the Newton-Raphson iterative formula. The iteration is begun with aninitial estimate of the root, x0, and continued to find x1, x2, ... until a suitablyaccurate estimate of the position of the root is obtained.

2.2 Analytical view point

We suppose that f is a C2 function on a given interval, then using Taylorsexpansion near x

f (x+ h) = f (x) + hf ′ (x) +O(h2)

and if we stop at the first order (linearization of the equation), we are lookingfor a small h such as

f (x+ h) = 0 ≈ f (x) + hf ′ (x)

giving

h = − f(x)f ′(x)

x+ h = x− f(x)f ′(x)

6

3 Selected examples

Example. f (x) = x − 2 + lnx has a root near x = 1.5. Use the Newton-Raphson formula to obtain a better estimate.Solution. Here

x0 = 1.5, f (1.5) = −0.5 + ln 1.5 = −0.0945

f ′ (x) = 1 +1

x, f ′ (1.5) = 1 +

1

1.5=

5

3Hence using the formula

x1 = 1.5− −0.0945

1.6667= 1.5567

The Newton-Raphson formula can be used again: this time beginning with1.5567 as our initial estimate. This time use:

x2 = x1 −f (x1)

f ′ (x1)= 1.5567− f (1.5567)

f ′ (1.5567)

= 1.5567− 1.5567− 2 + ln 1.5567

1 +1

1.5567

= 1.5571

This is in fact the correct value of the root to 4 d.p.

4 The Newton-Raphson method has fallen!

4.1 The Newton method can go bad

• Once the Newton Method catches scent of the root, it usually hunts itdown with amazing speed. But since the method is based on local infor-mation, namely f(xn) and f ′(xn), the Newton Method’s sense of smell isdeficient.

• If the initial estimate is not close enough to the root, the Newton Methodmay not converge, or may converge to the wrong root

• The successive estimates of the Newton Method may converge to the roottoo slowly, or may not converge at all.

4.2 Drawbacks

The Newton-Raphson methods has some drawbacks.

1. It cannot handle multiple roots.

2. It has slow convergence (compared with newer techniques).

3. The solution may diverge near a point of inflection.

4. The solution might oscillates new local minima or maxima.

5. With near-zero slope, the solution may diverge or reach a different root.

7

4.3 What is wrong with Newton-Raphson

The Newton-Raphson does not always work. Consider the function defined by

f (x) =

{x√|x|, if x 6= 0

0, if x = 0

Easy to prove that f is continuous. The derivative of this function is

f ′ (x) =1

2√|x|, x 6= 0

If we choose any starting point off the actual root, x1 = a 6= 0, then

x2 = a−a√|a|1

2√|a|

= a− 2a = −a

If follows that

xn =

{a, if n odd−a, if n even

So the Newton-Raphson is completely failed for this function.A more common occurrence is that NewtonRaphson works for some choices ofstarting point but not for others, and when it does work it does not necessarilytake you to the closest root. We consider the function defined by

f (x) = sinx− x

2, f ′ (x) = cosx− 1

2

This function has three roots.For this function, the Newton-Raphson method uses the iteration

xk+1 = xk −sinxk − xk

2

cosxk − 0.5

If we start with x1 = 2, we quickly approach the rightmost root:

x1 = 2x2 = 1.900995594x3 = 1.898679953x4 = 1.895500113x5 = 1.895494267x6 = 1.895494267

But if we start with x1 = 1, x1 = 1.1, x1 = 1.01, x1 = 1.02:This does not mean that the Newton-Raphson method is no good. Even todayit is one of the most useful and powerful tools available for finding roots. But aswe have seen, it can have problems. We need further analysis of how and whyit works if we want to determine when we can use it safely and when we must

8

proceed with caution.

Example 2. Take f : R → R, x 7→ x2 − x + 1 and x0 = 1. As f ′(x) = 2x− 1,

x1 = 1− f(1)f ′(1) = 1− 1

1 = 0 and x2 = 0− f(0)f ′(0) = 0− 1

−1 = 1. It follows that

xn =

{1, if n is even0, if n is odd

Thus {xn} does not converge.

4.4 Newton-Raphson method is not always applicable

Newton’s method has one small flaw, though. To apply the method you haveto be able to compute the derivative f ′(x). At first, you might think that thisnot such a big deal. Almost any reasonable function that one can write downcan be differentiated, so the derivative step doesn’t look like a problem.The problem in practice is that functions come in many forms, and not all ofthese forms lend themselves to computing derivatives. Here are several differentways that functions can be defined.

1. The function is defined via a closed form formula involving elementaryfunctions.

f (x) =ex + x2√

sinx

2. The function is defined via an integral

f (x) =

x∫1

sin t

tdt

3. The function is defined via a convergent power series.

f (x) =

∞∑n=0

(−1)n

n2 + 1xn

4. The function is the solution to a differential equation

y′′ + yy′ = ex; y (0) = 1; y′ (0) = 0

5. The function is defined recursively

f (x) =

√f (−2x− 2) + f (x+ 10), if − 10 < x < 10

2, if x ≤ −10−2, if x ≥ −10

Only about the first two and a half of these methods produce functions whosederivatives can be readily computed. In the absence of derivative informationwe can deploy some alternative algorithms.

9

5 Analysis and explain

Let r be the actual though unknown value of a root that we are trying toapproach, and let xk be our latest guess. We assume that it is ‘pretty close.’Wecan calculate f(xk) and f ′(xk). While we do not know the value of r, the factthat it is a root implies that f(r) = 0. Since xk is ‘pretty close’to r, f ′(xk) willbe ‘pretty close’to the slope of the line from (xk, f(xk)) to (r, 0):

f ′ (xk) ≈ f (r)− f (xk)

r − xk= − f (xk)

r − xk(5.1)

We can solve this to get something that is pretty close to r.

r ≈ xk −f (xk)

f ′ (xk)(5.2)

As our last example showed, ‘pretty close’can sometimes be not close at all.Equation (5.1) shows that we are using an approximation to the derivative.Equation (5.2) this approximation in the denominator , and there lies the cruxof our problem. We need to know the size of the error in equation (5.2). Wewill use Lagrange’s remainder for the Taylor series to solve this trouble.We use the equality

f (x) = f (a) + f ′ (a) (x− a) +f ′′ (c)

2!(x− a)

2(5.3)

where c is some unknown constant between a and x.Equivalent to

f ′ (a) =f (x)− f (a)

x− a− f ′′ (c)

2!(x− a) (5.4)

The error is precisely − f′′(c)2 (x− a). Although we do not know the value of c,

it may be possible to find bounds on f ′′(c) when c is between a and x, and thusfind bounds on the error.We replace a with xk and x with r, and then solve for r, keeping the rxk termin the error:

f ′ (xk) = −f(xk)r−xk

− f ′′(c)2! (r − xk)

r − xk = −f(xk)f ′(xk)

− f ′′(c)2f ′(xk)

(r − xk)2

r = xk − f(xk)f ′(xk)

− f ′′(c)2f ′(xk)

(r − xk)2

= xk+1 − f ′′(c)2f ′(xk)

(r − xk)2

(5.5)

Equation (5.5) gives us a relationship between |r − xk+1| and |r − xk|

|r − xk+1| =∣∣∣∣ f ′′ (c)2f ′ (xk)

∣∣∣∣ |r − xk|2 (5.6)

We shall get closer to r provided∣∣∣∣ f ′′ (c)2f ′ (xk)

∣∣∣∣ |r − xk| < 1

or, equivalently

|f ′′ (c)| < 2

∣∣∣∣f ′ (xk)

r − xk

∣∣∣∣10

6 Convergence’s conditions

Theorem. Let r be a root of f(x) = 0, where f is a C2 function on an inter-val containing r, and we suppose that |f ′ (r)| > 0, then Newton’s iteration willconverge to r if the starting point x0 is close enough to r.

This theorem insure that Newton’s method will always converge if the initialpoint is sufficiently close to the root and if this root if not singular (that is f(r)is non zero). This process has the local convergence property. A more construc-tive theorem was given by Kantorovich, we give it in the particular case of afunction of a single variable x.

Theorem. (Kantorovich). Let f be a C2 numerical function from an openinterval I of the real line, let x0 be a point in this interval, then if there existfinite positive constants (m0,M0,K0 such as∣∣∣ 1

f ′(x0)

∣∣∣ < m0∣∣∣ f(x0)f ′(x0)

∣∣∣ < M0

|f ′′ (x0)| < K0

and if h0 = 2m0M0K0 ≤ 1, Newton’s iteration will converge to a root r off(x) = 0, and

|xn − r| < 21−nM0h2n−1

0

This theorem gives sufficient conditions to insure the existence of a root andthe convergence of Newton’s process. More if h0 < 1, the last inequality showsthat the convergence is quadratic (the number of good digits is doubled at eachiteration) Note that if the starting point x0 tends to the root r, the constant M0

will tend to zero and h0 will become smaller than 1, so the local convergencetheorem is a consequence of Kantorovich’s theorem.

7 Improvements

We will focus on these things:

• Some improvements for Newton-Raphson method.

• Other iterations and method.

• Prepare for generalization.

7.1 Cubic iteration

Newton’s iteration may be seen as a first order method (or linearization method),it’s possible to go one step further and write the Taylor expansion of f to a higher

11

order

f (x+ h) = f (x) + hf ′ (x) +h2

2f ′′ (x) +O

(h3)

and we are looking for h such as

f (x+ h) = 0 ≈ f (x) + hf ′ (x) +h2

2f ′′ (x)

we take the smallest solution for h (we have to suppose that f ′(x) and f ′′(x)are non zero)

h = − f′ (x)

f ′′ (x)

(1−

√1− 2f (x) f ′′ (x)

(f ′ (x))2

)It’s not necessary to compute the square root, because if f(x) is small, usingthe expansion

1−√

1− α =α

2+α2

8+O

(α3)

h becomes

h = − f (x)

f ′ (x)

(1 +

f (x) f ′′ (x)

2(f ′ (x))2 + · · ·

)The first attempt to use the second order expansion is due to the astronomerE. Halley (1656-1743) in 1694.

7.2 Householder’s iteration

The previous expression for h, allows to derive the following cubic iteration (thenumber of digits triples at each iteration), starting with x0

xn+1 = xn −f (xn)

f ′ (xn)

(1 +

f (xn) f ′′ (xn)

2(f ′ (xn))2

)

It can be efficiently used to compute the inverse or the square root of a number.Another similar cubic iteration may be given by

xn+1 = xn −2f (xn) f ′ (xn)

2(f ′ (xn))2 − f (xn) f ′′ (xn)

sometimes known as Halley’s method. We may also write it as

xn+1 = xn − f (xn)

(1

f ′(xn)− f(xn)f′′(xn)

2f′(xn)

)= xn − f(xn)

f ′(xn)

(1− f(xn)f

′′(xn)

2(f ′(xn))2

)−1Note that if we replace (1− α)

−1by 1 + α+O

(α2), we retrieve Householder’s

iteration.

12

8 High order iteration

8.1 Householder’s methods

Under some conditions of regularity of f and its derivative, Householder gavethe general iteration

xn+1 = xn + (p+ 1)

(

1f

)(p)(

1f

)(p+1)

xn

where p is an integer and(

1f

)(p)is the derivative of order p of the inverse of

the function f . This iteration has convergence of order (p + 2). For examplep = 0 has quadratic convergence (order 2) and the formula gives back Newton’siteration while p = 1 has cubical convergence (order 3) and gives again Halley’smethod. Just like Newton’s method a good starting point is required to insureconvergence.Using the iteration with p = 2, gives the following iteration which has quarticalconvergence (order 4)

xn+1 = xn + f (xn)

((f ′ (xn))

2 − f(xn)f′′(xn)

2

(f ′ (xn))3 − f (xn) f ′ (xn) f ′′ (xn) + f(3)(xn)f2(xn)

6

)

8.2 Modified methods

Another idea is to write

xn+1 = xn + hn + an2hn22!

+ an3hn33!

+ · · ·

where hn = − f(xn)f ′(xn)

is given by the simple Newton’s iteration and (an2 , an3 , . . .)

are real parameters which we will estimate in order to minimize the value off (xn+1):

f (xn+1) = f

(xn + hn + an2

hn22!

+ an3hn33!

+ · · ·)

We assume that f is regular enough and hn + an2hn2

2! + an3hn3

3! + · · · is small, henceusing the expansion of f near xn

f (xn+1) = f (xn) +(hn + an2

hn2

2! + an3hn3

3! + · · ·)f ′ (xn) +

+(hn + an2

hn2

2! + an3hn3

3! + · · ·)f ′′(xn)

2 + · · ·

and because f (xn)+hnf′ (xn) = 0, we have f (xn+1) = (an2f

′ (xn) + f ′′ (xn))h2n

2! +(an3f

′ (xn) + 3an2f′′ (xn) + f (3) (xn)

) h3n

3! +O (hn4 ).

13

A good choice for the ani is clearly to cancel as many terms as possible in theprevious expansion, so we impose

an2 = − f′′(xn)f ′(xn)

an3 =−f ′(xn)f

(3)(xn)+3(f ′′(xn))2

(f ′(xn))2

an4 =−(f ′(xn))

2f(4)(xn)+10f ′(xn)f

′′(xn)f(3)(xn)−15(f ′′(xn))

3

(f ′(xn))2

an5 = . . .

The formal values of the ani may be computed for much larger values of i. Finallythe general iteration is

xn+1 = xn + hn(1 + an2

hn

2! + an3hn

3! + an4hn

4! + · · ·)

= xn − f(xn)f ′(xn)

(1 + f ′′(xn)

2!f ′(xn)

(f(xn)f ′(xn)

)+

3(f ′′(xn))2−f ′(xn)f

(3)(xn)

3!(f ′(xn))2

(f(xn)f ′(xn)

)2+ · · ·

)For example if we stop at an3 and set an4 = an5 = . . . = 0, we have the helpfulquartic modified iteration (note that this iteration is different than the previousHouseholders quartic method)

xn+1 = xn−f (xn)

f ′ (xn)

1 +f (xn) f ′′ (xn)

2!(f ′ (xn))2 +

(f (xn))2(

3(f ′′ (xn))2 − f ′ (xn) f (3) (xn)

)3!(f ′ (xn))

4

and if we omit an3 , we retrieve Householder’s cubic iteration. It’s also possibleto find the expressions for (an4 , a

n5 , a

n6 , a

n7 , . . .), and define quintic, sextic, septic,

octic ... iterations.

9 Generalization

Having indicated some of the difficulties with the Newton-Raphson method, weshow when the method can be applied.

Theorem. Let f : R → R be a continuous function which has precisely onezero, a, in the open interval (u, v). If f ′′ is continuous on (u, v) and f ′(a) 6= 0,then there is a δ > 0 such that if a−δ < x0 < a+δ, then {xn}, then sequence ofsuccessive approximations generated by the Newton-Raphson method, convergesto a.

Proof. Since f has a second derivative on (u, v), f ′ is continuous on (u, v).Since u < a < v and f ′(a) 6= 0:

∃µ > 0, u < a− µ < a+ µ < v, a− µ < x < a+ µ⇒ f ′ (x) 6= 0

The function

g : (a− µ, a+ µ)→ R, x 7→ x− f (x)

f ′ (x)

14

is continuously differentiable because f has a continuous second derivative andf ′(x) 6= 0 on (a− µ, a+ µ).Moreover, g(a) = a, and

g′ (x) = 1− (f ′ (x))2 − f ′′ (x) f (x)

(f ′ (x))2

whence g′(a) = 0. Since g′ is continuous, there is a δ with 0 < δ ≤ µ and|g′ (x)| < 1

2 whenever a− δ < x < a+ δ. Take such an x. Then

|g (x)− a| = |g (x)− g (a)| = g′ (c) |x− a| ≤ 1

2δ < δ

for some aδ < c < a + δ. Thus f (x) ∈ (a− δ, a+ δ), so that we may regard gas a function

(a− δ, a+ δ)→ (a− δ, a+ δ)

Thus a − δ < xn < a + δ for every term xn of the sequence {xn} where x0 ∈(a− δ, a+ δ) and xn+1 = g (xn) ,∀n ∈ N.For x < y ∈ (a− δ, a+ δ)

|g (x)− g (y)| = |g′ (c)| |x− y| ≤ 1

2|x− y|

for some x < c < y.In particular

|xn+2 − xn+1| = |g (xn+1)− g (xn)| ≤ 1

2|xn+1 − xn|

It follows that for all j ∈ N

|xn+j+1 − xn+j | ≤(

1

2

)j|xn+1 − xn|

Hence

|xn+k − xn| ≤k−1∑j=0

|xn+j+1 − xn+j | <k−1∑j=0

(12

)j |xn+1 − xn|

=1−( 1

2 )k

1− 12

|xn+1 − xn| < 2 |xn+1 − xn|< 2(12

)n |x1 − x0| = ( 12)n−1 |g (x0)− x0|

If g(x0) = x0, then f(x0) = 0 and so, by hypothesis, x0 = a.Otherwise |g (x0)− x0| 6= 0, and the above calculation shows that {xn} is aCauchy sequence. Hence, it has a limit b. Then

g (b) = g(

limn→∞

xn

)= limn→∞

g (xn) = limn→∞

xn+1 = b

Since g(b) = b, f(b) = 0 and so, by hypothesis, b = a. �

15

10 Convergence of Newton-Raphson method

Like all fixed point iteration methods, Newton’s method may or may not con-verge in the vicinity of a root. In addition, the convergence of fixed pointiteration methods is guaranteed only if g(x) < 1 in some neighborhood of theroot. Even Newton’s method can not always guarantee that condition. Whenthe condition is satisfied, Newton’s method converges, and it also convergesfaster than almost any other alternative iteration scheme based on other meth-ods of converting the original f(x) to a function with a fixed point.In order to start to get a handle on why Newton’s method is unusually effectivefor a fixed point iteration, we start with a couple of definitions.

Definition. A sequence of fixed-point iterates

pn = g (pn−1)

converges linearly to a limiting value p if there exists a constant 0 < λ < 1 anda positive integer N such that

|pn+1 − p| < λ |pn − p| ,∀n > N

Definition. A sequence of fixed-point iterates

pn = g (pn−1)

converges quadratically to a limiting value p if there exists a constant 0 < λ anda positive integer N such that

|pn+1 − p| < λ|pn − p|2,∀n > N

Both of these definitions state that the distance from pn to p shrinks as weprogress through the sequence. The shrinkage is much more dramatic in thesecond case due to the presence of the square term.The fixed point theorem we saw in the last lecture is sufficient to guaranteelinear convergence provided that certain simple conditions on g(x) are satisfied.Unfortunately, that theorem does not guarantee quadratic convergence. Forthat we need something special.

Theorem. Quadratic Convergence Theorem.Let p be a fixed point of a function g(x). If g′(p) = 0 and g′′(x) is continuouswith |g′′ (x)| < M on an open interval (p−δ, p+δ) any iterated sequence startingfrom a p0 ∈ (p− δ, p+ δ) will converge quadratically to p. Moreover, for largen we will have

|pn+1 − p| <M

2|pn − p|2

16

Proof. Expanding g(x) in a first order Taylor polynomial about x = p gives

g (x) = g (p) + g′ (p) (x− p) +g′′ (ξ)

2(x− p)2

where ξ is some point between x and p. Noting that p is fixed point and thatg′(p) = 0 gives

g (x) = p+g′′ (ξ)

2(x− p)2

Substituting pn for x and rearranging gives

pn+1 − p =g′′ (ξ)

2(pn − p)2

where ξn is a point between p and pn. Since g′(p) = 0 and g′′(x) is continuousnear p we can conclude that |g′ (x)| < 1 for all x in some neighborhood of p.If we choose δ to make the interval (p− δ, p+ δ) fit inside that interval we canuse the original fixed point theorem to conclude that the sequence of pn pointsconverges to p. Since the ξn points are trapped between p and pn they alsoconverges to p. Thus,

g′′ (ξn)

2<M

2

for n large enough. It follows that

|pn+1 − p| <M

2|pn − p|2

for n large enough and the sequence of pn points converges quadratically to p.�

Newton-Raphson method converges quadratically

The Newton-Raphson iteration function

g (x) = x− f (x)

f ′ (x)

satisfies the condition that g′(p) = 0 at the fixed point. In cases when it alsosatisfies the restriction that |g′′ (x)| < M on an open interval (p − δ, p + δ)we have enough to guarantee quadratic convergence of the Newton-Raphsonmethod sequence.

11 Newton’s method for several variables

Newton’s method may also be used to find a root of a system of two or morenon linear equations {

f (x, y) = 0g (x, y) = 0

17

where f and g are C2 functions on a given domain. Using Taylor’s expansionof the two functions near (x, y) we find

f (x+ h, y + k) = f (x, y) + h∂f∂x + k ∂f∂y +O(h2 + k2

)g (x+ h, y + k) = g (x, y) + h ∂g∂x + k ∂g∂y +O

(h2 + k2

)and if we keep only the first order terms, we are looking for a couple (h, k) suchas

f (x+ h, y + k) = 0 ≈ f (x, y) + h∂f∂x + k ∂f∂yg (x+ h, y + k) = 0 ≈ g (x, y) + h ∂g∂x + k ∂g∂y

hence its equivalent to the linear system(∂f∂x

∂f∂y

∂g∂x

∂g∂y

)(hk

)= −

(f (x, y)g (x, y)

)equivalent to

J (x, y)

(hk

)= −

(f (x, y)g (x, y)

)This suggest to define the new process(

xn+1

yn+1

)=

(xnyn

)− J−1 (xn, yn)

(f (xn, yn)g (xn, yn)

)starting with an initial guess (x0, y0) and under certain conditions (which arenot so easy to check and this is again the main disadvantage of the method),it’s possible to show that this process converges to a root of the system. Theconvergence remains quadratic.

Example. We are looking for a root near (x0 = 0.6, y0 = 0.6) of the followingsystem {

f (x, y) = x3 − 3xy2 − 1g (x, y) = 3x2y − y3

here the Jacobian and its inverse become

J (xn, yn) = 3

(x2n − y2n −2xnyn2xnyn x2n − y2n

)J−1 (xn, yn) = 1

3(x2n+y

2n)

2

(x2n − y2n 2xnyn−2xnyn x2n − y2n

)and the process gives

x1 = −0.40000000000000000000, y1 = 0.86296296296296296296x2 = −0.50478978186242263605, y2 = 0.85646430512069295697x3 = −0.49988539803643124722, y3 = 0.86603764032215486664x4 = −0.50000000406150565266, y4 = 0.86602539113638168322x5 = −0.49999999999999983928, y5 = 0.86602540378443871965. . .

18

Depending on your initial guess Newton’s process may converge to one of thethree roots of the system:(

−1

2,

√3

2

),

(−1

2,−√

3

2

), (1, 0)

and for some values of (x0, y0) the convergence of the process may be tricky!The study of the influence of this initial guess leads to aesthetic fractal pictures.

Cubic convergence also exists for several variables system of equations : Cheby-shev’s method.

12 Extension to systems of equations f1 (x1, . . . , xn) = 0. . .

fm (x1, . . . , xn) = 0

orf(x) = 0

The Newton-Raphson method becomes

xn+1 = xn − J−1 (xn) f (xn) , n = 0, 1, . . .

The end

19

References

[1] Adi Ben-Israel, A Newton-Raphson Method for the Solution of Systemsof Equations, Technion-Israel Institute of Technology and NorthwesternUniversity, Journal of Mathematical Analysis and Applications 15, 243-252 (1966).

[2] Helm, Workbook Level 1, The Newton-Raphson Method, March 18, 2004.

[3] Aaron Burton, Newton’s method and fractals.

[4] David M. Bressoud, Newton-Raphson Method, Appendix to A ‘RadicalApproach to Real analysis 2nd edition’, June 20, 2006.

[5] Pascal Sebah, Xavier Gourdon, Newton’s method and high order iterations,October 3, 2001.

[6] Ibrahim A. Assakkaf, Numerical methods for engineers, ComputationMethod in Civil Engineering II, Department of Civil and EnvironmentEngineering, University of Maryland, College Park, Spring 2001.

[7] Autar Kaw, Newton-Raphson method of solving nonlinear equations, De-cember 23, 2009.

20