optimization – lecture 1 review of differential calculus for functions of single...

Optimization – Lecture 1 Review of Differential Calculus for

Functions of Single Variable

http://users.encs.concordia.ca/~luisrod

Copyright © Luis Rodrigues, January 2014

Copyright © Luis Rodrigues

Outline

2

•  Optimization Problems

•  Real Numbers and Real Vectors

•  Open, Closed and Bounded sets

•  Real Functions

•  Limit and Continuity

•  Maximum and Minimum of Functions

•  Derivative, L’Hopital’s Rule and Stationary Points

Optimization Problems •  An optimization problem (OP) is a problem of the form

•  This is a minimization (we can consider a maximization of f as a minimization of –f), f is a function to be minimized, s.t means subject to, is the set of feasible values of variable

•  The obvious question is what kinds of functions, what kinds of sets and what kinds of variables can we handle

•  The variable can be a single real variable or a vector, the function f will be assumed to be of class (twice differentiable), the set will be an open set or a closed set

•  The next slides will explain what these concepts mean 3 Copyright © Luis Rodrigues

min f (x)s.t x ∈Ω

Ω x

xC2

Ω

Integer, Rational, Irrational Numbers •  We assume familiarity with addition +, subtraction -,division /,

multiplication , less <, greater >, less or equal , greater or equal , (approximately ) equal =,different ,equivalent , implies ,or , and , union , intersection , (not) member ( ), for all , there exists one ,

•  The integer numbers are the non-negative integers 0,1,2,… union with the negative integers -1,-2,-3,…

•  A rational number is the ratio of two integers (e.g, ½). There are numbers that cannot be represented as the ratio of two integers and are called irrational numbers. An example is the length of the circumference divided by its diameter that yields the number

•  Real numbers can be either rational or irrational •  Example: The area of a rectangle with sides measuring

2cm and 1cm is centimeters squared

4 Copyright © Luis Rodrigues

b = 2cma =1cm

2cm2

≤≥

π = 3.14159...

≠

A = 2 ⋅1= 2

a ⋅b = 0⇔ a = 0∨b = 0

⇔∨ ∧

∉∈

∀

≈

∃

⋅

⇒

Real Numbers •  The set of real numbers is denoted . In we can define

two closed operations (addition and multiplication) that have inverses (subtraction and division, respectively)

•  This means that two real numbers and can be added to yield a real and multiplied to yield a real or

•  Associative Property: •  Commutative Property: •  Existence of Identity: •  Existence of Inverse: •  Distributive Property: •  A set with two operations verifying these properties is

called a field and is represented by the triad •  All numbers form an open interval denoted (a,b) •  Notation for large products, sums:


ℜℜ

a+ (−a) = 0, b ⋅b−1 =1,b ≠ 0a+ 0 = a, a ⋅1= a

abbaabba .., =+=+

)..()..(),()( cbacbacbacba =++=++

a ⋅ (b+ c) = (a ⋅b)+ (a ⋅c)

ℜ,+, ⋅( )

aN = a ⋅aaN times , an

n=1

N

∑ = a1 + a2 ++ aN

a ⋅ba+ ba b

ab

a < x < b

Vectors and Vector Coordinates


•  A vector is a geometric object represented by a letter with an arrow, e.g., with magnitude and direction

•  A vector has coordinates in a given frame. A vector is not the same as its coordinates. Coordinates of the same vector can be different in different frames.

•  The coordinates of a vector in a frame E will be denoted by and will be represented as a column array of real numbers. The transpose of the coordinates of a vector will be a row array denoted by and the derivative of a vector with respect to t will be denoted by (more later)

•  Example: For a 2D frame E with unit vectors

x

x E x[ ]

[ ]TE x

[ ]htxhtxx h ))()((lim 0 −+= →

x = te1 + t2e2,

E x[ ] =x1x2

!

"##

$

%&&=

tt2

!

"##

$

%&&, E x[ ]T = t t2!

"#$%&,E x!" $%=

12t

!

"#

$

%&

x = x12 + x2

2

x

( )21,ee

x

e1

e2

Open, Closed, Compact Sets •  Let be the set of real numbers and the set of real

vectors with n coordinates •  An open ball in with center and radius is the

set of vectors verifying

•  An open set in is a set for which around each point one can find an open ball that is fully contained in the set. For example an open ball is an open set

•  A closed set is a set for which its complementary which contains all elements in that are not in is open. For example

•  A bounded set is a set that can be contained in an open ball with finite radius. For example

•  A compact set is a set that is closed and bounded 7

ℜ ℜn

ℜn x0 r > 0xBr (x0 ) =

x : x − x0 < r{ }x0 r

ℜn

ℜn \Ωℜn Ωx : x − x0 ≤ r{ }

x : x ≤1{ }


Real Functions •  A real function is a mapping of a real vector into a real

value . If is a scalar, f is a function of a real variable •  For each value of there can only be a value of f but

there could be more than one value of that have the same value of f. Example:

•  When different values of correspond to different values of f the function is injective or 1-to-1. Example:

•  The set of values of for which f exists is called the domain of f. The set of values f is called the image of f

•  When the image of f is equal to then f is called surjective or onto. Example:

•  A function that is both injective and surjective is called bijective. These functions can be inverted ( exists)

•  A function f is linear if Example: 8 Copyright © Luis Rodrigues

f (x) = x2 ⇒ x = ± f (x)

ℜ

f (α x1 +βx2 ) =α f (

x1)+β f (x2 ), ∀α,β ∈ℜ

1)( += xxf

xxf =)(

xxf 2)( =

f −1

f (x)

x

x x

x

x

x

Real Variable Functions: Trigonometric

CC

AA

B

B

ca

b

C A

B

c

b

a

A

Right Triangle

Oblique Triangle

222 bac +=

AabBA

caBA

cbB

baA

cbA

caA

cottan,sincos,cossin

tan,cos,sin

======

===

Pythagorean Theorem:

Law of the sines:

Law of the cosines:

Cc

Bb

Aa

sinsinsin==

Cabbac cos2222 −+=Copyright © Luis Rodrigues

Straight lines can be used to form triangles between three points A, B, C

9

Useful Trigonometric Relations


sin2θ + cos2θ =1sin2θ = 0.5− 0.5cos(2θ )sin(2θ ) = 2sin(θ )cos(θ )cos(2θ ) = cos2(θ )− sin2(θ )tan(2θ ) = 2 tan(θ ) / (1− tan2(θ ))sin(α +β) = sinα cosβ + cosα sinβsin(α −β) = sinα cosβ − cosα sinβcos(α +β) = cosα cosβ − sinα sinβcos(α −β) = cosα cosβ + sinα sinβ

sinα + sinβ = 2sinα +β2

cosα −β2

sinα − sinβ = 2cosα +β2

sinα −β2

cosα + cosβ = 2cosα +β2

cosα −β2

cosα − cosβ = −2sinα +β2

sinα −β2

Polynomials of a Real Variable •  A polynomial function of order n is a function of the form

•  If x is a real variable then we say that •  When n=1 then f is a polynomial of first order

We say that f is affine if and linear if •  If n=2 then f is second order. We say that f is quadratic •  Examples of polynomials of first and second order:


f (x) =α0 +α1x +α2x2 ++αnx

n,αi ∈ℜ

][xf ℜ∈

00 ≠α 00 =α

f (x) = x +1f (x) = 2xf (x) = (x +1)2 = x2 + 2x +1

Exponential & Logarithm Functions •  The Euler’s number e is approximately equal to 2.71828

and is represented by the infinite sum

•  The exponential function is defined by •  The logarithm function is the inverse of the exponential

function, is defined by and verifies the following important properties


e =1+11+11⋅2

+1

1⋅2 ⋅3+…

f (x) = ex

f (x) = ln(x)

ln(ex ) = x, eln(x ) = xln(xy) = ln(x)+ ln(y)ln(xa ) = a ln(x), a ∈ℜ

Inverse Function Property:

Limit of a Function and Continuity

13 Copyright © Luis Rodrigues x∞x1x 2x …

)(xf)( 1xf)( 2xf

)( ∞xf

•  Suppose we have a sequence of numbers getting closer and closer to some value. For example

•  The values get closer and closer to 1 as n increases. We say that converges and its limit is 1,i.e

•  The difference also gets smaller for any p as n increases and we say that is a Cauchy sequence

•  If f is a function of we sometimes want to determine if f also converges to a value when converges to a limit

•  This value, if it exists, is called . If then we say that f is continuous at . If f is continuous

at all points we say that f is a continuous function Example:

limx→x∞f (x)

,2,1, =nxn

,999.0,99.0,9.0 321 === xxx

1lim =∞→

nn

x

x

x∞

x∞

0)()(lim,sin)(

==

==

→

∞

π

π

πfxfxxxf

x

xn

xn+p − xnxn

xn

limx→x∞f (x) = f (x∞)

The Dirac Delta Function •  As an example of a function defined by a limit we will now

talk about the impulse function

•  An impulse function, also called Dirac delta function , is the mathematical model of an impulsive force like, for example, the one exerted in a billard ball: strong force in a short period of time

•  The function can be defined as the result of a limiting process as follows:

)(tδ

)(tδ

⎪⎩

⎪⎨⎧ Δ≤≤Δ== ΔΔ

→Δ otherwise

tifttt,0

0,1)(,)(lim)(0

δδδ

t

)(tΔδ

Δ

1

Δ

Area=1


Maximum and Minimum of a Function •  A value is called a local minimum (maximum) if ( ) •  If the result is true in all the domain of then it is a

global minimum (maximum)

•  Weierstrass Theorem: A continuous function defined on a compact set has a maximum and a minimum

15

f (x0 )f (x) ≤ f (x0 )∃δ > 0 : f (x) ≥ f (x0 ) ∀

x ≠ x0, x ∈ Bδ (x0 )D

xmaxxmin x

)(xf


fD

Derivative of a Scalar Function •  For a function f(x) represented by its graph one can

imagine connecting two points P, Q on the graph by a line. This line is called a secant.

•  If we now make converge to a value such that Q converges to P then the secant converges to the tangent line at P. The derivative f’ of f at is the slope of the tangent and is defined by the limit in the box. A function that has a derivative at all points is called differentiable. The second derivative f’’ is the derivative of the derivative

Example:


P

Q Q’

x

)(xf

1x2x0x

x 0x

0x

hxfhxf

xxxfxfxf

dxdf

hxxx

)()(lim)()(lim)(' 00

00

00

00

−+=

−

−==

→→

2)1)1((1)1(lim

1,1)(22

01

02

=+−++

=

=+=

→ hh

dxdf

xxxf

h

Useful Derivative Relations


f (x) = xα, f '(x) =αxα−1

f (x) = sin(x), f '(x) = cos(x)f (x) = cos(x), f '(x) = −sin(x)f (x) = tan(x), f '(x) =1/ cos2(x)f (x) = cos(x)sin(x), f '(x) = cos(2x)f (x) = ex, f '(x) = ex

f (x) = ln(x), f '(x) =1/ x( f (x)+ g(x))' = f '(x)+ g '(x)( f (x)g(x))' = f '(x)g(x)+ f (x)g '(x)( f (x) / g(x))' = ( f '(x)g(x)− f (x)g '(x)) / g2 (x)

•  Theorem (Chain Rule): If the derivatives g’(x) and f’(g(x)) exist and F(x)=f(g(x)) then

•  Example: )2sin()cos()sin(2)(',1sin))(()(

sin)(,1)(2

2

xxxxFxxgfxFxxgxxf

==+==

=+=

)('))((')(' xgxgfxF =

L’Hopital’s Rule •  As we saw a derivative is the limit of a ratio •  Sometimes when computing the limit of a ratio of two

functions one can have an indeterminate result 0/0 •  For these cases there exists a result called L’Hopital’s

rule stated in the following theorem •  Theorem: Let f and g be two functions that have a

derivative at each point of an open interval (a,b). For a point in this interval let

If exists then

•  Example:


)('),(' xgxf0)(lim)(lim

00

==→→

xgxfxxxx0x

)(')('lim

0 xgxf

xx→ )(')('lim

)()(lim

00 xgxf

xgxf

xxxx →→=

111)cos(lim,

00)sin(lim)0sin()0sin(lim

0,sin)(

0000

0

0

=⇒===−+

=

==

→→→ dxdfh

hh

hh

dxdf

xxxf

hhh

Stationary Points of Functions •  The derivative of a function at a point is the slope of the

tangent to the graph of the function at that point. If the derivative is zero then the slope of the tangent is zero

•  Points for which the derivative is zero are thus called stationary points of a function. They can be a maximum or a minimum (see figure below)


x

)(xf

xmaxxmin

f '(xmin ) = f '(xmax ) = 0Necessary Condition

optimization – lecture 1 review of differential calculus for functions of single...

Documents