abe425 engineering measurement systems ordinary least squares (ols) fitting dr. tony e. grift dept....
DESCRIPTION
Examples Multiplication by a constant c (linear)TRANSCRIPT
ABE425 Engineering Measurement Systems
Ordinary Least Squares (OLS) Fitting
Dr. Tony E. Grift
Dept. of Agricultural & Biological EngineeringUniversity of Illinois
First, we need to talk about linearity
In Mathematics, there are linear and non-linear operations:
If an operation is linear, the superposition principle can be applied:
a b
a b
Operation
Operation
1 1
2 2
a a b bOperation1 2 1 2
k a a k b bOperation1 2 1 2
Examples
Multiplication by a constant c (linear)
a a c
a a c
Operation
Operation
1 1
2 2
a a c a c a c a a cOperation1 2 1 2 1 2
Examples
Differentiation (linear)
a a c
a a c
Operation
Operation
1 1
2 2
d f x g x d f x d g xdx dx dx
Examples
Integration (linear)
f x g x dx f x dx g x dx
Examples
Squaring (non-linear)
a a
a a
Operation
Operation
1 12
2 22
a a a a a aOperation1 2
2
12
22
1 2
2
Examples
Square root (non-linear)
a a
a a
Operation
Operation
1 1
2 2
a a a a a aOperation1 2 1 2 1 2
Try these:
sin
ln
xe
x
x
You collected a set of data pairs (for example temperature versus time
>> x=[0:1:10]'>> y = [0.5 0.75 1.25 1.3 2.1 2.0 3.1 3.05 4.0 4.5 5]'
, , 1,i ix y i m
0 1 2 3 4 5 6 7 8 9 100.5
1
1.5
2
2.5
3
3.5
4
4.5
5
Time (s)
Tem
pera
ture
(deg
C)
Model is some function of the independent variable and the parameter vector. You choose the model!!
The error is the difference between a data point and the corresponding model
,iy f x
,i i ie y f x
Error = Data - Model
The idea of using the sum of least squared errors came from Gauss (first published by Legendre):
How can we minimize this error with respect to the parameter vector? In other words, how can we find the parameter vector that minimizes the error and maximizes the fit?
2
1
i m
ii
S e
“Sur la Méthode des moindres quarrés” in Legendre’s Nouvellesméthodes pour la détermination des orbites des comètes, Paris 1805.
The minimum value of the sum of squares is obtained by setting this partial derivative to zero
The derivative is partial, because the Sum of residuals S is a function of the error and the error itself is a function of the parameter vector (remember the chain rule):
* i
j i j
eS Se
2
1 1
2i m i m
i ii ii
SS e ee
,, ii
i i ij j
f xee y f x
S e
The minimum value of the sum of squares is obtained by setting this partial derivative to zero
Substitution of the results from the previous slide gives:
Now, we need to find out what is
1
,2i m
ii
ij j
f xS e
,ij
f x
The proposed model is linear in the parameters. Here is an polynomial example:
For each ith measurement this can be written using a matrix and a parameter vector as follows:
0 1 2 30 1 2 3 ..f x x x x x
This can also be written as:
0
10 1 2 3
2
3
,i i i i if x x x x x
1
,j n
i ij jj
f x A
For all measurement points we obtain:
3 2 1 01
3
2
1
03 2 1 0
| | | |, | | | |
| | | |
i i i i
m m m m m
A
x x x x x
f
x x x x x
This can also be written in vector form as:
Where A is the regressor matrix
,f x A
From the model definition we can obtain the partial derivative with respect to the parameter vector
By replacing the error with the (data – model) we get :
1
,,
j ni
i ij j ijj j
f xf x A A
1 1
,2 2i m i m
ii i ij
i ij j
f xS e e A
1
,j n
i i i i ij jj
e y f x y A
Upon rearrangement these become n simultaneous linear equations, the normal equations.
1 1
2 0i m k n
i ik k iji kj
S y A A
1 1 1
, 1,i m k n i m
ij ik k ij ii k i
A A A y j n
T TA A A y
1ˆ T TA A A y
Here a second order polynomial with intercept was applied
0 1 2 3 4 5 6 7 8 9 100
1
2
3
4
5
6
Time (s)
Tem
pera
ture
(deg
C)
2 1 02 1 0f x x x x
Determine whether the data needs an intercept. Often physical constraints demand that the fit curve passes through the origin!
2 12 1f x x x
0 1 2 3 4 5 6 7 8 9 100
0.5
1
1.5
2
2.5
3
3.5
4
4.5
5
No intercept!
OLS labfunction [theta, msq] = fitols(x,y,Ovec)% Fit polynomial function on data in OLS sense% Author :% Date :% Revision :%% Syntax : [theta,msq] = fitols(x,y,Ovec)%% theta : Parameter vector% msq : Mean square error% x : Independent variable% y : Dependent variable%% Ovec indicates terms [1 x x^2 ..]*Ovec'% Example Ovec = [1 0 1] gives [1 x^2] and not x % If vectors x,y are horizontal, transpose them to make them vertical % Make sure the x and y vector have the same length. If not alert the user% with an error dialog box (type help errordlg ) % Build the matrix of regressors. Check each entry of Ovec, and if it is a% 1, add another column to the regression matrix A. A = []; % Compute the parameter vector theta using the OLS formula % Compute the error vector % Compute the mean square error which indicates how good the fit is % Plot y (Temperature in C) versus x (Current in A). Add labels and title. % Your output should look as shown in the handout.
ABE425 Engineering Measurement Systems
Ordinary Least Squares (OLS) Fitting
The End
Dept. of Agricultural & Biological EngineeringUniversity of Illinois