higher derivatives in matlab using mad euroad workshop... · requests for higher derivatives of...
TRANSCRIPT
![Page 1: Higher Derivatives in Matlab Using MAD EuroAd Workshop... · Requests for higher derivatives of order 2 to 4 for uncertainty analysis and robust optimisation. Could implement a Taylor](https://reader034.vdocuments.mx/reader034/viewer/2022042621/5f7033f4c00904629a3f93e5/html5/thumbnails/1.jpg)
Higher Derivatives in Matlab Using MAD
Shaun Forth
Engineering Systems DepartmentCranfield University (DCMT Shrivenham)
Shrivenham, Swindon SN6 8LA, U.K.email: [email protected]
www.amorg.co.uk/AD/staff_SAF.html
5th European Workshop on Automatic DifferentiationUniversity of Hertfordshire, UK, May 21-22 2007
1/ 29 Higher Derivatives in Matlab Using MAD
![Page 2: Higher Derivatives in Matlab Using MAD EuroAd Workshop... · Requests for higher derivatives of order 2 to 4 for uncertainty analysis and robust optimisation. Could implement a Taylor](https://reader034.vdocuments.mx/reader034/viewer/2022042621/5f7033f4c00904629a3f93e5/html5/thumbnails/2.jpg)
Plan
1 Introduction
2 MAD’s Forward Modefmad classderivvec class
3 Higher DerivativesSecond DerivativesHow The Heck Does This Work?Required Changes to MAD
4 Reverse Mode AD in Matlab
5 ResultsMinpack Elastic-Plastic Torsion (EPT) ProblemObject-Oriented Test Case
6 Conclusions & Future Work
2/ 29 Higher Derivatives in Matlab Using MAD
![Page 3: Higher Derivatives in Matlab Using MAD EuroAd Workshop... · Requests for higher derivatives of order 2 to 4 for uncertainty analysis and robust optimisation. Could implement a Taylor](https://reader034.vdocuments.mx/reader034/viewer/2022042621/5f7033f4c00904629a3f93e5/html5/thumbnails/3.jpg)
Introduction
MAD’s overloaded forward mode is efficient for firstderivatives [For06].
Requests for higher derivatives of order 2 to 4 for uncertainty analysisand robust optimisation.
Could implement a Taylor series class a la ADOL-C [GJU96] -manpower expensive.
OR use MAD to recursively differentiate itself (c.f. EuroAD2 - BarakPerlmutter and Jeffrey Siskind, [Gri00, p109]).
3/ 29 Higher Derivatives in Matlab Using MAD
![Page 4: Higher Derivatives in Matlab Using MAD EuroAd Workshop... · Requests for higher derivatives of order 2 to 4 for uncertainty analysis and robust optimisation. Could implement a Taylor](https://reader034.vdocuments.mx/reader034/viewer/2022042621/5f7033f4c00904629a3f93e5/html5/thumbnails/4.jpg)
MAD’s Forward Mode
Based on two classes:
fmad class - forward mode AD by operator overloading
derivvec class - for storage and combination of multiple directionalderivatives.
4/ 29 Higher Derivatives in Matlab Using MAD
![Page 5: Higher Derivatives in Matlab Using MAD EuroAd Workshop... · Requests for higher derivatives of order 2 to 4 for uncertainty analysis and robust optimisation. Could implement a Taylor](https://reader034.vdocuments.mx/reader034/viewer/2022042621/5f7033f4c00904629a3f93e5/html5/thumbnails/5.jpg)
fmad Class Constructor
e.g. x=fmad([1.1 2 3],[4 5 6]);
Defines fmad object withI value component - row vector [1.1 2 3]I deriv component - single directional derivative [4 5 6]
Perform overloaded operations, e.g., element-wise multiplication viatimes
z=x.*xvalue = 1.2100 4.0000 9.0000derivatives = 8.8000 20.0000 36.0000
AD enabled by the times.m function of the fmad class.
5/ 29 Higher Derivatives in Matlab Using MAD
![Page 6: Higher Derivatives in Matlab Using MAD EuroAd Workshop... · Requests for higher derivatives of order 2 to 4 for uncertainty analysis and robust optimisation. Could implement a Taylor](https://reader034.vdocuments.mx/reader034/viewer/2022042621/5f7033f4c00904629a3f93e5/html5/thumbnails/6.jpg)
fmad Class Constructor
e.g. x=fmad([1.1 2 3],[4 5 6]);
Defines fmad object withI value component - row vector [1.1 2 3]I deriv component - single directional derivative [4 5 6]
Perform overloaded operations, e.g., element-wise multiplication viatimes
z=x.*xvalue = 1.2100 4.0000 9.0000derivatives = 8.8000 20.0000 36.0000
AD enabled by the times.m function of the fmad class.
5/ 29 Higher Derivatives in Matlab Using MAD
![Page 7: Higher Derivatives in Matlab Using MAD EuroAd Workshop... · Requests for higher derivatives of order 2 to 4 for uncertainty analysis and robust optimisation. Could implement a Taylor](https://reader034.vdocuments.mx/reader034/viewer/2022042621/5f7033f4c00904629a3f93e5/html5/thumbnails/7.jpg)
fmad Class Constructor
e.g. x=fmad([1.1 2 3],[4 5 6]);
Defines fmad object withI value component - row vector [1.1 2 3]I deriv component - single directional derivative [4 5 6]
Perform overloaded operations, e.g., element-wise multiplication viatimes
z=x.*xvalue = 1.2100 4.0000 9.0000derivatives = 8.8000 20.0000 36.0000
AD enabled by the times.m function of the fmad class.
5/ 29 Higher Derivatives in Matlab Using MAD
![Page 8: Higher Derivatives in Matlab Using MAD EuroAd Workshop... · Requests for higher derivatives of order 2 to 4 for uncertainty analysis and robust optimisation. Could implement a Taylor](https://reader034.vdocuments.mx/reader034/viewer/2022042621/5f7033f4c00904629a3f93e5/html5/thumbnails/8.jpg)
fmad times function for z=x.*y
function z=times(x,y)if isa(x,’fmad’)
z=x; % deep copy to avoid constructor, no refs. to xif isa(y,’fmad’)
z.deriv=y.value.*z.deriv+z.value.*y.deriv;z.value=z.value.*y.value;
elsez.value=z.value.*y;z.deriv=y.*z.deriv;
endelse
z=y; % deep copy to avoid constructor, no refs. to yz.value=x.*z.value;z.deriv=x.*z.deriv;
end
6/ 29 Higher Derivatives in Matlab Using MAD
![Page 9: Higher Derivatives in Matlab Using MAD EuroAd Workshop... · Requests for higher derivatives of order 2 to 4 for uncertainty analysis and robust optimisation. Could implement a Taylor](https://reader034.vdocuments.mx/reader034/viewer/2022042621/5f7033f4c00904629a3f93e5/html5/thumbnails/9.jpg)
Working with Multiple Directional Derivatives
What if we want the Jacobian?
Seed derivatives with identity I3
x=fmad([1.1 2 3],eye(3));
Overloaded operation with same times function gives
value = 1.2100 4.0000 9.0000DerivativesSize = 1 3No. of derivs = 3derivs(:,:,1) = 2.2000 0 0derivs(:,:,2) = 0 4 0derivs(:,:,3) = 0 0 6
7/ 29 Higher Derivatives in Matlab Using MAD
![Page 10: Higher Derivatives in Matlab Using MAD EuroAd Workshop... · Requests for higher derivatives of order 2 to 4 for uncertainty analysis and robust optimisation. Could implement a Taylor](https://reader034.vdocuments.mx/reader034/viewer/2022042621/5f7033f4c00904629a3f93e5/html5/thumbnails/10.jpg)
Working with Multiple Directional Derivatives
What if we want the Jacobian?
Seed derivatives with identity I3
x=fmad([1.1 2 3],eye(3));
Overloaded operation with same times function gives
value = 1.2100 4.0000 9.0000DerivativesSize = 1 3No. of derivs = 3derivs(:,:,1) = 2.2000 0 0derivs(:,:,2) = 0 4 0derivs(:,:,3) = 0 0 6
7/ 29 Higher Derivatives in Matlab Using MAD
![Page 11: Higher Derivatives in Matlab Using MAD EuroAd Workshop... · Requests for higher derivatives of order 2 to 4 for uncertainty analysis and robust optimisation. Could implement a Taylor](https://reader034.vdocuments.mx/reader034/viewer/2022042621/5f7033f4c00904629a3f93e5/html5/thumbnails/11.jpg)
Working with Multiple Directional Derivatives
What if we want the Jacobian?
Seed derivatives with identity I3
x=fmad([1.1 2 3],eye(3));
Overloaded operation with same times function gives
value = 1.2100 4.0000 9.0000DerivativesSize = 1 3No. of derivs = 3derivs(:,:,1) = 2.2000 0 0derivs(:,:,2) = 0 4 0derivs(:,:,3) = 0 0 6
7/ 29 Higher Derivatives in Matlab Using MAD
![Page 12: Higher Derivatives in Matlab Using MAD EuroAd Workshop... · Requests for higher derivatives of order 2 to 4 for uncertainty analysis and robust optimisation. Could implement a Taylor](https://reader034.vdocuments.mx/reader034/viewer/2022042621/5f7033f4c00904629a3f93e5/html5/thumbnails/12.jpg)
The fmad and derivvec classes
The fmad constructor function
function xad=fmad(x,dx)% FUNCTION: FMAD% SYNOPSIS: Class constructor for forward Matlab AD objectsxad.value=x;sx=size(xad.value);sd=size(dx);
if prod(sx)==prod(sd)% #values=#derivs => single directional derivativexad.deriv=reshape(dx,sx);
else% #values~=#derivs => multiple directional derivatives% Pass derivatives to derivvec constructorxad.deriv=derivvec(dx,size(xad.value));
end
8/ 29 Higher Derivatives in Matlab Using MAD
![Page 13: Higher Derivatives in Matlab Using MAD EuroAd Workshop... · Requests for higher derivatives of order 2 to 4 for uncertainty analysis and robust optimisation. Could implement a Taylor](https://reader034.vdocuments.mx/reader034/viewer/2022042621/5f7033f4c00904629a3f93e5/html5/thumbnails/13.jpg)
The fmad and derivvec classes
The fmad constructor function
function xad=fmad(x,dx)% FUNCTION: FMAD% SYNOPSIS: Class constructor for forward Matlab AD objectsxad.value=x;sx=size(xad.value);sd=size(dx);
if prod(sx)==prod(sd)% #values=#derivs => single directional derivativexad.deriv=reshape(dx,sx);
else% #values~=#derivs => multiple directional derivatives% Pass derivatives to derivvec constructorxad.deriv=derivvec(dx,size(xad.value));
end
8/ 29 Higher Derivatives in Matlab Using MAD
![Page 14: Higher Derivatives in Matlab Using MAD EuroAd Workshop... · Requests for higher derivatives of order 2 to 4 for uncertainty analysis and robust optimisation. Could implement a Taylor](https://reader034.vdocuments.mx/reader034/viewer/2022042621/5f7033f4c00904629a3f93e5/html5/thumbnails/14.jpg)
The derivvec class
Store derivatives as a matrix with each directional derivative unrolledinto a column.
e.g. derivvec(eye(3),[1 3]) derivatives conceptualised as as,[direc 1 direc 2 direc 3[1, 0, 0] [0, 1, 0] [0, 0, 1]
]But stored as,
direc 1 direc 2 direc 3 100
010
001
=
1 0 00 1 00 0 1
9/ 29 Higher Derivatives in Matlab Using MAD
![Page 15: Higher Derivatives in Matlab Using MAD EuroAd Workshop... · Requests for higher derivatives of order 2 to 4 for uncertainty analysis and robust optimisation. Could implement a Taylor](https://reader034.vdocuments.mx/reader034/viewer/2022042621/5f7033f4c00904629a3f93e5/html5/thumbnails/15.jpg)
The derivvec class
Store derivatives as a matrix with each directional derivative unrolledinto a column.
e.g. derivvec(eye(3),[1 3]) derivatives conceptualised as as,[direc 1 direc 2 direc 3[1, 0, 0] [0, 1, 0] [0, 0, 1]
]
But stored as,direc 1 direc 2 direc 3 1
00
010
001
=
1 0 00 1 00 0 1
9/ 29 Higher Derivatives in Matlab Using MAD
![Page 16: Higher Derivatives in Matlab Using MAD EuroAd Workshop... · Requests for higher derivatives of order 2 to 4 for uncertainty analysis and robust optimisation. Could implement a Taylor](https://reader034.vdocuments.mx/reader034/viewer/2022042621/5f7033f4c00904629a3f93e5/html5/thumbnails/16.jpg)
The derivvec class
Store derivatives as a matrix with each directional derivative unrolledinto a column.
e.g. derivvec(eye(3),[1 3]) derivatives conceptualised as as,[direc 1 direc 2 direc 3[1, 0, 0] [0, 1, 0] [0, 0, 1]
]But stored as,
direc 1 direc 2 direc 3 100
010
001
=
1 0 00 1 00 0 1
9/ 29 Higher Derivatives in Matlab Using MAD
![Page 17: Higher Derivatives in Matlab Using MAD EuroAd Workshop... · Requests for higher derivatives of order 2 to 4 for uncertainty analysis and robust optimisation. Could implement a Taylor](https://reader034.vdocuments.mx/reader034/viewer/2022042621/5f7033f4c00904629a3f93e5/html5/thumbnails/17.jpg)
The times operation of the derivvec class
e.g. Need to calculate,
[1.1 2 3
]. ∗
direc 1 direc 2 direc 3 1
00
010
001
with multiplication of each of the 3 directional derivatives.
Convert value to column matrix and replicate 3 times 1.1 1.1 1.12 2 23 3 3
. ∗
1 0 00 1 00 0 1
=
1.1 0 00 2 00 0 3
columns give required directional derivatives.
10/ 29 Higher Derivatives in Matlab Using MAD
![Page 18: Higher Derivatives in Matlab Using MAD EuroAd Workshop... · Requests for higher derivatives of order 2 to 4 for uncertainty analysis and robust optimisation. Could implement a Taylor](https://reader034.vdocuments.mx/reader034/viewer/2022042621/5f7033f4c00904629a3f93e5/html5/thumbnails/18.jpg)
Accessor Functions
Getting the value
getvalue(z)ans = 1.2100 4.0000 9.0000
Getting external representation of derivatives
getderivs(z)ans(:,:,1) = 2.2000 0 0ans(:,:,2) = 0 4 0ans(:,:,3) = 0 0 6
Getting unrolled internal representation
getinternalderivs(z)ans = 2.2000 0 0
0 4 00 0 6
11/ 29 Higher Derivatives in Matlab Using MAD
![Page 19: Higher Derivatives in Matlab Using MAD EuroAd Workshop... · Requests for higher derivatives of order 2 to 4 for uncertainty analysis and robust optimisation. Could implement a Taylor](https://reader034.vdocuments.mx/reader034/viewer/2022042621/5f7033f4c00904629a3f93e5/html5/thumbnails/19.jpg)
Higher Derivatives
Basic idea is to be able to use MAD recursively.
For first derivatives:
x=fmad([1.1 2 3],eye(3));z=x.*xzvalue=getvalue(z)
zvalue = 1.2100 4.0000 9.0000zderivs=getinternalderivs(z)
zderivs = 2.2000 0 00 4.0000 00 0 6.0000
Can we just differentiate the above process a second time with MAD?
12/ 29 Higher Derivatives in Matlab Using MAD
![Page 20: Higher Derivatives in Matlab Using MAD EuroAd Workshop... · Requests for higher derivatives of order 2 to 4 for uncertainty analysis and robust optimisation. Could implement a Taylor](https://reader034.vdocuments.mx/reader034/viewer/2022042621/5f7033f4c00904629a3f93e5/html5/thumbnails/20.jpg)
Second Derivatives
xx=fmad([1.1 2 3],eye(3)); % xx’s value and derivs. D1=I_3x=fmad(xx,eye(3)); % x’s value =xx, x’s derivs. D2=I_3z=x.*x;zvalue=getvalue(getvalue(z)) % z’s value’s value
zvalue = 1.2100 4.0000 9.0000zderivs=getinternalderivs(getvalue(z)) % z’s value’s derivs
% in D1 direcs.zderivs = 2.2000 0 0
0 4.0000 00 0 6.0000
zderivs=getvalue(getinternalderivs(z)) % value of z’s derivs% in D2 direc.
zderivs = 2.2000 0 00 4.0000 00 0 6.0000
13/ 29 Higher Derivatives in Matlab Using MAD
![Page 21: Higher Derivatives in Matlab Using MAD EuroAd Workshop... · Requests for higher derivatives of order 2 to 4 for uncertainty analysis and robust optimisation. Could implement a Taylor](https://reader034.vdocuments.mx/reader034/viewer/2022042621/5f7033f4c00904629a3f93e5/html5/thumbnails/21.jpg)
Second Derivatives (ctd)
z2ndderivs=reshape(...getinternalderivs(getinternalderivs(z)),[3 3 3])% derivs. in D1 direc. of z’s derivs. in D2 direc.z2ndderivs(:,:,1) = 2 0 0
0 0 00 0 0
z2ndderivs(:,:,2) = 0 0 00 2 00 0 0
z2ndderivs(:,:,3) = 0 0 00 0 00 0 2
14/ 29 Higher Derivatives in Matlab Using MAD
![Page 22: Higher Derivatives in Matlab Using MAD EuroAd Workshop... · Requests for higher derivatives of order 2 to 4 for uncertainty analysis and robust optimisation. Could implement a Taylor](https://reader034.vdocuments.mx/reader034/viewer/2022042621/5f7033f4c00904629a3f93e5/html5/thumbnails/22.jpg)
Why The Heck Does This Work?
xx=fmad([1.1 2 3],eye(3)); creates object,
xx
[1.1 2 3] eye(3)
value deriv
x=fmad(xx,eye(3)); creates object,
x
eye(3)xx
[1.1 2 3] eye(3)
value deriv
value deriv
15/ 29 Higher Derivatives in Matlab Using MAD
![Page 23: Higher Derivatives in Matlab Using MAD EuroAd Workshop... · Requests for higher derivatives of order 2 to 4 for uncertainty analysis and robust optimisation. Could implement a Taylor](https://reader034.vdocuments.mx/reader034/viewer/2022042621/5f7033f4c00904629a3f93e5/html5/thumbnails/23.jpg)
How The Heck (ctd.)
z=x.*x; forms object,z
2.*x.value.*x.derivx.value.*x.value
value deriv
xx
[1.1 2 3] eye(3)
value deriv.*
xx
[1.1 2 3] eye(3)
value deriv[1.21, 4.0, 9.0]
2.2 0 00 4 00 0 6
value
deriv
xx
[1.1 2 3] eye(3)
value deriv.* 2 eye(3)
xx
1.1 1.1 1.12 2 23 3 3
[I3, I3, I3]
value deriv.* 2 I32.2 0 0
0 4 00 0 6
[I3, I3, I3] .* 2
e1
e2
e3
value
deriv2.2 0 00 4 00 0 6
I3I3I3
.* 2
e1 e1 e1
e2 e2 e2
e3 e3 e3
value deriv2.2 0 0
0 4 00 0 6
e1 0 00 e2 00 0 e3
value deriv
deriv
16/ 29 Higher Derivatives in Matlab Using MAD
![Page 24: Higher Derivatives in Matlab Using MAD EuroAd Workshop... · Requests for higher derivatives of order 2 to 4 for uncertainty analysis and robust optimisation. Could implement a Taylor](https://reader034.vdocuments.mx/reader034/viewer/2022042621/5f7033f4c00904629a3f93e5/html5/thumbnails/24.jpg)
How The Heck (ctd.)
z=x.*x; forms object,z
2.*x.value.*x.deriv
x.value.*x.value
value deriv
xx
[1.1 2 3] eye(3)
value deriv.*
xx
[1.1 2 3] eye(3)
value deriv
[1.21, 4.0, 9.0]
2.2 0 00 4 00 0 6
value
deriv
xx
[1.1 2 3] eye(3)
value deriv.* 2 eye(3)
xx
1.1 1.1 1.12 2 23 3 3
[I3, I3, I3]
value deriv.* 2 I32.2 0 0
0 4 00 0 6
[I3, I3, I3] .* 2
e1
e2
e3
value
deriv2.2 0 00 4 00 0 6
I3I3I3
.* 2
e1 e1 e1
e2 e2 e2
e3 e3 e3
value deriv2.2 0 0
0 4 00 0 6
e1 0 00 e2 00 0 e3
value deriv
deriv
16/ 29 Higher Derivatives in Matlab Using MAD
![Page 25: Higher Derivatives in Matlab Using MAD EuroAd Workshop... · Requests for higher derivatives of order 2 to 4 for uncertainty analysis and robust optimisation. Could implement a Taylor](https://reader034.vdocuments.mx/reader034/viewer/2022042621/5f7033f4c00904629a3f93e5/html5/thumbnails/25.jpg)
How The Heck (ctd.)
z=x.*x; forms object,z
2.*x.value.*x.deriv
x.value.*x.value
value deriv
xx
[1.1 2 3] eye(3)
value deriv.*
xx
[1.1 2 3] eye(3)
value deriv
[1.21, 4.0, 9.0]
2.2 0 00 4 00 0 6
value
deriv
xx
[1.1 2 3] eye(3)
value deriv.* 2 eye(3)
xx
1.1 1.1 1.12 2 23 3 3
[I3, I3, I3]
value deriv.* 2 I32.2 0 0
0 4 00 0 6
[I3, I3, I3] .* 2
e1
e2
e3
value
deriv2.2 0 00 4 00 0 6
I3I3I3
.* 2
e1 e1 e1
e2 e2 e2
e3 e3 e3
value deriv2.2 0 0
0 4 00 0 6
e1 0 00 e2 00 0 e3
value deriv
deriv
16/ 29 Higher Derivatives in Matlab Using MAD
![Page 26: Higher Derivatives in Matlab Using MAD EuroAd Workshop... · Requests for higher derivatives of order 2 to 4 for uncertainty analysis and robust optimisation. Could implement a Taylor](https://reader034.vdocuments.mx/reader034/viewer/2022042621/5f7033f4c00904629a3f93e5/html5/thumbnails/26.jpg)
How The Heck (ctd.)
z=x.*x; forms object,z
2.*x.value.*x.derivx.value.*x.value
value deriv
xx
[1.1 2 3] eye(3)
value deriv.*
xx
[1.1 2 3] eye(3)
value deriv
[1.21, 4.0, 9.0]
2.2 0 00 4 00 0 6
value
deriv
xx
[1.1 2 3] eye(3)
value deriv.* 2 eye(3)
xx
1.1 1.1 1.12 2 23 3 3
[I3, I3, I3]
value deriv.* 2 I32.2 0 0
0 4 00 0 6
[I3, I3, I3] .* 2
e1
e2
e3
value
deriv2.2 0 00 4 00 0 6
I3I3I3
.* 2
e1 e1 e1
e2 e2 e2
e3 e3 e3
value deriv2.2 0 0
0 4 00 0 6
e1 0 00 e2 00 0 e3
value deriv
deriv
16/ 29 Higher Derivatives in Matlab Using MAD
![Page 27: Higher Derivatives in Matlab Using MAD EuroAd Workshop... · Requests for higher derivatives of order 2 to 4 for uncertainty analysis and robust optimisation. Could implement a Taylor](https://reader034.vdocuments.mx/reader034/viewer/2022042621/5f7033f4c00904629a3f93e5/html5/thumbnails/27.jpg)
How The Heck (ctd.)
z=x.*x; forms object,z
2.*x.value.*x.derivx.value.*x.value
value deriv
xx
[1.1 2 3] eye(3)
value deriv.*
xx
[1.1 2 3] eye(3)
value deriv
[1.21, 4.0, 9.0]
2.2 0 00 4 00 0 6
value
deriv
xx
[1.1 2 3] eye(3)
value deriv.* 2 eye(3)
xx
1.1 1.1 1.12 2 23 3 3
[I3, I3, I3]
value deriv.* 2 I3
2.2 0 00 4 00 0 6
[I3, I3, I3] .* 2
e1
e2
e3
value
deriv2.2 0 00 4 00 0 6
I3I3I3
.* 2
e1 e1 e1
e2 e2 e2
e3 e3 e3
value deriv2.2 0 0
0 4 00 0 6
e1 0 00 e2 00 0 e3
value deriv
deriv
16/ 29 Higher Derivatives in Matlab Using MAD
![Page 28: Higher Derivatives in Matlab Using MAD EuroAd Workshop... · Requests for higher derivatives of order 2 to 4 for uncertainty analysis and robust optimisation. Could implement a Taylor](https://reader034.vdocuments.mx/reader034/viewer/2022042621/5f7033f4c00904629a3f93e5/html5/thumbnails/28.jpg)
How The Heck (ctd.)
z=x.*x; forms object,z
2.*x.value.*x.derivx.value.*x.value
value deriv
xx
[1.1 2 3] eye(3)
value deriv.*
xx
[1.1 2 3] eye(3)
value deriv
[1.21, 4.0, 9.0]
2.2 0 00 4 00 0 6
value
deriv
xx
[1.1 2 3] eye(3)
value deriv.* 2 eye(3)
xx
1.1 1.1 1.12 2 23 3 3
[I3, I3, I3]
value deriv.* 2 I3
2.2 0 00 4 00 0 6
[I3, I3, I3] .* 2
e1
e2
e3
value
deriv
2.2 0 00 4 00 0 6
I3I3I3
.* 2
e1 e1 e1
e2 e2 e2
e3 e3 e3
value deriv2.2 0 0
0 4 00 0 6
e1 0 00 e2 00 0 e3
value deriv
deriv
16/ 29 Higher Derivatives in Matlab Using MAD
![Page 29: Higher Derivatives in Matlab Using MAD EuroAd Workshop... · Requests for higher derivatives of order 2 to 4 for uncertainty analysis and robust optimisation. Could implement a Taylor](https://reader034.vdocuments.mx/reader034/viewer/2022042621/5f7033f4c00904629a3f93e5/html5/thumbnails/29.jpg)
How The Heck (ctd.)
z=x.*x; forms object,z
2.*x.value.*x.derivx.value.*x.value
value deriv
xx
[1.1 2 3] eye(3)
value deriv.*
xx
[1.1 2 3] eye(3)
value deriv
[1.21, 4.0, 9.0]
2.2 0 00 4 00 0 6
value
deriv
xx
[1.1 2 3] eye(3)
value deriv.* 2 eye(3)
xx
1.1 1.1 1.12 2 23 3 3
[I3, I3, I3]
value deriv.* 2 I32.2 0 0
0 4 00 0 6
[I3, I3, I3] .* 2
e1
e2
e3
value
deriv
2.2 0 00 4 00 0 6
I3I3I3
.* 2
e1 e1 e1
e2 e2 e2
e3 e3 e3
value deriv
2.2 0 00 4 00 0 6
e1 0 00 e2 00 0 e3
value deriv
deriv
16/ 29 Higher Derivatives in Matlab Using MAD
![Page 30: Higher Derivatives in Matlab Using MAD EuroAd Workshop... · Requests for higher derivatives of order 2 to 4 for uncertainty analysis and robust optimisation. Could implement a Taylor](https://reader034.vdocuments.mx/reader034/viewer/2022042621/5f7033f4c00904629a3f93e5/html5/thumbnails/30.jpg)
How The Heck (ctd.)
z=x.*x; forms object,z
2.*x.value.*x.derivx.value.*x.value
value
deriv
xx
[1.1 2 3] eye(3)
value deriv.*
xx
[1.1 2 3] eye(3)
value deriv
[1.21, 4.0, 9.0]
2.2 0 00 4 00 0 6
value
deriv
xx
[1.1 2 3] eye(3)
value deriv.* 2 eye(3)
xx
1.1 1.1 1.12 2 23 3 3
[I3, I3, I3]
value deriv.* 2 I32.2 0 0
0 4 00 0 6
[I3, I3, I3] .* 2
e1
e2
e3
value
deriv2.2 0 00 4 00 0 6
I3I3I3
.* 2
e1 e1 e1
e2 e2 e2
e3 e3 e3
value deriv
2.2 0 00 4 00 0 6
e1 0 00 e2 00 0 e3
value deriv
deriv
16/ 29 Higher Derivatives in Matlab Using MAD
![Page 31: Higher Derivatives in Matlab Using MAD EuroAd Workshop... · Requests for higher derivatives of order 2 to 4 for uncertainty analysis and robust optimisation. Could implement a Taylor](https://reader034.vdocuments.mx/reader034/viewer/2022042621/5f7033f4c00904629a3f93e5/html5/thumbnails/31.jpg)
Required Changes to MAD
Few changes needed
Made derivvec class superiorto(’fmad’).
Subscript Referencing - make via direct calls to fmad’s subsreffunction.
For efficiency - eliminate unnecessary temporary fmad objects created.
17/ 29 Higher Derivatives in Matlab Using MAD
![Page 32: Higher Derivatives in Matlab Using MAD EuroAd Workshop... · Requests for higher derivatives of order 2 to 4 for uncertainty analysis and robust optimisation. Could implement a Taylor](https://reader034.vdocuments.mx/reader034/viewer/2022042621/5f7033f4c00904629a3f93e5/html5/thumbnails/32.jpg)
Made derivvec class superiorto(fmad)
derivvec Constructor Function
function dv=derivvec(x,shape,nd)..otherwise
error(’derivvec must have 1 or 2 inputs’)endsuperiorto(’fmad’)
Required for binary operations between fmad and derivvec objects.
e.g. xfmad.*yderivvec
Ensures object precedence is such that derivvec operation calledand not the fmad operation.
Enables differentiation through the derivvec class.
18/ 29 Higher Derivatives in Matlab Using MAD
![Page 33: Higher Derivatives in Matlab Using MAD EuroAd Workshop... · Requests for higher derivatives of order 2 to 4 for uncertainty analysis and robust optimisation. Could implement a Taylor](https://reader034.vdocuments.mx/reader034/viewer/2022042621/5f7033f4c00904629a3f93e5/html5/thumbnails/33.jpg)
Subscript Referencing
Cannot make subscript references such as x(1:5,2) within fmadclass if x is fmad class.
Have to replace overloaded subscripting with direct calls to fmad’ssubsref function.
In fmad’s prod.m Product Function
localderivs=pval(ones(s(1),1),:)./Aval;
replaced with
S.type=’()’;S.subs=ones(s(1),1),’:’;localderivs=subsref(pval,S)./Aval;
19/ 29 Higher Derivatives in Matlab Using MAD
![Page 34: Higher Derivatives in Matlab Using MAD EuroAd Workshop... · Requests for higher derivatives of order 2 to 4 for uncertainty analysis and robust optimisation. Could implement a Taylor](https://reader034.vdocuments.mx/reader034/viewer/2022042621/5f7033f4c00904629a3f93e5/html5/thumbnails/34.jpg)
Eliminate Unnecessary Temporary fmad Objects
A.value of fmad class:
Original Code in prod.m (fmad objects highlighted)
Aval=A.value;s=size(Aval);S.subs={ones(s(1),1),’:’};localderivs=subsref(pval,S)./Aval;
Modified Code in prod.m (fmad objects highlighted)
Aval=A.value;s=size(deactivate(Aval));S.subs={ones(s(1),1),’:’};localderivs=subsref(pval,S)./Aval;
Reduces number of fmad operations performed.
20/ 29 Higher Derivatives in Matlab Using MAD
![Page 35: Higher Derivatives in Matlab Using MAD EuroAd Workshop... · Requests for higher derivatives of order 2 to 4 for uncertainty analysis and robust optimisation. Could implement a Taylor](https://reader034.vdocuments.mx/reader034/viewer/2022042621/5f7033f4c00904629a3f93e5/html5/thumbnails/35.jpg)
Reverse Mode AD in Matlab
Uses a madtape class to build up 2 tapes:1 operations tape of all arithmetic operations, including subscripting.2 value tape of sizes of values involved in calculation, values of those
involved nonlinearly.
Functions to set adjoints and propagate back through thecomputation.
Ensure madtape class superior to fmad to enable forward-over-reverse.
21/ 29 Higher Derivatives in Matlab Using MAD
![Page 36: Higher Derivatives in Matlab Using MAD EuroAd Workshop... · Requests for higher derivatives of order 2 to 4 for uncertainty analysis and robust optimisation. Could implement a Taylor](https://reader034.vdocuments.mx/reader034/viewer/2022042621/5f7033f4c00904629a3f93e5/html5/thumbnails/36.jpg)
Use of Reverse Mode - Minpack EPT Gradient
x=madtape(x0); % initialise madtape object
f=MinpackEPT_F(x,Prob); % Evaluate the function% and create tapes
fmadtape=getvalue(f) % Extract the objective function value
setadjoint(f,1); % initialise adjoint of dependent var.
MADCalcAdjoints % propagate adjoints back% through the operations tape
gmadtape=getadjoint(x); % extract adjoint of independent
22/ 29 Higher Derivatives in Matlab Using MAD
![Page 37: Higher Derivatives in Matlab Using MAD EuroAd Workshop... · Requests for higher derivatives of order 2 to 4 for uncertainty analysis and robust optimisation. Could implement a Taylor](https://reader034.vdocuments.mx/reader034/viewer/2022042621/5f7033f4c00904629a3f93e5/html5/thumbnails/37.jpg)
Results
Very preliminary
Minpack EPT case.
Objected oriented application
23/ 29 Higher Derivatives in Matlab Using MAD
![Page 38: Higher Derivatives in Matlab Using MAD EuroAd Workshop... · Requests for higher derivatives of order 2 to 4 for uncertainty analysis and robust optimisation. Could implement a Taylor](https://reader034.vdocuments.mx/reader034/viewer/2022042621/5f7033f4c00904629a3f93e5/html5/thumbnails/38.jpg)
Minpack EPT Problem
Objective function with sparse Hessian
24/ 29 Higher Derivatives in Matlab Using MAD
![Page 39: Higher Derivatives in Matlab Using MAD EuroAd Workshop... · Requests for higher derivatives of order 2 to 4 for uncertainty analysis and robust optimisation. Could implement a Taylor](https://reader034.vdocuments.mx/reader034/viewer/2022042621/5f7033f4c00904629a3f93e5/html5/thumbnails/39.jpg)
Minpack EPT Problem
Ratio of Hessian time to function time.n = nx ∗ nx
4 16 64 256
hand-coded 12. 13. 12. 14.fmad(sparse) on fmad(sparse) 662. 664. 1154. 18553.fmad(sparse) on reverse 1160. 1147. 1235. 2897.
Modify Source Codehand-coded 13. 14. 14. 15.fmad(sparse) on fmad(sparse) 569. 559. 757. 4778.fmad(sparse) on reverse 1114. 1094. 1194. 2727.
n2
16 256 4096 65536
All times averaged over a minimum of 5s.
25/ 29 Higher Derivatives in Matlab Using MAD
![Page 40: Higher Derivatives in Matlab Using MAD EuroAd Workshop... · Requests for higher derivatives of order 2 to 4 for uncertainty analysis and robust optimisation. Could implement a Taylor](https://reader034.vdocuments.mx/reader034/viewer/2022042621/5f7033f4c00904629a3f93e5/html5/thumbnails/40.jpg)
Source Code Modification
[f,fgrad,fhess]=MinpackEPT_FGH(x,Prob)nx=Prob.user.nx; % nx is class doubleny=length(x)./nx; % x active => ny active with zero derivs.if ny~=Prob.user.ny
error(’Must have length(x)==nx*ny’)endhx = 1/(nx+1); % hx is inactivehy = 1/(ny+1); % hy is active with zero derivs.v = zeros(nx+2,ny+2); % ny active => v activev(2:nx+1,2:ny+1) = reshape(x,nx,ny);
% v active allows subscripted adssignment% computer dvdx and dvdy on each edge of the griddvdx = (v(2:nx+2,:)-v(1:nx+1,:))/hx;dvdy = (v(:,2:ny+2)-v(:,1:ny+1))/hy
% unnecessary differentiated division op.
26/ 29 Higher Derivatives in Matlab Using MAD
![Page 41: Higher Derivatives in Matlab Using MAD EuroAd Workshop... · Requests for higher derivatives of order 2 to 4 for uncertainty analysis and robust optimisation. Could implement a Taylor](https://reader034.vdocuments.mx/reader034/viewer/2022042621/5f7033f4c00904629a3f93e5/html5/thumbnails/41.jpg)
Source Code Modification
[f,fgrad,fhess]=MinpackEPT_FGH(x,Prob)nx=Prob.user.nx; % nx is class doubleny=length(x)./nx; % x active => ny active with zero derivs.if ny~=Prob.user.ny
error(’Must have length(x)==nx*ny’)endhx = 1/(nx+1); % hx is inactivehy = 1/(ny+1); % hy is active with zero derivs.v = zeros(nx+2,ny+2); % ny active => v activev(2:nx+1,2:ny+1) = reshape(x,nx,ny);
% v active allows subscripted adssignment% computer dvdx and dvdy on each edge of the griddvdx = (v(2:nx+2,:)-v(1:nx+1,:))/hx;dvdy = (v(:,2:ny+2)-v(:,1:ny+1))/hy
% unnecessary differentiated division op.
26/ 29 Higher Derivatives in Matlab Using MAD
![Page 42: Higher Derivatives in Matlab Using MAD EuroAd Workshop... · Requests for higher derivatives of order 2 to 4 for uncertainty analysis and robust optimisation. Could implement a Taylor](https://reader034.vdocuments.mx/reader034/viewer/2022042621/5f7033f4c00904629a3f93e5/html5/thumbnails/42.jpg)
Source Code Modification
[f,fgrad,fhess]=MinpackEPT_FGH(x,Prob)nx=Prob.user.nx; % nx is class doubleny=length(x)./nx; % x active => ny active with zero derivs.if ny~=Prob.user.ny
error(’Must have length(x)==nx*ny’)endhx = 1/(nx+1); % hx is inactivehy = 1/(ny+1); % hy is active with zero derivs.v = zeros(nx+2,ny+2); % ny active => v activev(2:nx+1,2:ny+1) = reshape(x,nx,ny);
% v active allows subscripted adssignment% computer dvdx and dvdy on each edge of the griddvdx = (v(2:nx+2,:)-v(1:nx+1,:))/hx;dvdy = (v(:,2:ny+2)-v(:,1:ny+1))/hy
% unnecessary differentiated division op.
26/ 29 Higher Derivatives in Matlab Using MAD
![Page 43: Higher Derivatives in Matlab Using MAD EuroAd Workshop... · Requests for higher derivatives of order 2 to 4 for uncertainty analysis and robust optimisation. Could implement a Taylor](https://reader034.vdocuments.mx/reader034/viewer/2022042621/5f7033f4c00904629a3f93e5/html5/thumbnails/43.jpg)
Source Code Modification
[f,fgrad,fhess]=MinpackEPT_FGH(x,Prob)nx=Prob.user.nx; % nx is class doubleny=length(x)./nx; % x active => ny active with zero derivs.if ny~=Prob.user.ny
error(’Must have length(x)==nx*ny’)endhx = 1/(nx+1); % hx is inactivehy = 1/(ny+1); % hy is active with zero derivs.v = zeros(nx+2,ny+2); % ny active => v activev(2:nx+1,2:ny+1) = reshape(x,nx,ny);
% v active allows subscripted adssignment% computer dvdx and dvdy on each edge of the griddvdx = (v(2:nx+2,:)-v(1:nx+1,:))/hx;dvdy = (v(:,2:ny+2)-v(:,1:ny+1))/deactivate(hy);
% deactivate hy => cheaper division
26/ 29 Higher Derivatives in Matlab Using MAD
![Page 44: Higher Derivatives in Matlab Using MAD EuroAd Workshop... · Requests for higher derivatives of order 2 to 4 for uncertainty analysis and robust optimisation. Could implement a Taylor](https://reader034.vdocuments.mx/reader034/viewer/2022042621/5f7033f4c00904629a3f93e5/html5/thumbnails/44.jpg)
Object-Oriented Test Case Preliminary Results
Test case with 4 classes from aerospace sector n = 21,m = 101.
max. 11 nonzeros/row of J.
Require derivatives for uncertainty propagation.
Users require derivatives to order 3 - results here to order 2 - workedfirst time!
technique cpu(JF)/cpu(F) cpu(∇2F)/cpu(F)
FD 22.0 486.7fmad 1.09 -fmad on fmad - 2.41
Very little computation performed - nearly all time creating objectsand some trivial computation - highlights benefits of OO!!
OO destroys performance of FD.
27/ 29 Higher Derivatives in Matlab Using MAD
![Page 45: Higher Derivatives in Matlab Using MAD EuroAd Workshop... · Requests for higher derivatives of order 2 to 4 for uncertainty analysis and robust optimisation. Could implement a Taylor](https://reader034.vdocuments.mx/reader034/viewer/2022042621/5f7033f4c00904629a3f93e5/html5/thumbnails/45.jpg)
Jacobian Sparsity
28/ 29 Higher Derivatives in Matlab Using MAD
![Page 46: Higher Derivatives in Matlab Using MAD EuroAd Workshop... · Requests for higher derivatives of order 2 to 4 for uncertainty analysis and robust optimisation. Could implement a Taylor](https://reader034.vdocuments.mx/reader034/viewer/2022042621/5f7033f4c00904629a3f93e5/html5/thumbnails/46.jpg)
Conclusions & Future Work
MAD can differentiate object-oriented code - including itself!
Further performance optimisation of fmad class needed:I Inactive fmad object is needed.I Store size of fmad object directly - don’t take size of it’s value.
Sparsity detection needed for first and higher derivatives - some proofof concept done using sparse logical matrices - summer 2007.
madtape class needs improvements:I Generate tape once and do multiple adjoints for same inputs x - need
to extend to different x with same control flow.I Performance analysis and optimisation.
29/ 29 Higher Derivatives in Matlab Using MAD
![Page 47: Higher Derivatives in Matlab Using MAD EuroAd Workshop... · Requests for higher derivatives of order 2 to 4 for uncertainty analysis and robust optimisation. Could implement a Taylor](https://reader034.vdocuments.mx/reader034/viewer/2022042621/5f7033f4c00904629a3f93e5/html5/thumbnails/47.jpg)
References
Shaun A. Forth.An efficient overloaded implementation of forward mode automatic differentiation inMATLAB.ACM Trans. Math. Softw., 32(2):195–222, June 2006.
Andreas Griewank, David Juedes, and Jean Utke.Algorithm 755: ADOL-C: A package for the automatic differentiation of algorithms writtenin C/C++.ACM Transactions on Mathematical Software, 22(2):131–167, 1996.
Andreas Griewank.Evaluating Derivatives: Principles and Techniques of Algorithmic Differentiation.Number 19 in Frontiers in Applied Mathematics. SIAM, Philadelphia, PA, 2000.
30/ 29 Higher Derivatives in Matlab Using MAD