metaprogramming patterns for physics simulation...
TRANSCRIPT
Metaprogramming Patterns for Physics Simulationand Verification
Roy H. Stogner
The University of Texas at Austin
April 16, 2012
Roy H. Stogner Metaprogramming April 16, 2012 1 / 52
Computational Modeling and Verification
PECOS ResearchVerification, Validation, Uncertainty Quantification• Atmospheric entry vehicle, two key quantities of interest:
I Rate of recession of TPS (throughout peak heating regime)I Local peak heat flux to after-body
• Validation and uncertainty quantification for these QoI
Roy H. Stogner Metaprogramming April 16, 2012 2 / 52
Computational Modeling and Verification
PECOS Development
Experimental Data
QUESO
FIN-SlibMeshBoost
Turbulence
SHOCKINGMUTATION
GSL
Chemistry/Shocktube
Plug FlowABLATION1D
GSL
Ablation
Calibration/Validation
FIN-S
Inve
rsio
n
Forw
ard
QUESO/Dakota
Multiphysics Analysis
MASA
QoI
GRVYExternal Libraries
LDVFlow
Measurements
NASA EAST Shocktube
Measurements
Molecular Beam (O2)Heated Flow Titration (N2)
New Codes
• New subphysicsmodels
• New developers
• New QoIpostprocessing
• New numerics
• New bugs
Roy H. Stogner Metaprogramming April 16, 2012 3 / 52
Computational Modeling and Verification
PECOS Development
Experimental Data
QUESO
FIN-SlibMeshBoost
Turbulence
SHOCKINGMUTATION
GSL
Chemistry/Shocktube
Plug FlowABLATION1D
GSL
Ablation
Calibration/Validation
FIN-S
Inve
rsio
n
Forw
ard
QUESO/Dakota
Multiphysics Analysis
MASA
QoI
GRVYExternal Libraries
LDVFlow
Measurements
NASA EAST Shocktube
Measurements
Molecular Beam (O2)Heated Flow Titration (N2)
New Codes• New subphysics
models
• New developers
• New QoIpostprocessing
• New numerics
• New bugs
Roy H. Stogner Metaprogramming April 16, 2012 3 / 52
Computational Modeling and Verification
PECOS Development
Experimental Data
QUESO
FIN-SlibMeshBoost
Turbulence
SHOCKINGMUTATION
GSL
Chemistry/Shocktube
Plug FlowABLATION1D
GSL
Ablation
Calibration/Validation
FIN-S
Inve
rsio
n
Forw
ard
QUESO/Dakota
Multiphysics Analysis
MASA
QoI
GRVYExternal Libraries
LDVFlow
Measurements
NASA EAST Shocktube
Measurements
Molecular Beam (O2)Heated Flow Titration (N2)
New Codes• New subphysics
models
• New developers
• New QoIpostprocessing
• New numerics
• New bugs
Roy H. Stogner Metaprogramming April 16, 2012 3 / 52
Computational Modeling and Verification
PECOS Development
Experimental Data
QUESO
FIN-SlibMeshBoost
Turbulence
SHOCKINGMUTATION
GSL
Chemistry/Shocktube
Plug FlowABLATION1D
GSL
Ablation
Calibration/Validation
FIN-S
Inve
rsio
n
Forw
ard
QUESO/Dakota
Multiphysics Analysis
MASA
QoI
GRVYExternal Libraries
LDVFlow
Measurements
NASA EAST Shocktube
Measurements
Molecular Beam (O2)Heated Flow Titration (N2)
New Codes• New subphysics
models
• New developers
• New QoIpostprocessing
• New numerics
• New bugs
Roy H. Stogner Metaprogramming April 16, 2012 3 / 52
Computational Modeling and Verification
PECOS Development
Experimental Data
QUESO
FIN-SlibMeshBoost
Turbulence
SHOCKINGMUTATION
GSL
Chemistry/Shocktube
Plug FlowABLATION1D
GSL
Ablation
Calibration/Validation
FIN-S
Inve
rsio
n
Forw
ard
QUESO/Dakota
Multiphysics Analysis
MASA
QoI
GRVYExternal Libraries
LDVFlow
Measurements
NASA EAST Shocktube
Measurements
Molecular Beam (O2)Heated Flow Titration (N2)
New Codes• New subphysics
models
• New developers
• New QoIpostprocessing
• New numerics
• New bugs
Roy H. Stogner Metaprogramming April 16, 2012 3 / 52
Computational Modeling and Verification
PECOS Development
Experimental Data
QUESO
FIN-SlibMeshBoost
Turbulence
SHOCKINGMUTATION
GSL
Chemistry/Shocktube
Plug FlowABLATION1D
GSL
Ablation
Calibration/Validation
FIN-S
Inve
rsio
n
Forw
ard
QUESO/Dakota
Multiphysics Analysis
MASA
QoI
GRVYExternal Libraries
LDVFlow
Measurements
NASA EAST Shocktube
Measurements
Molecular Beam (O2)Heated Flow Titration (N2)
New Codes• New subphysics
models
• New developers
• New QoIpostprocessing
• New numerics
• New bugs
Roy H. Stogner Metaprogramming April 16, 2012 3 / 52
Computational Modeling and Verification
Method of Manufactured Solutions
Verification Process• “Manufacture” a closed form exact solution u, e.g. parameterized trig
functions:(Roy, 2002)
u(x, y, z, t) = u0 + ux fs
(auxπxL
)+ uy fs
(auyπyL
)+
+ uz fs
(auzπzL
)+ ut fs
(autπt
L
)• Insert u into model equation system F , evaluate source terms
• Use simulation code to solve modified physics F (u) ≡ F (u)− F (u)• Verify that solution uh → u at expected rate
Roy H. Stogner Metaprogramming April 16, 2012 4 / 52
Computational Modeling and Verification
Maple MMS: 3D Navier-Stokes Energy Term
Qe =−apxπpx
L
γ
γ − 1sin
�apxπx
L
� �u0 + ux sin
�auxπx
L
�+ uy cos
�auyπy
L
�+ uz cos
�auzπz
L
��+
+apyπpy
L
γ
γ − 1cos
�apyπy
L
� �v0 + vx cos
�avxπx
L
�+ vy sin
�avyπy
L
�+ vz sin
�avzπz
L
��+
−apzπpz
L
γ
γ − 1sin
�apzπz
L
� �w0 + wx sin
�awxπx
L
�+ wy sin
�awyπy
L
�+ wz cos
�awzπz
L
��+
+aρxπρx2L
cos�aρxπx
L
� �u0 + ux sin
�auxπx
L
�+ uy cos
�auyπy
L
�+ uz cos
�auzπz
L
����u0 + ux sin
�auxπx
L
�+ uy cos
�auyπy
L
�+ uz cos
�auzπz
L
��2+
+�w0 + wx sin
�awxπx
L
�+ wy sin
�awyπy
L
�+ wz cos
�awzπz
L
��2+
�v0 + vx cos
�avxπx
L
�+ vy sin
�avyπy
L
�+ vz sin
�avzπz
L
��2�
+
−aρyπρy2L
sin�aρyπy
L
� �v0 + vx cos
�avxπx
L
�+ vy sin
�avyπy
L
�+ vz sin
�avzπz
L
����u0 + ux sin
�auxπx
L
�+ uy cos
�auyπy
L
�+ uz cos
�auzπz
L
��2+
+�w0 + wx sin
�awxπx
L
�+ wy sin
�awyπy
L
�+ wz cos
�awzπz
L
��2+
�v0 + vx cos
�avxπx
L
�+ vy sin
�avyπy
L
�+ vz sin
�avzπz
L
��2�
+
+aρzπρz2L
cos�aρzπz
L
� �w0 + wx sin
�awxπx
L
�+ wy sin
�awyπy
L
�+ wz cos
�awzπz
L
����u0 + ux sin
�auxπx
L
�+ uy cos
�auyπy
L
�+ uz cos
�auzπz
L
��2+
+�w0 + wx sin
�awxπx
L
�+ wy sin
�awyπy
L
�+ wz cos
�awzπz
L
��2+�v0 + vx cos
�avxπx
L
�+ vy sin
�avyπy
L
�+ vz sin
�avzπz
L
��2�
+
+auxπux
2Lcos
�auxπx
L
����w0 + wx sin
�awxπx
L
�+ wy sin
�awyπy
L
�+ wz cos
�awzπz
L
��2+
�v0 + vx cos
�avxπx
L
�+ vy sin
�avyπy
L
�+ vz sin
�avzπz
L
��2+
+3�u0 + ux sin
�auxπx
L
�+ uy cos
�auyπy
L
�+ uz cos
�auzπz
L
��2��
ρ0 + ρx sin�aρxπx
L
�+ ρy cos
�aρyπy
L
�+ ρz sin
�aρzπz
L
��+
+�p0 + px cos
�apxπx
L
�+ py sin
�apyπy
L
�+ pz cos
�apzπz
L
�� 2γ
(γ − 1)
�
+
−auyπuy
Lsin
�auyπy
L
� �v0 + vx cos
�avxπx
L
�+ vy sin
�avyπy
L
�+ vz sin
�avzπz
L
�� �ρ0 + ρx sin
�aρxπx
L
�+ ρy cos
�aρyπy
L
�+ ρz sin
�aρzπz
L
��·
·�u0 + ux sin
�auxπx
L
�+ uy cos
�auyπy
L
�+ uz cos
�auzπz
L
��+
−auzπuz
Lsin
�auzπz
L
� �w0 + wx sin
�awxπx
L
�+ wy sin
�awyπy
L
�+ wz cos
�awzπz
L
�� �ρ0 + ρx sin
�aρxπx
L
�+ ρy cos
�aρyπy
L
�+ ρz sin
�aρzπz
L
��·
·�u0 + ux sin
�auxπx
L
�+ uy cos
�auyπy
L
�+ uz cos
�auzπz
L
��+
Roy H. Stogner Metaprogramming April 16, 2012 5 / 52
Computational Modeling and Verification
And more 3D Navier-Stokes Energy Term
Roy H. Stogner Metaprogramming April 16, 2012 6 / 52
Computational Modeling and Verification
Combinatorial Explosion
Explosion in source term size• Exponential complexity increase with more sophisticated models
I Basic Sutherland viscosity starts at a million characters
• Large CAS RAM requirements (128 GB insufficient for Sutherland)
• segfaults, incorrect output
Hierarchic MMS?• Manual simplification of sub-terms, one-by-one
• Recombination into final forcing function
• Still labor-intensive
Automatic Differentiation?• Adding complex dependencies to MASA?
Roy H. Stogner Metaprogramming April 16, 2012 7 / 52
Computational Modeling and Verification
Combinatorial Explosion
Explosion in source term size• Exponential complexity increase with more sophisticated models
I Basic Sutherland viscosity starts at a million characters
• Large CAS RAM requirements (128 GB insufficient for Sutherland)
• segfaults, incorrect output
Hierarchic MMS?• Manual simplification of sub-terms, one-by-one
• Recombination into final forcing function
• Still labor-intensive
Automatic Differentiation?• Adding complex dependencies to MASA?
Roy H. Stogner Metaprogramming April 16, 2012 7 / 52
Computational Modeling and Verification
Combinatorial Explosion
Explosion in source term size• Exponential complexity increase with more sophisticated models
I Basic Sutherland viscosity starts at a million characters
• Large CAS RAM requirements (128 GB insufficient for Sutherland)
• segfaults, incorrect output
Hierarchic MMS?• Manual simplification of sub-terms, one-by-one
• Recombination into final forcing function
• Still labor-intensive
Automatic Differentiation?• Adding complex dependencies to MASA?
Roy H. Stogner Metaprogramming April 16, 2012 7 / 52
Computational Modeling and Verification Generic Programming
Generic Programming
• Or more informally, “Duck Typing”: templating around unspecifieddata types that expected to be overloaded to act like mathematicalobjects, creating new code and new data types at compile time.
• We do this all the time for float/double/long double.
• So we can duct tape our templated typename to any class thatbehaves like we expect math to behave
Roy H. Stogner Metaprogramming April 16, 2012 8 / 52
Computational Modeling and Verification Generic Programming
Duck Typing
If it walks like a duck, quacks likea duck, looks like a duck, it mustbe a duck.
Unless it’s a dragon doing a duckimpression...
We ran into a couple of those.
Roy H. Stogner Metaprogramming April 16, 2012 9 / 52
Computational Modeling and Verification Generic Programming
Duck Typing
If it walks like a duck, quacks likea duck, looks like a duck, it mustbe a duck.
Unless it’s a dragon doing a duckimpression...
We ran into a couple of those.
Roy H. Stogner Metaprogramming April 16, 2012 9 / 52
Computational Modeling and Verification Generic Programming
Duck Typing
If it walks like a duck, quacks likea duck, looks like a duck, it mustbe a duck.
Unless it’s a dragon doing a duckimpression...
We ran into a couple of those.
Roy H. Stogner Metaprogramming April 16, 2012 9 / 52
Computational Modeling and Verification Example Data Types
Operator-Overloaded Multi-precision
Example
template <typename T, typename S>
struct ShadowNumber {.........T _val;
S _shadow;
};
• Simultaneous calculation with multiple floating point representations,to estimate rounding error:
ExampleShadowNumber<float, double> shadowed_float;
ShadowNumber<double, long double> shadowed_double;
Roy H. Stogner Metaprogramming April 16, 2012 10 / 52
Computational Modeling and Verification Example Data Types
Operator-Overloaded Vectors
Example
template <unsigned int size, typename T>
struct NumberArray {.........
template <typename T2>
NumberArray<size,T>& operator+= (const NumberArray<size,T2>& a) {
for (unsigned int i = 0; i != size; ++i)
_data[i] += a[i]; return *this;
}.........T _data[size];
};
• Compile-time fixed array sizes - no heap memory
• Fixed length loops can unroll with no branching
• Mixed data subtypes
Roy H. Stogner Metaprogramming April 16, 2012 11 / 52
Computational Modeling and Verification Example Data Types
Overloaded Functions, Operators
Example
#define NumberArray_std_unary(funcname) \
template <unsigned int size, typename T> \
inline \
NumberArray<size, T> \
funcname (const NumberArray<size, T>& a) \
{ \
NumberArray<size, T> returnval; \
\
for (unsigned int i = 0; i != size; ++i) \
returnval[i] = funcname(a[i]); \
\
return returnval; \
}
NumberArray_std_unary(exp)
NumberArray_std_unary(sin)
NumberArray_std_unary(cos)
• Templates shouldreplace writing NTNf
repetitive codes usingunsafe macros
• Templated function andstruct names can’t betemplate arguments.
• We still need macros toavoid Nf repetitivetemplate functions.
• This gives usmultidimensional nth
derivatives in 300 linesof C++
Roy H. Stogner Metaprogramming April 16, 2012 12 / 52
Computational Modeling and Verification Example Data Types
Operator-Overloaded Vectors
Example
NumericArray<3,double> v;
v[1] = 2;
NumericArray<3,float> f;
f[2] = 1;
v += sin(f); // Great!
f += sin(v); // Tolerable!
v = v + sin(f); // Undefined?
What is the return type supposed to be for
template <unsigned int size, typename T1, typename T2>
operator+(const NumberArray<size,T1>& a,
const NumberArray<size,T2>& b)
std:: problem too - complex<float> + double won’t compile
Roy H. Stogner Metaprogramming April 16, 2012 13 / 52
Computational Modeling and Verification Example Data Types
Operator-Overloaded Vectors
Example
NumericArray<3,double> v;
v[1] = 2;
NumericArray<3,float> f;
f[2] = 1;
v += sin(f); // Great!
f += sin(v); // Tolerable!
v = v + sin(f); // Undefined?
What is the return type supposed to be for
template <unsigned int size, typename T1, typename T2>
operator+(const NumberArray<size,T1>& a,
const NumberArray<size,T2>& b)
std:: problem too - complex<float> + double won’t compile
Roy H. Stogner Metaprogramming April 16, 2012 13 / 52
Computational Modeling and Verification Example Data Types
Operator-Overloaded Vectors
Example
NumericArray<3,double> v;
v[1] = 2;
NumericArray<3,float> f;
f[2] = 1;
v += sin(f); // Great!
f += sin(v); // Tolerable!
v = v + sin(f); // Undefined?
What is the return type supposed to be for
template <unsigned int size, typename T1, typename T2>
operator+(const NumberArray<size,T1>& a,
const NumberArray<size,T2>& b)
std:: problem too - complex<float> + double won’t compile
Roy H. Stogner Metaprogramming April 16, 2012 13 / 52
Computational Modeling and Verification Example Data Types
Operator-Overloaded Vectors
Example
NumericArray<3,double> v;
v[1] = 2;
NumericArray<3,float> f;
f[2] = 1;
v += sin(f); // Great!
f += sin(v); // Tolerable!
v = v + sin(f); // Undefined?
What is the return type supposed to be for
template <unsigned int size, typename T1, typename T2>
operator+(const NumberArray<size,T1>& a,
const NumberArray<size,T2>& b)
std:: problem too - complex<float> + double won’t compile
Roy H. Stogner Metaprogramming April 16, 2012 13 / 52
Computational Modeling and Verification Example Data Types
Operator-Overloaded Vectors
Example
NumericArray<3,double> v;
v[1] = 2;
NumericArray<3,float> f;
f[2] = 1;
v += sin(f);
// Great!
f += sin(v); // Tolerable!
v = v + sin(f); // Undefined?
What is the return type supposed to be for
template <unsigned int size, typename T1, typename T2>
operator+(const NumberArray<size,T1>& a,
const NumberArray<size,T2>& b)
std:: problem too - complex<float> + double won’t compile
Roy H. Stogner Metaprogramming April 16, 2012 13 / 52
Computational Modeling and Verification Example Data Types
Operator-Overloaded Vectors
Example
NumericArray<3,double> v;
v[1] = 2;
NumericArray<3,float> f;
f[2] = 1;
v += sin(f); // Great!
f += sin(v); // Tolerable!
v = v + sin(f); // Undefined?
What is the return type supposed to be for
template <unsigned int size, typename T1, typename T2>
operator+(const NumberArray<size,T1>& a,
const NumberArray<size,T2>& b)
std:: problem too - complex<float> + double won’t compile
Roy H. Stogner Metaprogramming April 16, 2012 13 / 52
Computational Modeling and Verification Example Data Types
Operator-Overloaded Vectors
Example
NumericArray<3,double> v;
v[1] = 2;
NumericArray<3,float> f;
f[2] = 1;
v += sin(f); // Great!
f += sin(v);
// Tolerable!
v = v + sin(f); // Undefined?
What is the return type supposed to be for
template <unsigned int size, typename T1, typename T2>
operator+(const NumberArray<size,T1>& a,
const NumberArray<size,T2>& b)
std:: problem too - complex<float> + double won’t compile
Roy H. Stogner Metaprogramming April 16, 2012 13 / 52
Computational Modeling and Verification Example Data Types
Operator-Overloaded Vectors
Example
NumericArray<3,double> v;
v[1] = 2;
NumericArray<3,float> f;
f[2] = 1;
v += sin(f); // Great!
f += sin(v); // Tolerable!
v = v + sin(f); // Undefined?
What is the return type supposed to be for
template <unsigned int size, typename T1, typename T2>
operator+(const NumberArray<size,T1>& a,
const NumberArray<size,T2>& b)
std:: problem too - complex<float> + double won’t compile
Roy H. Stogner Metaprogramming April 16, 2012 13 / 52
Computational Modeling and Verification Example Data Types
Operator-Overloaded Vectors
Example
NumericArray<3,double> v;
v[1] = 2;
NumericArray<3,float> f;
f[2] = 1;
v += sin(f); // Great!
f += sin(v); // Tolerable!
v = v + sin(f);
// Undefined?
What is the return type supposed to be for
template <unsigned int size, typename T1, typename T2>
operator+(const NumberArray<size,T1>& a,
const NumberArray<size,T2>& b)
std:: problem too - complex<float> + double won’t compile
Roy H. Stogner Metaprogramming April 16, 2012 13 / 52
Computational Modeling and Verification Example Data Types
Operator-Overloaded Vectors
Example
NumericArray<3,double> v;
v[1] = 2;
NumericArray<3,float> f;
f[2] = 1;
v += sin(f); // Great!
f += sin(v); // Tolerable!
v = v + sin(f); // Undefined?
What is the return type supposed to be for
template <unsigned int size, typename T1, typename T2>
operator+(const NumberArray<size,T1>& a,
const NumberArray<size,T2>& b)
std:: problem too - complex<float> + double won’t compile
Roy H. Stogner Metaprogramming April 16, 2012 13 / 52
Computational Modeling and Verification Example Data Types
Operator-Overloaded Vectors
Example
NumericArray<3,double> v;
v[1] = 2;
NumericArray<3,float> f;
f[2] = 1;
v += sin(f); // Great!
f += sin(v); // Tolerable!
v = v + sin(f); // Undefined?
What is the return type supposed to be for
template <unsigned int size, typename T1, typename T2>
operator+(const NumberArray<size,T1>& a,
const NumberArray<size,T2>& b)
std:: problem too - complex<float> + double won’t compile
Roy H. Stogner Metaprogramming April 16, 2012 13 / 52
Computational Modeling and Verification Computing new Data Types
Return Type Selection, Template Specialization
Let’s give it a template:
template <typename S, typename T>
struct CompareTypes {
// typedef something supertype;
};
And then we can specialize
Exampletemplate <>
struct CompareTypes<float,double> {
typedef double
supertype;
};
Roy H. Stogner Metaprogramming April 16, 2012 14 / 52
Computational Modeling and Verification Computing new Data Types
Return Type Selection, Template Specialization
Let’s give it a template:
template <typename S, typename T>
struct CompareTypes {
// typedef something supertype;
};
Or, for templated classes, partially specialize
Exampletemplate <unsigned int size, typename T, typename T2>
struct CompareTypes<NumberArray<size,T>,T2> {
typedef NumberArray<size,typename CompareTypes<T,T2>::supertype>
supertype;
};
Roy H. Stogner Metaprogramming April 16, 2012 14 / 52
Computational Modeling and Verification Computing new Data Types
Return Types: Here Be Dragons
A NumberArray of something, combined with anything else we haven’tseen before
Example
template <unsigned int size, typename T, typename T2>
struct CompareTypes<NumberArray<size,T>,T2> {
typedef NumberArray<size,
typename CompareTypes<T,T2>::supertype>
supertype;
};
if(typeid(CompareTypes<
NumberArray<3,float>,double
>::supertype) ==
typeid(NumberArray<3,double>))
how_much_we_rock = 10;
Roy H. Stogner Metaprogramming April 16, 2012 15 / 52
Computational Modeling and Verification Computing new Data Types
Return Types: Here Be Dragons
A NumberArray of something, combined with anything else we haven’tseen before
Example
template <unsigned int size, typename T, typename T2>
struct CompareTypes<NumberArray<size,T>,T2> {
typedef NumberArray<size,
typename CompareTypes<T,T2>::supertype>
supertype;
};
if(typeid(CompareTypes<
NumberArray<3,NumberArray<float> >,double
>::supertype) ==
typeid(NumberArray<3,NumberArray<double> >))
how_much_we_rock = 11; // That extra push over the cliff
Roy H. Stogner Metaprogramming April 16, 2012 15 / 52
Computational Modeling and Verification Computing new Data Types
Return Types: Here Be Dragons
A ShadowNumber of something, combined with anything else we haven’tseen before
Example
template <typename T, typename S, typename T2>
struct CompareTypes<ShadowNumber<T,S>,T2> {
typedef ShadowNumber<typename CompareTypes<T,T2>::supertype,
typename CompareTypes<S,T2>::supertype>
supertype;
};
if(typeid(CompareTypes<
ShadowNumber<float,double>,double
>::supertype) ==
typeid(ShadowNumber<double,double>))
how_much_we_rock = 7; // C is still a passing grade
Roy H. Stogner Metaprogramming April 16, 2012 15 / 52
Computational Modeling and Verification Computing new Data Types
Return Types: Here Be Dragons
A ShadowNumber of something, combined with a NumberArray
Example
template <typename T, typename S, typename T2>
struct CompareTypes<ShadowNumber<T,S>,T2> {
typedef ShadowNumber<typename CompareTypes<T,T2>::supertype,
typename CompareTypes<S,T2>::supertype>
supertype;
};
if(typeid(CompareTypes<
NumberArray<3,float>,ShadowNumber<float,double>
>::supertype) ==
typeid(NumberArray<3,ShadowNumber<float,double> >))
how_much_we_rock = -1; // This didn’t even compile
Roy H. Stogner Metaprogramming April 16, 2012 15 / 52
Computational Modeling and Verification Computing new Data Types
Return Types: Here Be Dragons
A ShadowNumber of something, combined with a NumberArray
Example
template <typename T, typename S, typename T2>
struct CompareTypes<ShadowNumber<T,S>,T2> {
typedef ShadowNumber<typename CompareTypes<T,T2>::supertype,
typename CompareTypes<S,T2>::supertype>
supertype;
};
if(typeid(CompareTypes<
NumberArray<3,float>,ShadowNumber<float,double>
>::supertype) ==
typeid(ShadowNumber<NumberArray<3,float>,NumberArray<3,double> >))
how_much_we_rock = -1; // This didn’t even compile
Roy H. Stogner Metaprogramming April 16, 2012 15 / 52
Computational Modeling and Verification Computing new Data Types
Return Types: Killing Dragons• Mathematically, these are isomorphic:
I (V ⊗W )n
I V n ⊗Wn
• Your compiler doesn’t understandisomorphisms:
I {float, double}[size]I {float[size]}, {double[size]}
• Need partial template specialization,SFINAE to disambiguate
Example
template <unsigned int size, typename T, typename T2, typename S2>
struct CompareTypes<NumberArray<size,T>,ShadowNumber<T2,S2> > {
typedef NumberArray<size,
typename CompareTypes<T,ShadowNumber<T2,S2> >::supertype>
supertype;
};
Roy H. Stogner Metaprogramming April 16, 2012 16 / 52
Computational Modeling and Verification Computing new Data Types
Return Types: Killing Dragons• Mathematically, these are isomorphic:
I (V ⊗W )n
I V n ⊗Wn
• Your compiler doesn’t understandisomorphisms:
I {float, double}[size]I {float[size]}, {double[size]}
• Need partial template specialization,SFINAE to disambiguate
Example
template <unsigned int size, typename T, typename T2, typename S2>
struct CompareTypes<NumberArray<size,T>,ShadowNumber<T2,S2> > {
typedef NumberArray<size,
typename CompareTypes<T,ShadowNumber<T2,S2> >::supertype>
supertype;
};
Roy H. Stogner Metaprogramming April 16, 2012 16 / 52
Computational Modeling and Verification Computing new Data Types
Return Types: Killing Dragons• Mathematically, these are isomorphic:
I (V ⊗W )n
I V n ⊗Wn
• Your compiler doesn’t understandisomorphisms:
I {float, double}[size]I {float[size]}, {double[size]}
• Need partial template specialization,SFINAE to disambiguate
Example
template <unsigned int size, typename T, typename T2, typename S2>
struct CompareTypes<NumberArray<size,T>,ShadowNumber<T2,S2> > {
typedef NumberArray<size,
typename CompareTypes<T,ShadowNumber<T2,S2> >::supertype>
supertype;
};
Roy H. Stogner Metaprogramming April 16, 2012 16 / 52
Computational Modeling and Verification Computing new Data Types
Return Types: Killing Dragons• Mathematically, these are isomorphic:
I (V ⊗W )n
I V n ⊗Wn
• Your compiler doesn’t understandisomorphisms:
I {float, double}[size]I {float[size]}, {double[size]}
• Need partial template specialization,SFINAE to disambiguate
Example
template <unsigned int size, typename T, typename T2, typename S2>
struct CompareTypes<NumberArray<size,T>,ShadowNumber<T2,S2> > {
typedef NumberArray<size,
typename CompareTypes<T,ShadowNumber<T2,S2> >::supertype>
supertype;
};
Roy H. Stogner Metaprogramming April 16, 2012 16 / 52
Computational Modeling and Verification Automatic Differentiation
“Dual Numbers” - [Clifford 1873],[Study 1903]
• Add a new element ε to R, close the field:{m∑i=0
aiεi : ai ∈ R,m <∞
}
• Take the quotient with ε2 ≡
• (a+ bε)× (c+ dε) = (ac) + adε+ bcε+ bdε2 = ((ac) + (ad+ bc)ε)
• Extending to division, algebraic functions, transcendental functions,etc gives: for any function f(x),
f(a+ bε) = f(a) + bf ′(a)ε
• This time the isomorphism practically codes itself!
Roy H. Stogner Metaprogramming April 16, 2012 17 / 52
Computational Modeling and Verification Automatic Differentiation
“Dual Numbers” - [Clifford 1873],[Study 1903]
• Add a new element ε to R, close the field:{m∑i=0
aiεi : ai ∈ R,m <∞
}
• Take the quotient with ε2 ≡
• (a+ bε)× (c+ dε) = (ac) + adε+ bcε+ bdε2 = ((ac) + (ad+ bc)ε)
• Extending to division, algebraic functions, transcendental functions,etc gives: for any function f(x),
f(a+ bε) = f(a) + bf ′(a)ε
• This time the isomorphism practically codes itself!
Roy H. Stogner Metaprogramming April 16, 2012 17 / 52
Computational Modeling and Verification Automatic Differentiation
“Dual Numbers” - [Clifford 1873],[Study 1903]
• Add a new element ε to R, close the field:{m∑i=0
aiεi : ai ∈ R,m <∞
}
• Take the quotient with ε2 ≡ − 1
• (a+ bε)× (c+ dε) = (ac) + adε+ bcε+ bdε2 = ((ac) + (ad+ bc)ε)
• Extending to division, algebraic functions, transcendental functions,etc gives: for any function f(x),
f(a+ bε) = f(a) + bf ′(a)ε
• This time the isomorphism practically codes itself!
Roy H. Stogner Metaprogramming April 16, 2012 17 / 52
Computational Modeling and Verification Automatic Differentiation
“Dual Numbers” - [Clifford 1873],[Study 1903]
• Add a new element ε to R, close the field:{m∑i=0
aiεi : ai ∈ R,m <∞
}
• Take the quotient with ε2 ≡ − 1 ...no, wait, that’s C
• (a+ bε)× (c+ dε) = (ac) + adε+ bcε+ bdε2 = ((ac) + (ad+ bc)ε)
• Extending to division, algebraic functions, transcendental functions,etc gives: for any function f(x),
f(a+ bε) = f(a) + bf ′(a)ε
• This time the isomorphism practically codes itself!
Roy H. Stogner Metaprogramming April 16, 2012 17 / 52
Computational Modeling and Verification Automatic Differentiation
“Dual Numbers” - [Clifford 1873],[Study 1903]
• Add a new element ε to R, close the field:{m∑i=0
aiεi : ai ∈ R,m <∞
}
• Take the quotient with ε2 ≡ 0D[R] ≡ {a+ bε : a, b ∈ R}
• (a+ bε)× (c+ dε) = (ac) + adε+ bcε+ bdε2 = ((ac) + (ad+ bc)ε)
• Extending to division, algebraic functions, transcendental functions,etc gives: for any function f(x),
f(a+ bε) = f(a) + bf ′(a)ε
• This time the isomorphism practically codes itself!
Roy H. Stogner Metaprogramming April 16, 2012 17 / 52
Computational Modeling and Verification Automatic Differentiation
“Dual Numbers” - [Clifford 1873],[Study 1903]
• Add a new element ε to R, close the field:{m∑i=0
aiεi : ai ∈ R,m <∞
}
• Take the quotient with ε2 ≡ 0D[R] ≡ {a+ bε : a, b ∈ R}
• (a+ bε)× (c+ dε) = (ac) + adε+ bcε+ bdε2 = ((ac) + (ad+ bc)ε)
• Extending to division, algebraic functions, transcendental functions,etc gives: for any function f(x),
f(a+ bε) = f(a) + bf ′(a)ε
• This time the isomorphism practically codes itself!
Roy H. Stogner Metaprogramming April 16, 2012 17 / 52
Computational Modeling and Verification Automatic Differentiation
“Dual Numbers” - [Clifford 1873],[Study 1903]
• Add a new element ε to R, close the field:{m∑i=0
aiεi : ai ∈ R,m <∞
}
• Take the quotient with ε2 ≡ 0D[R] ≡ {a+ bε : a, b ∈ R}
• (a+ bε)× (c+ dε) = (ac) + adε+ bcε+ bdε2 = ((ac) + (ad+ bc)ε)
• Extending to division, algebraic functions, transcendental functions,etc gives: for any function f(x),
f(a+ bε) = f(a) + bf ′(a)ε
• This time the isomorphism practically codes itself!
Roy H. Stogner Metaprogramming April 16, 2012 17 / 52
Computational Modeling and Verification Automatic Differentiation
“Dual Numbers” - [Clifford 1873],[Study 1903]
• Add a new element ε to R, close the field:{m∑i=0
aiεi : ai ∈ R,m <∞
}
• Take the quotient with ε2 ≡ 0D[R] ≡ {a+ bε : a, b ∈ R}
• (a+ bε)× (c+ dε) = (ac) + adε+ bcε+ bdε2 = ((ac) + (ad+ bc)ε)
• Extending to division, algebraic functions, transcendental functions,etc gives: for any function f(x),
f(a+ bε) = f(a) + bf ′(a)ε
• This time the isomorphism practically codes itself!
Roy H. Stogner Metaprogramming April 16, 2012 17 / 52
Computational Modeling and Verification Automatic Differentiation
Operator-Overloaded Forward Differentiation
Example
typedef double RawType
// typedef ShadowNumber<double, long double> RawType;
typedef DualNumber<RawType,RawType> FirstDerivType;
const FirstDerivType x(pi/6,1); // Initializing independent var
const FirstDerivType sinx = sin(x); // Caching just like normal
const FirstDerivType y = sinx*sinx; // Smart pow would work too
const double raw_y = raw_value(y); // No implicit down-conversions
double deriv = raw_value(y.derivatives());
assert(deriv == 2*sin(pi/6)*cos(pi/6)); // FP ops match hand-code
Roy H. Stogner Metaprogramming April 16, 2012 18 / 52
Computational Modeling and Verification Automatic Differentiation
Operator-Overloaded Forward Differentiation
Example
typedef double RawType
// typedef ShadowNumber<double, long double> RawType;
typedef DualNumber<RawType,RawType> FirstDerivType;
const FirstDerivType x(pi/6,1); // Initializing independent var
const FirstDerivType sinx = sin(x); // Caching just like normal
const FirstDerivType y = sinx*sinx; // Smart pow would work too
const double raw_y = raw_value(y); // No implicit down-conversions
double deriv = raw_value(y.derivatives());
assert(deriv == 2*sin(pi/6)*cos(pi/6)); // FP ops match hand-code
Roy H. Stogner Metaprogramming April 16, 2012 18 / 52
Computational Modeling and Verification Automatic Differentiation
Operator-Overloaded Forward Differentiation
Example
typedef double RawType
// typedef ShadowNumber<double, long double> RawType;
typedef DualNumber<RawType,RawType> FirstDerivType;
const FirstDerivType x(pi/6,1); // Initializing independent var
const FirstDerivType sinx = sin(x); // Caching just like normal
const FirstDerivType y = sinx*sinx; // Smart pow would work too
const double raw_y = raw_value(y); // No implicit down-conversions
double deriv = raw_value(y.derivatives());
assert(deriv == 2*sin(pi/6)*cos(pi/6)); // FP ops match hand-code
Roy H. Stogner Metaprogramming April 16, 2012 18 / 52
Computational Modeling and Verification Automatic Differentiation
Operator-Overloaded Forward Differentiation
Example
typedef double RawType
// typedef ShadowNumber<double, long double> RawType;
typedef DualNumber<RawType,RawType> FirstDerivType;
const FirstDerivType x(pi/6,1); // Initializing independent var
const FirstDerivType sinx = sin(x); // Caching just like normal
const FirstDerivType y = sinx*sinx; // Smart pow would work too
const double raw_y = raw_value(y); // No implicit down-conversions
double deriv = raw_value(y.derivatives());
assert(deriv == 2*sin(pi/6)*cos(pi/6)); // FP ops match hand-code
Roy H. Stogner Metaprogramming April 16, 2012 18 / 52
Computational Modeling and Verification Automatic Differentiation
Operator-Overloaded Forward Differentiation
Example
typedef double RawType
// typedef ShadowNumber<double, long double> RawType;
typedef DualNumber<RawType,RawType> FirstDerivType;
const FirstDerivType x(pi/6,1); // Initializing independent var
const FirstDerivType sinx = sin(x); // Caching just like normal
const FirstDerivType y = sinx*sinx; // Smart pow would work too
const double raw_y = raw_value(y); // No implicit down-conversions
double deriv = raw_value(y.derivatives());
assert(deriv == 2*sin(pi/6)*cos(pi/6)); // FP ops match hand-code
Roy H. Stogner Metaprogramming April 16, 2012 18 / 52
Computational Modeling and Verification Automatic Differentiation
“Hyper-dual Numbers” - [Fike 2009]
• Add two new elements ε1, ε2 to R• Take the quotient with ε21 ≡ ε22 ≡ 0
H[R] ≡ {a+ bε1 + cε2 + dε1ε2 : a, b, c, d ∈ R}
• (a+ bε1 + cε2 + dε1ε2)× (e+ fε1 + gε2 + hε1ε2) =(ae) + (af + be)ε1 + (ag + ce)ε2 + (ah+ de+ bg + cf)ε1ε2
• Extending gives: for any function f(x),f(a+bε1+bε2+cε1ε2) = f(a)+bf ′(a)(ε1+ε2)+(b2f ′′(a)+cf ′(a))ε1ε2
• adding εiεj = 0 to quotient would give vector gradients instead...
Roy H. Stogner Metaprogramming April 16, 2012 19 / 52
Computational Modeling and Verification Automatic Differentiation
Hyper-dual Numbers via Recursive Mathematics
• RecallI D[X] ≡ {a+ bε : a, b ∈ X}I f(a+ bε) = f(a) + bf ′(a)ε
• Applied directly to R we get first derivatives• Applied recursively:
I D[D[R]] ≡ {a+ cε2 : a, c ∈ D[R]}I D[D[R]] ≡ {(a+ bε1) + (c+ dε1)ε2 : a, b, c, d ∈ R}I f((a+ bε1) + (c+ dε1)ε2) = f(a+ bε1) + (c+ dε1)f
′(a+ bε1)ε2I = f(a) + bf ′(a)ε1 + (c+ dε1)(f
′(a) + bf ′′(a)ε1)ε2I = f(a) + bf ′(a)ε1 + cf ′(a)ε2 + (cbf ′′(a) + df ′(a))ε1ε2
• In other words, D[D[R]] ≡ H[R]• This isomorphism works in software, too!
Roy H. Stogner Metaprogramming April 16, 2012 20 / 52
Computational Modeling and Verification Automatic Differentiation
Hessian Differentiation via Recursive Templates
Example
typedef DualNumber<RawType,NumberArray<3,RawType> > FAD1;
typedef DualNumber<FAD1,NumberArray<3,FAD1> > FAD2;
NumberArray<3, FAD2> xyz;
xyz[0] =
FAD2(x,NumberArrayUnitVector<3,0,RawType>::value());
xyz[1] =
FAD2(y,NumberArrayUnitVector<3,1,RawType>::value());
xyz[2] =
FAD2(z,NumberArrayUnitVector<3,2,RawType>::value());
const FAD2 u = some_function(xyz);
RawType laplacian_u = 0;
for (unsigned int i=0; i != 3; ++i)
laplacian_u += u.derivatives()[i].derivatives()[i];
Roy H. Stogner Metaprogramming April 16, 2012 21 / 52
Computational Modeling and Verification Automatic Differentiation
Hessian Differentiation via Recursive Templates
Example
typedef DualNumber<RawType,NumberArray<3,RawType> > FAD1;
typedef DualNumber<FAD1,NumberArray<3,FAD1> > FAD2;
NumberArray<3, FAD2> xyz;
xyz[0] =
FAD2(x,NumberArrayUnitVector<3,0,RawType>::value());
xyz[1] =
FAD2(y,NumberArrayUnitVector<3,1,RawType>::value());
xyz[2] =
FAD2(z,NumberArrayUnitVector<3,2,RawType>::value());
const FAD2 u = some_function(xyz);
RawType laplacian_u = 0;
for (unsigned int i=0; i != 3; ++i)
laplacian_u += u.derivatives()[i].derivatives()[i];
Roy H. Stogner Metaprogramming April 16, 2012 21 / 52
Computational Modeling and Verification Automatic Differentiation
Hessian Differentiation via Recursive Templates
Example
typedef DualNumber<RawType,NumberArray<3,RawType> > FAD1;
typedef DualNumber<FAD1,NumberArray<3,FAD1> > FAD2;
NumberArray<3, FAD2> xyz;
xyz[0] =
FAD2(x,NumberArrayUnitVector<3,0,RawType>::value());
xyz[1] =
FAD2(y,NumberArrayUnitVector<3,1,RawType>::value());
xyz[2] =
FAD2(z,NumberArrayUnitVector<3,2,RawType>::value());
const FAD2 u = some_function(xyz);
RawType laplacian_u = 0;
for (unsigned int i=0; i != 3; ++i)
laplacian_u += u.derivatives()[i].derivatives()[i];
Roy H. Stogner Metaprogramming April 16, 2012 21 / 52
Computational Modeling and Verification Automatic Differentiation
MASA PDE Examples
Manufactured Solution
// Arbitrary manufactured solution
U.template get<0>() = u_0 + u_x * sin(a_ux * PI * x / L) +
u_y * cos(a_uy * PI * y / L);
// Why not U[0] and U[1]? Explanation to come...
U.template get<1>() = v_0 + v_x * cos(a_vx * PI * x / L) +
v_y * sin(a_vy * PI * y / L);
ADScalar RHO = rho_0 + rho_x * sin(a_rhox * PI * x / L) +
rho_y * cos(a_rhoy * PI * y / L);
ADScalar P = p_0 + p_x * cos(a_px * PI * x / L) +
p_y * sin(a_py * PI * y / L);
// Constitutive laws
Tensor GradU = gradient(U);
Tensor Tau = mu * (GradU + transpose(GradU) -
2./3. * divergence(U) * RawArray::identity());
FullArray q = -k * T.derivatives();
Roy H. Stogner Metaprogramming April 16, 2012 22 / 52
Computational Modeling and Verification Automatic Differentiation
MASA PDE Examples
Euler
// Gas state
ADScalar T = P / RHO / R;
ADScalar E = 1. / (Gamma-1.) * P / RHO;
ADScalar ET = E + .5 * U.dot(U);
// Constitutive laws
Tensor GradU = gradient(U);
Tensor Tau = mu * (GradU + transpose(GradU) -
2./3. * divergence(U) * RawArray::identity());
FullArray q = -k * T.derivatives();
// Conservation equation residuals
Scalar Q_rho = raw_value(divergence(RHO*U));
RawArray Q_rho_u = raw_value(divergence(RHO*U.outerproduct(U)) +
P.derivatives());
Scalar Q_rho_e = raw_value(divergence((RHO*ET+P)*U));
Roy H. Stogner Metaprogramming April 16, 2012 22 / 52
Computational Modeling and Verification Automatic Differentiation
MASA PDE Examples
Navier-Stokes
// Gas state
ADScalar T = P / RHO / R;
ADScalar E = 1. / (Gamma-1.) * P / RHO;
ADScalar ET = E + .5 * U.dot(U);
// Constitutive laws
Tensor GradU = gradient(U);
Tensor Tau = mu * (GradU + transpose(GradU) -
2./3. * divergence(U) * RawArray::identity());
FullArray q = -k * T.derivatives();
// Conservation equation residuals
Scalar Q_rho = raw_value(divergence(RHO*U));
RawArray Q_rho_u = raw_value(divergence(RHO*U.outerproduct(U) - Tau) +
P.derivatives());
Scalar Q_rho_e = raw_value(divergence((RHO*ET+P)*U + q - Tau.dot(U)));
Roy H. Stogner Metaprogramming April 16, 2012 22 / 52
Computational Modeling and Verification Automatic Differentiation
Lessons Learned
• Automatic upconversion is dangerous!
Example
double problem(FirstDerivType x, SecondDerivType y) {
// This is desirable:
x *= 2.0;
// But now the compiler doesn’t see a problem with this!
return (x+y).derivatives().derivatives();
}
• Code with 2nd and higher derivatives still requires care
• For safe code: automatic upconversion only for built-in types,automatic downconversion for DualNumber, even to “HypodualNumber”
Roy H. Stogner Metaprogramming April 16, 2012 23 / 52
Computational Modeling and Verification Automatic Differentiation
Lessons Learned
• Automatic upconversion is dangerous!
Example
double problem(FirstDerivType x, SecondDerivType y) {
// This is desirable:
x *= 2.0;
// But now the compiler doesn’t see a problem with this!
return (x+y).derivatives().derivatives();
}
• Code with 2nd and higher derivatives still requires care
• For safe code: automatic upconversion only for built-in types,automatic downconversion for DualNumber, even to “HypodualNumber”
Roy H. Stogner Metaprogramming April 16, 2012 23 / 52
Computational Modeling and Verification Automatic Differentiation
Lessons Learned
• Automatic upconversion is dangerous!
Example
double problem(FirstDerivType x, SecondDerivType y) {
// This is desirable:
x *= 2.0;
// But now the compiler doesn’t see a problem with this!
return (x+y).derivatives().derivatives();
}
• Code with 2nd and higher derivatives still requires care
• For safe code: automatic upconversion only for built-in types,automatic downconversion for DualNumber, even to “HypodualNumber”
Roy H. Stogner Metaprogramming April 16, 2012 23 / 52
Computational Modeling and Verification Automatic Differentiation
Lessons Learned
• Automatic upconversion is dangerous!
Example
double problem(FirstDerivType x, SecondDerivType y) {
// This is desirable:
x *= 2.0;
// But now the compiler doesn’t see a problem with this!
return (x+y).derivatives().derivatives();
}
• Code with 2nd and higher derivatives still requires care
• For safe code: automatic upconversion only for built-in types,automatic downconversion for DualNumber, even to “HypodualNumber”
Roy H. Stogner Metaprogramming April 16, 2012 23 / 52
Computational Modeling and Verification Automatic Differentiation
Lessons Learned
• Automatic upconversion is dangerous!
Example
double problem(FirstDerivType x, SecondDerivType y) {
// This is desirable:
x *= 2.0;
// But now the compiler doesn’t see a problem with this!
return (x+y).derivatives().derivatives();
}
• Code with 2nd and higher derivatives still requires care
• For safe code: automatic upconversion only for built-in types,automatic downconversion for DualNumber, even to “HypodualNumber”
Roy H. Stogner Metaprogramming April 16, 2012 23 / 52
Computational Modeling and Verification Automatic Differentiation
Lessons Learned
• Hyperduals are inefficient!
I (1 +m)n data for n derivatives in m dimensions (e.g. (1 + 3)2 = 16)I Should require less than (e.g. 10)
n∑i=0
(m+ i− 1)!
(m− 1)!i!
• For efficient higher derivatives: partial specialization of recursiveDualNumber
• For efficient higher dimensions:
I Sparse vector data typesI Functional metaprogramming for efficient compile-time optimizationsI Functional-to-imperative APIs for run-time operations
Roy H. Stogner Metaprogramming April 16, 2012 24 / 52
Computational Modeling and Verification Automatic Differentiation
Lessons Learned
• Hyperduals are inefficient!I (1 +m)n data for n derivatives in m dimensions (e.g. (1 + 3)2 = 16)
I Should require less than (e.g. 10)
n∑i=0
(m+ i− 1)!
(m− 1)!i!
• For efficient higher derivatives: partial specialization of recursiveDualNumber
• For efficient higher dimensions:
I Sparse vector data typesI Functional metaprogramming for efficient compile-time optimizationsI Functional-to-imperative APIs for run-time operations
Roy H. Stogner Metaprogramming April 16, 2012 24 / 52
Computational Modeling and Verification Automatic Differentiation
Lessons Learned
• Hyperduals are inefficient!I (1 +m)n data for n derivatives in m dimensions (e.g. (1 + 3)2 = 16)I Should require less than (e.g. 10)
n∑i=0
(m+ i− 1)!
(m− 1)!i!
• For efficient higher derivatives: partial specialization of recursiveDualNumber
• For efficient higher dimensions:
I Sparse vector data typesI Functional metaprogramming for efficient compile-time optimizationsI Functional-to-imperative APIs for run-time operations
Roy H. Stogner Metaprogramming April 16, 2012 24 / 52
Computational Modeling and Verification Automatic Differentiation
Lessons Learned
• Hyperduals are inefficient!I (1 +m)n data for n derivatives in m dimensions (e.g. (1 + 3)2 = 16)I Should require less than (e.g. 10)
n∑i=0
(m+ i− 1)!
(m− 1)!i!
• For efficient higher derivatives: partial specialization of recursiveDualNumber
• For efficient higher dimensions:
I Sparse vector data typesI Functional metaprogramming for efficient compile-time optimizationsI Functional-to-imperative APIs for run-time operations
Roy H. Stogner Metaprogramming April 16, 2012 24 / 52
Computational Modeling and Verification Automatic Differentiation
Lessons Learned
• Hyperduals are inefficient!I (1 +m)n data for n derivatives in m dimensions (e.g. (1 + 3)2 = 16)I Should require less than (e.g. 10)
n∑i=0
(m+ i− 1)!
(m− 1)!i!
• For efficient higher derivatives: partial specialization of recursiveDualNumber
• For efficient higher dimensions:
I Sparse vector data typesI Functional metaprogramming for efficient compile-time optimizationsI Functional-to-imperative APIs for run-time operations
Roy H. Stogner Metaprogramming April 16, 2012 24 / 52
Computational Modeling and Verification Automatic Differentiation
Lessons Learned
• Hyperduals are inefficient!I (1 +m)n data for n derivatives in m dimensions (e.g. (1 + 3)2 = 16)I Should require less than (e.g. 10)
n∑i=0
(m+ i− 1)!
(m− 1)!i!
• For efficient higher derivatives: partial specialization of recursiveDualNumber
• For efficient higher dimensions:I Sparse vector data types
I Functional metaprogramming for efficient compile-time optimizationsI Functional-to-imperative APIs for run-time operations
Roy H. Stogner Metaprogramming April 16, 2012 24 / 52
Computational Modeling and Verification Automatic Differentiation
Lessons Learned
• Hyperduals are inefficient!I (1 +m)n data for n derivatives in m dimensions (e.g. (1 + 3)2 = 16)I Should require less than (e.g. 10)
n∑i=0
(m+ i− 1)!
(m− 1)!i!
• For efficient higher derivatives: partial specialization of recursiveDualNumber
• For efficient higher dimensions:I Sparse vector data typesI Functional metaprogramming for efficient compile-time optimizations
I Functional-to-imperative APIs for run-time operations
Roy H. Stogner Metaprogramming April 16, 2012 24 / 52
Computational Modeling and Verification Automatic Differentiation
Lessons Learned
• Hyperduals are inefficient!I (1 +m)n data for n derivatives in m dimensions (e.g. (1 + 3)2 = 16)I Should require less than (e.g. 10)
n∑i=0
(m+ i− 1)!
(m− 1)!i!
• For efficient higher derivatives: partial specialization of recursiveDualNumber
• For efficient higher dimensions:I Sparse vector data typesI Functional metaprogramming for efficient compile-time optimizationsI Functional-to-imperative APIs for run-time operations
Roy H. Stogner Metaprogramming April 16, 2012 24 / 52
Numeric Functional Metaprogramming Functional Data Structures
Data Structures with Templates: Type ContainerExample
template <typename HeadType,
typename TailContainer=NullContainer,
typename Comparison=ValueLessThan>
struct Container
{
typedef HeadType head_type; // car and cdr would scare
typedef TailContainer tail_set; // off the Fortran folks
typedef Comparison comparison;
// These may be empty types or may have data
HeadType head;
TailContainer tail;.........};
struct NullContainer
{.........};
Roy H. Stogner Metaprogramming April 16, 2012 25 / 52
Numeric Functional Metaprogramming Functional Data Structures
Data Structures with Templates: Type Container
Example
Container<IntType<1> > no_data_one_static_int;
Container<IntType<1>,
Container<IntType<4> > no_data_two_static_ints;
Container<IntType<1,float>,
Container<IntType<4,float> > float_data_two_static_ints;
Container<IntType<1,double>,
Container<IntType<4>,NumberArray<3,double> > heterogenous_data;
Roy H. Stogner Metaprogramming April 16, 2012 26 / 52
Numeric Functional Metaprogramming Functional Data Structures
Data Structures with Templates: Type Container
Example
Container<IntType<1> > no_data_one_static_int;
Container<IntType<1>,
Container<IntType<4> > no_data_two_static_ints;
Container<IntType<1,float>,
Container<IntType<4,float> > float_data_two_static_ints;
Container<IntType<1,double>,
Container<IntType<4>,NumberArray<3,double> > heterogenous_data;
Roy H. Stogner Metaprogramming April 16, 2012 26 / 52
Numeric Functional Metaprogramming Functional Data Structures
Data Structures with Templates: Type Container
Example
Container<IntType<1> > no_data_one_static_int;
Container<IntType<1>,
Container<IntType<4> > no_data_two_static_ints;
Container<IntType<1,float>,
Container<IntType<4,float> > float_data_two_static_ints;
Container<IntType<1,double>,
Container<IntType<4>,NumberArray<3,double> > heterogenous_data;
Roy H. Stogner Metaprogramming April 16, 2012 26 / 52
Numeric Functional Metaprogramming Functional Data Structures
Data Structures with Templates: Type Container
Example
Container<IntType<1> > no_data_one_static_int;
Container<IntType<1>,
Container<IntType<4> > no_data_two_static_ints;
Container<IntType<1,float>,
Container<IntType<4,float> > float_data_two_static_ints;
Container<IntType<1,double>,
Container<IntType<4>,NumberArray<3,double> > heterogenous_data;
Roy H. Stogner Metaprogramming April 16, 2012 26 / 52
Numeric Functional Metaprogramming Functional Data Structures
Set Operations: Insert
Inserting into a NullContainer is easy:
• If we’re inserting null, we’re still left with a NullContainer
• If not, we have a one-element container with the inserted element.
Examplestruct NullContainer
{.........template <typename ValueType, typename NewComparison>
struct Insert
{
typedef
typename IfElse<
(TypesEqual<ValueType,NullContainer>::value),
NullContainer,
Container<ValueType, NullContainer, NewComparison>
>::type type;
};.........};
C++ template syntax: karmic retribution for engineering majors whomocked CS majors’ Lisp syntax?
Roy H. Stogner Metaprogramming April 16, 2012 27 / 52
Numeric Functional Metaprogramming Functional Data Structures
Set Operations: Insert
Inserting into a NullContainer is easy:
• If we’re inserting null, we’re still left with a NullContainer
• If not, we have a one-element container with the inserted element.
Examplestruct NullContainer
{.........template <typename ValueType, typename NewComparison>
struct Insert
{
typedef
typename IfElse<
(TypesEqual<ValueType,NullContainer>::value),
NullContainer,
Container<ValueType, NullContainer, NewComparison>
>::type type;
};.........};
C++ template syntax: karmic retribution for engineering majors whomocked CS majors’ Lisp syntax?
Roy H. Stogner Metaprogramming April 16, 2012 27 / 52
Numeric Functional Metaprogramming Functional Data Structures
Set Operations: Insert
Inserting into a NullContainer is easy:
• If we’re inserting null, we’re still left with a NullContainer
• If not, we have a one-element container with the inserted element.
Examplestruct NullContainer
{.........template <typename ValueType, typename NewComparison>
struct Insert
{
typedef
typename IfElse<
(TypesEqual<ValueType,NullContainer>::value),
NullContainer,
Container<ValueType, NullContainer, NewComparison>
>::type type;
};.........};
C++ template syntax: karmic retribution for engineering majors whomocked CS majors’ Lisp syntax?
Roy H. Stogner Metaprogramming April 16, 2012 27 / 52
Numeric Functional Metaprogramming Functional Data Structures
Set Operations: Insert
Inserting into a NullContainer is easy:
• If we’re inserting null, we’re still left with a NullContainer
• If not, we have a one-element container with the inserted element.
Examplestruct NullContainer
{.........template <typename ValueType, typename NewComparison>
struct Insert
{
typedef
typename IfElse<
(TypesEqual<ValueType,NullContainer>::value),
NullContainer,
Container<ValueType, NullContainer, NewComparison>
>::type type;
};.........};
C++ template syntax: karmic retribution for engineering majors whomocked CS majors’ Lisp syntax?
Roy H. Stogner Metaprogramming April 16, 2012 27 / 52
Numeric Functional Metaprogramming Functional Data Structures
Set Operations: Insert
Inserting into a NullContainer is easy:
• If we’re inserting null, we’re still left with a NullContainer
• If not, we have a one-element container with the inserted element.
Examplestruct NullContainer
{.........template <typename ValueType, typename NewComparison>
struct Insert
{
typedef
typename IfElse<
(TypesEqual<ValueType,NullContainer>::value),
NullContainer,
Container<ValueType, NullContainer, NewComparison>
>::type type;
};.........};
C++ template syntax: karmic retribution for engineering majors whomocked CS majors’ Lisp syntax?
Roy H. Stogner Metaprogramming April 16, 2012 27 / 52
Numeric Functional Metaprogramming Functional Data Structures
Set Operations: Insert
Inserting in nonempty sorted sets is trickier:
• If we’re inserting null, we return our same set
• If we’re inserting a type smaller than our set head, then it becomesthe new head and our set becomes the new tail
• If we’re inserting a type larger than our set head, then the head staysunchanged and we insert into the tail
• If we’re inserting something equivalent to our set head, then the headmay get “upgraded”
• Templating around a comparison type to define “smaller”, “larger” and“equivalent” can give us set-like, multiset-like, or vector-like behavior
Roy H. Stogner Metaprogramming April 16, 2012 28 / 52
Numeric Functional Metaprogramming Functional Data Structures
Set Operations: Insert
Inserting in nonempty sorted sets is trickier:
• If we’re inserting null, we return our same set
• If we’re inserting a type smaller than our set head, then it becomesthe new head and our set becomes the new tail
• If we’re inserting a type larger than our set head, then the head staysunchanged and we insert into the tail
• If we’re inserting something equivalent to our set head, then the headmay get “upgraded”
• Templating around a comparison type to define “smaller”, “larger” and“equivalent” can give us set-like, multiset-like, or vector-like behavior
Roy H. Stogner Metaprogramming April 16, 2012 28 / 52
Numeric Functional Metaprogramming Functional Data Structures
Set Operations: Insert
Inserting in nonempty sorted sets is trickier:
• If we’re inserting null, we return our same set
• If we’re inserting a type smaller than our set head, then it becomesthe new head and our set becomes the new tail
• If we’re inserting a type larger than our set head, then the head staysunchanged and we insert into the tail
• If we’re inserting something equivalent to our set head, then the headmay get “upgraded”
• Templating around a comparison type to define “smaller”, “larger” and“equivalent” can give us set-like, multiset-like, or vector-like behavior
Roy H. Stogner Metaprogramming April 16, 2012 28 / 52
Numeric Functional Metaprogramming Functional Data Structures
Set Operations: Insert
Inserting in nonempty sorted sets is trickier:
• If we’re inserting null, we return our same set
• If we’re inserting a type smaller than our set head, then it becomesthe new head and our set becomes the new tail
• If we’re inserting a type larger than our set head, then the head staysunchanged and we insert into the tail
• If we’re inserting something equivalent to our set head, then the headmay get “upgraded”
• Templating around a comparison type to define “smaller”, “larger” and“equivalent” can give us set-like, multiset-like, or vector-like behavior
Roy H. Stogner Metaprogramming April 16, 2012 28 / 52
Numeric Functional Metaprogramming Functional Data Structures
Set Operations: Insert
Inserting in nonempty sorted sets is trickier:
• If we’re inserting null, we return our same set
• If we’re inserting a type smaller than our set head, then it becomesthe new head and our set becomes the new tail
• If we’re inserting a type larger than our set head, then the head staysunchanged and we insert into the tail
• If we’re inserting something equivalent to our set head, then the headmay get “upgraded”
• Templating around a comparison type to define “smaller”, “larger” and“equivalent” can give us set-like, multiset-like, or vector-like behavior
Roy H. Stogner Metaprogramming April 16, 2012 28 / 52
Numeric Functional Metaprogramming Functional Data Structures
Set Operations: Insert
Inserting in nonempty sorted sets is trickier:
• If we’re inserting null, we return our same set
• If we’re inserting a type smaller than our set head, then it becomesthe new head and our set becomes the new tail
• If we’re inserting a type larger than our set head, then the head staysunchanged and we insert into the tail
• If we’re inserting something equivalent to our set head, then the headmay get “upgraded”
• Templating around a comparison type to define “smaller”, “larger” and“equivalent” can give us set-like, multiset-like, or vector-like behavior
Roy H. Stogner Metaprogramming April 16, 2012 28 / 52
Numeric Functional Metaprogramming Functional Data Structures
Set Operations
• Union, Intersection, Difference with an empty container is trivial.
• With non-empty containers, Insert and Contains methods makerecursive definitions easy:
Example
template <typename Set2>
struct Union {
typedef typename
tail_set::Sorted::template Union<
typename Set2::Sorted::template Insert<
head_type
>::type
>::type type;
};
Roy H. Stogner Metaprogramming April 16, 2012 29 / 52
Numeric Functional Metaprogramming Functional Data Structures
Set Operations
• Union, Intersection, Difference with an empty container is trivial.
• With non-empty containers, Insert and Contains methods makerecursive definitions easy:
Example
template <typename Set2>
struct Union {
typedef typename
tail_set::Sorted::template Union<
typename Set2::Sorted::template Insert<
head_type
>::type
>::type type;
};
Roy H. Stogner Metaprogramming April 16, 2012 29 / 52
Numeric Functional Metaprogramming Functional Data Structures
Set Operations
• Union, Intersection, Difference with an empty container is trivial.
• With non-empty containers, Insert and Contains methods makerecursive definitions easy:
Example
template <typename Set2>
struct Union {
typedef typename
tail_set::Sorted::template Union<
typename Set2::Sorted::template Insert<
head_type
>::type
>::type type;
};
Roy H. Stogner Metaprogramming April 16, 2012 29 / 52
Numeric Functional Metaprogramming Functional Data Structures
Set Operations
• Union, Intersection, Difference with an empty container is trivial.
• With non-empty containers, Insert and Contains methods makerecursive definitions easy:
Example
template <typename Set2>
struct Union {
typedef typename
tail_set::Sorted::template Union<
typename Set2::Sorted::template Insert<
head_type
>::type
>::type type;
};
Roy H. Stogner Metaprogramming April 16, 2012 29 / 52
Numeric Functional Metaprogramming Functional Data Structures
Runtime Indexed ForEach
• Applying a standard functor to every set element
• Recursive compile-time metaprogram unrolls into a simple linearrun-time program
• f receives a non-compile-time-const v
Example
struct RuntimeForEach
{
template <typename Functor>
void operator()(const Functor &f) {
// f might want a reference, so we pass in a non-static copy
const typename head_type::value_type v = head_type::value;
f(v);
typename tail_set::RuntimeForEach()(f);
}
};
Roy H. Stogner Metaprogramming April 16, 2012 30 / 52
Numeric Functional Metaprogramming Functional Data Structures
Compile-time Indexed ForEach
• Applying a “template functor” to every set element
• Recursive compile-time metaprogram unrolls into a simple linearrun-time program
• f receives a compile-time-const HeadType
Example
struct ForEach
{
template <typename Functor>
void operator()(const Functor &f) {
f.operator()<HeadType>(v);
typename tail_set::ForEach()(f);
}
};
Roy H. Stogner Metaprogramming April 16, 2012 31 / 52
Numeric Functional Metaprogramming Sparse Data Structures
Operator-Overloaded Sparse Vectors• An IndexSet lists sparse indices with potentially non-zero data• e.g. the vector [0 0 1 0 2 0] will have {3, 5} or a superset as index set
• Const-index O (1) access via template arguments
Example
template <typename T, typename IndexSet>
class SparseNumberArray {
public:
typedef IndexSet index_set;
static const unsigned int size = IndexSet::size;.........
T& raw_at(unsigned int i)
{ return _data[i]; }
.........private:
T _data[size];
};
Roy H. Stogner Metaprogramming April 16, 2012 32 / 52
Numeric Functional Metaprogramming Sparse Data Structures
Operator-Overloaded Sparse Vectors• An IndexSet lists sparse indices with potentially non-zero data• Variable-index component access via runtime search through indices
• Const-index O (1) access via template arguments
Example
template <typename T, typename IndexSet>
class SparseNumberArray {
public:
typedef IndexSet index_set;
static const unsigned int size = IndexSet::size;.........
T& operator[](index_value_type i)
{ return _data[IndexSet::runtime_index_of(i)]; }
.........private:
T _data[size];
};
Roy H. Stogner Metaprogramming April 16, 2012 32 / 52
Numeric Functional Metaprogramming Sparse Data Structures
Operator-Overloaded Sparse Vectors• An IndexSet lists sparse indices with potentially non-zero data• Variable-index component access via runtime search through indices• Const-index O (1) access via template arguments
Example
template <typename T, typename IndexSet>
class SparseNumberArray {
public:
typedef IndexSet index_set;
static const unsigned int size = IndexSet::size;.........template <unsigned int i>
typename entry_type<i>::type& get() {
return _data[IndexSet::template IndexOf<i>::index];
}.........private:
T _data[size];
};
Roy H. Stogner Metaprogramming April 16, 2012 32 / 52
Numeric Functional Metaprogramming Sparse Data Structures
Operator-Overloaded Sparse Vector OperationsSome operations are easily defined, no index set operations needed:
• Multiplication by a constant: for loop over all data, multiply datum
Other operations require simple set operations:
• Element-wise addition takes a Union of index sets
• Element-wise multiplication takes an Intersection of index sets
• Element-wise division takes the numerator’s index set, but asserts(compile-time!) that the divisor’s is a superset.
Others require repeated set operations:
• Consider a typical binary function, max(a,b) , applied element-wise totwo sparse vectors:
• For each element in the intersection of the index sets, we apply thefunction to the corresponding pair of vector data.
• For each element in each asymmetric difference of the index sets, weapply the function to a pair of the present vector datum and 0.
Roy H. Stogner Metaprogramming April 16, 2012 33 / 52
Numeric Functional Metaprogramming Sparse Data Structures
Operator-Overloaded Sparse Vector OperationsSome operations are easily defined, no index set operations needed:
• Multiplication by a constant: for loop over all data, multiply datum
Other operations require simple set operations:
• Element-wise addition takes a Union of index sets
• Element-wise multiplication takes an Intersection of index sets
• Element-wise division takes the numerator’s index set, but asserts(compile-time!) that the divisor’s is a superset.
Others require repeated set operations:
• Consider a typical binary function, max(a,b) , applied element-wise totwo sparse vectors:
• For each element in the intersection of the index sets, we apply thefunction to the corresponding pair of vector data.
• For each element in each asymmetric difference of the index sets, weapply the function to a pair of the present vector datum and 0.
Roy H. Stogner Metaprogramming April 16, 2012 33 / 52
Numeric Functional Metaprogramming Sparse Data Structures
Operator-Overloaded Sparse Vector OperationsSome operations are easily defined, no index set operations needed:
• Multiplication by a constant: for loop over all data, multiply datum
Other operations require simple set operations:
• Element-wise addition takes a Union of index sets
• Element-wise multiplication takes an Intersection of index sets
• Element-wise division takes the numerator’s index set, but asserts(compile-time!) that the divisor’s is a superset.
Others require repeated set operations:
• Consider a typical binary function, max(a,b) , applied element-wise totwo sparse vectors:
• For each element in the intersection of the index sets, we apply thefunction to the corresponding pair of vector data.
• For each element in each asymmetric difference of the index sets, weapply the function to a pair of the present vector datum and 0.
Roy H. Stogner Metaprogramming April 16, 2012 33 / 52
Numeric Functional Metaprogramming Sparse Data Structures
Operator-Overloaded Sparse Vector OperationsSome operations are easily defined, no index set operations needed:
• Multiplication by a constant: for loop over all data, multiply datum
Other operations require simple set operations:
• Element-wise addition takes a Union of index sets
• Element-wise multiplication takes an Intersection of index sets
• Element-wise division takes the numerator’s index set, but asserts(compile-time!) that the divisor’s is a superset.
Others require repeated set operations:
• Consider a typical binary function, max(a,b) , applied element-wise totwo sparse vectors:
• For each element in the intersection of the index sets, we apply thefunction to the corresponding pair of vector data.
• For each element in each asymmetric difference of the index sets, weapply the function to a pair of the present vector datum and 0.
Roy H. Stogner Metaprogramming April 16, 2012 33 / 52
Numeric Functional Metaprogramming Sparse Data Structures
Operator-Overloaded Sparse Vector OperationsSome operations are easily defined, no index set operations needed:
• Multiplication by a constant: for loop over all data, multiply datum
Other operations require simple set operations:
• Element-wise addition takes a Union of index sets
• Element-wise multiplication takes an Intersection of index sets
• Element-wise division takes the numerator’s index set, but asserts(compile-time!) that the divisor’s is a superset.
Others require repeated set operations:
• Consider a typical binary function, max(a,b) , applied element-wise totwo sparse vectors:
• For each element in the intersection of the index sets, we apply thefunction to the corresponding pair of vector data.
• For each element in each asymmetric difference of the index sets, weapply the function to a pair of the present vector datum and 0.
Roy H. Stogner Metaprogramming April 16, 2012 33 / 52
Numeric Functional Metaprogramming Sparse Data Structures
Operator-Overloaded Sparse Vector OperationsSome operations are easily defined, no index set operations needed:
• Multiplication by a constant: for loop over all data, multiply datum
Other operations require simple set operations:
• Element-wise addition takes a Union of index sets
• Element-wise multiplication takes an Intersection of index sets
• Element-wise division takes the numerator’s index set, but asserts(compile-time!) that the divisor’s is a superset.
Others require repeated set operations:
• Consider a typical binary function, max(a,b) , applied element-wise totwo sparse vectors:
• For each element in the intersection of the index sets, we apply thefunction to the corresponding pair of vector data.
• For each element in each asymmetric difference of the index sets, weapply the function to a pair of the present vector datum and 0.
Roy H. Stogner Metaprogramming April 16, 2012 33 / 52
Numeric Functional Metaprogramming Sparse Data Structures
Operator-Overloaded Sparse Vector OperationsSome operations are easily defined, no index set operations needed:
• Multiplication by a constant: for loop over all data, multiply datum
Other operations require simple set operations:
• Element-wise addition takes a Union of index sets
• Element-wise multiplication takes an Intersection of index sets
• Element-wise division takes the numerator’s index set, but asserts(compile-time!) that the divisor’s is a superset.
Others require repeated set operations:
• Consider a typical binary function, max(a,b) , applied element-wise totwo sparse vectors:
• For each element in the intersection of the index sets, we apply thefunction to the corresponding pair of vector data.
• For each element in each asymmetric difference of the index sets, weapply the function to a pair of the present vector datum and 0.
Roy H. Stogner Metaprogramming April 16, 2012 33 / 52
Numeric Functional Metaprogramming Sparse Data Structures
Operator-Overloaded Sparse Vectors
Consider recursion with dual numbers, for automatic differentiation
• x is only non-constant w.r.t. x; y is only non-constant w.r.t. y
• DualNumber x and y are given unit vector gradients
• Their sums, products, etc. then all have optimally sparse gradients.
We now have excellent sparse vectors-of-T!
[0 0 0 0 2 0] + [0 0 1 0 0 0] = [0 0 1 0 2 0]
[· · · · 2 ·] + [· · 1 · · ·] = [· · 1 · 2 ·]
• From 6 FLOPs down to 0
• But not good enough yet
Roy H. Stogner Metaprogramming April 16, 2012 34 / 52
Numeric Functional Metaprogramming Sparse Data Structures
Operator-Overloaded Sparse Vectors
Consider recursion with dual numbers, for automatic differentiation
• x is only non-constant w.r.t. x; y is only non-constant w.r.t. y
• DualNumber x and y are given unit vector gradients
• Their sums, products, etc. then all have optimally sparse gradients.
We now have excellent sparse vectors-of-T!
[0 0 0 0 2 0] + [0 0 1 0 0 0] = [0 0 1 0 2 0]
[· · · · 2 ·] + [· · 1 · · ·] = [· · 1 · 2 ·]
• From 6 FLOPs down to 0
• But not good enough yet
Roy H. Stogner Metaprogramming April 16, 2012 34 / 52
Numeric Functional Metaprogramming Sparse Data Structures
Operator-Overloaded Sparse Vectors
Consider recursion with dual numbers, for automatic differentiation
• x is only non-constant w.r.t. x; y is only non-constant w.r.t. y
• DualNumber x and y are given unit vector gradients
• Their sums, products, etc. then all have optimally sparse gradients.
We now have excellent sparse vectors-of-T!
[0 0 0 0 2 0] + [0 0 1 0 0 0] = [0 0 1 0 2 0]
[· · · · 2 ·] + [· · 1 · · ·] = [· · 1 · 2 ·]
• From 6 FLOPs down to 0
• But not good enough yet
Roy H. Stogner Metaprogramming April 16, 2012 34 / 52
Numeric Functional Metaprogramming Sparse Data Structures
Operator-Overloaded Sparse Vectors
Consider recursion with dual numbers, for automatic differentiation
• x is only non-constant w.r.t. x; y is only non-constant w.r.t. y
• DualNumber x and y are given unit vector gradients
• Their sums, products, etc. then all have optimally sparse gradients.
We now have excellent sparse vectors-of-T!
[0 0 0 0 2 0] + [0 0 1 0 0 0] = [0 0 1 0 2 0]
[· · · · 2 ·] + [· · 1 · · ·] = [· · 1 · 2 ·]
• From 6 FLOPs down to 0
• But not good enough yet
Roy H. Stogner Metaprogramming April 16, 2012 34 / 52
Numeric Functional Metaprogramming Sparse Data Structures
Operator-Overloaded Sparse Vectors
Consider recursion with dual numbers, for automatic differentiation
• x is only non-constant w.r.t. x; y is only non-constant w.r.t. y
• DualNumber x and y are given unit vector gradients
• Their sums, products, etc. then all have optimally sparse gradients.
We now have excellent sparse vectors-of-T!
[0 0 0 0 2 0] + [0 0 1 0 0 0] = [0 0 1 0 2 0]
[· · · · 2 ·] + [· · 1 · · ·]
= [· · 1 · 2 ·]
• From 6 FLOPs down to 0
• But not good enough yet
Roy H. Stogner Metaprogramming April 16, 2012 34 / 52
Numeric Functional Metaprogramming Sparse Data Structures
Operator-Overloaded Sparse Vectors
Consider recursion with dual numbers, for automatic differentiation
• x is only non-constant w.r.t. x; y is only non-constant w.r.t. y
• DualNumber x and y are given unit vector gradients
• Their sums, products, etc. then all have optimally sparse gradients.
We now have excellent sparse vectors-of-T!
[0 0 0 0 2 0] + [0 0 1 0 0 0] = [0 0 1 0 2 0]
[· · · · 2 ·] + [· · 1 · · ·] = [· · 1 · 2 ·]
• From 6 FLOPs down to 0
• But not good enough yet
Roy H. Stogner Metaprogramming April 16, 2012 34 / 52
Numeric Functional Metaprogramming Sparse Data Structures
Operator-Overloaded Sparse Vectors
Consider recursion with dual numbers, for automatic differentiation
• x is only non-constant w.r.t. x; y is only non-constant w.r.t. y
• DualNumber x and y are given unit vector gradients
• Their sums, products, etc. then all have optimally sparse gradients.
We now have excellent sparse vectors-of-T!
[0 0 0 0 2 0] + [0 0 1 0 0 0] = [0 0 1 0 2 0]
[· · · · 2 ·] + [· · 1 · · ·] = [· · 1 · 2 ·]
• From 6 FLOPs down to 0
• But not good enough yet
Roy H. Stogner Metaprogramming April 16, 2012 34 / 52
Numeric Functional Metaprogramming Sparse Data Structures
Operator-Overloaded Sparse Vectors
Consider recursion with dual numbers, for automatic differentiation
• x is only non-constant w.r.t. x; y is only non-constant w.r.t. y
• DualNumber x and y are given unit vector gradients
• Their sums, products, etc. then all have optimally sparse gradients.
We now have excellent sparse vectors-of-T!
[0 0 0 0 2 0] + [0 0 1 0 0 0] = [0 0 1 0 2 0]
[· · · · 2 ·] + [· · 1 · · ·] = [· · 1 · 2 ·]
• From 6 FLOPs down to 0
• But not good enough yet
Roy H. Stogner Metaprogramming April 16, 2012 34 / 52
Numeric Functional Metaprogramming Sparse Data Structures
Operator-Overloaded Sparse StructuresConsider recursion with arrays, for matrices, hessians
0 0 0 00 0 2 00 0 0 00 0 0 0
+
0 0 0 00 0 0 00 0 0 01 0 0 0
=
0 0 0 00 0 2 00 0 0 01 0 0 0
· · · ·· · 2 ·· · · ·· · · ·
+
· · · ·· · · ·· · · ·1 · · ·
=
· · · ·0 · 2 ·· · · ·1 · 0 ·
• A SparseNumberArray-of-SparseNumberArray is sparse...• But only with a tensor-product sparsity structure.
I Triangular matrices become full?I Diagonal matrices become full??
Roy H. Stogner Metaprogramming April 16, 2012 35 / 52
Numeric Functional Metaprogramming Sparse Data Structures
Operator-Overloaded Sparse StructuresConsider recursion with arrays, for matrices, hessians
0 0 0 00 0 2 00 0 0 00 0 0 0
+
0 0 0 00 0 0 00 0 0 01 0 0 0
=
0 0 0 00 0 2 00 0 0 01 0 0 0
· · · ·· · 2 ·· · · ·· · · ·
+
· · · ·· · · ·· · · ·1 · · ·
=
· · · ·0 · 2 ·· · · ·1 · 0 ·
• A SparseNumberArray-of-SparseNumberArray is sparse...• But only with a tensor-product sparsity structure.
I Triangular matrices become full?I Diagonal matrices become full??
Roy H. Stogner Metaprogramming April 16, 2012 35 / 52
Numeric Functional Metaprogramming Sparse Data Structures
Operator-Overloaded Sparse StructuresConsider recursion with arrays, for matrices, hessians
0 0 0 00 0 2 00 0 0 00 0 0 0
+
0 0 0 00 0 0 00 0 0 01 0 0 0
=
0 0 0 00 0 2 00 0 0 01 0 0 0
· · · ·· · 2 ·· · · ·· · · ·
+
· · · ·· · · ·· · · ·1 · · ·
=
· · · ·0 · 2 ·· · · ·1 · 0 ·
• A SparseNumberArray-of-SparseNumberArray is sparse...• But only with a tensor-product sparsity structure.
I Triangular matrices become full?I Diagonal matrices become full??
Roy H. Stogner Metaprogramming April 16, 2012 35 / 52
Numeric Functional Metaprogramming Sparse Data Structures
Operator-Overloaded Sparse StructuresConsider recursion with arrays, for matrices, hessians
0 0 0 00 0 2 00 0 0 00 0 0 0
+
0 0 0 00 0 0 00 0 0 01 0 0 0
=
0 0 0 00 0 2 00 0 0 01 0 0 0
· · · ·· · 2 ·· · · ·· · · ·
+
· · · ·· · · ·· · · ·1 · · ·
=
· · · ·0 · 2 ·· · · ·1 · 0 ·
• A SparseNumberArray-of-SparseNumberArray is sparse...• But only with a tensor-product sparsity structure.
I Triangular matrices become full?I Diagonal matrices become full??
Roy H. Stogner Metaprogramming April 16, 2012 35 / 52
Numeric Functional Metaprogramming Sparse Data Structures
Operator-Overloaded Sparse StructuresConsider recursion with arrays, for matrices, hessians
0 0 0 00 0 2 00 0 0 00 0 0 0
+
0 0 0 00 0 0 00 0 0 01 0 0 0
=
0 0 0 00 0 2 00 0 0 01 0 0 0
· · · ·· · 2 ·· · · ·· · · ·
+
· · · ·· · · ·· · · ·1 · · ·
=
· · · ·0 · 2 ·· · · ·1 · 0 ·
• A SparseNumberArray-of-SparseNumberArray is sparse...• But only with a tensor-product sparsity structure.
I Triangular matrices become full?I Diagonal matrices become full??
Roy H. Stogner Metaprogramming April 16, 2012 35 / 52
Numeric Functional Metaprogramming Sparse Data Structures
Operator-Overloaded Sparse Structures• An IndexSet now stores data, not just static indices
• Index-dependent data type makes variable-index component accessimpossible!
Example
template <typename IndexSet>
class SparseNumberStruct {
public:
typedef IndexSet index_set;
static const unsigned int size = IndexSet::size;
template <unsigned int i>
typename entry_type<i>::type& get() {
return _data.template data<UnsignedIntType<i> >(); }.........private:
IndexSet _data;
};
Roy H. Stogner Metaprogramming April 16, 2012 36 / 52
Numeric Functional Metaprogramming Sparse Data Structures
Operator-Overloaded Sparse Structures• An IndexSet now stores data, not just static indices• Index-dependent data type makes variable-index component access
impossible!
Example
template <typename IndexSet>
class SparseNumberStruct {
public:
typedef IndexSet index_set;
static const unsigned int size = IndexSet::size;
template <unsigned int i>
typename entry_type<i>::type& get() {
return _data.template data<UnsignedIntType<i> >(); }.........private:
IndexSet _data;
};
Roy H. Stogner Metaprogramming April 16, 2012 36 / 52
Numeric Functional Metaprogramming Sparse Data Structures
Operator-Overloaded Sparse Structure Operations
No operations are easily defined.
• C for loops don’t work over heterogenous structures.
• Templated functors must be defined for every operation.
Multiplication by a constant, using a functor and subfunctor:
Exampletemplate <typename T2, typename IndexSet>
inline
typename MultipliesType<SparseNumberStruct<IndexSet>,T2>::supertype
operator * (const SparseNumberStruct<IndexSet>& a, const T2& b)
{
typedef typename MultipliesType<SparseNumberStruct<IndexSet>,T2>::supertype type;
typename CompareTypes<SparseNumberStruct<IndexSet>,T2>::supertype
returnval;
typename IndexSet::ForEach()
(BinaryFunctor<MultipliesSubfunctor, IndexSet, ConstantDataSet<T2>, typename type::index_set>
(MultipliesSubfunctor(), a.raw_data(), ConstantDataSet<T2>(b), returnval.raw_data()));
return returnval;
}
Roy H. Stogner Metaprogramming April 16, 2012 37 / 52
Numeric Functional Metaprogramming Sparse Data Structures
Operator-Overloaded Sparse Structure Operations
No operations are easily defined.
• C for loops don’t work over heterogenous structures.
• Templated functors must be defined for every operation.
Multiplication by a constant, using a functor and subfunctor:
Exampletemplate <typename T2, typename IndexSet>
inline
typename MultipliesType<SparseNumberStruct<IndexSet>,T2>::supertype
operator * (const SparseNumberStruct<IndexSet>& a, const T2& b)
{
typedef typename MultipliesType<SparseNumberStruct<IndexSet>,T2>::supertype type;
typename CompareTypes<SparseNumberStruct<IndexSet>,T2>::supertype
returnval;
typename IndexSet::ForEach()
(BinaryFunctor<MultipliesSubfunctor, IndexSet, ConstantDataSet<T2>, typename type::index_set>
(MultipliesSubfunctor(), a.raw_data(), ConstantDataSet<T2>(b), returnval.raw_data()));
return returnval;
}
Roy H. Stogner Metaprogramming April 16, 2012 37 / 52
Numeric Functional Metaprogramming Sparse Data Structures
Operator-Overloaded Sparse Structure Operations
No operations are easily defined.
• C for loops don’t work over heterogenous structures.
• Templated functors must be defined for every operation.
Multiplication by a constant, using a functor and subfunctor:
Exampletemplate <typename T2, typename IndexSet>
inline
typename MultipliesType<SparseNumberStruct<IndexSet>,T2>::supertype
operator * (const SparseNumberStruct<IndexSet>& a, const T2& b)
{
typedef typename MultipliesType<SparseNumberStruct<IndexSet>,T2>::supertype type;
typename CompareTypes<SparseNumberStruct<IndexSet>,T2>::supertype
returnval;
typename IndexSet::ForEach()
(BinaryFunctor<MultipliesSubfunctor, IndexSet, ConstantDataSet<T2>, typename type::index_set>
(MultipliesSubfunctor(), a.raw_data(), ConstantDataSet<T2>(b), returnval.raw_data()));
return returnval;
}
Roy H. Stogner Metaprogramming April 16, 2012 37 / 52
Numeric Functional Metaprogramming Sparse Data Structures
Operator-Overloaded Sparse Structure Operations
No operations are easily defined.
• C for loops don’t work over heterogenous structures.
• Templated functors must be defined for every operation.
Multiplication by a constant, using a functor and subfunctor:
Exampletemplate <typename T2, typename IndexSet>
inline
typename MultipliesType<SparseNumberStruct<IndexSet>,T2>::supertype
operator * (const SparseNumberStruct<IndexSet>& a, const T2& b)
{
typedef typename MultipliesType<SparseNumberStruct<IndexSet>,T2>::supertype type;
typename CompareTypes<SparseNumberStruct<IndexSet>,T2>::supertype
returnval;
typename IndexSet::ForEach()
(BinaryFunctor<MultipliesSubfunctor, IndexSet, ConstantDataSet<T2>, typename type::index_set>
(MultipliesSubfunctor(), a.raw_data(), ConstantDataSet<T2>(b), returnval.raw_data()));
return returnval;
}
Roy H. Stogner Metaprogramming April 16, 2012 37 / 52
Numeric Functional Metaprogramming Sparse Data Structures
Operator-Overloaded Sparse Structure Operations
No operations are easily defined.
• C for loops don’t work over heterogenous structures.
• Templated functors must be defined for every operation.
Multiplication by a constant, using a functor and subfunctor:
Exampletemplate <typename T2, typename IndexSet>
inline
typename MultipliesType<SparseNumberStruct<IndexSet>,T2>::supertype
operator * (const SparseNumberStruct<IndexSet>& a, const T2& b)
{
typedef typename MultipliesType<SparseNumberStruct<IndexSet>,T2>::supertype type;
typename CompareTypes<SparseNumberStruct<IndexSet>,T2>::supertype
returnval;
typename IndexSet::ForEach()
(BinaryFunctor<MultipliesSubfunctor, IndexSet, ConstantDataSet<T2>, typename type::index_set>
(MultipliesSubfunctor(), a.raw_data(), ConstantDataSet<T2>(b), returnval.raw_data()));
return returnval;
}
Roy H. Stogner Metaprogramming April 16, 2012 37 / 52
Numeric Functional Metaprogramming Sparse Data Structures
Operator-Overloaded Sparse Structure Operations
No operations are easily defined.
• C for loops don’t work over heterogenous structures.
• Templated functors must be defined for every operation.
Multiplication by a constant, using a functor and subfunctor:
Exampletemplate <typename T2, typename IndexSet>
inline
typename MultipliesType<SparseNumberStruct<IndexSet>,T2>::supertype
operator * (const SparseNumberStruct<IndexSet>& a, const T2& b)
{
typedef typename MultipliesType<SparseNumberStruct<IndexSet>,T2>::supertype type;
typename CompareTypes<SparseNumberStruct<IndexSet>,T2>::supertype
returnval;
typename IndexSet::ForEach()
(BinaryFunctor<MultipliesSubfunctor, IndexSet, ConstantDataSet<T2>, typename type::index_set>
(MultipliesSubfunctor(), a.raw_data(), ConstantDataSet<T2>(b), returnval.raw_data()));
return returnval;
}
Roy H. Stogner Metaprogramming April 16, 2012 37 / 52
Numeric Functional Metaprogramming Sparse Data Structures
Templated Functors• A subfunctor class represents a single operation
• Template overloading of operator() accepts arbitrary input types• CompareTypes or similar classes (or C++11 decltype) provides
proper return types
Example
struct MultipliesSubfunctor {
template <typename T1, typename T2>
struct Return {
typedef typename
SymmetricMultipliesType<T1,T2>::supertype type;
};
template <typename T1, typename T2>
typename Return<T1,T2>::type
operator()(const T1& x, const T2& y) const { return x * y; }
};
Roy H. Stogner Metaprogramming April 16, 2012 38 / 52
Numeric Functional Metaprogramming Sparse Data Structures
Templated Functors• A subfunctor class represents a single operation• Template overloading of operator() accepts arbitrary input types
• CompareTypes or similar classes (or C++11 decltype) providesproper return types
Example
struct MultipliesSubfunctor {
template <typename T1, typename T2>
struct Return {
typedef typename
SymmetricMultipliesType<T1,T2>::supertype type;
};
template <typename T1, typename T2>
typename Return<T1,T2>::type
operator()(const T1& x, const T2& y) const { return x * y; }
};
Roy H. Stogner Metaprogramming April 16, 2012 38 / 52
Numeric Functional Metaprogramming Sparse Data Structures
Templated Functors• A subfunctor class represents a single operation• Template overloading of operator() accepts arbitrary input types• CompareTypes or similar classes (or C++11 decltype) provides
proper return types
Example
struct MultipliesSubfunctor {
template <typename T1, typename T2>
struct Return {
typedef typename
SymmetricMultipliesType<T1,T2>::supertype type;
};
template <typename T1, typename T2>
typename Return<T1,T2>::type
operator()(const T1& x, const T2& y) const { return x * y; }
};
Roy H. Stogner Metaprogramming April 16, 2012 38 / 52
Numeric Functional Metaprogramming Sparse Data Structures
Templated Functors• A subfunctor class represents a single operation• Template overloading of operator() accepts arbitrary input types• CompareTypes or similar classes (or C++11 decltype) provides
proper return types
Example
struct MultipliesSubfunctor {
template <typename T1, typename T2>
struct Return {
typedef typename
SymmetricMultipliesType<T1,T2>::supertype type;
};
template <typename T1, typename T2>
typename Return<T1,T2>::type
operator()(const T1& x, const T2& y) const { return x * y; }
};
Roy H. Stogner Metaprogramming April 16, 2012 38 / 52
Numeric Functional Metaprogramming Sparse Data Structures
Templated Functors• A subfunctor class represents a single operation• Template overloading of operator() accepts arbitrary input types• CompareTypes or similar classes (or C++11 decltype) provides
proper return types
Example
struct MultipliesSubfunctor {
template <typename T1, typename T2>
struct Return {
typedef typename
SymmetricMultipliesType<T1,T2>::supertype type;
};
template <typename T1, typename T2>
typename Return<T1,T2>::type
operator()(const T1& x, const T2& y) const { return x * y; }
};
Roy H. Stogner Metaprogramming April 16, 2012 38 / 52
Numeric Functional Metaprogramming Sparse Data Structures
Sparse Matrix Operations
Example
A ≡[a bT
c D
]{L,U} ≡ {LUL(A), LUU (A)}
L =
[a 0T
c LUL(D− ca−1bT)
]U =
[1 a−1bT
0 LUU (D− ca−1bT)
]Then apply to sparse arrays of arrays:
• One compile-time pass to allocate output structure
• One run-time pass to do calculations
• Asymptotic complexity drops beautifully
• But now we have asymptotic code generation to worry about!
Roy H. Stogner Metaprogramming April 16, 2012 39 / 52
Numeric Functional Metaprogramming Sparse Data Structures
Sparse Matrix Operations
Example
A ≡[a bT
c D
]
{L,U} ≡ {LUL(A), LUU (A)}
L =
[a 0T
c LUL(D− ca−1bT)
]U =
[1 a−1bT
0 LUU (D− ca−1bT)
]Then apply to sparse arrays of arrays:
• One compile-time pass to allocate output structure
• One run-time pass to do calculations
• Asymptotic complexity drops beautifully
• But now we have asymptotic code generation to worry about!
Roy H. Stogner Metaprogramming April 16, 2012 39 / 52
Numeric Functional Metaprogramming Sparse Data Structures
Sparse Matrix Operations
Example
A ≡[a bT
c D
]{L,U} ≡ {LUL(A), LUU (A)}
L =
[a 0T
c LUL(D− ca−1bT)
]U =
[1 a−1bT
0 LUU (D− ca−1bT)
]Then apply to sparse arrays of arrays:
• One compile-time pass to allocate output structure
• One run-time pass to do calculations
• Asymptotic complexity drops beautifully
• But now we have asymptotic code generation to worry about!
Roy H. Stogner Metaprogramming April 16, 2012 39 / 52
Numeric Functional Metaprogramming Sparse Data Structures
Sparse Matrix Operations
Example
A ≡[a bT
c D
]{L,U} ≡ {LUL(A), LUU (A)}
L =
[a 0T
c LUL(D− ca−1bT)
]U =
[1 a−1bT
0 LUU (D− ca−1bT)
]
Then apply to sparse arrays of arrays:
• One compile-time pass to allocate output structure
• One run-time pass to do calculations
• Asymptotic complexity drops beautifully
• But now we have asymptotic code generation to worry about!
Roy H. Stogner Metaprogramming April 16, 2012 39 / 52
Numeric Functional Metaprogramming Sparse Data Structures
Sparse Matrix Operations
Example
A ≡[a bT
c D
]{L,U} ≡ {LUL(A), LUU (A)}
L =
[a 0T
c LUL(D− ca−1bT)
]U =
[1 a−1bT
0 LUU (D− ca−1bT)
]Then apply to sparse arrays of arrays:
• One compile-time pass to allocate output structure
• One run-time pass to do calculations
• Asymptotic complexity drops beautifully
• But now we have asymptotic code generation to worry about!
Roy H. Stogner Metaprogramming April 16, 2012 39 / 52
Numeric Functional Metaprogramming Sparse Data Structures
Sparse Matrix Operations
Example
A ≡[a bT
c D
]{L,U} ≡ {LUL(A), LUU (A)}
L =
[a 0T
c LUL(D− ca−1bT)
]U =
[1 a−1bT
0 LUU (D− ca−1bT)
]Then apply to sparse arrays of arrays:
• One compile-time pass to allocate output structure
• One run-time pass to do calculations
• Asymptotic complexity drops beautifully
• But now we have asymptotic code generation to worry about!
Roy H. Stogner Metaprogramming April 16, 2012 39 / 52
Numeric Functional Metaprogramming Sparse Data Structures
Sparse Matrix Operations
Example
A ≡[a bT
c D
]{L,U} ≡ {LUL(A), LUU (A)}
L =
[a 0T
c LUL(D− ca−1bT)
]U =
[1 a−1bT
0 LUU (D− ca−1bT)
]Then apply to sparse arrays of arrays:
• One compile-time pass to allocate output structure
• One run-time pass to do calculations
• Asymptotic complexity drops beautifully
• But now we have asymptotic code generation to worry about!
Roy H. Stogner Metaprogramming April 16, 2012 39 / 52
Numeric Functional Metaprogramming Sparse Data Structures
Sparse Matrix Operations
Example
A ≡[a bT
c D
]{L,U} ≡ {LUL(A), LUU (A)}
L =
[a 0T
c LUL(D− ca−1bT)
]U =
[1 a−1bT
0 LUU (D− ca−1bT)
]Then apply to sparse arrays of arrays:
• One compile-time pass to allocate output structure
• One run-time pass to do calculations
• Asymptotic complexity drops beautifully
• But now we have asymptotic code generation to worry about!
Roy H. Stogner Metaprogramming April 16, 2012 39 / 52
Numeric Functional Metaprogramming Sparse Data Structures
Sparse Matrix Operations
Example
A ≡[a bT
c D
]{L,U} ≡ {LUL(A), LUU (A)}
L =
[a 0T
c LUL(D− ca−1bT)
]U =
[1 a−1bT
0 LUU (D− ca−1bT)
]Then apply to sparse arrays of arrays:
• One compile-time pass to allocate output structure
• One run-time pass to do calculations
• Asymptotic complexity drops beautifully
• But now we have asymptotic code generation to worry about!
Roy H. Stogner Metaprogramming April 16, 2012 39 / 52
Numeric Functional Metaprogramming Physics as a Graph Problem
Equations• A templated functor is almost a full physics equation!
I Define which physical quantities are inputs and outputs• Enumerated types are metaprogramming compatible
I Equations can be split into minimal chunks for modularityI Equations must be split into minimal chunks for Jacobian optimization?
Example
#define DeclareUnaryPhysics(physicsname, input1enum, outputenum, code) \
struct physicsname { \
static const char* name() { return #physicsname; } \
typedef UIntSetConstructor<input1enum>::type inputset; \
typedef UIntSetConstructor<outputenum>::type outputset; \
\
template <typename inputtype> \
auto operator() (const inputtype& input1enum##_NAME) \
-> decltype(code) { \
return code; \
} \
}
Roy H. Stogner Metaprogramming April 16, 2012 40 / 52
Numeric Functional Metaprogramming Physics as a Graph Problem
Equations• A templated functor is almost a full physics equation!
I Define which physical quantities are inputs and outputs• Enumerated types are metaprogramming compatible
I Equations can be split into minimal chunks for modularityI Equations must be split into minimal chunks for Jacobian optimization?
Example
#define DeclareUnaryPhysics(physicsname, input1enum, outputenum, code) \
struct physicsname { \
static const char* name() { return #physicsname; } \
typedef UIntSetConstructor<input1enum>::type inputset; \
typedef UIntSetConstructor<outputenum>::type outputset; \
\
template <typename inputtype> \
auto operator() (const inputtype& input1enum##_NAME) \
-> decltype(code) { \
return code; \
} \
}
Roy H. Stogner Metaprogramming April 16, 2012 40 / 52
Numeric Functional Metaprogramming Physics as a Graph Problem
Equations
Example
DeclareUnaryPhysics(DensityFromSpeciesDensities,
DENSITIES_VAR, DENSITY_VAR,
rhoi.sum());
DeclareBinaryPhysics(MomentumFromVelocity,
DENSITY_VAR, VELOCITY_VAR, MOMENTUM_VAR,
U * rho);
DeclareBinaryPhysics(VelocityFromMomentum,
DENSITY_VAR, MOMENTUM_VAR, VELOCITY_VAR,
rhoU / rho);
DeclareBinaryPhysics(SpecificEnergyFromSpecificInternalEnergy,
SPECIFIC_INTERNAL_ENERGY_VAR, SPEED_SQUARED_VAR,
SPECIFIC_ENERGY_VAR,
e + UdotU / 2);
• Functors get reused in every Physics set, for all input argument types
• Minimizing how much code must be entrusted to physics people
Roy H. Stogner Metaprogramming April 16, 2012 41 / 52
Numeric Functional Metaprogramming Physics as a Graph Problem
Equations
Example
DeclareUnaryPhysics(DensityFromSpeciesDensities,
DENSITIES_VAR, DENSITY_VAR,
rhoi.sum());
DeclareBinaryPhysics(MomentumFromVelocity,
DENSITY_VAR, VELOCITY_VAR, MOMENTUM_VAR,
U * rho);
DeclareBinaryPhysics(VelocityFromMomentum,
DENSITY_VAR, MOMENTUM_VAR, VELOCITY_VAR,
rhoU / rho);
DeclareBinaryPhysics(SpecificEnergyFromSpecificInternalEnergy,
SPECIFIC_INTERNAL_ENERGY_VAR, SPEED_SQUARED_VAR,
SPECIFIC_ENERGY_VAR,
e + UdotU / 2);
• Functors get reused in every Physics set, for all input argument types
• Minimizing how much code must be entrusted to physics people
Roy H. Stogner Metaprogramming April 16, 2012 41 / 52
Numeric Functional Metaprogramming Physics as a Graph Problem
Equations
Example
DeclareUnaryPhysics(DensityFromSpeciesDensities,
DENSITIES_VAR, DENSITY_VAR,
rhoi.sum());
DeclareBinaryPhysics(MomentumFromVelocity,
DENSITY_VAR, VELOCITY_VAR, MOMENTUM_VAR,
U * rho);
DeclareBinaryPhysics(VelocityFromMomentum,
DENSITY_VAR, MOMENTUM_VAR, VELOCITY_VAR,
rhoU / rho);
DeclareBinaryPhysics(SpecificEnergyFromSpecificInternalEnergy,
SPECIFIC_INTERNAL_ENERGY_VAR, SPEED_SQUARED_VAR,
SPECIFIC_ENERGY_VAR,
e + UdotU / 2);
• Functors get reused in every Physics set, for all input argument types
• Minimizing how much code must be entrusted to physics people
Roy H. Stogner Metaprogramming April 16, 2012 41 / 52
Numeric Functional Metaprogramming Physics as a Graph Problem
Equation Sets
• Equations are now represented by data types
• Data types can be stuck into metaprogramming containers!
Example
typedef VectorConstructor<
DensityFromSpeciesDensities,
VelocityFromMomentum,
MomentumFromVelocity,
SpeedSquaredFromVelocity,
SpecificEnergyFromConserved,
SpecificInternalEnergyFromSpecificEnergy,
SpecificEnergyFromSpecificInternalEnergy,
LinearTranslationalRotationalEnergyFromTemperature,
SpecificInternalEnergyFromOnlyTransationalRotationalEnergy
>::type AllPhysics;
Roy H. Stogner Metaprogramming April 16, 2012 42 / 52
Numeric Functional Metaprogramming Physics as a Graph Problem
Equation Sets
• Equations are now represented by data types
• Data types can be stuck into metaprogramming containers!
Example
typedef VectorConstructor<
DensityFromSpeciesDensities,
VelocityFromMomentum,
MomentumFromVelocity,
SpeedSquaredFromVelocity,
SpecificEnergyFromConserved,
SpecificInternalEnergyFromSpecificEnergy,
SpecificEnergyFromSpecificInternalEnergy,
LinearTranslationalRotationalEnergyFromTemperature,
SpecificInternalEnergyFromOnlyTransationalRotationalEnergy
>::type AllPhysics;
Roy H. Stogner Metaprogramming April 16, 2012 42 / 52
Numeric Functional Metaprogramming Physics as a Graph Problem
Equation Sets
• Equations are now represented by data types
• Data types can be stuck into metaprogramming containers!
Example
typedef VectorConstructor<
DensityFromSpeciesDensities,
VelocityFromMomentum,
MomentumFromVelocity,
SpeedSquaredFromVelocity,
SpecificEnergyFromConserved,
SpecificInternalEnergyFromSpecificEnergy,
SpecificEnergyFromSpecificInternalEnergy,
LinearTranslationalRotationalEnergyFromTemperature,
SpecificInternalEnergyFromOnlyTransationalRotationalEnergy
>::type AllPhysics;
Roy H. Stogner Metaprogramming April 16, 2012 42 / 52
Numeric Functional Metaprogramming Physics as a Graph Problem
Equation Sets
• Equations are now represented by data types
• Data types can be stuck into metaprogramming containers!
Example
typedef VectorConstructor<
DensityFromSpeciesDensities,
VelocityFromMomentum,
MomentumFromVelocity,
SpeedSquaredFromVelocity,
SpecificEnergyFromConserved,
SpecificInternalEnergyFromSpecificEnergy,
SpecificEnergyFromSpecificInternalEnergy,
LinearTranslationalRotationalEnergyFromTemperature,
SpecificInternalEnergyFromOnlyTransationalRotationalEnergy
>::type AllPhysics;
Roy H. Stogner Metaprogramming April 16, 2012 42 / 52
Numeric Functional Metaprogramming Physics as a Graph Problem
Variable Sets
• Variables are now represented by enumerated values
• Enums can be stuck into metaprogramming containers!
Example
typedef UIntSetConstructor<
DENSITIES_VAR,
VELOCITY_VAR,
TEMPERATURE_VAR
>::type PrimitiveVars;
typedef UIntSetConstructor<
DENSITIES_VAR,
MOMENTUM_VAR,
ENERGY_VAR
>::type ConservedVars;
Roy H. Stogner Metaprogramming April 16, 2012 43 / 52
Numeric Functional Metaprogramming Physics as a Graph Problem
Variable Sets
• Variables are now represented by enumerated values
• Enums can be stuck into metaprogramming containers!
Example
typedef UIntSetConstructor<
DENSITIES_VAR,
VELOCITY_VAR,
TEMPERATURE_VAR
>::type PrimitiveVars;
typedef UIntSetConstructor<
DENSITIES_VAR,
MOMENTUM_VAR,
ENERGY_VAR
>::type ConservedVars;
Roy H. Stogner Metaprogramming April 16, 2012 43 / 52
Numeric Functional Metaprogramming Physics as a Graph Problem
Variable Sets
• Variables are now represented by enumerated values
• Enums can be stuck into metaprogramming containers!
Example
typedef UIntSetConstructor<
DENSITIES_VAR,
VELOCITY_VAR,
TEMPERATURE_VAR
>::type PrimitiveVars;
typedef UIntSetConstructor<
DENSITIES_VAR,
MOMENTUM_VAR,
ENERGY_VAR
>::type ConservedVars;
Roy H. Stogner Metaprogramming April 16, 2012 43 / 52
Numeric Functional Metaprogramming Physics as a Graph Problem
Variable Sets
• Variables are now represented by enumerated values
• Enums can be stuck into metaprogramming containers!
Example
typedef UIntSetConstructor<
DENSITIES_VAR,
VELOCITY_VAR,
TEMPERATURE_VAR
>::type PrimitiveVars;
typedef UIntSetConstructor<
DENSITIES_VAR,
MOMENTUM_VAR,
ENERGY_VAR
>::type ConservedVars;
Roy H. Stogner Metaprogramming April 16, 2012 43 / 52
Numeric Functional Metaprogramming Physics as a Graph Problem
Equation Sets as Bicolored Graphs
• Consider each variable as a rednode
• Consider each equation as ablack node
I Each output is a directededge from equation to variable
I Each input is a directed edgefrom variable to equation
Then our physics now looks like agraph!
T
ei
ρi
(ρe)i
ρe
ρ
~u
~u · ~u
ρ~u · ~u
ρE
Roy H. Stogner Metaprogramming April 16, 2012 44 / 52
Numeric Functional Metaprogramming Physics as a Graph Problem
Equation Sets as Bicolored Graphs
• Consider each variable as a rednode
• Consider each equation as ablack node
I Each output is a directededge from equation to variable
I Each input is a directed edgefrom variable to equation
Then our physics now looks like agraph!
T
ei
ρi
(ρe)i
ρe
ρ
~u
~u · ~u
ρ~u · ~u
ρE
Roy H. Stogner Metaprogramming April 16, 2012 44 / 52
Numeric Functional Metaprogramming Physics as a Graph Problem
Equation Sets as Bicolored Graphs
• Consider each variable as a rednode
• Consider each equation as ablack node
I Each output is a directededge from equation to variable
I Each input is a directed edgefrom variable to equation
Then our physics now looks like agraph!
T
ei
ρi
(ρe)i
ρe
ρ
~u
~u · ~u
ρ~u · ~u
ρE
Roy H. Stogner Metaprogramming April 16, 2012 44 / 52
Numeric Functional Metaprogramming Physics as a Graph Problem
Equation Sets as Bicolored Graphs
• Consider each variable as a rednode
• Consider each equation as ablack node
I Each output is a directededge from equation to variable
I Each input is a directed edgefrom variable to equation
Then our physics now looks like agraph!
T
ei
ρi
(ρe)i
ρe
ρ
~u
~u · ~u
ρ~u · ~u
ρE
Roy H. Stogner Metaprogramming April 16, 2012 44 / 52
Numeric Functional Metaprogramming Physics as a Graph Problem
Equation Sets as Bicolored Graphs
• Consider each variable as a rednode
• Consider each equation as ablack node
I Each output is a directededge from equation to variable
I Each input is a directed edgefrom variable to equation
Then our physics now looks like agraph!
T
ei
ρi
(ρe)i
ρe
ρ
~u
~u · ~u
ρ~u · ~u
ρE
Roy H. Stogner Metaprogramming April 16, 2012 44 / 52
Numeric Functional Metaprogramming Physics as a Graph Problem
Equation Sets as Bicolored Graphs
• Consider each variable as a rednode
• Consider each equation as ablack node
I Each output is a directededge from equation to variable
I Each input is a directed edgefrom variable to equation
Then our physics now looks like agraph!
T
ei
ρi
(ρe)i
ρe
ρ
~u
~u · ~u
ρ~u · ~u
ρE
Roy H. Stogner Metaprogramming April 16, 2012 44 / 52
Numeric Functional Metaprogramming Physics as a Graph Problem
Equation Sets as Bicolored Graphs
T
ei
ρi
(ρe)i
ρe
ρ
~u
~u · ~u
ρ~u · ~u
ρE
We can use metaprogramming toanswer questions about the graph.
• Given a set of input variables:I Which non-input variables in
the graph are nowdetermined?
I What are the dependencies ofeach variable?
• And a set of output variables:I Which equations must be
solved to evaluate them?I In which order?
Roy H. Stogner Metaprogramming April 16, 2012 45 / 52
Numeric Functional Metaprogramming Physics as a Graph Problem
Equation Sets as Bicolored Graphs
T
ei
ρi
(ρe)i
ρe
ρ
~u
~u · ~u
ρ~u · ~u
ρE
We can use metaprogramming toanswer questions about the graph.
• Given a set of input variables:I Which non-input variables in
the graph are nowdetermined?
I What are the dependencies ofeach variable?
• And a set of output variables:I Which equations must be
solved to evaluate them?I In which order?
Roy H. Stogner Metaprogramming April 16, 2012 45 / 52
Numeric Functional Metaprogramming Evaluations on the Graph
SolveList
T
ei
ρi
(ρe)i
ρe
ρ
~u
~u · ~u
ρ~u · ~u
ρE
45
6
12
3
7
Given input variables, outputvariables, and equation functor classtypes connecting them, we want aSolveList: an ordered set of whichequations to solve in what order.
1 Find all the dependencies ofyour output variables
I Unreachable output variablesbecome a compile-time error
I Unnecessary non-inputnon-output variables are leftout
2 Greedy search throughequations (in the same order asfor dependency finding!), addingsolveable equations to list
Roy H. Stogner Metaprogramming April 16, 2012 46 / 52
Numeric Functional Metaprogramming Evaluations on the Graph
SolveList
T
ei
ρi
(ρe)i
ρe
ρ
~u
~u · ~u
ρ~u · ~u
ρE
45
6
12
3
7
Given input variables, outputvariables, and equation functor classtypes connecting them, we want aSolveList: an ordered set of whichequations to solve in what order.
1 Find all the dependencies ofyour output variables
I Unreachable output variablesbecome a compile-time error
I Unnecessary non-inputnon-output variables are leftout
2 Greedy search throughequations (in the same order asfor dependency finding!), addingsolveable equations to list
Roy H. Stogner Metaprogramming April 16, 2012 46 / 52
Numeric Functional Metaprogramming Evaluations on the Graph
SolveList
T
ei
ρi
(ρe)i
ρe
ρ
~u
~u · ~u
ρ~u · ~u
ρE
45
6
12
3
7
Given input variables, outputvariables, and equation functor classtypes connecting them, we want aSolveList: an ordered set of whichequations to solve in what order.
1 Find all the dependencies ofyour output variables
I Unreachable output variablesbecome a compile-time error
I Unnecessary non-inputnon-output variables are leftout
2 Greedy search throughequations (in the same order asfor dependency finding!), addingsolveable equations to list
Roy H. Stogner Metaprogramming April 16, 2012 46 / 52
Numeric Functional Metaprogramming Evaluations on the Graph
SolveList
T
ei
ρi
(ρe)i
ρe
ρ
~u
~u · ~u
ρ~u · ~u
ρE
45
6
12
3
7
Given input variables, outputvariables, and equation functor classtypes connecting them, we want aSolveList: an ordered set of whichequations to solve in what order.
1 Find all the dependencies ofyour output variables
I Unreachable output variablesbecome a compile-time error
I Unnecessary non-inputnon-output variables are leftout
2 Greedy search throughequations (in the same order asfor dependency finding!), addingsolveable equations to list
Roy H. Stogner Metaprogramming April 16, 2012 46 / 52
Numeric Functional Metaprogramming Evaluations on the Graph
SolveList
T
ei
ρi
(ρe)i
ρe
ρ
~u
~u · ~u
ρ~u · ~u
ρE
45
6
12
3
7
Given input variables, outputvariables, and equation functor classtypes connecting them, we want aSolveList: an ordered set of whichequations to solve in what order.
1 Find all the dependencies ofyour output variables
I Unreachable output variablesbecome a compile-time error
I Unnecessary non-inputnon-output variables are leftout
2 Greedy search throughequations (in the same order asfor dependency finding!), addingsolveable equations to list
Roy H. Stogner Metaprogramming April 16, 2012 46 / 52
Numeric Functional Metaprogramming Evaluations on the Graph
State Updating
With user-specified types on all variables, SolveList evaluation is easy:
1 Create a Container type holding the specified variable types
2 Instantiate this State state; set values on the user input variables3 Do a ForEach loop over the SolveList
I Pass input variables from state, assign output variables to state foreach equation
With user-specified types on input variables only, we first need ametaprogramming pass:
1 Create a Container type holding the specified input variable types2 Do a metaprogramming loop over the SolveList
I Pass input variable types to each equationI Insert output variable types into the State
3 Do a ForEach loop at runtime as before
Roy H. Stogner Metaprogramming April 16, 2012 47 / 52
Numeric Functional Metaprogramming Evaluations on the Graph
State Updating
With user-specified types on all variables, SolveList evaluation is easy:
1 Create a Container type holding the specified variable types
2 Instantiate this State state; set values on the user input variables3 Do a ForEach loop over the SolveList
I Pass input variables from state, assign output variables to state foreach equation
With user-specified types on input variables only, we first need ametaprogramming pass:
1 Create a Container type holding the specified input variable types2 Do a metaprogramming loop over the SolveList
I Pass input variable types to each equationI Insert output variable types into the State
3 Do a ForEach loop at runtime as before
Roy H. Stogner Metaprogramming April 16, 2012 47 / 52
Numeric Functional Metaprogramming Evaluations on the Graph
State Updating
With user-specified types on all variables, SolveList evaluation is easy:
1 Create a Container type holding the specified variable types
2 Instantiate this State state; set values on the user input variables
3 Do a ForEach loop over the SolveList
I Pass input variables from state, assign output variables to state foreach equation
With user-specified types on input variables only, we first need ametaprogramming pass:
1 Create a Container type holding the specified input variable types2 Do a metaprogramming loop over the SolveList
I Pass input variable types to each equationI Insert output variable types into the State
3 Do a ForEach loop at runtime as before
Roy H. Stogner Metaprogramming April 16, 2012 47 / 52
Numeric Functional Metaprogramming Evaluations on the Graph
State Updating
With user-specified types on all variables, SolveList evaluation is easy:
1 Create a Container type holding the specified variable types
2 Instantiate this State state; set values on the user input variables3 Do a ForEach loop over the SolveList
I Pass input variables from state, assign output variables to state foreach equation
With user-specified types on input variables only, we first need ametaprogramming pass:
1 Create a Container type holding the specified input variable types2 Do a metaprogramming loop over the SolveList
I Pass input variable types to each equationI Insert output variable types into the State
3 Do a ForEach loop at runtime as before
Roy H. Stogner Metaprogramming April 16, 2012 47 / 52
Numeric Functional Metaprogramming Evaluations on the Graph
State Updating
With user-specified types on all variables, SolveList evaluation is easy:
1 Create a Container type holding the specified variable types
2 Instantiate this State state; set values on the user input variables3 Do a ForEach loop over the SolveList
I Pass input variables from state, assign output variables to state foreach equation
With user-specified types on input variables only, we first need ametaprogramming pass:
1 Create a Container type holding the specified input variable types2 Do a metaprogramming loop over the SolveList
I Pass input variable types to each equationI Insert output variable types into the State
3 Do a ForEach loop at runtime as before
Roy H. Stogner Metaprogramming April 16, 2012 47 / 52
Numeric Functional Metaprogramming Evaluations on the Graph
State Updating
With user-specified types on all variables, SolveList evaluation is easy:
1 Create a Container type holding the specified variable types
2 Instantiate this State state; set values on the user input variables3 Do a ForEach loop over the SolveList
I Pass input variables from state, assign output variables to state foreach equation
With user-specified types on input variables only, we first need ametaprogramming pass:
1 Create a Container type holding the specified input variable types2 Do a metaprogramming loop over the SolveList
I Pass input variable types to each equationI Insert output variable types into the State
3 Do a ForEach loop at runtime as before
Roy H. Stogner Metaprogramming April 16, 2012 47 / 52
Numeric Functional Metaprogramming Evaluations on the Graph
State Updating
With user-specified types on all variables, SolveList evaluation is easy:
1 Create a Container type holding the specified variable types
2 Instantiate this State state; set values on the user input variables3 Do a ForEach loop over the SolveList
I Pass input variables from state, assign output variables to state foreach equation
With user-specified types on input variables only, we first need ametaprogramming pass:
1 Create a Container type holding the specified input variable types2 Do a metaprogramming loop over the SolveList
I Pass input variable types to each equationI Insert output variable types into the State
3 Do a ForEach loop at runtime as before
Roy H. Stogner Metaprogramming April 16, 2012 47 / 52
Numeric Functional Metaprogramming Evaluations on the Graph
State Updating
With user-specified types on all variables, SolveList evaluation is easy:
1 Create a Container type holding the specified variable types
2 Instantiate this State state; set values on the user input variables3 Do a ForEach loop over the SolveList
I Pass input variables from state, assign output variables to state foreach equation
With user-specified types on input variables only, we first need ametaprogramming pass:
1 Create a Container type holding the specified input variable types
2 Do a metaprogramming loop over the SolveList
I Pass input variable types to each equationI Insert output variable types into the State
3 Do a ForEach loop at runtime as before
Roy H. Stogner Metaprogramming April 16, 2012 47 / 52
Numeric Functional Metaprogramming Evaluations on the Graph
State Updating
With user-specified types on all variables, SolveList evaluation is easy:
1 Create a Container type holding the specified variable types
2 Instantiate this State state; set values on the user input variables3 Do a ForEach loop over the SolveList
I Pass input variables from state, assign output variables to state foreach equation
With user-specified types on input variables only, we first need ametaprogramming pass:
1 Create a Container type holding the specified input variable types2 Do a metaprogramming loop over the SolveList
I Pass input variable types to each equationI Insert output variable types into the State
3 Do a ForEach loop at runtime as before
Roy H. Stogner Metaprogramming April 16, 2012 47 / 52
Numeric Functional Metaprogramming Evaluations on the Graph
State Updating
With user-specified types on all variables, SolveList evaluation is easy:
1 Create a Container type holding the specified variable types
2 Instantiate this State state; set values on the user input variables3 Do a ForEach loop over the SolveList
I Pass input variables from state, assign output variables to state foreach equation
With user-specified types on input variables only, we first need ametaprogramming pass:
1 Create a Container type holding the specified input variable types2 Do a metaprogramming loop over the SolveList
I Pass input variable types to each equation
I Insert output variable types into the State
3 Do a ForEach loop at runtime as before
Roy H. Stogner Metaprogramming April 16, 2012 47 / 52
Numeric Functional Metaprogramming Evaluations on the Graph
State Updating
With user-specified types on all variables, SolveList evaluation is easy:
1 Create a Container type holding the specified variable types
2 Instantiate this State state; set values on the user input variables3 Do a ForEach loop over the SolveList
I Pass input variables from state, assign output variables to state foreach equation
With user-specified types on input variables only, we first need ametaprogramming pass:
1 Create a Container type holding the specified input variable types2 Do a metaprogramming loop over the SolveList
I Pass input variable types to each equationI Insert output variable types into the State
3 Do a ForEach loop at runtime as before
Roy H. Stogner Metaprogramming April 16, 2012 47 / 52
Numeric Functional Metaprogramming Evaluations on the Graph
State Updating
With user-specified types on all variables, SolveList evaluation is easy:
1 Create a Container type holding the specified variable types
2 Instantiate this State state; set values on the user input variables3 Do a ForEach loop over the SolveList
I Pass input variables from state, assign output variables to state foreach equation
With user-specified types on input variables only, we first need ametaprogramming pass:
1 Create a Container type holding the specified input variable types2 Do a metaprogramming loop over the SolveList
I Pass input variable types to each equationI Insert output variable types into the State
3 Do a ForEach loop at runtime as before
Roy H. Stogner Metaprogramming April 16, 2012 47 / 52
Numeric Functional Metaprogramming Differentiation on the Graph
Equation Derivatives
Differentiation looks already solved!
1 Collect your Physics
2 Choose your input variables
3 Evaluate the SolveList
4 Use DualNumber input variable types
5 Take derivatives() of output variables
6 Cheer
Roy H. Stogner Metaprogramming April 16, 2012 48 / 52
Numeric Functional Metaprogramming Differentiation on the Graph
Equation Derivatives
Differentiation looks already solved!
1 Collect your Physics
2 Choose your input variables
3 Evaluate the SolveList
4 Use DualNumber input variable types
5 Take derivatives() of output variables
6 Cheer
Roy H. Stogner Metaprogramming April 16, 2012 48 / 52
Numeric Functional Metaprogramming Differentiation on the Graph
Equation Derivatives
Differentiation looks already solved!
1 Collect your Physics
2 Choose your input variables
3 Evaluate the SolveList
4 Use DualNumber input variable types
5 Take derivatives() of output variables
6 Cheer
Roy H. Stogner Metaprogramming April 16, 2012 48 / 52
Numeric Functional Metaprogramming Differentiation on the Graph
Equation Derivatives
Differentiation looks already solved!
1 Collect your Physics
2 Choose your input variables
3 Evaluate the SolveList
4 Use DualNumber input variable types
5 Take derivatives() of output variables
6 Cheer
Roy H. Stogner Metaprogramming April 16, 2012 48 / 52
Numeric Functional Metaprogramming Differentiation on the Graph
Equation Derivatives
Differentiation looks already solved!
1 Collect your Physics
2 Choose your input variables
3 Evaluate the SolveList
4 Use DualNumber input variable types
5 Take derivatives() of output variables
6 Cheer
Roy H. Stogner Metaprogramming April 16, 2012 48 / 52
Numeric Functional Metaprogramming Differentiation on the Graph
Equation Derivatives
Differentiation looks already solved!
1 Collect your Physics
2 Choose your input variables
3 Evaluate the SolveList
4 Use DualNumber input variable types
5 Take derivatives() of output variables
6 Cheer
Roy H. Stogner Metaprogramming April 16, 2012 48 / 52
Numeric Functional Metaprogramming Differentiation on the Graph
Equation Derivatives
But how well does that solve differentiation?• Forward differentiation issues:
1 Needlessly expensive for many-input-variable, few-output variablegraphs
• or subgraphs!
• Adjoint differentiation issues:1 Needlessly expensive for many-output-variable, few-input variable
graphs• or subgraphs!
2 Typically requires additional temporary data• particularly with operator overloading...
• Optimal Jacobian Accumulation issues:1 NP-complete!
• don’t want to stress-test compilers...
• Heuristic: “Eh, Good Enough” Jacobian Accumulation
Roy H. Stogner Metaprogramming April 16, 2012 49 / 52
Numeric Functional Metaprogramming Differentiation on the Graph
Chain Rule Application: Condensing one NodeEach equation node can be used to define local partial derivatives
• Ni inputs, No outputs gives Ni ·No partial derivatives
• Small equations are efficiently evaluated with sparse forward AD
• Equation nodes are out of the picture
• Partial derivatives are directed edges with weights D(Vi, Vj)connecting variables Vi and Vj
An intermediate variable M can be “condensed away”:
• For all outgoing edges to output variable O:
• For all incoming edges from input variable I:• D(I,O)+ = D(I,M) ·D(M,O)
I (starting from D(I,O) ≡ 0 when creating a new edge)
Keep condensing until all remaining edges are directly from user inputs touser outputs
Roy H. Stogner Metaprogramming April 16, 2012 50 / 52
Numeric Functional Metaprogramming Differentiation on the Graph
Chain Rule Application: Condensing one NodeEach equation node can be used to define local partial derivatives
• Ni inputs, No outputs gives Ni ·No partial derivatives
• Small equations are efficiently evaluated with sparse forward AD
• Equation nodes are out of the picture
• Partial derivatives are directed edges with weights D(Vi, Vj)connecting variables Vi and Vj
An intermediate variable M can be “condensed away”:
• For all outgoing edges to output variable O:
• For all incoming edges from input variable I:• D(I,O)+ = D(I,M) ·D(M,O)
I (starting from D(I,O) ≡ 0 when creating a new edge)
Keep condensing until all remaining edges are directly from user inputs touser outputs
Roy H. Stogner Metaprogramming April 16, 2012 50 / 52
Numeric Functional Metaprogramming Differentiation on the Graph
Chain Rule Application: Condensing one NodeEach equation node can be used to define local partial derivatives
• Ni inputs, No outputs gives Ni ·No partial derivatives
• Small equations are efficiently evaluated with sparse forward AD
• Equation nodes are out of the picture
• Partial derivatives are directed edges with weights D(Vi, Vj)connecting variables Vi and Vj
An intermediate variable M can be “condensed away”:
• For all outgoing edges to output variable O:
• For all incoming edges from input variable I:• D(I,O)+ = D(I,M) ·D(M,O)
I (starting from D(I,O) ≡ 0 when creating a new edge)
Keep condensing until all remaining edges are directly from user inputs touser outputs
Roy H. Stogner Metaprogramming April 16, 2012 50 / 52
Numeric Functional Metaprogramming Differentiation on the Graph
Chain Rule Application: Condensing one NodeEach equation node can be used to define local partial derivatives
• Ni inputs, No outputs gives Ni ·No partial derivatives
• Small equations are efficiently evaluated with sparse forward AD
• Equation nodes are out of the picture
• Partial derivatives are directed edges with weights D(Vi, Vj)connecting variables Vi and Vj
An intermediate variable M can be “condensed away”:
• For all outgoing edges to output variable O:
• For all incoming edges from input variable I:• D(I,O)+ = D(I,M) ·D(M,O)
I (starting from D(I,O) ≡ 0 when creating a new edge)
Keep condensing until all remaining edges are directly from user inputs touser outputs
Roy H. Stogner Metaprogramming April 16, 2012 50 / 52
Numeric Functional Metaprogramming Differentiation on the Graph
Chain Rule Application: Condensing one NodeEach equation node can be used to define local partial derivatives
• Ni inputs, No outputs gives Ni ·No partial derivatives
• Small equations are efficiently evaluated with sparse forward AD
• Equation nodes are out of the picture
• Partial derivatives are directed edges with weights D(Vi, Vj)connecting variables Vi and Vj
An intermediate variable M can be “condensed away”:
• For all outgoing edges to output variable O:
• For all incoming edges from input variable I:• D(I,O)+ = D(I,M) ·D(M,O)
I (starting from D(I,O) ≡ 0 when creating a new edge)
Keep condensing until all remaining edges are directly from user inputs touser outputs
Roy H. Stogner Metaprogramming April 16, 2012 50 / 52
Numeric Functional Metaprogramming Differentiation on the Graph
Chain Rule Application: Condensing the Graph
Ia
Ib
Ic
Ma
Mb
Mc Md
Me
Mf
Oa
Ob
Oc
A
B
C D
E
F
G
H
I
• In what order do we condense these equations?I N -factorial possibilitiesI Some possibilities are cheaper than othersI NP -complete optimization problem
Eh, good enough
Roy H. Stogner Metaprogramming April 16, 2012 51 / 52
Numeric Functional Metaprogramming Differentiation on the Graph
Chain Rule Application: Condensing the Graph
Ia
Ib
Ic
Ma
Mb
Mc Md
Me
Mf
Oa
Ob
Oc
A
B
C D
E
F
G
H
I
• Forward differentiation:I Evaluates equations inputs-to-outputsI Effectively does Ma,Mb,Mc,Md,Me,Mf
I 3 + 3 + 3 + 6 + 9 + 9 = 33 multiply+add operations
Eh, good enough
Roy H. Stogner Metaprogramming April 16, 2012 51 / 52
Numeric Functional Metaprogramming Differentiation on the Graph
Chain Rule Application: Condensing the Graph
Ia
Ib
Ic
Ma
Mb
Mc Md
Me
Mf
Oa
Ob
Oc
A
B
C D
E
F
G
H
I
• Reverse differentiation:I Evaluates equations outputs-to-inputsI Effectively does Mf ,Me,Md,Mc,Mb,Ma
I Still 3 + 3 + 3 + 6 + 9 + 9 = 33 multiply+add operations
Eh, good enough
Roy H. Stogner Metaprogramming April 16, 2012 51 / 52
Numeric Functional Metaprogramming Differentiation on the Graph
Chain Rule Application: Condensing the Graph
Ia
Ib
Ic
Ma
Mb
Mc Md
Me
Mf
Oa
Ob
Oc
A
B
C D
E
F
G
H
I
• Greedy (lazy?) differentiation:I Pick cheapest condensations firstI Does Mc,Ma,Mb,Me,Mf ,Md
I Still 2 + 3 + 3 + 3 + 3 + 9 = 23 multiply+add operations: 30% cheaper!
Eh, good enough
Roy H. Stogner Metaprogramming April 16, 2012 51 / 52
Numeric Functional Metaprogramming Differentiation on the Graph
Chain Rule Application: Condensing the Graph
Ia
Ib
Ic
Ma
Mb
Mc Md
Me
Mf
Oa
Ob
Oc
A
B
C D
E
F
G
H
I
• Greedy (lazy?) differentiation:I Pick cheapest condensations firstI Does Mc,Ma,Mb,Me,Mf ,Md
I Still 2 + 3 + 3 + 3 + 3 + 9 = 23 multiply+add operations: 30% cheaper!
Eh, good enough
Roy H. Stogner Metaprogramming April 16, 2012 51 / 52
Numeric Functional Metaprogramming Differentiation on the Graph
Any Questions?
Lots of My Questions:• How much code for non-MASA
apps can be replaced bystandard libraries?
I Boost MTLI SacadoI Phalanx
• Can physics functors bedecomposed automatically?
I Precompilation script?
• Can we “roll up” sparsestructure algorithms?
Roy H. Stogner Metaprogramming April 16, 2012 52 / 52
Numeric Functional Metaprogramming Differentiation on the Graph
Any Questions?
Lots of My Questions:• How much code for non-MASA
apps can be replaced bystandard libraries?
I Boost MTLI SacadoI Phalanx
• Can physics functors bedecomposed automatically?
I Precompilation script?
• Can we “roll up” sparsestructure algorithms?
Roy H. Stogner Metaprogramming April 16, 2012 52 / 52