numerical generic programming with c++ templatesusers.ices.utexas.edu/~roystgnr/dualnumber.pdf ·...

Numerical Generic Programming with C++ Templates

Roy H. Stogner

The University of Texas at Austin

Jan 18, 2012

Roy H. Stogner Generic Jan 18, 2012 1 / 44

Generic Programming Data Types

Outline

1 Generic ProgrammingData TypesMulti-precision verificationArray OperationsComputation with Data TypesAutomatic Differentiation

2 Stuff We Won’t Get To This TimeAdvanced Functional Programming with TemplatesFunctional Data StructuresPhysics as a Graph ProblemEvaluations on the GraphDifferentiation on the Graph



Primitive Data Types

Exampleconst float theta = 0.5;

double onethird = 1.0/3.0; // Not 1/3!

int power = 2;

int* ptr_to_power = &power;

bool goodexamples = false;

const char* nexttask = "do better";

• Data types native to the CPU

• Data addresses in memory

• Logical extensions, reinterpretations of those types

• “Constant” correctness checking



Data Type Safety

Weakly Typed Languages

Example

a = 2;

b = "2";

concatenate(a, b) # "22"

add(a, b) # 4

Bash, Perl, etc.

Strongly Typed Languages

Example

int a = 2;

string b = "2";

concatenate(a, b); // error

add(a, b); // error

concatenate(string(a), b); // "22"

add(a, int(b)); // 4

C++, Fortran, Python, etc.

• But even C++ lets you write implicit conversion operators

• And even Bash screams at runtime if it can’t reinterpret data asrequested



Structured Types

Example

class ComplexFloat {

ComplexFloat(float re, float im);

ComplexFloat(float re);

typedef float value_type;

.........

float norm() const {

return sqrt(realpart*realpart +

imagpart*imagpart);

}

private:

float realpart, imagpart;

static const bool is_complex = true;

};

• DataI Private?I Static?

• MethodsI ConstructorsI DestructorsI AccessorsI Inline?

• Subtype DefinitionsI Way more powerful

than they look...



Structured Types

Example




typedef float value_type;

.........float norm() const {


imagpart*imagpart);

}

private:



};




than they look...



Structured Types

Example




typedef float value_type;.........float norm() const {


imagpart*imagpart);

}

private:



};




than they look...



Overloading

Example

inline ComplexFloat operator*(const ComplexFloat& a,

const ComplexFloat& b) {

return ComplexFloat(a.realpart*b.realpart-a.imagpart*b.imagpart,

a.realpart*b.imagpart+a.imagpart*b.realpart);

}

const ComplexFloat i(0,1), three(3,0);

ComplexFloat old_syntax_3i, function_overloaded_3i;

old_syntax_3i.equals(ComplexFloat_times(three,i));

function_overloaded_3i.equals(times(three,i));

const ComplexFloat new_3i = three * i;

• To computer scientists: overloading is “syntactic sugar”, which justlets you write the exact same program a little less bitterly.

• To engineers: overloading lets your math look like math!



Data Type Dependence

Example

class VectorFloat {.........float dot(const VectorFloat& v) const {

float r = 0;

for(int i = 0; i != size; ++i)

r += data[i] * v[i];

return r;.........float* data;

};

class VectorDouble {.........class VectorComplexDouble {.........class VectorTendonitis {.........

But not having to write the same code over and over again is critical forfeeling superior to Fortran programmers!




Example


float r = 0;




};

class VectorDouble {.........

class VectorComplexDouble {.........class VectorTendonitis {.........





Example


float r = 0;




};

class VectorDouble {.........class VectorComplexDouble {.........

class VectorTendonitis {.........





Example


float r = 0;




};

class VectorDouble {.........class VectorComplexDouble {.........class VectorTendonitis {.........




Data Type Independence: Object Oriented Programming

Example

class Number {.........virtual Number&

operator+=(const Number& n)

= 0; // Pure virtual.........};

• Create an abstract interface

• Derive concrete subclassesfrom it

• Implement the interface’sfunctions in each subclass

• Hold pointers or references tothe base class

• Write code to use the interface




Example



= 0; // Pure virtual.........};

class FloatNumber :

public Number {.........};

class DoubleNumber :










Example



= 0; // Pure virtual.........};

class FloatNumber :

public Number {.........virtual Number&

operator+=(const Number& n) {

datum += n.data();

return *this;

}.........float datum;.........};








Data Type Independence: Object Oriented ProgrammingExample

class Number {.........};

class FloatNumber :

public Number {.........virtual Number&

operator+=(const Number& n) {

datum += n.data();

return *this;

}.........float datum;.........};

class VectorNumber {.........

Number** data;

};









Example


class FloatNumber :


class VectorNumber {.........Number dot(const VectorNumber& v)

const {

Number r = 0;


r += (*data)[i] * v[i];

return r;

}.........Number** data;

};









Example


class FloatNumber :



const {

Number r = 0;


r += (*data)[i] * v[i];

return r;


};

• Advantages:I Works correctlyI Defines concept relations

• Number is an Abstract BaseClass

• FloatNumber is-a Number

I Heterogenous containers!• data[0] can be double,• data[1] can be complex...

• Disadvantages:I Can require “shim” classes

• A float isn’t aFloatNumber!

I Slow as hell• Fragmented dynamic

memory• Virtual function calls for

every FLOP!




Example


class FloatNumber :



const {

Number r = 0;


r += (*data)[i] * v[i];

return r;


};

• Advantages:

I Works correctlyI Defines concept relations








every FLOP!




Example


class FloatNumber :



const {

Number r = 0;


r += (*data)[i] * v[i];

return r;


};

• Advantages:I Works correctly

I Defines concept relations• Number is an Abstract Base

Class• FloatNumber is-a Number






every FLOP!




Example


class FloatNumber :



const {

Number r = 0;


r += (*data)[i] * v[i];

return r;


};









every FLOP!




Example


class FloatNumber :



const {

Number r = 0;


r += (*data)[i] * v[i];

return r;


};





• Disadvantages:

I Can require “shim” classes• A float isn’t a

FloatNumber!I Slow as hell

• Fragmented dynamicmemory

• Virtual function calls forevery FLOP!




Example


class FloatNumber :



const {

Number r = 0;


r += (*data)[i] * v[i];

return r;


};









every FLOP!



Data Type Independence: Generic Programming

Example

template <typename T>

• Template arguments can bedata types!

• Class data members candepend on template argument

• Class methods can depend ontemplate argument

• Templated classes can beinstantiated with plain datatypes

I ... or with other instantiatedtemplated types

I ... or with other instantiationsof themselves!




Example


class NumericVector {.........T* data;

};










Example


class NumericVector {.........T dot(const NumericVector<T>& v)

const {

T r = 0;



return r;.........T* data;

};










Example



const {

T r = 0;




};

NumericVector<float>

floatvec;










Example



const {

T r = 0;




};


floatvec;

NumericVector<complex<double> >

complexdoublevec;










Example



const {

T r = 0;




};


floatvec;


complexdoublevec;

NumericVector<NumericVector<float> >

floatmatrix;










Example



const {

T r = 0;




};


floatvec;


complexdoublevec;


floatmatrix;

• Advantages

I Works with existing typesI Functions can be inlined, not

just virtualI Intermediate calculation types

can be implicit• Disadvantages

I Makes assumptions aboutinput types

• Is that NumericVectoroperator* an element byelement multiplication or adot product?

I Functions are undefined untilinstantiation

• Compile-time bugs goundetected until triggered




Example



const {

T r = 0;




};


floatvec;


complexdoublevec;


floatmatrix;

• AdvantagesI Works with existing types

I Functions can be inlined, notjust virtual

I Intermediate calculation typescan be implicit

• DisadvantagesI Makes assumptions about

input types• Is that NumericVector

operator* an element byelement multiplication or adot product?






Example



const {

T r = 0;




};


floatvec;


complexdoublevec;


floatmatrix;

• AdvantagesI Works with existing typesI Functions can be inlined, not

just virtual

I Intermediate calculation typescan be implicit









Example



const {

T r = 0;




};


floatvec;


complexdoublevec;


floatmatrix;



can be implicit









Example



const {

T r = 0;




};


floatvec;


complexdoublevec;


floatmatrix;



can be implicit• Disadvantages

I Makes assumptions aboutinput types

• Is that NumericVectoroperator* an element byelement multiplication or adot product?





Duck Typing

So we can duct tape our templated type to any type T that behaves like weexpect math to behave? Is that safe?

If it walks like a duck, quacks likea duck, looks like a duck, it mustbe a duck.

Unless it’s a dragon doing a duckimpression...

We’ll bump into some of thoselater.



Duck Typing

If it walks like a duck, quacks likea duck, looks like a duck, it mustbe a duck.

Unless it’s a dragon doing a duckimpression...

We’ll bump into some of thoselater.


Generic Programming Multi-precision verification

Outline





Operator-Overloaded Multi-precision

Example

template <typename T, typename S>

struct ShadowNumber {.........T _val;

S _shadow;

};

• Simultaneous calculation with multiple floating point representations,to estimate rounding error:

ExampleShadowNumber<float, double> shadowed_float;

ShadowNumber<double, long double> shadowed_double;



Operator-Overloaded Multi-precisionExample


struct ShadowNumber {.........

template <typename T2>

ShadowNumber(const T2& val) : _val(val), _shadow(val) {}

template <typename T2, typename S2>

ShadowNumber(ShadowNumber<T2, S2>& other) :

_val(other._val), _shadow(other._shadow) {}.........T _val;

S _shadow;

};

• Functions can be templated, and templates can be nested, too!





Operator-Overloaded Multi-precisionExample


struct ShadowNumber {.........


ShadowNumber(const T2& val) : _val(val), _shadow(val) {}


ShadowNumber(ShadowNumber<T2, S2>& other) :

_val(other._val), _shadow(other._shadow) {}.........


ShadowNumber<T,S>& operator+= (const T2& a)

{ _val += a; _shadow += a; return *this; }


ShadowNumber<T,S>& operator+= (const ShadowNumber<T2,S2>& a)

{ _val += a.value(); _shadow += a.shadow(); return *this; }.........T _val;

S _shadow;

};

•




Generic Programming Array Operations

Outline





Operator-Overloaded Vectors

Example

template <unsigned int size, typename T>

• Class templates can have integer (long, short, bool, etc.) arguments,not just data type arguments

• This lets us fix array sizes at compile time - no heap memory

• Fixed length loops can unroll with no branching




Example


struct NumberArray {.........

T _data[size];

};







Example


struct NumberArray {.........


NumberArray<size,T>& operator+= (const NumberArray<size,T2>& a) {

for (unsigned int i = 0; i != size; ++i)

_data[i] += a[i]; return *this;

}.........T _data[size];

};






Element-by-element Vector Functions

Example

#define NumberArray_std_unary(funcname) \

template <unsigned int size, typename T> \

inline \

NumberArray<size, T> \

funcname (const NumberArray<size, T>& a) \

{ \

NumberArray<size, T> returnval; \

\

for (unsigned int i = 0; i != size; ++i) \

returnval[i] = funcname(a[i]); \

\

return returnval; \

}

NumberArray_std_unary(exp)

NumberArray_std_unary(sin)

NumberArray_std_unary(cos)

• Templated function andstruct definitions can’t beinstantiated with thename to be defined as atemplate argument.

• We instantiate templatesto avoid writing NTNf

repetitive algorithms withunsafe macros

• We still need unsafemacros to avoid writingNf repetitive templateinstantiations.



Operator-Overloaded VectorsExample

NumericArray<3,double> v;

v[1] = 2;

NumericArray<3,float> f;

f[2] = 1;

v += sin(f); // Great!

f += sin(v); // Tolerable!

v = v + sin(f); // Uh-oh!

What is the return type supposed to be for

template <unsigned int size1, typename T1,

unsigned int size2, typename T2>

operator+(const NumberArray<size1,T1>& a,

const NumberArray<size2,T2>& b)

???





v[1] = 2;


f[2] = 1;

v += sin(f);

// Great!








???





v[1] = 2;


f[2] = 1;









???





v[1] = 2;


f[2] = 1;


f += sin(v);

// Tolerable!







???





v[1] = 2;


f[2] = 1;









???





v[1] = 2;


f[2] = 1;



v = v + sin(f);

// Uh-oh!






???





v[1] = 2;


f[2] = 1;









???


Generic Programming Computation with Data Types

Outline





Return Type Selection, Template Specialization

Let’s give it a template:

template <typename S, typename T>

struct CompareTypes {

// typedef something supertype;

};

And then we can specialize!

Exampletemplate <>

struct CompareTypes<float,double> {

typedef double

supertype;

};



Return Type Selection, Template Specialization

Let’s give it a template:

template <typename S, typename T>

struct CompareTypes {

// typedef something supertype;

};

Or, for class templates, partially specialize!

Exampletemplate <unsigned int size, typename T, typename T2>

struct CompareTypes<NumberArray<size,T>,T2> {

typedef NumberArray<size,typename CompareTypes<T,T2>::supertype>

supertype;

};



Interlude: Functional Programming

• Wait, we can partially specialize class templates?

• Oops, we just created a new Turing-complete metalanguage.

• Anything computable with integers is now computable at compile time!

• ...with atrocious syntax...



Functional Programming

Imperative, run-time C++ calculation of N !:

Exampleunsigned int factorial(unsigned int N) {

unsigned int f = 1;

for (unsigned int i = 2; i <= N; ++i)

f *= i; Value is variable

return f;

}

// Three FP ops, four test-and-branches

unsigned int twelve = factorial(4);



Functional Programming

Declarative, compile-time C++ calculation of N !:

Exampletemplate<unsigned int N>

struct Factorial {

static const unsigned int value = Each value

N * Factorial<N-1>::value; is constant

};

template<>

struct Factorial<0> {

static const unsigned int value = 1;

};

// Zero FP ops, zero tests/branches at runtime

unsigned int twelve = Factorial<4>::value;



Back to Return Types: Here Be Dragons

A NumberArray of something, combined with anything else we haven’tseen before

Example

template <unsigned int size, typename T, typename T2>


typedef NumberArray<size,

typename CompareTypes<T,T2>::supertype>

supertype;

};

if(typeid(CompareTypes<

NumberArray<3,float>,double

>::supertype) ==

typeid(NumberArray<3,double>))

how_much_we_rock = 10;




A NumberArray of something, combined with anything else we haven’tseen before

Example

template <unsigned int size, typename T, typename T2>



typename CompareTypes<T,T2>::supertype>

supertype;

};


NumberArray<3,NumberArray<float> >,double

>::supertype) ==

typeid(NumberArray<3,NumberArray<double> >))

how_much_we_rock = 11; // That extra push over the cliff




A ShadowNumber of something, combined with anything else we haven’tseen before

Example

template <typename T, typename S, typename T2>

struct CompareTypes<ShadowNumber<T,S>,T2> {

typedef ShadowNumber<typename CompareTypes<T,T2>::supertype,

typename CompareTypes<S,T2>::supertype>

supertype;

};


ShadowNumber<float,double>,double

>::supertype) ==

typeid(ShadowNumber<double,double>))

how_much_we_rock = 7; // C is still a passing grade





Example





supertype;

};


NumberArray<3,float>,ShadowNumber<float,double>

>::supertype) ==

typeid(NumberArray<3,ShadowNumber<float,double> >))

how_much_we_rock = -1; // This didn’t even compile





Example





supertype;

};


NumberArray<3,float>,ShadowNumber<float,double>

>::supertype) ==

typeid(ShadowNumber<NumberArray<3,float>,NumberArray<3,double> >))

how_much_we_rock = -1; // This didn’t even compile



Back to Return Types: Killing Dragons• Mathematically, these are isomorphic:

I (V ⊗W )n

I V n ⊗Wn

• Your compiler doesn’t understandisomorphisms:

I {float, double}[size]I {float[size]}, {double[size]}

• Need partial template specialization todisambiguate every combination

Example

template <unsigned int size, typename T, typename T2, typename S2>

struct CompareTypes<NumberArray<size,T>,ShadowNumber<T2,S2> > {


typename CompareTypes<T,ShadowNumber<T2,S2> >::supertype>

supertype;

};



Return Type Magic: SFINAE

Example


typename T2>

typename CompareTypes<NumberArray<size,T1>,T2>::supertype


const T2& b)


typename T2>

typename CompareTypes<ShadowNumber<T2,S2>,T1>::supertype

operator+(const T1& a,

const ShadowNumber<T2,S2>& b)

• Substitution Failure Is Not An Error

• These ambiguous functions are no longer ambiguous!


Generic Programming Automatic Differentiation

Outline





“Dual Numbers” - [Clifford 1873],[Study 1903]

• Add a single new element ε to R

• Close under addition and multiplication:{m∑i=0

aiεi : ai ∈ R,m <∞

}

• Take the quotient with ε2 ≡

• Arithmetic:

(a+ bε) + (c+ dε) = ((a+ c) + (b+ d)ε)

(a+ bε)− (c+ dε) = ((a+ c)− (b+ d)ε)

(a+ bε)× (c+ dε) = (ac) + adε+ bcε+ bdε2

= ((ac) + (ad+ bc)ε)



“Dual Numbers” - [Clifford 1873],[Study 1903]

• Add a single new element ε to R• Close under addition and multiplication:{

m∑i=0


}

• Take the quotient with ε2 ≡

• Arithmetic:

(a+ bε) + (c+ dε) = ((a+ c) + (b+ d)ε)

(a+ bε)− (c+ dε) = ((a+ c)− (b+ d)ε)


= ((ac) + (ad+ bc)ε)



“Dual Numbers” - [Clifford 1873],[Study 1903]• Add a single new element ε to R• Close under addition and multiplication:{

m∑i=0


}• Take the quotient with ε2 ≡ − 1

D[R] ≡ {a+ bε : a, b ∈ R}

• Arithmetic:

(a+ bε) + (c+ dε) = ((a+ c) + (b+ d)ε)

(a+ bε)− (c+ dε) = ((a+ c)− (b+ d)ε)


= ((ac) + (ad+ bc)ε)




m∑i=0


}• Take the quotient with ε2 ≡ − 1 ...no, wait, that’s C

D[R] ≡ {a+ bε : a, b ∈ R}

• Arithmetic:

(a+ bε) + (c+ dε) = ((a+ c) + (b+ d)ε)

(a+ bε)− (c+ dε) = ((a+ c)− (b+ d)ε)


= ((ac) + (ad+ bc)ε)




m∑i=0


}• Take the quotient with ε2 ≡ 0

D[R] ≡ {a+ bε : a, b ∈ R}

• Arithmetic:

(a+ bε) + (c+ dε) = ((a+ c) + (b+ d)ε)

(a+ bε)− (c+ dε) = ((a+ c)− (b+ d)ε)


= ((ac) + (ad+ bc)ε)



“Dual Numbers”

For each simple operation s(x, y),

s(a+ bε, c+ dε) = s(a, c) +

(b∂s

∂x(a, c) + d

∂s

∂y(a, c)

)ε

Extending to division, algebraic functions, transcendental functions, etcalso gives: for any function f(x),

f(a+ bε) = f(a) + bf ′(a)ε

Composition of functions preserves these properties, so with independentvariable x ∈ R, we set X ≡ x+ ε and get:

f(X) = f(x) + f ′(x)ε

This time the isomorphism practically codes itself!



Operator-Overloaded Forward Differentiation

Example

template <typename T, typename D>

struct DualNumber {.........T _value;

D _deriv;

};




Example


struct DualNumber {.........


DualNumber(const T2& val);

template <typename T2, typename D2>

DualNumber(const DualNumber<D2, T2>& val);.........T _value;

D _deriv;

};



Operator-Overloaded Forward DifferentiationExample


struct DualNumber {.........


DualNumber<T,S>& operator*= (const T2& a)

{ _value *= a; _deriv *= a; return *this; }

template <typename T2, typename D2>

DualNumber<T,S>& operator*= (const DualNumber<T2,D2>& a)

{

_value *= a.value();

_deriv *= a.value();

_deriv += _value * a.derivatives();

return *this;

}.........T _value;

D _deriv;

};




Example

typedef double RawType

// typedef ShadowNumber<double, long double> RawType;

typedef DualNumber<RawType,RawType> FirstDerivType;

const FirstDerivType x(pi/6,1); // Initializing independent var

const FirstDerivType sinx = sin(x); // Caching just like normal

const FirstDerivType y = sinx*sinx; // Smart pow would work too

const double raw_y = raw_value(y); // No implicit down-conversions!

double deriv = raw_value(y.derivatives());

assert(deriv == 2*sin(pi/6)*cos(pi/6)); // FP ops match hand-code!



“Hyper-dual Numbers” - [Fike 2009]

• Add two new elements ε1, ε2 to R

• Take the quotient with ε21 ≡ ε22 ≡ 0

H[R] ≡ {a+ bε1 + cε2 + dε1ε2 : a, b, c, d ∈ R}

• Arithmetic:

(a+ bε1 + cε2 + dε1ε2) + (e+ fε1 + gε2 + hε1ε2) =

((a+ e) + (b+ f)ε1 + (c+ g)ε2 + (d+ h)ε1ε2)

(a+ bε1 + cε2 + dε1ε2)− (e+ fε1 + gε2 + hε1ε2) =

((a− e) + (b− f)ε1 + (c− g)ε2 + (d− h)ε1ε2)(a+ bε1 + cε2 + dε1ε2)× (e+ fε1 + gε2 + hε1ε2) =

(ae) + (af + be)ε1 + (ag + ce)ε2 + (ah+ de+ bg + cf)ε1ε2



“Hyper-dual Numbers” - [Fike 2009]

• Add two new elements ε1, ε2 to R• Take the quotient with ε21 ≡ ε22 ≡ 0

H[R] ≡ {a+ bε1 + cε2 + dε1ε2 : a, b, c, d ∈ R}

• Arithmetic:

(a+ bε1 + cε2 + dε1ε2) + (e+ fε1 + gε2 + hε1ε2) =

((a+ e) + (b+ f)ε1 + (c+ g)ε2 + (d+ h)ε1ε2)

(a+ bε1 + cε2 + dε1ε2)− (e+ fε1 + gε2 + hε1ε2) =

((a− e) + (b− f)ε1 + (c− g)ε2 + (d− h)ε1ε2)(a+ bε1 + cε2 + dε1ε2)× (e+ fε1 + gε2 + hε1ε2) =

(ae) + (af + be)ε1 + (ag + ce)ε2 + (ah+ de+ bg + cf)ε1ε2



“Hyper-Dual Numbers”

For each simple operation s(x, y),

s(a+ bε1 + bε2 + cε1ε2, d+ eε1 + eε2 + fε1ε2) =

s(a, d) +

(b∂s

∂x(a, d) + e

∂s

∂y(a, d)

)(ε1 + ε2)+(

c∂s

∂x+ b2

∂2s

∂2x+ 2be

∂2s

∂x∂y+ f

∂s

∂y+ e2

∂2s

∂2y

)ε1ε2

Extending gives: for any function f(x),

f(a+bε1+bε2+cε1ε2) = f(a)+bf ′(a)ε1+bf′(a)ε2+(b2f ′′(a)+cf ′(a))ε1ε2

With independent x ∈ R, set X ≡ x+ ε1 + ε2 and get:

f(X) = f(x) + f ′(x)ε1 + f ′(x)ε2 + f ′′(x)ε1ε2



Hyper-dual Numbers via Recursive Mathematics

• Recall

I D[X] ≡ {a+ bε : a, b ∈ X}I f(a+ bε) = f(a) + bf ′(a)ε

• Applied directly to R we get first derivatives• Applied recursively:

I D[D[R]] ≡ {a+ cε2 : a, c ∈ D[R]}I D[D[R]] ≡ {(a+ bε1) + (c+ dε1)ε2 : a, b, c, d ∈ R}I f((a+ bε1) + (c+ dε1)ε2) = f(a+ bε1) + (c+ dε1)f

′(a+ bε1)ε2I = f(a) + bf ′(a)ε1 + (c+ dε1)(f

′(a) + bf ′′(a)ε1)ε2I = f(a) + bf ′(a)ε1 + cf ′(a)ε2 + (cbf ′′(a) + df ′(a))ε1ε2

• In other words, D[D[R]] ≡ H[R]• This isomorphism works in software, too!




• RecallI D[X] ≡ {a+ bε : a, b ∈ X}

I f(a+ bε) = f(a) + bf ′(a)ε









• RecallI D[X] ≡ {a+ bε : a, b ∈ X}I f(a+ bε) = f(a) + bf ′(a)ε










• Applied directly to R we get first derivatives

• Applied recursively:










I D[D[R]] ≡ {a+ cε2 : a, c ∈ D[R]}

I D[D[R]] ≡ {(a+ bε1) + (c+ dε1)ε2 : a, b, c, d ∈ R}I f((a+ bε1) + (c+ dε1)ε2) = f(a+ bε1) + (c+ dε1)f









I D[D[R]] ≡ {a+ cε2 : a, c ∈ D[R]}I D[D[R]] ≡ {(a+ bε1) + (c+ dε1)ε2 : a, b, c, d ∈ R}

I f((a+ bε1) + (c+ dε1)ε2) = f(a+ bε1) + (c+ dε1)f′(a+ bε1)ε2

I = f(a) + bf ′(a)ε1 + (c+ dε1)(f′(a) + bf ′′(a)ε1)ε2

I = f(a) + bf ′(a)ε1 + cf ′(a)ε2 + (cbf ′′(a) + df ′(a))ε1ε2








′(a+ bε1)ε2

I = f(a) + bf ′(a)ε1 + (c+ dε1)(f′(a) + bf ′′(a)ε1)ε2










′(a) + bf ′′(a)ε1)ε2











• In other words, D[D[R]] ≡ H[R]

• This isomorphism works in software, too!



Hyper-dual Numbers via Recursive Templates

Example

typedef DualNumber<RawType,RawType> FirstDerivType;

typedef DualNumber<FirstDerivType,FirstDerivType> SecondDerivType;

// Initializing independent vars would be ugly:

const FirstDerivType x1(pi/6,1);

const FirstDerivType one(1,0);

const SecondDerivType x(x1,one);

// But smart constructor partial specialization

// makes it independent of recursion level:

const SecondDerivType x(pi/6,1);

const FirstDerivType sinx = sin(x); // Calculations like normal

const FirstDerivType y = sinx*sinx;

const double raw_y = raw_value(y); // Recursive down-conversion

double deriv = raw_value(y.derivatives());

double deriv2 = raw_value(y.derivatives().derivatives());



Gradient Differentiation via Dual NumbersDefine:

• D[X;n] ≡ {a+∑n

i=1 biεi : a, bi ∈ X}

• Quotient ideal: εiεj = 0 ∀i, j• f(a+

∑biεi) = f(a) +

∑bif

′(a)εi

E.g. with independent variables e.g. x, y, z ∈ R, we set

X ≡ x+ ε1

Y ≡ y + ε2

Z ≡ z + ε3

and get:

f(X,Y, Z) = f(x, y, z) +∂f

∂x(x, y, z)ε1 +

∂f

∂y(x, y, z)ε2 +

∂f

∂z(x, y, z)ε3




• D[X;n] ≡ {a+∑n

i=1 biεi : a, bi ∈ X}• Quotient ideal: εiεj = 0 ∀i, j

• f(a+∑biεi) = f(a) +

∑bif

′(a)εi


X ≡ x+ ε1

Y ≡ y + ε2

Z ≡ z + ε3

and get:

f(X,Y, Z) = f(x, y, z) +∂f

∂x(x, y, z)ε1 +

∂f

∂y(x, y, z)ε2 +

∂f

∂z(x, y, z)ε3




• D[X;n] ≡ {a+∑n

i=1 biεi : a, bi ∈ X}• Quotient ideal: εiεj = 0 ∀i, j• f(a+

∑biεi) = f(a) +

∑bif

′(a)εi


X ≡ x+ ε1

Y ≡ y + ε2

Z ≡ z + ε3

and get:

f(X,Y, Z) = f(x, y, z) +∂f

∂x(x, y, z)ε1 +

∂f

∂y(x, y, z)ε2 +

∂f

∂z(x, y, z)ε3



Gradient Differentiation via Recursive TemplatesExample

typedef DualNumber<RawType,NumberArray<3,RawType> > FirstDerivType;

typedef DualNumber<FirstDerivType,NumberArray<3,FirstDerivType> >

SecondDerivType;

NumberArray<3, FirstDerivType> xyz;

xyz[0] =

FirstDerivType(x,NumberArrayUnitVector<3,0,RawType>::value());

xyz[1] =

FirstDerivType(y,NumberArrayUnitVector<3,1,RawType>::value());

xyz[2] =

FirstDerivType(z,NumberArrayUnitVector<3,2,RawType>::value());

const NumberArray<3, FirstDerivType> vec = some_function(xyz);

RawType divergence_vec = 0;

for (unsigned int i=0; i != 3; ++i)

divergence_vec += vec[i].derivatives()[i];



Vector Calculus via Recursive Templates

Playing with vectors becomes easier with various helper functions:

Example

// divergence of both vectors and tensors

template <std::size_t size, typename T>

inline

typename DerivativeType<T>::type

divergence(const NumberArray<size, T>& a)

{

typename DerivativeType<T>::type returnval = 0;

for (unsigned int i=0; i != size; ++i)

returnval += DerivativesOf<T>::derivative(a[i], i);

return returnval;

}



MASA PDE Examples

Manufactured Solution

// Arbitrary manufactured solution

U.template get<0>() = u_0 + u_x * sin(a_ux * PI * x / L) +

u_y * cos(a_uy * PI * y / L);

// Why not U[0] and U[1]? To be explained in the future...

U.template get<1>() = v_0 + v_x * cos(a_vx * PI * x / L) +

v_y * sin(a_vy * PI * y / L);

ADScalar RHO = rho_0 + rho_x * sin(a_rhox * PI * x / L) +

rho_y * cos(a_rhoy * PI * y / L);

ADScalar P = p_0 + p_x * cos(a_px * PI * x / L) +

p_y * sin(a_py * PI * y / L);

// Constitutive laws

Tensor GradU = gradient(U);

Tensor Tau = mu * (GradU + transpose(GradU) -

2./3. * divergence(U) * RawArray::identity());

FullArray q = -k * T.derivatives();



MASA PDE Examples

Euler

// Gas state

ADScalar T = P / RHO / R;

ADScalar E = 1. / (Gamma-1.) * P / RHO;

ADScalar ET = E + .5 * U.dot(U);






// Conservation equation residuals

Scalar Q_rho = raw_value(divergence(RHO*U));

RawArray Q_rho_u = raw_value(divergence(RHO*U.outerproduct(U)) +

P.derivatives());

Scalar Q_rho_e = raw_value(divergence((RHO*ET+P)*U));



MASA PDE Examples

Navier-Stokes

// Gas state

ADScalar T = P / RHO / R;

ADScalar E = 1. / (Gamma-1.) * P / RHO;

ADScalar ET = E + .5 * U.dot(U);






// Conservation equation residuals

Scalar Q_rho = raw_value(divergence(RHO*U));

RawArray Q_rho_u = raw_value(divergence(RHO*U.outerproduct(U) - Tau) +

P.derivatives());

Scalar Q_rho_e = raw_value(divergence((RHO*ET+P)*U + q - Tau.dot(U)));



Remaining Problems

• Automatic upconversion is dangerous!

Example

double problem(FirstDerivType x, SecondDerivType y) {

// This is desirable:

x *= 2.0;

// But now the compiler doesn’t see a problem with this!

return (x+y).derivatives().derivatives();

}

• Code with 2nd and higher derivatives still requires care

• Long-term solution: automatic upconversion only for built-in types,automatic downconversion for DualNumber, even to “HypodualNumber”

• Current solution: don’t use insufficiently differentiable types, or don’twrite buggy second derivative evaluations!



Remaining Problems

• Hyperduals are inefficient!

I (1 +m)n data for n derivatives in m dimensions (e.g. (1 + 3)2 = 16)I Should require less than (e.g. 10)

n∑i=0

(m+ i− 1)!

(m− 1)!i!

• Long-term solution: partial specialization of recursive DualNumber

• Current solution for large n: it’s just going to slow down MASA, so whocares?

I Pin blame on Nick somehow?

• Current solution for first derivatives in higher dimensions:

I Sparse vector data typesI Functional programming for efficient compile-time optimizationsI Functional-to-imperative APIs for efficient run-time operations



Remaining Problems

• Hyperduals are inefficient!I (1 +m)n data for n derivatives in m dimensions (e.g. (1 + 3)2 = 16)

I Should require less than (e.g. 10)

n∑i=0

(m+ i− 1)!

(m− 1)!i!








Remaining Problems

• Hyperduals are inefficient!I (1 +m)n data for n derivatives in m dimensions (e.g. (1 + 3)2 = 16)I Should require less than (e.g. 10)

n∑i=0

(m+ i− 1)!

(m− 1)!i!








Remaining Problems


n∑i=0

(m+ i− 1)!

(m− 1)!i!



I Pin blame on Nick somehow?• Current solution for first derivatives in higher dimensions:




Remaining Problems


n∑i=0

(m+ i− 1)!

(m− 1)!i!








Remaining Problems


n∑i=0

(m+ i− 1)!

(m− 1)!i!




I Sparse vector data types

I Functional programming for efficient compile-time optimizationsI Functional-to-imperative APIs for efficient run-time operations



Remaining Problems


n∑i=0

(m+ i− 1)!

(m− 1)!i!




I Sparse vector data typesI Functional programming for efficient compile-time optimizations

I Functional-to-imperative APIs for efficient run-time operations



Remaining Problems


n∑i=0

(m+ i− 1)!

(m− 1)!i!







To Be Continued


Stuff We Won’t Get To This Time Advanced Functional Programming with Templates

Outline




Stuff We Won’t Get To This Time Functional Data Structures

Outline




Stuff We Won’t Get To This Time Physics as a Graph Problem

Outline




Stuff We Won’t Get To This Time Evaluations on the Graph

Outline




Stuff We Won’t Get To This Time Differentiation on the Graph

Outline




numerical generic programming with c++ templatesusers.ices.utexas.edu/~roystgnr/dualnumber.pdf ·...

Documents