(An attempt of)Introduction to ad-3.4,

an automatic differentiation library in Haskell

by Nebuta

3/31/2013Ekmett study meeting

in Shibuya, Tokyo

Any comments or correction to the material are welcome

About myself

Nebuta (@nebutalab) https://github.com/nebuta

My research areas:A graduate student, studying biophysical chemistry and quantitative biology (2010−)Imaging live cells, analyzing microscopy images by Scala on ImageJ

My interest in softwares:Programming languages (Haskell, Scala, Ruby, etc)Image processing, data visualization, web designBrainstorming and lifehack methods that take advantage of IT, etc.

Where my interest in Haskell came from:MATLAB、ImageJで細胞の顕微鏡画像の解析 (2010年) → MATLAB, Javaはいまいち使いづらい → Scalaっていうイケてる言語がある（2011年） → 関数型？ → Haskell 面白い！（2011年）

ad-3.4, an automatic differentiation library

Installation

array (≥0.2 & <0.5), base (4.*), comonad (≥3), containers (≥0.2 & <0.6), data-reify (0.6.*), erf (2.0.*),free (≥3), mtl (≥2), reflection (≥1.1.6),tagged (≥0.4.2.1), template-haskell (≥2.5 & <2.9)

Dependencies

Differentiation of arbitrary mathematical functions

$ sudo cabal install ad simple-reflect

For symbolic differentiation

What you can do

Taylor expansionCalculation of gradient, Jacobian, and Hessian, etc.

記号で微分するのに使う

How to use ad-3.4

>> :m + Numeric.AD>> diff sin 01.0>> :m + Debug.SimpleReflect>> diff sin x -- x :: Expr is defined in Debug.SimpleReflectcos x * 1>> diff (\x -> if x >= 0 then 1 else 0) 00.0

https://github.com/ekmett/ad/blob/master/README.markdown#examples

Not delta function nor undefined.

Derivative with a symbol!

※Derivative of atrigonometric function

Differentiation of a single-variable scalar function

Gradient>> grad (\[x,y] -> exp (x * y)) [x,y][0 + (0 + y * (0 + exp (x * y) * 1)),0 + (0 + x * (0 + exp (x * y) * 1))]>> grad (\[x,y] -> exp (x * y)) [1,1][2.718281828459045,2.718281828459045]

How to use (continued)

Taylor expansionPrelude Numeric.AD Debug.SimpleReflect> take 3 $ taylor exp 0 d[exp 0 * 1,1 * exp 0 * (1 * d / 1),(0 * exp 0 + 1 * exp 0 * 1) * (1 * d / 1 * d / (1 + 1))]Prelude Numeric.AD Debug.SimpleReflect> take 3 $ taylor exp x d[exp x * 1,1 * exp x * (1 * d / 1),(0 * exp x + 1 * exp x * 1) * (1 * d / 1 * d / (1 + 1))]Prelude Numeric.AD Debug.SimpleReflect> take 3 $ taylor exp x 0[exp x * 1,1 * exp x * (1 * 0 / 1),(0 * exp x + 1 * exp x * 1) * (1 * 0 / 1 * 0 / (1 + 1))]

•Taylor expansion is an infinite list!•No simplification, and slow in higher order terms

Taylor expansion (general)

Exponential function

Equality of functions>> sin x == sin xTrue>> diff sin xcos x * 1>> diff sin x == cos x * 1True>> diff sin x == cos x * 0.5 * 2False

And so on.

Cool! (no simplification, though...)

How to use (continued)

Cf. Mechanism of automatic differentiationhttp://en.wikipedia.org/wiki/Automatic_differentiation

？？

(f + g)’ = f’ + g’

要は、合成関数の微分を機械的に順次適用していく、という認識で良いかと思われる

I don’t understand it yet.（What’s the difference from symbolic differentiation?）

Read a Wikipedia article

It seems to be mechanical, successive application of rules of differentiation for

composite functions.

I’ll try to appreciate the type and class organization

注意：Githubに上がっている最新バージョン（https://github.com/ekmett/ad）は4.0で、Hackage(http://hackage.haskell.org/package/ad-3.4)とは違います。ライブラリの構造が若干違うようです。

cabal unpack ad-3.4 でver. 3.4のソースをダウンロードするか、Hackageを見て下さい。

I’ll use ad-3.4 on Hackage, not ad-4.0 on Github

Package structure

いろいろな自動微分の”モード”の実装

ここでは複数のモードの実装のうち、デフォルトのモードがインポート＆再エクスポートされている

型の定義

クラスの定義

http://new-hackage.haskell.org/package/ad-3.4

The starting point for exploration: diff functionNumeric.AD.Mode.Forward{-# LANGUAGE Rank2Types #-}diff :: Num a => (forall s. Mode s => AD s a -> AD s a) -> a -> adiff f a = tangent $ apply f a

Numeric.AD.Internal.Forward

{-# LANGUAGE Rank2Types, TypeFamilies, DeriveDataTypeable, TemplateHaskell, UndecidableInstances, BangPatterns #-}tangent :: Num a => AD Forward a -> atangent (AD (Forward _ da)) = datangent _ = 0

bundle :: a -> a -> AD Forward abundle a da = AD (Forward a da)

apply :: Num a => (AD Forward a -> b) -> a -> bapply f a = f (bundle a 1)

この部分の型：AD Forward a

微分対象の関数fは(AD s a->AD s a)型として表される

１変数スカラー関数の微分

どうやらAD型が鍵になるようだ。

AD typeNumeric.AD.Types

An instance of Mode class determines the behavior.

{-# LANGUAGE CPP #-}{-# LANGUAGE Rank2Types, GeneralizedNewtypeDeriving, TemplateHaskell, FlexibleContexts, FlexibleInstances, MultiParamTypeClasses, UndecidableInstances #-}

{-# ANN module "HLint: ignore Eta reduce" #-}

newtype AD f a = AD { runAD :: f a } deriving (Iso (f a), Lifted, Mode, Primal)

e.g.) AD (Forward 10 1) :: Num a => AD Forward a

Classes that have AD type as an instanceNumeric.AD.Types http://hackage.haskell.org/packages/archive/ad/3.4/doc/

html/Numeric-AD-Types.html#t:AD

（fがLiftedのインスタンス &&aがFloatingのインスタンス）のとき、　AD f a　はFloatingのインスタンス。

AD s a -> AD s adiffの第一引数の型

は、以下の型に適合する＊

Floating a => a -> a

＊こういう言い方が正確か分からないが。

例：１変数実数スカラー関数 y = f(x)

Let’s look at some examples of values with AD type

import Numeric.ADimport Numeric.AD.Typesimport Numeric.AD.Internal.Classesimport Numeric.AD.Internal.Forward

f x = x + 3g x | x > 0 = x | otherwise = 0

d = diff (g . f)

*Main> :t (g . f)(g . f) :: (Num c, Ord c) => c -> c*Main> :t (g . f) :: (Num a, Ord a, Mode s) => AD s a -> AD s a(g . f) :: (Num a, Ord a, Mode s) => AD s a -> AD s a :: (Num a, Ord a, Mode s) => AD s a -> AD s a*Main> :t ff :: Num a => a -> a*Main> :t f :: (Num a, Lifted s) => AD s a -> AD s af :: (Num a, Lifted s) => AD s a -> AD s a :: (Num a, Lifted s) => AD s a -> AD s a*Main> :t gg :: (Num a, Ord a) => a -> a*Main> :t dd :: Integer -> Integer

GHCi

adtest.hs

> :l adtest.hs[1 of 1] Compiling Main ( adtest.hs, interpreted )Ok, modules loaded: Main.[*Main]> :t applyapply :: Num a => (AD Forward a -> b) -> a -> b[*Main]> :t apply fapply f :: Num a => a -> AD Forward a[*Main]> :t apply f 0apply f 0 :: Num a => AD Forward a[*Main]> apply f 03

Let’s play around with apply and tangentGHCi

[*Main]> :t tangenttangent :: Num a => AD Forward a -> a[*Main]> :t tangent $ apply f 0tangent $ apply f 0 :: Num a => a[*Main]> tangent $ apply f 01

Since f :: Num t0 => t0 -> t0,b in the type of apply is restricted to AD Forward a

http://hackage.haskell.org/packages/archive/ad/3.4/doc/html/Numeric-AD-Internal-Classes.html#t:Mode

http://hackage.haskell.org/packages/archive/ad/3.4/doc/html/Numeric-AD-Internal-Classes.html#t:Lifted

Lifted class and Mode classChain rule of differentiation

Definition of different modes.Mode is a subclass of Lifted

Wrap a (Num a => a) value into t

e.g.) Forward is an instance of Lifted and Mode

Lifted class defines chain rules of differentiation

deriveLifted (in Numeric.AD.Internal.Classes)というTemplate Haskellの関数がLiftedのインスタンスを作ってくれる。

http://hackage.haskell.org/packages/archive/ad/3.4/doc/html/src/Numeric-AD-Internal-Classes.html#deriveLifted

exp1 = lift1_ exp const log1 = lift1 log recip1

sin1 = lift1 sin cos1 cos1 = lift1 cos $ negate1 . sin1 tan1 = lift1 tan $ recip1 . square1 . cos1

A part of deriveLifted

deriveNumeric generates instances of Num, etc.

-- | @'deriveNumeric' f g@ provides the following instances:---- > instance ('Lifted' $f, 'Num' a, 'Enum' a) => 'Enum' ($g a)-- > instance ('Lifted' $f, 'Num' a, 'Eq' a) => 'Eq' ($g a)-- > instance ('Lifted' $f, 'Num' a, 'Ord' a) => 'Ord' ($g a)-- > instance ('Lifted' $f, 'Num' a, 'Bounded' a) => 'Bounded' ($g a)---- > instance ('Lifted' $f, 'Show' a) => 'Show' ($g a)-- > instance ('Lifted' $f, 'Num' a) => 'Num' ($g a)-- > instance ('Lifted' $f, 'Fractional' a) => 'Fractional' ($g a)-- > instance ('Lifted' $f, 'Floating' a) => 'Floating' ($g a)-- > instance ('Lifted' $f, 'RealFloat' a) => 'RealFloat' ($g a)-- > instance ('Lifted' $f, 'RealFrac' a) => 'RealFrac' ($g a)-- > instance ('Lifted' $f, 'Real' a) => 'Real' ($g a)

deriveNumeric (in Numeric.AD.Internal.Classes)

http://hackage.haskell.org/packages/archive/ad/3.4/doc/html/src/Numeric-AD-Internal-Classes.html#deriveNumeric

Definition of deriveNumericderiveNumeric :: ([Q Pred] -> [Q Pred]) -> Q Type -> Q [Dec]deriveNumeric f t = do members <- liftedMembers let keep n = nameBase n `elem` members xs <- lowerInstance keep ((classP ''Num [varA]:) . f) t `mapM` [''Enum, ''Eq, ''Ord, ''Bounded, ''Show] ys <- lowerInstance keep f t `mapM` [''Num, ''Fractional, ''Floating, ''RealFloat,''RealFrac, ''Real, ''Erf, ''InvErf] return (xs ++ ys)

lowerInstance :: (Name -> Bool) -> ([Q Pred] -> [Q Pred]) -> Q Type -> Name -> Q DeclowerInstance p f t n = do#ifdef OldClassI ClassI (ClassD _ _ _ _ ds) <- reify n#else ClassI (ClassD _ _ _ _ ds) _ <- reify n#endif instanceD (cxt (f [classP n [varA]])) (conT n `appT` (t `appT` varA)) (concatMap lower1 ds) where lower1 :: Dec -> [Q Dec] lower1 (SigD n' _) | p n'' = [valD (varP n') (normalB (varE n'')) []] where n'' = primed n' lower1 _ = []

primed n' = mkName $ base ++ [prime] where base = nameBase n' h = head base prime | isSymbol h || h `elem` "/*-<>" = '!' | otherwise = '1'

This looks an important part, but a bit difficult....

A pitfall?>> :t diff ((**2) :: Floating a => a -> a)diff ((**2) :: Floating a => a -> a) :: Floating a => a -> a>> :t diff ((**2) :: Double -> Double)

<interactive>:1:7: Couldn't match expected type `ad-3.4:Numeric.AD.Internal.Types.AD s a0' with actual type `Double' Expected type: ad-3.4:Numeric.AD.Internal.Types.AD s a0 -> ad-3.4:Numeric.AD.Internal.Types.AD s a0 Actual type: Double -> Double In the first argument of `diff', namely `((** 2) :: Double -> Double)' In the expression: diff ((** 2) :: Double -> Double)

You need to keep functions polymorphic for differentiation.

import Numeric.AD

func x = x ** 2func2 x = x ** 3

fs = [func, func2]test = map (\f -> diff f 1.0) fs

So, this does not compile. (there is a GHC extension to accept this, isn’t there..?)

Summary

Things that might be improved for application

• Simplification of terms

These might be a good practice for seasoned Haskellers (I’m not...)

• Improving the performance of higher order taylor expansion.

• Is automatic integration possible, maybe?

Sorry, but I’m still unclear about how Lifted actually executes differentiation

It seems to be done by Num instance of AD f a, viainstance (Lifted f, Num a) => Num (AD f a)...

An interesting library that shows how to use types and typeclasses effectively and beautifully.

感想

•他のEkmett氏のライブラリよりは具体的な対象がはっきりしている分、少しは分かりやすい。

•Haskellならではのコードの簡潔さのおかげで、ソースのどこを追っていけば良いかはわりと明確。

•とはいっても、私には若干（相当？）手に余る感じで、深いところまでは解説出来ませんでした。知らないGHC

拡張も多し。ただ、人に説明しようとすることで少しは理解が進んで、嬉しい。