Maximum Entropy,
Maximum Entropy Production
and their
Application to Physics and Biology
Roderick C. Dewar
Research School of Biological Sciences
The Australian National University
Part 1: Maximum Entropy (MaxEnt) – an overview
Part 2: Applying MaxEnt to ecology
Part 3: Maximum Entropy Production (MEP)
Part 4: Applying MEP to physics & biology
• The problem: to predict “complex system” behaviour
• The solution: statistical mechanics
- Boltzmann microstate counting (maximum probability)
- Gibbs algorithm (MaxEnt)
• Applications of MaxEnt to equilibrium systems
- micro-canonical, canonical, grand-canonical distributions
• Physical interpretation of MaxEnt
- frequency interpretation
- information theory interpretation (Jaynes)
• Extension to non-equilibrium systems (Jaynes)
• General properties of MaxEnt distributions
Part 1: MaxEnt – an overview
• The problem: to predict “complex system” behaviour
• The solution: statistical mechanics
- Boltzmann microstate counting (maximum probability)
- Gibbs algorithm (MaxEnt)
• Applications of MaxEnt to equilibrium systems
- micro-canonical, canonical, grand-canonical distributions
• Physical interpretation of MaxEnt
- frequency interpretation
- information theory interpretation (Jaynes)
• Extension to non-equilibrium systems (Jaynes)
• General properties of MaxEnt distributions
Part 1: MaxEnt – an overview
system
energy in
What is the problem? - to predict the macroscopic behaviour of systems having many interacting degrees of freedom cells, plants, ecosystems, economies, climates …
environment
matter in
many interacting degrees of freedom
energy out
matter out
opennon-equilibrium
Poleward heat transport
SW
LW
Latitudinal heat
transport H = ?
170 W m-2
300 W m-2
T
Cold plate, Tc
Hot plate, Th
Ra < 1760
conduction
T
Cold plate, Tc
Hot plate, Th
Ra > 1760
convection
H = ?
Turbulent heat flow (Raleigh-Bénard convection)
Fsw
Flw + H + E C, H20, O2, N
T,
Ecosystem energy & mass fluxes
system
energy in
What is the problem? - to predict the macroscopic behaviour of systems having many interacting degrees of freedom cells, plants, ecosystems, economies, climates …
environment
matter in
many interacting degrees of freedom
energy out
matter out
opennon-equilibrium
many degrees of freedom statistical mechanicsstatistical mechanics
Global Circulation Models, Dynamic Ecosystem Models ….
• The problem: to predict “complex system” behaviour
• The solution: statistical mechanics
- Boltzmann microstate counting (maximum probability)
- Gibbs algorithm (MaxEnt)
• Applications of MaxEnt to equilibrium systems
- micro-canonical, canonical, grand-canonical distributions
• Physical interpretation of MaxEnt
- frequency interpretation
- information theory interpretation (Jaynes)
• Extension to non-equilibrium systems (Jaynes)
• General properties of MaxEnt distributions
Part 1: MaxEnt – an overview
W(A) = number of microstates that give macrostate A
Microstate i1 Macrostate A = less detailed description
Ludwig Boltzmann (1844 - 1906)
The most probable macrostate A is the one with the largest W(A) (assume microstates are a priori equiprobable)
SB(A) = kBlog W(A) = Boltzmann entropy of macrostate A
The most probable macrostate is the one of maximum entropy
Boltzmann microstate counting
Microstate i2
Example: N independent distinguishable particles with fixed total energy E
Macrostate A = {nj particles are in state j}
Microstate i = {the mth particle is in state jm}ε3
ε2
ε1
j
jjj
jj
jj nnnnNN εβαloglogΦ
j
jn
NAW
!
!Nn
jj
Enj
jj ε: maximise S = kBlog W subject to
j
jj
jj
ZN
nβεexp
1
βεexp
βεexp
0
Φ
jn
(large N)
Boltzmann entropy Clausius entropy
β 1/kBT
Given E, Smax = kBlogWmax = kB(βE + NlogZ)
under δE = δQ, Smax changes by δSmax = kBβ(δQ)
cf. Clausius thermodynamic entropy δSTD = δQ/T
Smax STD
BUT: microstate counting only works for non-interacting particles
pi = probability that system is in microstate i Macroscopic predictions via
J Willard Gibbs (1839 - 1903)
Gibbs algorithm
i
iiQpQ
The Gibbs algorithm
(MaxEnt)
Maximise H = -i pi log pi with respect to {pi} subject to the
constraints (C) on the system
But how do we construct pi ?
‘minimise the index of probability of
phase’
• The problem: to predict “complex system” behaviour
• The solution: statistical mechanics
- Boltzmann microstate counting (maximum probability)
- Gibbs algorithm (MaxEnt)
• Applications of MaxEnt to equilibrium systems
- micro-canonical, canonical, grand-canonical distributions
• Physical interpretation of MaxEnt
- frequency interpretation
- information theory interpretation (Jaynes)
• Extension to non-equilibrium systems (Jaynes)
• General properties of MaxEnt distributions
Part 1: MaxEnt – an overview
• Closed, isolated
• Closed
• Open
Three applications of MaxEnt (equilibrium systems)
• Microcanonical
• Canonical
• Grand-canonical
System constraints (C) Distribution (pi)
Example 1: closed, isolated system in equilibrium
C: N and E fixed
Microstate i = any N-particle state with total energy Ei restricted to E
Precise description of i and Ei depends on microscopic physics (CAN include particle interactions)
1
1
NE
iip
,ΩMaximise
i
NE
ii ppH log
,Ω
1
subject to
NE
iii
NE
ii ppp
,Ω,ΩαlogΦ
11
NEpi ,Ω
10
ip
Φbasis for Boltzmann’s microstate counting
Example 2: closed system in equilibrium
Microstate i = any N-particle state (no restriction on Ei)
E
Maximise subject to
C: N and fixedE
iC
i EZ
p βexp 1
0
ip
Φ
β 1/kBT
N
iii ppH
Ωlog
1
1
1
N
iip
Ω
EEp
N
iii
Ω
1
N
iii
N
iii
N
ii Epppp
ΩΩΩβαlogΦ
111
Hmax STD
Example 3: open system in equilibrium
Microstate i = any physically allowed microscopic state (no restriction on Ei or Ni)
E N
Maximise
1i
ip
Ω
log1i
ii ppH subject to EEpi
ii
NNpi
ii
C: and fixedEN
iiGC
i NEZ
p γβexp 1
0
ip
Φ β 1/kBT
γ -μ/kBT
ΩΩΩΩ
γβαlogΦ1111 i
iii
iii
iii
i NpEpppp
Hmax STD
• The problem: to predict “complex system” behaviour
• The solution: statistical mechanics
- Boltzmann microstate counting (maximum probability)
- Gibbs algorithm (MaxEnt)
• Applications of MaxEnt to equilibrium systems
- micro-canonical, canonical, grand-canonical distributions
• Physical interpretation of MaxEnt
- frequency interpretation
- information theory interpretation (Jaynes)
• Extension to non-equilibrium systems (Jaynes)
• General properties of MaxEnt distributions
Part 1: MaxEnt – an overview
Frequency interpretation (Venn, Pearson, Fisher …)
System has Ω a priori equiprobable microstates
N independent identical systems, ni = no. of systems in state i
pi describes a physical property of the real world (frequency)
! ... !!
!
Ωnnn
N
21
W = no. of microstates giving {n1,n2 … nΩ}
pi = ni /N = frequency of microstate i
NHppWN i
ii as loglogΩ
1
1
MaxEnt coincides with large-N limit of Maximum Probability for multinomial W
• pi represents our state of knowledge of the real world
• basic axioms for uncertainty H associated with pi the unique uncertainty function is
• Applies to any discrete set of outcomes i
pi = 1/6 (i = 1…6) H = log 6maximum uncertainty :
pi = 0 (i = 1,2...5), p6 = 1 H = 0
minimum uncertainty :
Ω
1log
iii ppH
Information theory interpretation (Shannon 1948, Jaynes 1957 …)
Claude Shannon(1916-2001)
Jaynes (1957b, 1978)
Q (= ΣipiQi) reproducible under C
it is sufficient to encode only the information C into pi …
all information other than C is thrown away
… but this is precisely what MaxEnt does!
MaxEnt = max H subject to C
H = -i pi log pi = missing information about i
Behaviour that is experimentally reproducible under conditions C must be
theoretically predictable from C aloneEdwin Jaynes (1922-1998)
Assumed constraints C
experimental conditionsconservation laws
microstates (e.g. QM)
Reproducible behaviour Q
Max H subject to C pi
The prediction game
i
iiQpQ
Observed behaviour Qobs
Qobs Q missing constraint
MaxEnt
testC C'
ESSENTIAL PHYSICS PREDICTION
OBSERVATION
• The problem: to predict “complex system” behaviour
• The solution: statistical mechanics
- Boltzmann microstate counting (maximum probability)
- Gibbs algorithm (MaxEnt)
• Applications of MaxEnt to equilibrium systems
- micro-canonical, canonical, grand-canonical distributions
• Physical interpretation of MaxEnt
- frequency interpretation
- information theory interpretation (Jaynes)
• Extension to non-equilibrium systems (Jaynes)
• General properties of MaxEnt distributions
Part 1: MaxEnt – an overview
Edwin Jaynes (aged 14 months)
Information theory interpretation of MaxEnt
general algorithm for predicting reproducible behaviour
under given constraints
can be extended to non-equilibrium systems
(same principle, different constraints)
‘Maximum caliber principle’ (Jaynes 1980, 1996)
Γ micropaths
ΓΓ logmax ppH
cf. Feynman path integral formalism of QM!
A B
The second law in a nutshell
AB reproducible WB WA' = WA SB SA
WB
WA WA'
. .microscopic
path in phase-space
after Jaynes 1963, 1988
S = kBlog W
Liouville Theorem(Hamiltonian dynamics)
reproducible macroscopic change
• The problem: to predict “complex system” behaviour
• The solution: statistical mechanics
- Boltzmann microstate counting (maximum probability)
- Gibbs algorithm (MaxEnt)
• Applications of MaxEnt to equilibrium systems
- micro-canonical, canonical, grand-canonical distributions
• Physical interpretation of MaxEnt
- frequency interpretation
- information theory interpretation (Jaynes)
• Extension to non-equilibrium systems (Jaynes)
• General properties of MaxEnt distributions
Part 1: MaxEnt – an overview
Some general properties of MaxEnt distributions
λexpλ
1
kkiki f
Zp
1i
ip
m k,Ffpf ki
kiik 1
k
kk FλZH λlogmax
k
kZ
Fλ
λlog
kjkjkj
kjk
jjk Cffff
ZFC
λλ
λlog
λ
2
subject to m + 1 constraints C
i
ii ppH logmax
Fλ
k
k F
FHF
maxλ FHmax
i kkik fZ λexpλ
;
λ max2
kjkjj
kjk B
FF
FH
FB
1CB
Response-fluctuation & reciprocity relations:
Stability-convexity relation:
Constitutive relation: Orthogonality:
Partition function:
Summary of Lecture 1 …
The problem
to predict the behaviour of non-equilibrium systems with many degrees of freedom
The proposed solution
MaxEnt: a general information-theoretical algorithm for predicting reproducible behaviour under given constraints
Boltzmann
Gibbs
Shannon
Jaynes