from variable elimination to junction trees
DESCRIPTION
From Variable Elimination to Junction Trees. Yaniv Hamo and Mark Silberstein. Variable Elimination – what is it and why we need it. R. Reference. S. Submit HW. P. Pass course. Variable elimination is needed for answering questions such as “ so, do I pass this course or not? ”. - PowerPoint PPT PresentationTRANSCRIPT
1
From Variable Elimination to From Variable Elimination to Junction TreesJunction Trees
Yaniv Hamo and Mark Silberstein
2
Variable Elimination ndash what is it and Variable Elimination ndash what is it and why we need itwhy we need it
R
S
P
Reference
Submit HW
Pass course
exists01
not exists09
existsnot exists
yes0804
no0206
yesno
pass0905
fail0105
Variable elimination is needed for answering questions such as ldquoso do I pass this course or notrdquo
3
So do I pass this course or notSo do I pass this course or not
bull We want to compute P(p)
bull By definition
bull In our case (chain)
r s r s
srpPrsPrPpsrPpP )|()|()()()(
r s
spPrsPrPpP )|()|()()(
r s
spPrsPrPpP )|()|()()(
P(p) = 01(0809+0205)+09(0409+0605) = 0676
We essentially eliminated nodes R and S
4
The General Case ndash InferenceThe General Case ndash Inference
bull Network describes a unique probability distribution Pbull We use inference as a name for the process of computing
answers to queries about Pbull There are many types of queries we might ask bull Most of these involve evidence
ndash An evidence e is an assignment of values to a set E variables in the domain
ndash Without loss of generality E = Xk+1 hellip Xn
bull Simplest query compute probability of evidence
bull This is often referred to as computing the likelihood of the evidence
1x
1 )( )(kx
kxxPP ee
5
Another example of Variable Another example of Variable EliminationElimination
bull The ldquoAsiardquo network
Visit to Asia
Smoking
Lung CancerTuberculosis
Abnormalityin Chest
Bronchitis
X-Ray Dyspnea
6
V S
LT
A B
X D
)|( )|( )|( )|( )|( )|( )( )( badPaxPltaPsbPslPvtPsPvP
We are interested in P(d)
- Need to eliminate vsxtlab
Initial factors
Brute force
v s x t l a b
badPaxPltaPsbPslPvtPsPvPdP )|( )|( )|( )|( )|( )|( )( )()(
7
V S
LT
A B
X D
)|( )|( )|( )|( )|( )|( )( )( badPaxPltaPsbPslPvtPsPvP
Eliminate variables in order
Initial factors
v
v vtPvPtf )|()()(
baltxsv
)|()|()|()|()|()()( badPaxPltaPsbPslPsPtfv
[ Note fv(t) = P(t) In general result of elimination is not necessarily a probability term ]
8
)|( )|( )|( )|( )|( )|( )( )( badPaxPltaPsbPslPvtPsPvP
Eliminate variables in order
Initial factors
V S
LT
A B
X D
baltxsv
)|( )|( )|( )|( )|( )( )( badPaxPltaPsbPslPsPtfv
s
s slPsbPsPlbf )|()|()()(
)|()|()|()()( badPaxPltaPlbftf sv
[ Note result of elimination may be a function of several variables ]
9
)|()|()()()( badPltaPaflbftf xsv
)|( )|( )|( )|( )|( )|( )( )( badPaxPltaPsbPslPvtPsPvP
Eliminate variables in order
Initial factors
V S
LT
A B
X D
baltxsv
)|( )|( )|( )|( )|( )( )( badPaxPltaPsbPslPsPtfv)|( )|( )|( )( )( badPaxPltaPlbftf sv
x
x axPaf )|()([ Note fx(a) = 1 for all values of a ]
10
)|( )|( )( )( )( badPltaPaflbftf xsv
)|( )|( )|( )|( )|( )|( )( )( badPaxPltaPsbPslPvtPsPvP
Eliminate variables in order
Initial factors
V S
LT
A B
X D
baltxsv
)|( )|( )|( )|( )|( )( )( badPaxPltaPsbPslPsPtfv)|( )|( )|( )( )( badPaxPltaPlbftf sv
t
vt ltaPtflaf )|()()(
)|()()()( badPlafaflbf txs
11
)|( )|( )( )( )( badPltaPaflbftf xsv
)|( )|( )|( )|( )|( )|( )( )( badPaxPltaPsbPslPvtPsPvP
Eliminate variables in order
Initial factors
V S
LT
A B
X D
baltxsv
)|( )|( )|( )|( )|( )( )( badPaxPltaPsbPslPsPtfv)|( )|( )|( )( )( badPaxPltaPlbftf sv
)|( )( )( )( badPlafaflbf txs
l
tsl laflbfbaf )()()( )|()()( badPafbaf xl
12
)|( )( )( badPafbaf xl
)|( )|( )( )( )( badPltaPaflbftf xsv
)|( )|( )|( )|( )|( )|( )( )( badPaxPltaPsbPslPvtPsPvP
Eliminate variables in order
Initial factors
V S
LT
A B
X D
baltxsv
)|( )|( )|( )|( )|( )( )( badPaxPltaPsbPslPsPtfv)|( )|( )|( )( )( badPaxPltaPlbftf sv
)|( )( )( )( badPlafaflbf txs
a
xla badpafbafdbf )|()()()( )( dbfa
13
)|( )( )( badPafbaf xl
)|( )|( )( )( )( badPltaPaflbftf xsv
)|( )|( )|( )|( )|( )|( )( )( badPaxPltaPsbPslPvtPsPvP
Eliminate variables in order
Initial factors
V S
LT
A B
X D
baltxsv
)|( )|( )|( )|( )|( )( )( badPaxPltaPsbPslPsPtfv)|( )|( )|( )( )( badPaxPltaPlbftf sv
)|( )( )( )( badPlafaflbf txs
)( dbfa )()()( dfdbfdf bb
ab
14
Intermediate factorsIntermediate factors
baltxsv
ga (ltdbx)
gb (ltdxs)
gx (ltds)
gt (ltsv)
gv (lds)
gs(ld)
gl (d))()()()(
)()(
)(
dfdbfbaflaf
aflbf
tf
b
a
l
t
x
s
v
lsvtxba
In our previous example With a different ordering
V S
LT
A B
X D
Complexity is exponential in the size of these factors
15
Notes about variable Notes about variable eliminationelimination
bull Actual computation is done in the elimination steps
bull Computation depends on the order of elimination
bull For each query we need to compute everything againndash Many redundant calculations
16
The ideaThe idea
bull Compute joint over partitions of U
ndash 1048708 small subset of U (typically made of a variable and its parents) - clusters
ndash not necessary disjoint
ndash Calculate
ndash To compute P(X) need far less operations
iC
iiCU
XC
i
i
CPXP
)()(
iC)( iCP
17
Junction TreesJunction Trees
bull The junction tree algorithms generalize Variable Elimination to the efficient simultaneous execution of a large class of queries
bull Theoretical background was shown in the previous lecture
18
Constructing Junction TreesConstructing Junction Trees
1 Moralize the graph (if directed)
2 Choose a node ordering and find the cliques generated by variable elimination This gives a triangulation of the graph
3 Build a junction graph from the eliminated cliques
4 Find an appropriate spanning tree
19
Step 1 MoralizationStep 1 Moralization
a
b c
d e
f
g
h
a
b c
d e
f
g
h
a
b c
d e
f
g
h
1 For all w Vbull For all uvpa(w) add an edge e=u-v
2 Undirect all edges
GMG = ( V E )
20
Step 2 TriangulationStep 2 Triangulation
Add edges to GM such that there is no cyclewith length 4 that does not contain a chord
NO YES
a
b c
d e
f
g
h
a
b c
d e
f
g
h
GM GT
21
Step 2 Triangulation (cont)Step 2 Triangulation (cont)
bull Each elimination ordering triangulates the graph not necessarily in the same way
A
H
B
D
F
C
E
G
A
H
B
D
F
C
E
G
A
H
B
D
F
C
E
G
A
H
B
D
F
C
E
G
A
H
B
D
F
C
E
G
A
H
B
D
F
C
E
G
A
H
B
D
F
C
E
G
A
H
B
D
F
C
E
G
A
H
B
D
F
C
E
G
22
Step 2 Triangulation (cont)Step 2 Triangulation (cont)
bull Intuitively triangulations with as few fill-ins as possible are preferredndash Leaves us with small cliques (small probability tables)
bull A common heuristic Repeat until no nodes remain
ndash Find the node whose elimination would require the least number of fill-ins (may be zero)
ndash Eliminate that node and note the need for a fill-in edge between any two non-adjacent neighbors
bull Add the fill-in edges to the original graph
23
a
b c
d e
f
g
h
a
b c
d e
f
g
h
a
b c
d e
f
g
a
b c
d e
f
a
b c
d e
a
b
d e
aa
e
a
d e
vertex induced addedremoved clique edges
1 h egh -2 g ceg -3 f def -4 c ace a-e
vertex induced added removed clique edges
5 b abd a-d6 d ade -7 e ae -8 a a -
GT
GM
Eliminate the vertex that requires least number of edges to be added
24
Step 3 Junction GraphStep 3 Junction Graph
bull A junction graph for an undirected graph G is an undirected labeled graph
bull The nodes are the cliques in G
bull If two cliques intersect they are joined in the junction graph by an edge labeled with their intersection
25
a
b
d
a
c
e
d e
f
a
d e
e
g
h
c
e
g
a
b c
d e
f
g
h
Bayesian NetworkG = ( V E )
a
b c
d e
f
g
h
a
b c
d e
f
g
h
Moral graph GM Triangulated graph GT
abd
ade
ace
ceg
eghdef
ad ae ce
de eg
seperators
Junction graph GJ (not complete)eg ceg egh = eg Cliques
e
e
e
a
e
26
Step 4 Junction TreeStep 4 Junction Tree
bull A junction tree is a sub-graph of the junction graph that ndash Is a tree
ndash Contains all the cliques (spanning tree)
ndash Satisfies the running intersection property
for each pair of nodes U V all nodes on the path between U and V contain
(as seen in the previous part of the lecture)
VU
27
Step 4 Junction Tree (cont)Step 4 Junction Tree (cont)
bull Theorem An undirected graph is triangulated if and only if its junction graph has a junction tree
bull Definition The weight of a link in a junction graph is the number of variable in the label The weight of a junction tree is the sum of weights of the labels
bull Theorem A sub-tree of the junction graph of a triangulated graph is a junction tree if and only if it is a spanning of maximal weight
28
Junction graph GJ (not complete)
abd
ade
ace
ceg
eghdef
ad ae ce
de eg
e
e
e
a
e
abd
ade
ace
ceg
eghdef
ad ae ce
de eg
Junction tree GJT
There are several methods to find MST
Kruskalrsquos algorithm choose successively a link of
maximal weight unless it creates a cycle
29
Another exampleAnother example
bull Compute the elimination cliques(the order here is f d e c b a)
bull Form the complete junction graph over the maximal elimination cliques and find a maximum-weight spanning tree
30
Junction Trees and Elimination Junction Trees and Elimination OrderOrder
bull We can use different orderings in variable elimination - affects efficiency
bull Each ordering corresponds to a junction tree
bull Just as some elimination orderings are more efficient than others some junction trees are better than others (Recall our mention of heuristics for triangulation)
31
OK I have this tree now whatOK I have this tree now what
bull A separator S divides the remaining variables into two groupsndash Variables in each group appear
on one side in the cluster tree
bull Examplesndash AB L S T V amp D X
ndash AL T V amp BDSX
ndash BL S amp A DT V X
ndash A X amp BDL S T V
ndash T V amp A B D K S X
LT
A B
X
V S
D
TV
ALTBLS
XAALB
ABD
AAB
BL
T
AL
32
Elimination in Junction TreesElimination in Junction Trees
bull Let X and Y be the partition induced by S
Observation
bull Eliminating all variables in X results in a factor fX(S)
bull Proof Since S is a separator only variables in S are adjacentto variables in X
bull NoteThe same factor would result regardless of theelimination order
x
y
A BSfX(S)
fY(S)
33
Recursive Elimination in Recursive Elimination in Junction TreesJunction Trees
bull How do we compute fX(S)
bull By recursive decomposition alongcluster tree
ndash Let X1 and X2 be the disjoint partitioning of X C implied by theseparators S1 and S2
ndash Eliminate X1 to get fX1(S1)
ndash Eliminate X2 to get fX2(S2)
ndash Eliminate variables in C S toget fX(S)
C
S
S2S1
x1
x2
y
- Slide 6
- Slide 7
- Slide 8
- Slide 9
- Slide 10
- Slide 11
- Slide 12
- Slide 13
- Slide 23
- Slide 25
- Slide 28
-
2
Variable Elimination ndash what is it and Variable Elimination ndash what is it and why we need itwhy we need it
R
S
P
Reference
Submit HW
Pass course
exists01
not exists09
existsnot exists
yes0804
no0206
yesno
pass0905
fail0105
Variable elimination is needed for answering questions such as ldquoso do I pass this course or notrdquo
3
So do I pass this course or notSo do I pass this course or not
bull We want to compute P(p)
bull By definition
bull In our case (chain)
r s r s
srpPrsPrPpsrPpP )|()|()()()(
r s
spPrsPrPpP )|()|()()(
r s
spPrsPrPpP )|()|()()(
P(p) = 01(0809+0205)+09(0409+0605) = 0676
We essentially eliminated nodes R and S
4
The General Case ndash InferenceThe General Case ndash Inference
bull Network describes a unique probability distribution Pbull We use inference as a name for the process of computing
answers to queries about Pbull There are many types of queries we might ask bull Most of these involve evidence
ndash An evidence e is an assignment of values to a set E variables in the domain
ndash Without loss of generality E = Xk+1 hellip Xn
bull Simplest query compute probability of evidence
bull This is often referred to as computing the likelihood of the evidence
1x
1 )( )(kx
kxxPP ee
5
Another example of Variable Another example of Variable EliminationElimination
bull The ldquoAsiardquo network
Visit to Asia
Smoking
Lung CancerTuberculosis
Abnormalityin Chest
Bronchitis
X-Ray Dyspnea
6
V S
LT
A B
X D
)|( )|( )|( )|( )|( )|( )( )( badPaxPltaPsbPslPvtPsPvP
We are interested in P(d)
- Need to eliminate vsxtlab
Initial factors
Brute force
v s x t l a b
badPaxPltaPsbPslPvtPsPvPdP )|( )|( )|( )|( )|( )|( )( )()(
7
V S
LT
A B
X D
)|( )|( )|( )|( )|( )|( )( )( badPaxPltaPsbPslPvtPsPvP
Eliminate variables in order
Initial factors
v
v vtPvPtf )|()()(
baltxsv
)|()|()|()|()|()()( badPaxPltaPsbPslPsPtfv
[ Note fv(t) = P(t) In general result of elimination is not necessarily a probability term ]
8
)|( )|( )|( )|( )|( )|( )( )( badPaxPltaPsbPslPvtPsPvP
Eliminate variables in order
Initial factors
V S
LT
A B
X D
baltxsv
)|( )|( )|( )|( )|( )( )( badPaxPltaPsbPslPsPtfv
s
s slPsbPsPlbf )|()|()()(
)|()|()|()()( badPaxPltaPlbftf sv
[ Note result of elimination may be a function of several variables ]
9
)|()|()()()( badPltaPaflbftf xsv
)|( )|( )|( )|( )|( )|( )( )( badPaxPltaPsbPslPvtPsPvP
Eliminate variables in order
Initial factors
V S
LT
A B
X D
baltxsv
)|( )|( )|( )|( )|( )( )( badPaxPltaPsbPslPsPtfv)|( )|( )|( )( )( badPaxPltaPlbftf sv
x
x axPaf )|()([ Note fx(a) = 1 for all values of a ]
10
)|( )|( )( )( )( badPltaPaflbftf xsv
)|( )|( )|( )|( )|( )|( )( )( badPaxPltaPsbPslPvtPsPvP
Eliminate variables in order
Initial factors
V S
LT
A B
X D
baltxsv
)|( )|( )|( )|( )|( )( )( badPaxPltaPsbPslPsPtfv)|( )|( )|( )( )( badPaxPltaPlbftf sv
t
vt ltaPtflaf )|()()(
)|()()()( badPlafaflbf txs
11
)|( )|( )( )( )( badPltaPaflbftf xsv
)|( )|( )|( )|( )|( )|( )( )( badPaxPltaPsbPslPvtPsPvP
Eliminate variables in order
Initial factors
V S
LT
A B
X D
baltxsv
)|( )|( )|( )|( )|( )( )( badPaxPltaPsbPslPsPtfv)|( )|( )|( )( )( badPaxPltaPlbftf sv
)|( )( )( )( badPlafaflbf txs
l
tsl laflbfbaf )()()( )|()()( badPafbaf xl
12
)|( )( )( badPafbaf xl
)|( )|( )( )( )( badPltaPaflbftf xsv
)|( )|( )|( )|( )|( )|( )( )( badPaxPltaPsbPslPvtPsPvP
Eliminate variables in order
Initial factors
V S
LT
A B
X D
baltxsv
)|( )|( )|( )|( )|( )( )( badPaxPltaPsbPslPsPtfv)|( )|( )|( )( )( badPaxPltaPlbftf sv
)|( )( )( )( badPlafaflbf txs
a
xla badpafbafdbf )|()()()( )( dbfa
13
)|( )( )( badPafbaf xl
)|( )|( )( )( )( badPltaPaflbftf xsv
)|( )|( )|( )|( )|( )|( )( )( badPaxPltaPsbPslPvtPsPvP
Eliminate variables in order
Initial factors
V S
LT
A B
X D
baltxsv
)|( )|( )|( )|( )|( )( )( badPaxPltaPsbPslPsPtfv)|( )|( )|( )( )( badPaxPltaPlbftf sv
)|( )( )( )( badPlafaflbf txs
)( dbfa )()()( dfdbfdf bb
ab
14
Intermediate factorsIntermediate factors
baltxsv
ga (ltdbx)
gb (ltdxs)
gx (ltds)
gt (ltsv)
gv (lds)
gs(ld)
gl (d))()()()(
)()(
)(
dfdbfbaflaf
aflbf
tf
b
a
l
t
x
s
v
lsvtxba
In our previous example With a different ordering
V S
LT
A B
X D
Complexity is exponential in the size of these factors
15
Notes about variable Notes about variable eliminationelimination
bull Actual computation is done in the elimination steps
bull Computation depends on the order of elimination
bull For each query we need to compute everything againndash Many redundant calculations
16
The ideaThe idea
bull Compute joint over partitions of U
ndash 1048708 small subset of U (typically made of a variable and its parents) - clusters
ndash not necessary disjoint
ndash Calculate
ndash To compute P(X) need far less operations
iC
iiCU
XC
i
i
CPXP
)()(
iC)( iCP
17
Junction TreesJunction Trees
bull The junction tree algorithms generalize Variable Elimination to the efficient simultaneous execution of a large class of queries
bull Theoretical background was shown in the previous lecture
18
Constructing Junction TreesConstructing Junction Trees
1 Moralize the graph (if directed)
2 Choose a node ordering and find the cliques generated by variable elimination This gives a triangulation of the graph
3 Build a junction graph from the eliminated cliques
4 Find an appropriate spanning tree
19
Step 1 MoralizationStep 1 Moralization
a
b c
d e
f
g
h
a
b c
d e
f
g
h
a
b c
d e
f
g
h
1 For all w Vbull For all uvpa(w) add an edge e=u-v
2 Undirect all edges
GMG = ( V E )
20
Step 2 TriangulationStep 2 Triangulation
Add edges to GM such that there is no cyclewith length 4 that does not contain a chord
NO YES
a
b c
d e
f
g
h
a
b c
d e
f
g
h
GM GT
21
Step 2 Triangulation (cont)Step 2 Triangulation (cont)
bull Each elimination ordering triangulates the graph not necessarily in the same way
A
H
B
D
F
C
E
G
A
H
B
D
F
C
E
G
A
H
B
D
F
C
E
G
A
H
B
D
F
C
E
G
A
H
B
D
F
C
E
G
A
H
B
D
F
C
E
G
A
H
B
D
F
C
E
G
A
H
B
D
F
C
E
G
A
H
B
D
F
C
E
G
22
Step 2 Triangulation (cont)Step 2 Triangulation (cont)
bull Intuitively triangulations with as few fill-ins as possible are preferredndash Leaves us with small cliques (small probability tables)
bull A common heuristic Repeat until no nodes remain
ndash Find the node whose elimination would require the least number of fill-ins (may be zero)
ndash Eliminate that node and note the need for a fill-in edge between any two non-adjacent neighbors
bull Add the fill-in edges to the original graph
23
a
b c
d e
f
g
h
a
b c
d e
f
g
h
a
b c
d e
f
g
a
b c
d e
f
a
b c
d e
a
b
d e
aa
e
a
d e
vertex induced addedremoved clique edges
1 h egh -2 g ceg -3 f def -4 c ace a-e
vertex induced added removed clique edges
5 b abd a-d6 d ade -7 e ae -8 a a -
GT
GM
Eliminate the vertex that requires least number of edges to be added
24
Step 3 Junction GraphStep 3 Junction Graph
bull A junction graph for an undirected graph G is an undirected labeled graph
bull The nodes are the cliques in G
bull If two cliques intersect they are joined in the junction graph by an edge labeled with their intersection
25
a
b
d
a
c
e
d e
f
a
d e
e
g
h
c
e
g
a
b c
d e
f
g
h
Bayesian NetworkG = ( V E )
a
b c
d e
f
g
h
a
b c
d e
f
g
h
Moral graph GM Triangulated graph GT
abd
ade
ace
ceg
eghdef
ad ae ce
de eg
seperators
Junction graph GJ (not complete)eg ceg egh = eg Cliques
e
e
e
a
e
26
Step 4 Junction TreeStep 4 Junction Tree
bull A junction tree is a sub-graph of the junction graph that ndash Is a tree
ndash Contains all the cliques (spanning tree)
ndash Satisfies the running intersection property
for each pair of nodes U V all nodes on the path between U and V contain
(as seen in the previous part of the lecture)
VU
27
Step 4 Junction Tree (cont)Step 4 Junction Tree (cont)
bull Theorem An undirected graph is triangulated if and only if its junction graph has a junction tree
bull Definition The weight of a link in a junction graph is the number of variable in the label The weight of a junction tree is the sum of weights of the labels
bull Theorem A sub-tree of the junction graph of a triangulated graph is a junction tree if and only if it is a spanning of maximal weight
28
Junction graph GJ (not complete)
abd
ade
ace
ceg
eghdef
ad ae ce
de eg
e
e
e
a
e
abd
ade
ace
ceg
eghdef
ad ae ce
de eg
Junction tree GJT
There are several methods to find MST
Kruskalrsquos algorithm choose successively a link of
maximal weight unless it creates a cycle
29
Another exampleAnother example
bull Compute the elimination cliques(the order here is f d e c b a)
bull Form the complete junction graph over the maximal elimination cliques and find a maximum-weight spanning tree
30
Junction Trees and Elimination Junction Trees and Elimination OrderOrder
bull We can use different orderings in variable elimination - affects efficiency
bull Each ordering corresponds to a junction tree
bull Just as some elimination orderings are more efficient than others some junction trees are better than others (Recall our mention of heuristics for triangulation)
31
OK I have this tree now whatOK I have this tree now what
bull A separator S divides the remaining variables into two groupsndash Variables in each group appear
on one side in the cluster tree
bull Examplesndash AB L S T V amp D X
ndash AL T V amp BDSX
ndash BL S amp A DT V X
ndash A X amp BDL S T V
ndash T V amp A B D K S X
LT
A B
X
V S
D
TV
ALTBLS
XAALB
ABD
AAB
BL
T
AL
32
Elimination in Junction TreesElimination in Junction Trees
bull Let X and Y be the partition induced by S
Observation
bull Eliminating all variables in X results in a factor fX(S)
bull Proof Since S is a separator only variables in S are adjacentto variables in X
bull NoteThe same factor would result regardless of theelimination order
x
y
A BSfX(S)
fY(S)
33
Recursive Elimination in Recursive Elimination in Junction TreesJunction Trees
bull How do we compute fX(S)
bull By recursive decomposition alongcluster tree
ndash Let X1 and X2 be the disjoint partitioning of X C implied by theseparators S1 and S2
ndash Eliminate X1 to get fX1(S1)
ndash Eliminate X2 to get fX2(S2)
ndash Eliminate variables in C S toget fX(S)
C
S
S2S1
x1
x2
y
- Slide 6
- Slide 7
- Slide 8
- Slide 9
- Slide 10
- Slide 11
- Slide 12
- Slide 13
- Slide 23
- Slide 25
- Slide 28
-
3
So do I pass this course or notSo do I pass this course or not
bull We want to compute P(p)
bull By definition
bull In our case (chain)
r s r s
srpPrsPrPpsrPpP )|()|()()()(
r s
spPrsPrPpP )|()|()()(
r s
spPrsPrPpP )|()|()()(
P(p) = 01(0809+0205)+09(0409+0605) = 0676
We essentially eliminated nodes R and S
4
The General Case ndash InferenceThe General Case ndash Inference
bull Network describes a unique probability distribution Pbull We use inference as a name for the process of computing
answers to queries about Pbull There are many types of queries we might ask bull Most of these involve evidence
ndash An evidence e is an assignment of values to a set E variables in the domain
ndash Without loss of generality E = Xk+1 hellip Xn
bull Simplest query compute probability of evidence
bull This is often referred to as computing the likelihood of the evidence
1x
1 )( )(kx
kxxPP ee
5
Another example of Variable Another example of Variable EliminationElimination
bull The ldquoAsiardquo network
Visit to Asia
Smoking
Lung CancerTuberculosis
Abnormalityin Chest
Bronchitis
X-Ray Dyspnea
6
V S
LT
A B
X D
)|( )|( )|( )|( )|( )|( )( )( badPaxPltaPsbPslPvtPsPvP
We are interested in P(d)
- Need to eliminate vsxtlab
Initial factors
Brute force
v s x t l a b
badPaxPltaPsbPslPvtPsPvPdP )|( )|( )|( )|( )|( )|( )( )()(
7
V S
LT
A B
X D
)|( )|( )|( )|( )|( )|( )( )( badPaxPltaPsbPslPvtPsPvP
Eliminate variables in order
Initial factors
v
v vtPvPtf )|()()(
baltxsv
)|()|()|()|()|()()( badPaxPltaPsbPslPsPtfv
[ Note fv(t) = P(t) In general result of elimination is not necessarily a probability term ]
8
)|( )|( )|( )|( )|( )|( )( )( badPaxPltaPsbPslPvtPsPvP
Eliminate variables in order
Initial factors
V S
LT
A B
X D
baltxsv
)|( )|( )|( )|( )|( )( )( badPaxPltaPsbPslPsPtfv
s
s slPsbPsPlbf )|()|()()(
)|()|()|()()( badPaxPltaPlbftf sv
[ Note result of elimination may be a function of several variables ]
9
)|()|()()()( badPltaPaflbftf xsv
)|( )|( )|( )|( )|( )|( )( )( badPaxPltaPsbPslPvtPsPvP
Eliminate variables in order
Initial factors
V S
LT
A B
X D
baltxsv
)|( )|( )|( )|( )|( )( )( badPaxPltaPsbPslPsPtfv)|( )|( )|( )( )( badPaxPltaPlbftf sv
x
x axPaf )|()([ Note fx(a) = 1 for all values of a ]
10
)|( )|( )( )( )( badPltaPaflbftf xsv
)|( )|( )|( )|( )|( )|( )( )( badPaxPltaPsbPslPvtPsPvP
Eliminate variables in order
Initial factors
V S
LT
A B
X D
baltxsv
)|( )|( )|( )|( )|( )( )( badPaxPltaPsbPslPsPtfv)|( )|( )|( )( )( badPaxPltaPlbftf sv
t
vt ltaPtflaf )|()()(
)|()()()( badPlafaflbf txs
11
)|( )|( )( )( )( badPltaPaflbftf xsv
)|( )|( )|( )|( )|( )|( )( )( badPaxPltaPsbPslPvtPsPvP
Eliminate variables in order
Initial factors
V S
LT
A B
X D
baltxsv
)|( )|( )|( )|( )|( )( )( badPaxPltaPsbPslPsPtfv)|( )|( )|( )( )( badPaxPltaPlbftf sv
)|( )( )( )( badPlafaflbf txs
l
tsl laflbfbaf )()()( )|()()( badPafbaf xl
12
)|( )( )( badPafbaf xl
)|( )|( )( )( )( badPltaPaflbftf xsv
)|( )|( )|( )|( )|( )|( )( )( badPaxPltaPsbPslPvtPsPvP
Eliminate variables in order
Initial factors
V S
LT
A B
X D
baltxsv
)|( )|( )|( )|( )|( )( )( badPaxPltaPsbPslPsPtfv)|( )|( )|( )( )( badPaxPltaPlbftf sv
)|( )( )( )( badPlafaflbf txs
a
xla badpafbafdbf )|()()()( )( dbfa
13
)|( )( )( badPafbaf xl
)|( )|( )( )( )( badPltaPaflbftf xsv
)|( )|( )|( )|( )|( )|( )( )( badPaxPltaPsbPslPvtPsPvP
Eliminate variables in order
Initial factors
V S
LT
A B
X D
baltxsv
)|( )|( )|( )|( )|( )( )( badPaxPltaPsbPslPsPtfv)|( )|( )|( )( )( badPaxPltaPlbftf sv
)|( )( )( )( badPlafaflbf txs
)( dbfa )()()( dfdbfdf bb
ab
14
Intermediate factorsIntermediate factors
baltxsv
ga (ltdbx)
gb (ltdxs)
gx (ltds)
gt (ltsv)
gv (lds)
gs(ld)
gl (d))()()()(
)()(
)(
dfdbfbaflaf
aflbf
tf
b
a
l
t
x
s
v
lsvtxba
In our previous example With a different ordering
V S
LT
A B
X D
Complexity is exponential in the size of these factors
15
Notes about variable Notes about variable eliminationelimination
bull Actual computation is done in the elimination steps
bull Computation depends on the order of elimination
bull For each query we need to compute everything againndash Many redundant calculations
16
The ideaThe idea
bull Compute joint over partitions of U
ndash 1048708 small subset of U (typically made of a variable and its parents) - clusters
ndash not necessary disjoint
ndash Calculate
ndash To compute P(X) need far less operations
iC
iiCU
XC
i
i
CPXP
)()(
iC)( iCP
17
Junction TreesJunction Trees
bull The junction tree algorithms generalize Variable Elimination to the efficient simultaneous execution of a large class of queries
bull Theoretical background was shown in the previous lecture
18
Constructing Junction TreesConstructing Junction Trees
1 Moralize the graph (if directed)
2 Choose a node ordering and find the cliques generated by variable elimination This gives a triangulation of the graph
3 Build a junction graph from the eliminated cliques
4 Find an appropriate spanning tree
19
Step 1 MoralizationStep 1 Moralization
a
b c
d e
f
g
h
a
b c
d e
f
g
h
a
b c
d e
f
g
h
1 For all w Vbull For all uvpa(w) add an edge e=u-v
2 Undirect all edges
GMG = ( V E )
20
Step 2 TriangulationStep 2 Triangulation
Add edges to GM such that there is no cyclewith length 4 that does not contain a chord
NO YES
a
b c
d e
f
g
h
a
b c
d e
f
g
h
GM GT
21
Step 2 Triangulation (cont)Step 2 Triangulation (cont)
bull Each elimination ordering triangulates the graph not necessarily in the same way
A
H
B
D
F
C
E
G
A
H
B
D
F
C
E
G
A
H
B
D
F
C
E
G
A
H
B
D
F
C
E
G
A
H
B
D
F
C
E
G
A
H
B
D
F
C
E
G
A
H
B
D
F
C
E
G
A
H
B
D
F
C
E
G
A
H
B
D
F
C
E
G
22
Step 2 Triangulation (cont)Step 2 Triangulation (cont)
bull Intuitively triangulations with as few fill-ins as possible are preferredndash Leaves us with small cliques (small probability tables)
bull A common heuristic Repeat until no nodes remain
ndash Find the node whose elimination would require the least number of fill-ins (may be zero)
ndash Eliminate that node and note the need for a fill-in edge between any two non-adjacent neighbors
bull Add the fill-in edges to the original graph
23
a
b c
d e
f
g
h
a
b c
d e
f
g
h
a
b c
d e
f
g
a
b c
d e
f
a
b c
d e
a
b
d e
aa
e
a
d e
vertex induced addedremoved clique edges
1 h egh -2 g ceg -3 f def -4 c ace a-e
vertex induced added removed clique edges
5 b abd a-d6 d ade -7 e ae -8 a a -
GT
GM
Eliminate the vertex that requires least number of edges to be added
24
Step 3 Junction GraphStep 3 Junction Graph
bull A junction graph for an undirected graph G is an undirected labeled graph
bull The nodes are the cliques in G
bull If two cliques intersect they are joined in the junction graph by an edge labeled with their intersection
25
a
b
d
a
c
e
d e
f
a
d e
e
g
h
c
e
g
a
b c
d e
f
g
h
Bayesian NetworkG = ( V E )
a
b c
d e
f
g
h
a
b c
d e
f
g
h
Moral graph GM Triangulated graph GT
abd
ade
ace
ceg
eghdef
ad ae ce
de eg
seperators
Junction graph GJ (not complete)eg ceg egh = eg Cliques
e
e
e
a
e
26
Step 4 Junction TreeStep 4 Junction Tree
bull A junction tree is a sub-graph of the junction graph that ndash Is a tree
ndash Contains all the cliques (spanning tree)
ndash Satisfies the running intersection property
for each pair of nodes U V all nodes on the path between U and V contain
(as seen in the previous part of the lecture)
VU
27
Step 4 Junction Tree (cont)Step 4 Junction Tree (cont)
bull Theorem An undirected graph is triangulated if and only if its junction graph has a junction tree
bull Definition The weight of a link in a junction graph is the number of variable in the label The weight of a junction tree is the sum of weights of the labels
bull Theorem A sub-tree of the junction graph of a triangulated graph is a junction tree if and only if it is a spanning of maximal weight
28
Junction graph GJ (not complete)
abd
ade
ace
ceg
eghdef
ad ae ce
de eg
e
e
e
a
e
abd
ade
ace
ceg
eghdef
ad ae ce
de eg
Junction tree GJT
There are several methods to find MST
Kruskalrsquos algorithm choose successively a link of
maximal weight unless it creates a cycle
29
Another exampleAnother example
bull Compute the elimination cliques(the order here is f d e c b a)
bull Form the complete junction graph over the maximal elimination cliques and find a maximum-weight spanning tree
30
Junction Trees and Elimination Junction Trees and Elimination OrderOrder
bull We can use different orderings in variable elimination - affects efficiency
bull Each ordering corresponds to a junction tree
bull Just as some elimination orderings are more efficient than others some junction trees are better than others (Recall our mention of heuristics for triangulation)
31
OK I have this tree now whatOK I have this tree now what
bull A separator S divides the remaining variables into two groupsndash Variables in each group appear
on one side in the cluster tree
bull Examplesndash AB L S T V amp D X
ndash AL T V amp BDSX
ndash BL S amp A DT V X
ndash A X amp BDL S T V
ndash T V amp A B D K S X
LT
A B
X
V S
D
TV
ALTBLS
XAALB
ABD
AAB
BL
T
AL
32
Elimination in Junction TreesElimination in Junction Trees
bull Let X and Y be the partition induced by S
Observation
bull Eliminating all variables in X results in a factor fX(S)
bull Proof Since S is a separator only variables in S are adjacentto variables in X
bull NoteThe same factor would result regardless of theelimination order
x
y
A BSfX(S)
fY(S)
33
Recursive Elimination in Recursive Elimination in Junction TreesJunction Trees
bull How do we compute fX(S)
bull By recursive decomposition alongcluster tree
ndash Let X1 and X2 be the disjoint partitioning of X C implied by theseparators S1 and S2
ndash Eliminate X1 to get fX1(S1)
ndash Eliminate X2 to get fX2(S2)
ndash Eliminate variables in C S toget fX(S)
C
S
S2S1
x1
x2
y
- Slide 6
- Slide 7
- Slide 8
- Slide 9
- Slide 10
- Slide 11
- Slide 12
- Slide 13
- Slide 23
- Slide 25
- Slide 28
-
4
The General Case ndash InferenceThe General Case ndash Inference
bull Network describes a unique probability distribution Pbull We use inference as a name for the process of computing
answers to queries about Pbull There are many types of queries we might ask bull Most of these involve evidence
ndash An evidence e is an assignment of values to a set E variables in the domain
ndash Without loss of generality E = Xk+1 hellip Xn
bull Simplest query compute probability of evidence
bull This is often referred to as computing the likelihood of the evidence
1x
1 )( )(kx
kxxPP ee
5
Another example of Variable Another example of Variable EliminationElimination
bull The ldquoAsiardquo network
Visit to Asia
Smoking
Lung CancerTuberculosis
Abnormalityin Chest
Bronchitis
X-Ray Dyspnea
6
V S
LT
A B
X D
)|( )|( )|( )|( )|( )|( )( )( badPaxPltaPsbPslPvtPsPvP
We are interested in P(d)
- Need to eliminate vsxtlab
Initial factors
Brute force
v s x t l a b
badPaxPltaPsbPslPvtPsPvPdP )|( )|( )|( )|( )|( )|( )( )()(
7
V S
LT
A B
X D
)|( )|( )|( )|( )|( )|( )( )( badPaxPltaPsbPslPvtPsPvP
Eliminate variables in order
Initial factors
v
v vtPvPtf )|()()(
baltxsv
)|()|()|()|()|()()( badPaxPltaPsbPslPsPtfv
[ Note fv(t) = P(t) In general result of elimination is not necessarily a probability term ]
8
)|( )|( )|( )|( )|( )|( )( )( badPaxPltaPsbPslPvtPsPvP
Eliminate variables in order
Initial factors
V S
LT
A B
X D
baltxsv
)|( )|( )|( )|( )|( )( )( badPaxPltaPsbPslPsPtfv
s
s slPsbPsPlbf )|()|()()(
)|()|()|()()( badPaxPltaPlbftf sv
[ Note result of elimination may be a function of several variables ]
9
)|()|()()()( badPltaPaflbftf xsv
)|( )|( )|( )|( )|( )|( )( )( badPaxPltaPsbPslPvtPsPvP
Eliminate variables in order
Initial factors
V S
LT
A B
X D
baltxsv
)|( )|( )|( )|( )|( )( )( badPaxPltaPsbPslPsPtfv)|( )|( )|( )( )( badPaxPltaPlbftf sv
x
x axPaf )|()([ Note fx(a) = 1 for all values of a ]
10
)|( )|( )( )( )( badPltaPaflbftf xsv
)|( )|( )|( )|( )|( )|( )( )( badPaxPltaPsbPslPvtPsPvP
Eliminate variables in order
Initial factors
V S
LT
A B
X D
baltxsv
)|( )|( )|( )|( )|( )( )( badPaxPltaPsbPslPsPtfv)|( )|( )|( )( )( badPaxPltaPlbftf sv
t
vt ltaPtflaf )|()()(
)|()()()( badPlafaflbf txs
11
)|( )|( )( )( )( badPltaPaflbftf xsv
)|( )|( )|( )|( )|( )|( )( )( badPaxPltaPsbPslPvtPsPvP
Eliminate variables in order
Initial factors
V S
LT
A B
X D
baltxsv
)|( )|( )|( )|( )|( )( )( badPaxPltaPsbPslPsPtfv)|( )|( )|( )( )( badPaxPltaPlbftf sv
)|( )( )( )( badPlafaflbf txs
l
tsl laflbfbaf )()()( )|()()( badPafbaf xl
12
)|( )( )( badPafbaf xl
)|( )|( )( )( )( badPltaPaflbftf xsv
)|( )|( )|( )|( )|( )|( )( )( badPaxPltaPsbPslPvtPsPvP
Eliminate variables in order
Initial factors
V S
LT
A B
X D
baltxsv
)|( )|( )|( )|( )|( )( )( badPaxPltaPsbPslPsPtfv)|( )|( )|( )( )( badPaxPltaPlbftf sv
)|( )( )( )( badPlafaflbf txs
a
xla badpafbafdbf )|()()()( )( dbfa
13
)|( )( )( badPafbaf xl
)|( )|( )( )( )( badPltaPaflbftf xsv
)|( )|( )|( )|( )|( )|( )( )( badPaxPltaPsbPslPvtPsPvP
Eliminate variables in order
Initial factors
V S
LT
A B
X D
baltxsv
)|( )|( )|( )|( )|( )( )( badPaxPltaPsbPslPsPtfv)|( )|( )|( )( )( badPaxPltaPlbftf sv
)|( )( )( )( badPlafaflbf txs
)( dbfa )()()( dfdbfdf bb
ab
14
Intermediate factorsIntermediate factors
baltxsv
ga (ltdbx)
gb (ltdxs)
gx (ltds)
gt (ltsv)
gv (lds)
gs(ld)
gl (d))()()()(
)()(
)(
dfdbfbaflaf
aflbf
tf
b
a
l
t
x
s
v
lsvtxba
In our previous example With a different ordering
V S
LT
A B
X D
Complexity is exponential in the size of these factors
15
Notes about variable Notes about variable eliminationelimination
bull Actual computation is done in the elimination steps
bull Computation depends on the order of elimination
bull For each query we need to compute everything againndash Many redundant calculations
16
The ideaThe idea
bull Compute joint over partitions of U
ndash 1048708 small subset of U (typically made of a variable and its parents) - clusters
ndash not necessary disjoint
ndash Calculate
ndash To compute P(X) need far less operations
iC
iiCU
XC
i
i
CPXP
)()(
iC)( iCP
17
Junction TreesJunction Trees
bull The junction tree algorithms generalize Variable Elimination to the efficient simultaneous execution of a large class of queries
bull Theoretical background was shown in the previous lecture
18
Constructing Junction TreesConstructing Junction Trees
1 Moralize the graph (if directed)
2 Choose a node ordering and find the cliques generated by variable elimination This gives a triangulation of the graph
3 Build a junction graph from the eliminated cliques
4 Find an appropriate spanning tree
19
Step 1 MoralizationStep 1 Moralization
a
b c
d e
f
g
h
a
b c
d e
f
g
h
a
b c
d e
f
g
h
1 For all w Vbull For all uvpa(w) add an edge e=u-v
2 Undirect all edges
GMG = ( V E )
20
Step 2 TriangulationStep 2 Triangulation
Add edges to GM such that there is no cyclewith length 4 that does not contain a chord
NO YES
a
b c
d e
f
g
h
a
b c
d e
f
g
h
GM GT
21
Step 2 Triangulation (cont)Step 2 Triangulation (cont)
bull Each elimination ordering triangulates the graph not necessarily in the same way
A
H
B
D
F
C
E
G
A
H
B
D
F
C
E
G
A
H
B
D
F
C
E
G
A
H
B
D
F
C
E
G
A
H
B
D
F
C
E
G
A
H
B
D
F
C
E
G
A
H
B
D
F
C
E
G
A
H
B
D
F
C
E
G
A
H
B
D
F
C
E
G
22
Step 2 Triangulation (cont)Step 2 Triangulation (cont)
bull Intuitively triangulations with as few fill-ins as possible are preferredndash Leaves us with small cliques (small probability tables)
bull A common heuristic Repeat until no nodes remain
ndash Find the node whose elimination would require the least number of fill-ins (may be zero)
ndash Eliminate that node and note the need for a fill-in edge between any two non-adjacent neighbors
bull Add the fill-in edges to the original graph
23
a
b c
d e
f
g
h
a
b c
d e
f
g
h
a
b c
d e
f
g
a
b c
d e
f
a
b c
d e
a
b
d e
aa
e
a
d e
vertex induced addedremoved clique edges
1 h egh -2 g ceg -3 f def -4 c ace a-e
vertex induced added removed clique edges
5 b abd a-d6 d ade -7 e ae -8 a a -
GT
GM
Eliminate the vertex that requires least number of edges to be added
24
Step 3 Junction GraphStep 3 Junction Graph
bull A junction graph for an undirected graph G is an undirected labeled graph
bull The nodes are the cliques in G
bull If two cliques intersect they are joined in the junction graph by an edge labeled with their intersection
25
a
b
d
a
c
e
d e
f
a
d e
e
g
h
c
e
g
a
b c
d e
f
g
h
Bayesian NetworkG = ( V E )
a
b c
d e
f
g
h
a
b c
d e
f
g
h
Moral graph GM Triangulated graph GT
abd
ade
ace
ceg
eghdef
ad ae ce
de eg
seperators
Junction graph GJ (not complete)eg ceg egh = eg Cliques
e
e
e
a
e
26
Step 4 Junction TreeStep 4 Junction Tree
bull A junction tree is a sub-graph of the junction graph that ndash Is a tree
ndash Contains all the cliques (spanning tree)
ndash Satisfies the running intersection property
for each pair of nodes U V all nodes on the path between U and V contain
(as seen in the previous part of the lecture)
VU
27
Step 4 Junction Tree (cont)Step 4 Junction Tree (cont)
bull Theorem An undirected graph is triangulated if and only if its junction graph has a junction tree
bull Definition The weight of a link in a junction graph is the number of variable in the label The weight of a junction tree is the sum of weights of the labels
bull Theorem A sub-tree of the junction graph of a triangulated graph is a junction tree if and only if it is a spanning of maximal weight
28
Junction graph GJ (not complete)
abd
ade
ace
ceg
eghdef
ad ae ce
de eg
e
e
e
a
e
abd
ade
ace
ceg
eghdef
ad ae ce
de eg
Junction tree GJT
There are several methods to find MST
Kruskalrsquos algorithm choose successively a link of
maximal weight unless it creates a cycle
29
Another exampleAnother example
bull Compute the elimination cliques(the order here is f d e c b a)
bull Form the complete junction graph over the maximal elimination cliques and find a maximum-weight spanning tree
30
Junction Trees and Elimination Junction Trees and Elimination OrderOrder
bull We can use different orderings in variable elimination - affects efficiency
bull Each ordering corresponds to a junction tree
bull Just as some elimination orderings are more efficient than others some junction trees are better than others (Recall our mention of heuristics for triangulation)
31
OK I have this tree now whatOK I have this tree now what
bull A separator S divides the remaining variables into two groupsndash Variables in each group appear
on one side in the cluster tree
bull Examplesndash AB L S T V amp D X
ndash AL T V amp BDSX
ndash BL S amp A DT V X
ndash A X amp BDL S T V
ndash T V amp A B D K S X
LT
A B
X
V S
D
TV
ALTBLS
XAALB
ABD
AAB
BL
T
AL
32
Elimination in Junction TreesElimination in Junction Trees
bull Let X and Y be the partition induced by S
Observation
bull Eliminating all variables in X results in a factor fX(S)
bull Proof Since S is a separator only variables in S are adjacentto variables in X
bull NoteThe same factor would result regardless of theelimination order
x
y
A BSfX(S)
fY(S)
33
Recursive Elimination in Recursive Elimination in Junction TreesJunction Trees
bull How do we compute fX(S)
bull By recursive decomposition alongcluster tree
ndash Let X1 and X2 be the disjoint partitioning of X C implied by theseparators S1 and S2
ndash Eliminate X1 to get fX1(S1)
ndash Eliminate X2 to get fX2(S2)
ndash Eliminate variables in C S toget fX(S)
C
S
S2S1
x1
x2
y
- Slide 6
- Slide 7
- Slide 8
- Slide 9
- Slide 10
- Slide 11
- Slide 12
- Slide 13
- Slide 23
- Slide 25
- Slide 28
-
5
Another example of Variable Another example of Variable EliminationElimination
bull The ldquoAsiardquo network
Visit to Asia
Smoking
Lung CancerTuberculosis
Abnormalityin Chest
Bronchitis
X-Ray Dyspnea
6
V S
LT
A B
X D
)|( )|( )|( )|( )|( )|( )( )( badPaxPltaPsbPslPvtPsPvP
We are interested in P(d)
- Need to eliminate vsxtlab
Initial factors
Brute force
v s x t l a b
badPaxPltaPsbPslPvtPsPvPdP )|( )|( )|( )|( )|( )|( )( )()(
7
V S
LT
A B
X D
)|( )|( )|( )|( )|( )|( )( )( badPaxPltaPsbPslPvtPsPvP
Eliminate variables in order
Initial factors
v
v vtPvPtf )|()()(
baltxsv
)|()|()|()|()|()()( badPaxPltaPsbPslPsPtfv
[ Note fv(t) = P(t) In general result of elimination is not necessarily a probability term ]
8
)|( )|( )|( )|( )|( )|( )( )( badPaxPltaPsbPslPvtPsPvP
Eliminate variables in order
Initial factors
V S
LT
A B
X D
baltxsv
)|( )|( )|( )|( )|( )( )( badPaxPltaPsbPslPsPtfv
s
s slPsbPsPlbf )|()|()()(
)|()|()|()()( badPaxPltaPlbftf sv
[ Note result of elimination may be a function of several variables ]
9
)|()|()()()( badPltaPaflbftf xsv
)|( )|( )|( )|( )|( )|( )( )( badPaxPltaPsbPslPvtPsPvP
Eliminate variables in order
Initial factors
V S
LT
A B
X D
baltxsv
)|( )|( )|( )|( )|( )( )( badPaxPltaPsbPslPsPtfv)|( )|( )|( )( )( badPaxPltaPlbftf sv
x
x axPaf )|()([ Note fx(a) = 1 for all values of a ]
10
)|( )|( )( )( )( badPltaPaflbftf xsv
)|( )|( )|( )|( )|( )|( )( )( badPaxPltaPsbPslPvtPsPvP
Eliminate variables in order
Initial factors
V S
LT
A B
X D
baltxsv
)|( )|( )|( )|( )|( )( )( badPaxPltaPsbPslPsPtfv)|( )|( )|( )( )( badPaxPltaPlbftf sv
t
vt ltaPtflaf )|()()(
)|()()()( badPlafaflbf txs
11
)|( )|( )( )( )( badPltaPaflbftf xsv
)|( )|( )|( )|( )|( )|( )( )( badPaxPltaPsbPslPvtPsPvP
Eliminate variables in order
Initial factors
V S
LT
A B
X D
baltxsv
)|( )|( )|( )|( )|( )( )( badPaxPltaPsbPslPsPtfv)|( )|( )|( )( )( badPaxPltaPlbftf sv
)|( )( )( )( badPlafaflbf txs
l
tsl laflbfbaf )()()( )|()()( badPafbaf xl
12
)|( )( )( badPafbaf xl
)|( )|( )( )( )( badPltaPaflbftf xsv
)|( )|( )|( )|( )|( )|( )( )( badPaxPltaPsbPslPvtPsPvP
Eliminate variables in order
Initial factors
V S
LT
A B
X D
baltxsv
)|( )|( )|( )|( )|( )( )( badPaxPltaPsbPslPsPtfv)|( )|( )|( )( )( badPaxPltaPlbftf sv
)|( )( )( )( badPlafaflbf txs
a
xla badpafbafdbf )|()()()( )( dbfa
13
)|( )( )( badPafbaf xl
)|( )|( )( )( )( badPltaPaflbftf xsv
)|( )|( )|( )|( )|( )|( )( )( badPaxPltaPsbPslPvtPsPvP
Eliminate variables in order
Initial factors
V S
LT
A B
X D
baltxsv
)|( )|( )|( )|( )|( )( )( badPaxPltaPsbPslPsPtfv)|( )|( )|( )( )( badPaxPltaPlbftf sv
)|( )( )( )( badPlafaflbf txs
)( dbfa )()()( dfdbfdf bb
ab
14
Intermediate factorsIntermediate factors
baltxsv
ga (ltdbx)
gb (ltdxs)
gx (ltds)
gt (ltsv)
gv (lds)
gs(ld)
gl (d))()()()(
)()(
)(
dfdbfbaflaf
aflbf
tf
b
a
l
t
x
s
v
lsvtxba
In our previous example With a different ordering
V S
LT
A B
X D
Complexity is exponential in the size of these factors
15
Notes about variable Notes about variable eliminationelimination
bull Actual computation is done in the elimination steps
bull Computation depends on the order of elimination
bull For each query we need to compute everything againndash Many redundant calculations
16
The ideaThe idea
bull Compute joint over partitions of U
ndash 1048708 small subset of U (typically made of a variable and its parents) - clusters
ndash not necessary disjoint
ndash Calculate
ndash To compute P(X) need far less operations
iC
iiCU
XC
i
i
CPXP
)()(
iC)( iCP
17
Junction TreesJunction Trees
bull The junction tree algorithms generalize Variable Elimination to the efficient simultaneous execution of a large class of queries
bull Theoretical background was shown in the previous lecture
18
Constructing Junction TreesConstructing Junction Trees
1 Moralize the graph (if directed)
2 Choose a node ordering and find the cliques generated by variable elimination This gives a triangulation of the graph
3 Build a junction graph from the eliminated cliques
4 Find an appropriate spanning tree
19
Step 1 MoralizationStep 1 Moralization
a
b c
d e
f
g
h
a
b c
d e
f
g
h
a
b c
d e
f
g
h
1 For all w Vbull For all uvpa(w) add an edge e=u-v
2 Undirect all edges
GMG = ( V E )
20
Step 2 TriangulationStep 2 Triangulation
Add edges to GM such that there is no cyclewith length 4 that does not contain a chord
NO YES
a
b c
d e
f
g
h
a
b c
d e
f
g
h
GM GT
21
Step 2 Triangulation (cont)Step 2 Triangulation (cont)
bull Each elimination ordering triangulates the graph not necessarily in the same way
A
H
B
D
F
C
E
G
A
H
B
D
F
C
E
G
A
H
B
D
F
C
E
G
A
H
B
D
F
C
E
G
A
H
B
D
F
C
E
G
A
H
B
D
F
C
E
G
A
H
B
D
F
C
E
G
A
H
B
D
F
C
E
G
A
H
B
D
F
C
E
G
22
Step 2 Triangulation (cont)Step 2 Triangulation (cont)
bull Intuitively triangulations with as few fill-ins as possible are preferredndash Leaves us with small cliques (small probability tables)
bull A common heuristic Repeat until no nodes remain
ndash Find the node whose elimination would require the least number of fill-ins (may be zero)
ndash Eliminate that node and note the need for a fill-in edge between any two non-adjacent neighbors
bull Add the fill-in edges to the original graph
23
a
b c
d e
f
g
h
a
b c
d e
f
g
h
a
b c
d e
f
g
a
b c
d e
f
a
b c
d e
a
b
d e
aa
e
a
d e
vertex induced addedremoved clique edges
1 h egh -2 g ceg -3 f def -4 c ace a-e
vertex induced added removed clique edges
5 b abd a-d6 d ade -7 e ae -8 a a -
GT
GM
Eliminate the vertex that requires least number of edges to be added
24
Step 3 Junction GraphStep 3 Junction Graph
bull A junction graph for an undirected graph G is an undirected labeled graph
bull The nodes are the cliques in G
bull If two cliques intersect they are joined in the junction graph by an edge labeled with their intersection
25
a
b
d
a
c
e
d e
f
a
d e
e
g
h
c
e
g
a
b c
d e
f
g
h
Bayesian NetworkG = ( V E )
a
b c
d e
f
g
h
a
b c
d e
f
g
h
Moral graph GM Triangulated graph GT
abd
ade
ace
ceg
eghdef
ad ae ce
de eg
seperators
Junction graph GJ (not complete)eg ceg egh = eg Cliques
e
e
e
a
e
26
Step 4 Junction TreeStep 4 Junction Tree
bull A junction tree is a sub-graph of the junction graph that ndash Is a tree
ndash Contains all the cliques (spanning tree)
ndash Satisfies the running intersection property
for each pair of nodes U V all nodes on the path between U and V contain
(as seen in the previous part of the lecture)
VU
27
Step 4 Junction Tree (cont)Step 4 Junction Tree (cont)
bull Theorem An undirected graph is triangulated if and only if its junction graph has a junction tree
bull Definition The weight of a link in a junction graph is the number of variable in the label The weight of a junction tree is the sum of weights of the labels
bull Theorem A sub-tree of the junction graph of a triangulated graph is a junction tree if and only if it is a spanning of maximal weight
28
Junction graph GJ (not complete)
abd
ade
ace
ceg
eghdef
ad ae ce
de eg
e
e
e
a
e
abd
ade
ace
ceg
eghdef
ad ae ce
de eg
Junction tree GJT
There are several methods to find MST
Kruskalrsquos algorithm choose successively a link of
maximal weight unless it creates a cycle
29
Another exampleAnother example
bull Compute the elimination cliques(the order here is f d e c b a)
bull Form the complete junction graph over the maximal elimination cliques and find a maximum-weight spanning tree
30
Junction Trees and Elimination Junction Trees and Elimination OrderOrder
bull We can use different orderings in variable elimination - affects efficiency
bull Each ordering corresponds to a junction tree
bull Just as some elimination orderings are more efficient than others some junction trees are better than others (Recall our mention of heuristics for triangulation)
31
OK I have this tree now whatOK I have this tree now what
bull A separator S divides the remaining variables into two groupsndash Variables in each group appear
on one side in the cluster tree
bull Examplesndash AB L S T V amp D X
ndash AL T V amp BDSX
ndash BL S amp A DT V X
ndash A X amp BDL S T V
ndash T V amp A B D K S X
LT
A B
X
V S
D
TV
ALTBLS
XAALB
ABD
AAB
BL
T
AL
32
Elimination in Junction TreesElimination in Junction Trees
bull Let X and Y be the partition induced by S
Observation
bull Eliminating all variables in X results in a factor fX(S)
bull Proof Since S is a separator only variables in S are adjacentto variables in X
bull NoteThe same factor would result regardless of theelimination order
x
y
A BSfX(S)
fY(S)
33
Recursive Elimination in Recursive Elimination in Junction TreesJunction Trees
bull How do we compute fX(S)
bull By recursive decomposition alongcluster tree
ndash Let X1 and X2 be the disjoint partitioning of X C implied by theseparators S1 and S2
ndash Eliminate X1 to get fX1(S1)
ndash Eliminate X2 to get fX2(S2)
ndash Eliminate variables in C S toget fX(S)
C
S
S2S1
x1
x2
y
- Slide 6
- Slide 7
- Slide 8
- Slide 9
- Slide 10
- Slide 11
- Slide 12
- Slide 13
- Slide 23
- Slide 25
- Slide 28
-
6
V S
LT
A B
X D
)|( )|( )|( )|( )|( )|( )( )( badPaxPltaPsbPslPvtPsPvP
We are interested in P(d)
- Need to eliminate vsxtlab
Initial factors
Brute force
v s x t l a b
badPaxPltaPsbPslPvtPsPvPdP )|( )|( )|( )|( )|( )|( )( )()(
7
V S
LT
A B
X D
)|( )|( )|( )|( )|( )|( )( )( badPaxPltaPsbPslPvtPsPvP
Eliminate variables in order
Initial factors
v
v vtPvPtf )|()()(
baltxsv
)|()|()|()|()|()()( badPaxPltaPsbPslPsPtfv
[ Note fv(t) = P(t) In general result of elimination is not necessarily a probability term ]
8
)|( )|( )|( )|( )|( )|( )( )( badPaxPltaPsbPslPvtPsPvP
Eliminate variables in order
Initial factors
V S
LT
A B
X D
baltxsv
)|( )|( )|( )|( )|( )( )( badPaxPltaPsbPslPsPtfv
s
s slPsbPsPlbf )|()|()()(
)|()|()|()()( badPaxPltaPlbftf sv
[ Note result of elimination may be a function of several variables ]
9
)|()|()()()( badPltaPaflbftf xsv
)|( )|( )|( )|( )|( )|( )( )( badPaxPltaPsbPslPvtPsPvP
Eliminate variables in order
Initial factors
V S
LT
A B
X D
baltxsv
)|( )|( )|( )|( )|( )( )( badPaxPltaPsbPslPsPtfv)|( )|( )|( )( )( badPaxPltaPlbftf sv
x
x axPaf )|()([ Note fx(a) = 1 for all values of a ]
10
)|( )|( )( )( )( badPltaPaflbftf xsv
)|( )|( )|( )|( )|( )|( )( )( badPaxPltaPsbPslPvtPsPvP
Eliminate variables in order
Initial factors
V S
LT
A B
X D
baltxsv
)|( )|( )|( )|( )|( )( )( badPaxPltaPsbPslPsPtfv)|( )|( )|( )( )( badPaxPltaPlbftf sv
t
vt ltaPtflaf )|()()(
)|()()()( badPlafaflbf txs
11
)|( )|( )( )( )( badPltaPaflbftf xsv
)|( )|( )|( )|( )|( )|( )( )( badPaxPltaPsbPslPvtPsPvP
Eliminate variables in order
Initial factors
V S
LT
A B
X D
baltxsv
)|( )|( )|( )|( )|( )( )( badPaxPltaPsbPslPsPtfv)|( )|( )|( )( )( badPaxPltaPlbftf sv
)|( )( )( )( badPlafaflbf txs
l
tsl laflbfbaf )()()( )|()()( badPafbaf xl
12
)|( )( )( badPafbaf xl
)|( )|( )( )( )( badPltaPaflbftf xsv
)|( )|( )|( )|( )|( )|( )( )( badPaxPltaPsbPslPvtPsPvP
Eliminate variables in order
Initial factors
V S
LT
A B
X D
baltxsv
)|( )|( )|( )|( )|( )( )( badPaxPltaPsbPslPsPtfv)|( )|( )|( )( )( badPaxPltaPlbftf sv
)|( )( )( )( badPlafaflbf txs
a
xla badpafbafdbf )|()()()( )( dbfa
13
)|( )( )( badPafbaf xl
)|( )|( )( )( )( badPltaPaflbftf xsv
)|( )|( )|( )|( )|( )|( )( )( badPaxPltaPsbPslPvtPsPvP
Eliminate variables in order
Initial factors
V S
LT
A B
X D
baltxsv
)|( )|( )|( )|( )|( )( )( badPaxPltaPsbPslPsPtfv)|( )|( )|( )( )( badPaxPltaPlbftf sv
)|( )( )( )( badPlafaflbf txs
)( dbfa )()()( dfdbfdf bb
ab
14
Intermediate factorsIntermediate factors
baltxsv
ga (ltdbx)
gb (ltdxs)
gx (ltds)
gt (ltsv)
gv (lds)
gs(ld)
gl (d))()()()(
)()(
)(
dfdbfbaflaf
aflbf
tf
b
a
l
t
x
s
v
lsvtxba
In our previous example With a different ordering
V S
LT
A B
X D
Complexity is exponential in the size of these factors
15
Notes about variable Notes about variable eliminationelimination
bull Actual computation is done in the elimination steps
bull Computation depends on the order of elimination
bull For each query we need to compute everything againndash Many redundant calculations
16
The ideaThe idea
bull Compute joint over partitions of U
ndash 1048708 small subset of U (typically made of a variable and its parents) - clusters
ndash not necessary disjoint
ndash Calculate
ndash To compute P(X) need far less operations
iC
iiCU
XC
i
i
CPXP
)()(
iC)( iCP
17
Junction TreesJunction Trees
bull The junction tree algorithms generalize Variable Elimination to the efficient simultaneous execution of a large class of queries
bull Theoretical background was shown in the previous lecture
18
Constructing Junction TreesConstructing Junction Trees
1 Moralize the graph (if directed)
2 Choose a node ordering and find the cliques generated by variable elimination This gives a triangulation of the graph
3 Build a junction graph from the eliminated cliques
4 Find an appropriate spanning tree
19
Step 1 MoralizationStep 1 Moralization
a
b c
d e
f
g
h
a
b c
d e
f
g
h
a
b c
d e
f
g
h
1 For all w Vbull For all uvpa(w) add an edge e=u-v
2 Undirect all edges
GMG = ( V E )
20
Step 2 TriangulationStep 2 Triangulation
Add edges to GM such that there is no cyclewith length 4 that does not contain a chord
NO YES
a
b c
d e
f
g
h
a
b c
d e
f
g
h
GM GT
21
Step 2 Triangulation (cont)Step 2 Triangulation (cont)
bull Each elimination ordering triangulates the graph not necessarily in the same way
A
H
B
D
F
C
E
G
A
H
B
D
F
C
E
G
A
H
B
D
F
C
E
G
A
H
B
D
F
C
E
G
A
H
B
D
F
C
E
G
A
H
B
D
F
C
E
G
A
H
B
D
F
C
E
G
A
H
B
D
F
C
E
G
A
H
B
D
F
C
E
G
22
Step 2 Triangulation (cont)Step 2 Triangulation (cont)
bull Intuitively triangulations with as few fill-ins as possible are preferredndash Leaves us with small cliques (small probability tables)
bull A common heuristic Repeat until no nodes remain
ndash Find the node whose elimination would require the least number of fill-ins (may be zero)
ndash Eliminate that node and note the need for a fill-in edge between any two non-adjacent neighbors
bull Add the fill-in edges to the original graph
23
a
b c
d e
f
g
h
a
b c
d e
f
g
h
a
b c
d e
f
g
a
b c
d e
f
a
b c
d e
a
b
d e
aa
e
a
d e
vertex induced addedremoved clique edges
1 h egh -2 g ceg -3 f def -4 c ace a-e
vertex induced added removed clique edges
5 b abd a-d6 d ade -7 e ae -8 a a -
GT
GM
Eliminate the vertex that requires least number of edges to be added
24
Step 3 Junction GraphStep 3 Junction Graph
bull A junction graph for an undirected graph G is an undirected labeled graph
bull The nodes are the cliques in G
bull If two cliques intersect they are joined in the junction graph by an edge labeled with their intersection
25
a
b
d
a
c
e
d e
f
a
d e
e
g
h
c
e
g
a
b c
d e
f
g
h
Bayesian NetworkG = ( V E )
a
b c
d e
f
g
h
a
b c
d e
f
g
h
Moral graph GM Triangulated graph GT
abd
ade
ace
ceg
eghdef
ad ae ce
de eg
seperators
Junction graph GJ (not complete)eg ceg egh = eg Cliques
e
e
e
a
e
26
Step 4 Junction TreeStep 4 Junction Tree
bull A junction tree is a sub-graph of the junction graph that ndash Is a tree
ndash Contains all the cliques (spanning tree)
ndash Satisfies the running intersection property
for each pair of nodes U V all nodes on the path between U and V contain
(as seen in the previous part of the lecture)
VU
27
Step 4 Junction Tree (cont)Step 4 Junction Tree (cont)
bull Theorem An undirected graph is triangulated if and only if its junction graph has a junction tree
bull Definition The weight of a link in a junction graph is the number of variable in the label The weight of a junction tree is the sum of weights of the labels
bull Theorem A sub-tree of the junction graph of a triangulated graph is a junction tree if and only if it is a spanning of maximal weight
28
Junction graph GJ (not complete)
abd
ade
ace
ceg
eghdef
ad ae ce
de eg
e
e
e
a
e
abd
ade
ace
ceg
eghdef
ad ae ce
de eg
Junction tree GJT
There are several methods to find MST
Kruskalrsquos algorithm choose successively a link of
maximal weight unless it creates a cycle
29
Another exampleAnother example
bull Compute the elimination cliques(the order here is f d e c b a)
bull Form the complete junction graph over the maximal elimination cliques and find a maximum-weight spanning tree
30
Junction Trees and Elimination Junction Trees and Elimination OrderOrder
bull We can use different orderings in variable elimination - affects efficiency
bull Each ordering corresponds to a junction tree
bull Just as some elimination orderings are more efficient than others some junction trees are better than others (Recall our mention of heuristics for triangulation)
31
OK I have this tree now whatOK I have this tree now what
bull A separator S divides the remaining variables into two groupsndash Variables in each group appear
on one side in the cluster tree
bull Examplesndash AB L S T V amp D X
ndash AL T V amp BDSX
ndash BL S amp A DT V X
ndash A X amp BDL S T V
ndash T V amp A B D K S X
LT
A B
X
V S
D
TV
ALTBLS
XAALB
ABD
AAB
BL
T
AL
32
Elimination in Junction TreesElimination in Junction Trees
bull Let X and Y be the partition induced by S
Observation
bull Eliminating all variables in X results in a factor fX(S)
bull Proof Since S is a separator only variables in S are adjacentto variables in X
bull NoteThe same factor would result regardless of theelimination order
x
y
A BSfX(S)
fY(S)
33
Recursive Elimination in Recursive Elimination in Junction TreesJunction Trees
bull How do we compute fX(S)
bull By recursive decomposition alongcluster tree
ndash Let X1 and X2 be the disjoint partitioning of X C implied by theseparators S1 and S2
ndash Eliminate X1 to get fX1(S1)
ndash Eliminate X2 to get fX2(S2)
ndash Eliminate variables in C S toget fX(S)
C
S
S2S1
x1
x2
y
- Slide 6
- Slide 7
- Slide 8
- Slide 9
- Slide 10
- Slide 11
- Slide 12
- Slide 13
- Slide 23
- Slide 25
- Slide 28
-
7
V S
LT
A B
X D
)|( )|( )|( )|( )|( )|( )( )( badPaxPltaPsbPslPvtPsPvP
Eliminate variables in order
Initial factors
v
v vtPvPtf )|()()(
baltxsv
)|()|()|()|()|()()( badPaxPltaPsbPslPsPtfv
[ Note fv(t) = P(t) In general result of elimination is not necessarily a probability term ]
8
)|( )|( )|( )|( )|( )|( )( )( badPaxPltaPsbPslPvtPsPvP
Eliminate variables in order
Initial factors
V S
LT
A B
X D
baltxsv
)|( )|( )|( )|( )|( )( )( badPaxPltaPsbPslPsPtfv
s
s slPsbPsPlbf )|()|()()(
)|()|()|()()( badPaxPltaPlbftf sv
[ Note result of elimination may be a function of several variables ]
9
)|()|()()()( badPltaPaflbftf xsv
)|( )|( )|( )|( )|( )|( )( )( badPaxPltaPsbPslPvtPsPvP
Eliminate variables in order
Initial factors
V S
LT
A B
X D
baltxsv
)|( )|( )|( )|( )|( )( )( badPaxPltaPsbPslPsPtfv)|( )|( )|( )( )( badPaxPltaPlbftf sv
x
x axPaf )|()([ Note fx(a) = 1 for all values of a ]
10
)|( )|( )( )( )( badPltaPaflbftf xsv
)|( )|( )|( )|( )|( )|( )( )( badPaxPltaPsbPslPvtPsPvP
Eliminate variables in order
Initial factors
V S
LT
A B
X D
baltxsv
)|( )|( )|( )|( )|( )( )( badPaxPltaPsbPslPsPtfv)|( )|( )|( )( )( badPaxPltaPlbftf sv
t
vt ltaPtflaf )|()()(
)|()()()( badPlafaflbf txs
11
)|( )|( )( )( )( badPltaPaflbftf xsv
)|( )|( )|( )|( )|( )|( )( )( badPaxPltaPsbPslPvtPsPvP
Eliminate variables in order
Initial factors
V S
LT
A B
X D
baltxsv
)|( )|( )|( )|( )|( )( )( badPaxPltaPsbPslPsPtfv)|( )|( )|( )( )( badPaxPltaPlbftf sv
)|( )( )( )( badPlafaflbf txs
l
tsl laflbfbaf )()()( )|()()( badPafbaf xl
12
)|( )( )( badPafbaf xl
)|( )|( )( )( )( badPltaPaflbftf xsv
)|( )|( )|( )|( )|( )|( )( )( badPaxPltaPsbPslPvtPsPvP
Eliminate variables in order
Initial factors
V S
LT
A B
X D
baltxsv
)|( )|( )|( )|( )|( )( )( badPaxPltaPsbPslPsPtfv)|( )|( )|( )( )( badPaxPltaPlbftf sv
)|( )( )( )( badPlafaflbf txs
a
xla badpafbafdbf )|()()()( )( dbfa
13
)|( )( )( badPafbaf xl
)|( )|( )( )( )( badPltaPaflbftf xsv
)|( )|( )|( )|( )|( )|( )( )( badPaxPltaPsbPslPvtPsPvP
Eliminate variables in order
Initial factors
V S
LT
A B
X D
baltxsv
)|( )|( )|( )|( )|( )( )( badPaxPltaPsbPslPsPtfv)|( )|( )|( )( )( badPaxPltaPlbftf sv
)|( )( )( )( badPlafaflbf txs
)( dbfa )()()( dfdbfdf bb
ab
14
Intermediate factorsIntermediate factors
baltxsv
ga (ltdbx)
gb (ltdxs)
gx (ltds)
gt (ltsv)
gv (lds)
gs(ld)
gl (d))()()()(
)()(
)(
dfdbfbaflaf
aflbf
tf
b
a
l
t
x
s
v
lsvtxba
In our previous example With a different ordering
V S
LT
A B
X D
Complexity is exponential in the size of these factors
15
Notes about variable Notes about variable eliminationelimination
bull Actual computation is done in the elimination steps
bull Computation depends on the order of elimination
bull For each query we need to compute everything againndash Many redundant calculations
16
The ideaThe idea
bull Compute joint over partitions of U
ndash 1048708 small subset of U (typically made of a variable and its parents) - clusters
ndash not necessary disjoint
ndash Calculate
ndash To compute P(X) need far less operations
iC
iiCU
XC
i
i
CPXP
)()(
iC)( iCP
17
Junction TreesJunction Trees
bull The junction tree algorithms generalize Variable Elimination to the efficient simultaneous execution of a large class of queries
bull Theoretical background was shown in the previous lecture
18
Constructing Junction TreesConstructing Junction Trees
1 Moralize the graph (if directed)
2 Choose a node ordering and find the cliques generated by variable elimination This gives a triangulation of the graph
3 Build a junction graph from the eliminated cliques
4 Find an appropriate spanning tree
19
Step 1 MoralizationStep 1 Moralization
a
b c
d e
f
g
h
a
b c
d e
f
g
h
a
b c
d e
f
g
h
1 For all w Vbull For all uvpa(w) add an edge e=u-v
2 Undirect all edges
GMG = ( V E )
20
Step 2 TriangulationStep 2 Triangulation
Add edges to GM such that there is no cyclewith length 4 that does not contain a chord
NO YES
a
b c
d e
f
g
h
a
b c
d e
f
g
h
GM GT
21
Step 2 Triangulation (cont)Step 2 Triangulation (cont)
bull Each elimination ordering triangulates the graph not necessarily in the same way
A
H
B
D
F
C
E
G
A
H
B
D
F
C
E
G
A
H
B
D
F
C
E
G
A
H
B
D
F
C
E
G
A
H
B
D
F
C
E
G
A
H
B
D
F
C
E
G
A
H
B
D
F
C
E
G
A
H
B
D
F
C
E
G
A
H
B
D
F
C
E
G
22
Step 2 Triangulation (cont)Step 2 Triangulation (cont)
bull Intuitively triangulations with as few fill-ins as possible are preferredndash Leaves us with small cliques (small probability tables)
bull A common heuristic Repeat until no nodes remain
ndash Find the node whose elimination would require the least number of fill-ins (may be zero)
ndash Eliminate that node and note the need for a fill-in edge between any two non-adjacent neighbors
bull Add the fill-in edges to the original graph
23
a
b c
d e
f
g
h
a
b c
d e
f
g
h
a
b c
d e
f
g
a
b c
d e
f
a
b c
d e
a
b
d e
aa
e
a
d e
vertex induced addedremoved clique edges
1 h egh -2 g ceg -3 f def -4 c ace a-e
vertex induced added removed clique edges
5 b abd a-d6 d ade -7 e ae -8 a a -
GT
GM
Eliminate the vertex that requires least number of edges to be added
24
Step 3 Junction GraphStep 3 Junction Graph
bull A junction graph for an undirected graph G is an undirected labeled graph
bull The nodes are the cliques in G
bull If two cliques intersect they are joined in the junction graph by an edge labeled with their intersection
25
a
b
d
a
c
e
d e
f
a
d e
e
g
h
c
e
g
a
b c
d e
f
g
h
Bayesian NetworkG = ( V E )
a
b c
d e
f
g
h
a
b c
d e
f
g
h
Moral graph GM Triangulated graph GT
abd
ade
ace
ceg
eghdef
ad ae ce
de eg
seperators
Junction graph GJ (not complete)eg ceg egh = eg Cliques
e
e
e
a
e
26
Step 4 Junction TreeStep 4 Junction Tree
bull A junction tree is a sub-graph of the junction graph that ndash Is a tree
ndash Contains all the cliques (spanning tree)
ndash Satisfies the running intersection property
for each pair of nodes U V all nodes on the path between U and V contain
(as seen in the previous part of the lecture)
VU
27
Step 4 Junction Tree (cont)Step 4 Junction Tree (cont)
bull Theorem An undirected graph is triangulated if and only if its junction graph has a junction tree
bull Definition The weight of a link in a junction graph is the number of variable in the label The weight of a junction tree is the sum of weights of the labels
bull Theorem A sub-tree of the junction graph of a triangulated graph is a junction tree if and only if it is a spanning of maximal weight
28
Junction graph GJ (not complete)
abd
ade
ace
ceg
eghdef
ad ae ce
de eg
e
e
e
a
e
abd
ade
ace
ceg
eghdef
ad ae ce
de eg
Junction tree GJT
There are several methods to find MST
Kruskalrsquos algorithm choose successively a link of
maximal weight unless it creates a cycle
29
Another exampleAnother example
bull Compute the elimination cliques(the order here is f d e c b a)
bull Form the complete junction graph over the maximal elimination cliques and find a maximum-weight spanning tree
30
Junction Trees and Elimination Junction Trees and Elimination OrderOrder
bull We can use different orderings in variable elimination - affects efficiency
bull Each ordering corresponds to a junction tree
bull Just as some elimination orderings are more efficient than others some junction trees are better than others (Recall our mention of heuristics for triangulation)
31
OK I have this tree now whatOK I have this tree now what
bull A separator S divides the remaining variables into two groupsndash Variables in each group appear
on one side in the cluster tree
bull Examplesndash AB L S T V amp D X
ndash AL T V amp BDSX
ndash BL S amp A DT V X
ndash A X amp BDL S T V
ndash T V amp A B D K S X
LT
A B
X
V S
D
TV
ALTBLS
XAALB
ABD
AAB
BL
T
AL
32
Elimination in Junction TreesElimination in Junction Trees
bull Let X and Y be the partition induced by S
Observation
bull Eliminating all variables in X results in a factor fX(S)
bull Proof Since S is a separator only variables in S are adjacentto variables in X
bull NoteThe same factor would result regardless of theelimination order
x
y
A BSfX(S)
fY(S)
33
Recursive Elimination in Recursive Elimination in Junction TreesJunction Trees
bull How do we compute fX(S)
bull By recursive decomposition alongcluster tree
ndash Let X1 and X2 be the disjoint partitioning of X C implied by theseparators S1 and S2
ndash Eliminate X1 to get fX1(S1)
ndash Eliminate X2 to get fX2(S2)
ndash Eliminate variables in C S toget fX(S)
C
S
S2S1
x1
x2
y
- Slide 6
- Slide 7
- Slide 8
- Slide 9
- Slide 10
- Slide 11
- Slide 12
- Slide 13
- Slide 23
- Slide 25
- Slide 28
-
8
)|( )|( )|( )|( )|( )|( )( )( badPaxPltaPsbPslPvtPsPvP
Eliminate variables in order
Initial factors
V S
LT
A B
X D
baltxsv
)|( )|( )|( )|( )|( )( )( badPaxPltaPsbPslPsPtfv
s
s slPsbPsPlbf )|()|()()(
)|()|()|()()( badPaxPltaPlbftf sv
[ Note result of elimination may be a function of several variables ]
9
)|()|()()()( badPltaPaflbftf xsv
)|( )|( )|( )|( )|( )|( )( )( badPaxPltaPsbPslPvtPsPvP
Eliminate variables in order
Initial factors
V S
LT
A B
X D
baltxsv
)|( )|( )|( )|( )|( )( )( badPaxPltaPsbPslPsPtfv)|( )|( )|( )( )( badPaxPltaPlbftf sv
x
x axPaf )|()([ Note fx(a) = 1 for all values of a ]
10
)|( )|( )( )( )( badPltaPaflbftf xsv
)|( )|( )|( )|( )|( )|( )( )( badPaxPltaPsbPslPvtPsPvP
Eliminate variables in order
Initial factors
V S
LT
A B
X D
baltxsv
)|( )|( )|( )|( )|( )( )( badPaxPltaPsbPslPsPtfv)|( )|( )|( )( )( badPaxPltaPlbftf sv
t
vt ltaPtflaf )|()()(
)|()()()( badPlafaflbf txs
11
)|( )|( )( )( )( badPltaPaflbftf xsv
)|( )|( )|( )|( )|( )|( )( )( badPaxPltaPsbPslPvtPsPvP
Eliminate variables in order
Initial factors
V S
LT
A B
X D
baltxsv
)|( )|( )|( )|( )|( )( )( badPaxPltaPsbPslPsPtfv)|( )|( )|( )( )( badPaxPltaPlbftf sv
)|( )( )( )( badPlafaflbf txs
l
tsl laflbfbaf )()()( )|()()( badPafbaf xl
12
)|( )( )( badPafbaf xl
)|( )|( )( )( )( badPltaPaflbftf xsv
)|( )|( )|( )|( )|( )|( )( )( badPaxPltaPsbPslPvtPsPvP
Eliminate variables in order
Initial factors
V S
LT
A B
X D
baltxsv
)|( )|( )|( )|( )|( )( )( badPaxPltaPsbPslPsPtfv)|( )|( )|( )( )( badPaxPltaPlbftf sv
)|( )( )( )( badPlafaflbf txs
a
xla badpafbafdbf )|()()()( )( dbfa
13
)|( )( )( badPafbaf xl
)|( )|( )( )( )( badPltaPaflbftf xsv
)|( )|( )|( )|( )|( )|( )( )( badPaxPltaPsbPslPvtPsPvP
Eliminate variables in order
Initial factors
V S
LT
A B
X D
baltxsv
)|( )|( )|( )|( )|( )( )( badPaxPltaPsbPslPsPtfv)|( )|( )|( )( )( badPaxPltaPlbftf sv
)|( )( )( )( badPlafaflbf txs
)( dbfa )()()( dfdbfdf bb
ab
14
Intermediate factorsIntermediate factors
baltxsv
ga (ltdbx)
gb (ltdxs)
gx (ltds)
gt (ltsv)
gv (lds)
gs(ld)
gl (d))()()()(
)()(
)(
dfdbfbaflaf
aflbf
tf
b
a
l
t
x
s
v
lsvtxba
In our previous example With a different ordering
V S
LT
A B
X D
Complexity is exponential in the size of these factors
15
Notes about variable Notes about variable eliminationelimination
bull Actual computation is done in the elimination steps
bull Computation depends on the order of elimination
bull For each query we need to compute everything againndash Many redundant calculations
16
The ideaThe idea
bull Compute joint over partitions of U
ndash 1048708 small subset of U (typically made of a variable and its parents) - clusters
ndash not necessary disjoint
ndash Calculate
ndash To compute P(X) need far less operations
iC
iiCU
XC
i
i
CPXP
)()(
iC)( iCP
17
Junction TreesJunction Trees
bull The junction tree algorithms generalize Variable Elimination to the efficient simultaneous execution of a large class of queries
bull Theoretical background was shown in the previous lecture
18
Constructing Junction TreesConstructing Junction Trees
1 Moralize the graph (if directed)
2 Choose a node ordering and find the cliques generated by variable elimination This gives a triangulation of the graph
3 Build a junction graph from the eliminated cliques
4 Find an appropriate spanning tree
19
Step 1 MoralizationStep 1 Moralization
a
b c
d e
f
g
h
a
b c
d e
f
g
h
a
b c
d e
f
g
h
1 For all w Vbull For all uvpa(w) add an edge e=u-v
2 Undirect all edges
GMG = ( V E )
20
Step 2 TriangulationStep 2 Triangulation
Add edges to GM such that there is no cyclewith length 4 that does not contain a chord
NO YES
a
b c
d e
f
g
h
a
b c
d e
f
g
h
GM GT
21
Step 2 Triangulation (cont)Step 2 Triangulation (cont)
bull Each elimination ordering triangulates the graph not necessarily in the same way
A
H
B
D
F
C
E
G
A
H
B
D
F
C
E
G
A
H
B
D
F
C
E
G
A
H
B
D
F
C
E
G
A
H
B
D
F
C
E
G
A
H
B
D
F
C
E
G
A
H
B
D
F
C
E
G
A
H
B
D
F
C
E
G
A
H
B
D
F
C
E
G
22
Step 2 Triangulation (cont)Step 2 Triangulation (cont)
bull Intuitively triangulations with as few fill-ins as possible are preferredndash Leaves us with small cliques (small probability tables)
bull A common heuristic Repeat until no nodes remain
ndash Find the node whose elimination would require the least number of fill-ins (may be zero)
ndash Eliminate that node and note the need for a fill-in edge between any two non-adjacent neighbors
bull Add the fill-in edges to the original graph
23
a
b c
d e
f
g
h
a
b c
d e
f
g
h
a
b c
d e
f
g
a
b c
d e
f
a
b c
d e
a
b
d e
aa
e
a
d e
vertex induced addedremoved clique edges
1 h egh -2 g ceg -3 f def -4 c ace a-e
vertex induced added removed clique edges
5 b abd a-d6 d ade -7 e ae -8 a a -
GT
GM
Eliminate the vertex that requires least number of edges to be added
24
Step 3 Junction GraphStep 3 Junction Graph
bull A junction graph for an undirected graph G is an undirected labeled graph
bull The nodes are the cliques in G
bull If two cliques intersect they are joined in the junction graph by an edge labeled with their intersection
25
a
b
d
a
c
e
d e
f
a
d e
e
g
h
c
e
g
a
b c
d e
f
g
h
Bayesian NetworkG = ( V E )
a
b c
d e
f
g
h
a
b c
d e
f
g
h
Moral graph GM Triangulated graph GT
abd
ade
ace
ceg
eghdef
ad ae ce
de eg
seperators
Junction graph GJ (not complete)eg ceg egh = eg Cliques
e
e
e
a
e
26
Step 4 Junction TreeStep 4 Junction Tree
bull A junction tree is a sub-graph of the junction graph that ndash Is a tree
ndash Contains all the cliques (spanning tree)
ndash Satisfies the running intersection property
for each pair of nodes U V all nodes on the path between U and V contain
(as seen in the previous part of the lecture)
VU
27
Step 4 Junction Tree (cont)Step 4 Junction Tree (cont)
bull Theorem An undirected graph is triangulated if and only if its junction graph has a junction tree
bull Definition The weight of a link in a junction graph is the number of variable in the label The weight of a junction tree is the sum of weights of the labels
bull Theorem A sub-tree of the junction graph of a triangulated graph is a junction tree if and only if it is a spanning of maximal weight
28
Junction graph GJ (not complete)
abd
ade
ace
ceg
eghdef
ad ae ce
de eg
e
e
e
a
e
abd
ade
ace
ceg
eghdef
ad ae ce
de eg
Junction tree GJT
There are several methods to find MST
Kruskalrsquos algorithm choose successively a link of
maximal weight unless it creates a cycle
29
Another exampleAnother example
bull Compute the elimination cliques(the order here is f d e c b a)
bull Form the complete junction graph over the maximal elimination cliques and find a maximum-weight spanning tree
30
Junction Trees and Elimination Junction Trees and Elimination OrderOrder
bull We can use different orderings in variable elimination - affects efficiency
bull Each ordering corresponds to a junction tree
bull Just as some elimination orderings are more efficient than others some junction trees are better than others (Recall our mention of heuristics for triangulation)
31
OK I have this tree now whatOK I have this tree now what
bull A separator S divides the remaining variables into two groupsndash Variables in each group appear
on one side in the cluster tree
bull Examplesndash AB L S T V amp D X
ndash AL T V amp BDSX
ndash BL S amp A DT V X
ndash A X amp BDL S T V
ndash T V amp A B D K S X
LT
A B
X
V S
D
TV
ALTBLS
XAALB
ABD
AAB
BL
T
AL
32
Elimination in Junction TreesElimination in Junction Trees
bull Let X and Y be the partition induced by S
Observation
bull Eliminating all variables in X results in a factor fX(S)
bull Proof Since S is a separator only variables in S are adjacentto variables in X
bull NoteThe same factor would result regardless of theelimination order
x
y
A BSfX(S)
fY(S)
33
Recursive Elimination in Recursive Elimination in Junction TreesJunction Trees
bull How do we compute fX(S)
bull By recursive decomposition alongcluster tree
ndash Let X1 and X2 be the disjoint partitioning of X C implied by theseparators S1 and S2
ndash Eliminate X1 to get fX1(S1)
ndash Eliminate X2 to get fX2(S2)
ndash Eliminate variables in C S toget fX(S)
C
S
S2S1
x1
x2
y
- Slide 6
- Slide 7
- Slide 8
- Slide 9
- Slide 10
- Slide 11
- Slide 12
- Slide 13
- Slide 23
- Slide 25
- Slide 28
-
9
)|()|()()()( badPltaPaflbftf xsv
)|( )|( )|( )|( )|( )|( )( )( badPaxPltaPsbPslPvtPsPvP
Eliminate variables in order
Initial factors
V S
LT
A B
X D
baltxsv
)|( )|( )|( )|( )|( )( )( badPaxPltaPsbPslPsPtfv)|( )|( )|( )( )( badPaxPltaPlbftf sv
x
x axPaf )|()([ Note fx(a) = 1 for all values of a ]
10
)|( )|( )( )( )( badPltaPaflbftf xsv
)|( )|( )|( )|( )|( )|( )( )( badPaxPltaPsbPslPvtPsPvP
Eliminate variables in order
Initial factors
V S
LT
A B
X D
baltxsv
)|( )|( )|( )|( )|( )( )( badPaxPltaPsbPslPsPtfv)|( )|( )|( )( )( badPaxPltaPlbftf sv
t
vt ltaPtflaf )|()()(
)|()()()( badPlafaflbf txs
11
)|( )|( )( )( )( badPltaPaflbftf xsv
)|( )|( )|( )|( )|( )|( )( )( badPaxPltaPsbPslPvtPsPvP
Eliminate variables in order
Initial factors
V S
LT
A B
X D
baltxsv
)|( )|( )|( )|( )|( )( )( badPaxPltaPsbPslPsPtfv)|( )|( )|( )( )( badPaxPltaPlbftf sv
)|( )( )( )( badPlafaflbf txs
l
tsl laflbfbaf )()()( )|()()( badPafbaf xl
12
)|( )( )( badPafbaf xl
)|( )|( )( )( )( badPltaPaflbftf xsv
)|( )|( )|( )|( )|( )|( )( )( badPaxPltaPsbPslPvtPsPvP
Eliminate variables in order
Initial factors
V S
LT
A B
X D
baltxsv
)|( )|( )|( )|( )|( )( )( badPaxPltaPsbPslPsPtfv)|( )|( )|( )( )( badPaxPltaPlbftf sv
)|( )( )( )( badPlafaflbf txs
a
xla badpafbafdbf )|()()()( )( dbfa
13
)|( )( )( badPafbaf xl
)|( )|( )( )( )( badPltaPaflbftf xsv
)|( )|( )|( )|( )|( )|( )( )( badPaxPltaPsbPslPvtPsPvP
Eliminate variables in order
Initial factors
V S
LT
A B
X D
baltxsv
)|( )|( )|( )|( )|( )( )( badPaxPltaPsbPslPsPtfv)|( )|( )|( )( )( badPaxPltaPlbftf sv
)|( )( )( )( badPlafaflbf txs
)( dbfa )()()( dfdbfdf bb
ab
14
Intermediate factorsIntermediate factors
baltxsv
ga (ltdbx)
gb (ltdxs)
gx (ltds)
gt (ltsv)
gv (lds)
gs(ld)
gl (d))()()()(
)()(
)(
dfdbfbaflaf
aflbf
tf
b
a
l
t
x
s
v
lsvtxba
In our previous example With a different ordering
V S
LT
A B
X D
Complexity is exponential in the size of these factors
15
Notes about variable Notes about variable eliminationelimination
bull Actual computation is done in the elimination steps
bull Computation depends on the order of elimination
bull For each query we need to compute everything againndash Many redundant calculations
16
The ideaThe idea
bull Compute joint over partitions of U
ndash 1048708 small subset of U (typically made of a variable and its parents) - clusters
ndash not necessary disjoint
ndash Calculate
ndash To compute P(X) need far less operations
iC
iiCU
XC
i
i
CPXP
)()(
iC)( iCP
17
Junction TreesJunction Trees
bull The junction tree algorithms generalize Variable Elimination to the efficient simultaneous execution of a large class of queries
bull Theoretical background was shown in the previous lecture
18
Constructing Junction TreesConstructing Junction Trees
1 Moralize the graph (if directed)
2 Choose a node ordering and find the cliques generated by variable elimination This gives a triangulation of the graph
3 Build a junction graph from the eliminated cliques
4 Find an appropriate spanning tree
19
Step 1 MoralizationStep 1 Moralization
a
b c
d e
f
g
h
a
b c
d e
f
g
h
a
b c
d e
f
g
h
1 For all w Vbull For all uvpa(w) add an edge e=u-v
2 Undirect all edges
GMG = ( V E )
20
Step 2 TriangulationStep 2 Triangulation
Add edges to GM such that there is no cyclewith length 4 that does not contain a chord
NO YES
a
b c
d e
f
g
h
a
b c
d e
f
g
h
GM GT
21
Step 2 Triangulation (cont)Step 2 Triangulation (cont)
bull Each elimination ordering triangulates the graph not necessarily in the same way
A
H
B
D
F
C
E
G
A
H
B
D
F
C
E
G
A
H
B
D
F
C
E
G
A
H
B
D
F
C
E
G
A
H
B
D
F
C
E
G
A
H
B
D
F
C
E
G
A
H
B
D
F
C
E
G
A
H
B
D
F
C
E
G
A
H
B
D
F
C
E
G
22
Step 2 Triangulation (cont)Step 2 Triangulation (cont)
bull Intuitively triangulations with as few fill-ins as possible are preferredndash Leaves us with small cliques (small probability tables)
bull A common heuristic Repeat until no nodes remain
ndash Find the node whose elimination would require the least number of fill-ins (may be zero)
ndash Eliminate that node and note the need for a fill-in edge between any two non-adjacent neighbors
bull Add the fill-in edges to the original graph
23
a
b c
d e
f
g
h
a
b c
d e
f
g
h
a
b c
d e
f
g
a
b c
d e
f
a
b c
d e
a
b
d e
aa
e
a
d e
vertex induced addedremoved clique edges
1 h egh -2 g ceg -3 f def -4 c ace a-e
vertex induced added removed clique edges
5 b abd a-d6 d ade -7 e ae -8 a a -
GT
GM
Eliminate the vertex that requires least number of edges to be added
24
Step 3 Junction GraphStep 3 Junction Graph
bull A junction graph for an undirected graph G is an undirected labeled graph
bull The nodes are the cliques in G
bull If two cliques intersect they are joined in the junction graph by an edge labeled with their intersection
25
a
b
d
a
c
e
d e
f
a
d e
e
g
h
c
e
g
a
b c
d e
f
g
h
Bayesian NetworkG = ( V E )
a
b c
d e
f
g
h
a
b c
d e
f
g
h
Moral graph GM Triangulated graph GT
abd
ade
ace
ceg
eghdef
ad ae ce
de eg
seperators
Junction graph GJ (not complete)eg ceg egh = eg Cliques
e
e
e
a
e
26
Step 4 Junction TreeStep 4 Junction Tree
bull A junction tree is a sub-graph of the junction graph that ndash Is a tree
ndash Contains all the cliques (spanning tree)
ndash Satisfies the running intersection property
for each pair of nodes U V all nodes on the path between U and V contain
(as seen in the previous part of the lecture)
VU
27
Step 4 Junction Tree (cont)Step 4 Junction Tree (cont)
bull Theorem An undirected graph is triangulated if and only if its junction graph has a junction tree
bull Definition The weight of a link in a junction graph is the number of variable in the label The weight of a junction tree is the sum of weights of the labels
bull Theorem A sub-tree of the junction graph of a triangulated graph is a junction tree if and only if it is a spanning of maximal weight
28
Junction graph GJ (not complete)
abd
ade
ace
ceg
eghdef
ad ae ce
de eg
e
e
e
a
e
abd
ade
ace
ceg
eghdef
ad ae ce
de eg
Junction tree GJT
There are several methods to find MST
Kruskalrsquos algorithm choose successively a link of
maximal weight unless it creates a cycle
29
Another exampleAnother example
bull Compute the elimination cliques(the order here is f d e c b a)
bull Form the complete junction graph over the maximal elimination cliques and find a maximum-weight spanning tree
30
Junction Trees and Elimination Junction Trees and Elimination OrderOrder
bull We can use different orderings in variable elimination - affects efficiency
bull Each ordering corresponds to a junction tree
bull Just as some elimination orderings are more efficient than others some junction trees are better than others (Recall our mention of heuristics for triangulation)
31
OK I have this tree now whatOK I have this tree now what
bull A separator S divides the remaining variables into two groupsndash Variables in each group appear
on one side in the cluster tree
bull Examplesndash AB L S T V amp D X
ndash AL T V amp BDSX
ndash BL S amp A DT V X
ndash A X amp BDL S T V
ndash T V amp A B D K S X
LT
A B
X
V S
D
TV
ALTBLS
XAALB
ABD
AAB
BL
T
AL
32
Elimination in Junction TreesElimination in Junction Trees
bull Let X and Y be the partition induced by S
Observation
bull Eliminating all variables in X results in a factor fX(S)
bull Proof Since S is a separator only variables in S are adjacentto variables in X
bull NoteThe same factor would result regardless of theelimination order
x
y
A BSfX(S)
fY(S)
33
Recursive Elimination in Recursive Elimination in Junction TreesJunction Trees
bull How do we compute fX(S)
bull By recursive decomposition alongcluster tree
ndash Let X1 and X2 be the disjoint partitioning of X C implied by theseparators S1 and S2
ndash Eliminate X1 to get fX1(S1)
ndash Eliminate X2 to get fX2(S2)
ndash Eliminate variables in C S toget fX(S)
C
S
S2S1
x1
x2
y
- Slide 6
- Slide 7
- Slide 8
- Slide 9
- Slide 10
- Slide 11
- Slide 12
- Slide 13
- Slide 23
- Slide 25
- Slide 28
-
10
)|( )|( )( )( )( badPltaPaflbftf xsv
)|( )|( )|( )|( )|( )|( )( )( badPaxPltaPsbPslPvtPsPvP
Eliminate variables in order
Initial factors
V S
LT
A B
X D
baltxsv
)|( )|( )|( )|( )|( )( )( badPaxPltaPsbPslPsPtfv)|( )|( )|( )( )( badPaxPltaPlbftf sv
t
vt ltaPtflaf )|()()(
)|()()()( badPlafaflbf txs
11
)|( )|( )( )( )( badPltaPaflbftf xsv
)|( )|( )|( )|( )|( )|( )( )( badPaxPltaPsbPslPvtPsPvP
Eliminate variables in order
Initial factors
V S
LT
A B
X D
baltxsv
)|( )|( )|( )|( )|( )( )( badPaxPltaPsbPslPsPtfv)|( )|( )|( )( )( badPaxPltaPlbftf sv
)|( )( )( )( badPlafaflbf txs
l
tsl laflbfbaf )()()( )|()()( badPafbaf xl
12
)|( )( )( badPafbaf xl
)|( )|( )( )( )( badPltaPaflbftf xsv
)|( )|( )|( )|( )|( )|( )( )( badPaxPltaPsbPslPvtPsPvP
Eliminate variables in order
Initial factors
V S
LT
A B
X D
baltxsv
)|( )|( )|( )|( )|( )( )( badPaxPltaPsbPslPsPtfv)|( )|( )|( )( )( badPaxPltaPlbftf sv
)|( )( )( )( badPlafaflbf txs
a
xla badpafbafdbf )|()()()( )( dbfa
13
)|( )( )( badPafbaf xl
)|( )|( )( )( )( badPltaPaflbftf xsv
)|( )|( )|( )|( )|( )|( )( )( badPaxPltaPsbPslPvtPsPvP
Eliminate variables in order
Initial factors
V S
LT
A B
X D
baltxsv
)|( )|( )|( )|( )|( )( )( badPaxPltaPsbPslPsPtfv)|( )|( )|( )( )( badPaxPltaPlbftf sv
)|( )( )( )( badPlafaflbf txs
)( dbfa )()()( dfdbfdf bb
ab
14
Intermediate factorsIntermediate factors
baltxsv
ga (ltdbx)
gb (ltdxs)
gx (ltds)
gt (ltsv)
gv (lds)
gs(ld)
gl (d))()()()(
)()(
)(
dfdbfbaflaf
aflbf
tf
b
a
l
t
x
s
v
lsvtxba
In our previous example With a different ordering
V S
LT
A B
X D
Complexity is exponential in the size of these factors
15
Notes about variable Notes about variable eliminationelimination
bull Actual computation is done in the elimination steps
bull Computation depends on the order of elimination
bull For each query we need to compute everything againndash Many redundant calculations
16
The ideaThe idea
bull Compute joint over partitions of U
ndash 1048708 small subset of U (typically made of a variable and its parents) - clusters
ndash not necessary disjoint
ndash Calculate
ndash To compute P(X) need far less operations
iC
iiCU
XC
i
i
CPXP
)()(
iC)( iCP
17
Junction TreesJunction Trees
bull The junction tree algorithms generalize Variable Elimination to the efficient simultaneous execution of a large class of queries
bull Theoretical background was shown in the previous lecture
18
Constructing Junction TreesConstructing Junction Trees
1 Moralize the graph (if directed)
2 Choose a node ordering and find the cliques generated by variable elimination This gives a triangulation of the graph
3 Build a junction graph from the eliminated cliques
4 Find an appropriate spanning tree
19
Step 1 MoralizationStep 1 Moralization
a
b c
d e
f
g
h
a
b c
d e
f
g
h
a
b c
d e
f
g
h
1 For all w Vbull For all uvpa(w) add an edge e=u-v
2 Undirect all edges
GMG = ( V E )
20
Step 2 TriangulationStep 2 Triangulation
Add edges to GM such that there is no cyclewith length 4 that does not contain a chord
NO YES
a
b c
d e
f
g
h
a
b c
d e
f
g
h
GM GT
21
Step 2 Triangulation (cont)Step 2 Triangulation (cont)
bull Each elimination ordering triangulates the graph not necessarily in the same way
A
H
B
D
F
C
E
G
A
H
B
D
F
C
E
G
A
H
B
D
F
C
E
G
A
H
B
D
F
C
E
G
A
H
B
D
F
C
E
G
A
H
B
D
F
C
E
G
A
H
B
D
F
C
E
G
A
H
B
D
F
C
E
G
A
H
B
D
F
C
E
G
22
Step 2 Triangulation (cont)Step 2 Triangulation (cont)
bull Intuitively triangulations with as few fill-ins as possible are preferredndash Leaves us with small cliques (small probability tables)
bull A common heuristic Repeat until no nodes remain
ndash Find the node whose elimination would require the least number of fill-ins (may be zero)
ndash Eliminate that node and note the need for a fill-in edge between any two non-adjacent neighbors
bull Add the fill-in edges to the original graph
23
a
b c
d e
f
g
h
a
b c
d e
f
g
h
a
b c
d e
f
g
a
b c
d e
f
a
b c
d e
a
b
d e
aa
e
a
d e
vertex induced addedremoved clique edges
1 h egh -2 g ceg -3 f def -4 c ace a-e
vertex induced added removed clique edges
5 b abd a-d6 d ade -7 e ae -8 a a -
GT
GM
Eliminate the vertex that requires least number of edges to be added
24
Step 3 Junction GraphStep 3 Junction Graph
bull A junction graph for an undirected graph G is an undirected labeled graph
bull The nodes are the cliques in G
bull If two cliques intersect they are joined in the junction graph by an edge labeled with their intersection
25
a
b
d
a
c
e
d e
f
a
d e
e
g
h
c
e
g
a
b c
d e
f
g
h
Bayesian NetworkG = ( V E )
a
b c
d e
f
g
h
a
b c
d e
f
g
h
Moral graph GM Triangulated graph GT
abd
ade
ace
ceg
eghdef
ad ae ce
de eg
seperators
Junction graph GJ (not complete)eg ceg egh = eg Cliques
e
e
e
a
e
26
Step 4 Junction TreeStep 4 Junction Tree
bull A junction tree is a sub-graph of the junction graph that ndash Is a tree
ndash Contains all the cliques (spanning tree)
ndash Satisfies the running intersection property
for each pair of nodes U V all nodes on the path between U and V contain
(as seen in the previous part of the lecture)
VU
27
Step 4 Junction Tree (cont)Step 4 Junction Tree (cont)
bull Theorem An undirected graph is triangulated if and only if its junction graph has a junction tree
bull Definition The weight of a link in a junction graph is the number of variable in the label The weight of a junction tree is the sum of weights of the labels
bull Theorem A sub-tree of the junction graph of a triangulated graph is a junction tree if and only if it is a spanning of maximal weight
28
Junction graph GJ (not complete)
abd
ade
ace
ceg
eghdef
ad ae ce
de eg
e
e
e
a
e
abd
ade
ace
ceg
eghdef
ad ae ce
de eg
Junction tree GJT
There are several methods to find MST
Kruskalrsquos algorithm choose successively a link of
maximal weight unless it creates a cycle
29
Another exampleAnother example
bull Compute the elimination cliques(the order here is f d e c b a)
bull Form the complete junction graph over the maximal elimination cliques and find a maximum-weight spanning tree
30
Junction Trees and Elimination Junction Trees and Elimination OrderOrder
bull We can use different orderings in variable elimination - affects efficiency
bull Each ordering corresponds to a junction tree
bull Just as some elimination orderings are more efficient than others some junction trees are better than others (Recall our mention of heuristics for triangulation)
31
OK I have this tree now whatOK I have this tree now what
bull A separator S divides the remaining variables into two groupsndash Variables in each group appear
on one side in the cluster tree
bull Examplesndash AB L S T V amp D X
ndash AL T V amp BDSX
ndash BL S amp A DT V X
ndash A X amp BDL S T V
ndash T V amp A B D K S X
LT
A B
X
V S
D
TV
ALTBLS
XAALB
ABD
AAB
BL
T
AL
32
Elimination in Junction TreesElimination in Junction Trees
bull Let X and Y be the partition induced by S
Observation
bull Eliminating all variables in X results in a factor fX(S)
bull Proof Since S is a separator only variables in S are adjacentto variables in X
bull NoteThe same factor would result regardless of theelimination order
x
y
A BSfX(S)
fY(S)
33
Recursive Elimination in Recursive Elimination in Junction TreesJunction Trees
bull How do we compute fX(S)
bull By recursive decomposition alongcluster tree
ndash Let X1 and X2 be the disjoint partitioning of X C implied by theseparators S1 and S2
ndash Eliminate X1 to get fX1(S1)
ndash Eliminate X2 to get fX2(S2)
ndash Eliminate variables in C S toget fX(S)
C
S
S2S1
x1
x2
y
- Slide 6
- Slide 7
- Slide 8
- Slide 9
- Slide 10
- Slide 11
- Slide 12
- Slide 13
- Slide 23
- Slide 25
- Slide 28
-
11
)|( )|( )( )( )( badPltaPaflbftf xsv
)|( )|( )|( )|( )|( )|( )( )( badPaxPltaPsbPslPvtPsPvP
Eliminate variables in order
Initial factors
V S
LT
A B
X D
baltxsv
)|( )|( )|( )|( )|( )( )( badPaxPltaPsbPslPsPtfv)|( )|( )|( )( )( badPaxPltaPlbftf sv
)|( )( )( )( badPlafaflbf txs
l
tsl laflbfbaf )()()( )|()()( badPafbaf xl
12
)|( )( )( badPafbaf xl
)|( )|( )( )( )( badPltaPaflbftf xsv
)|( )|( )|( )|( )|( )|( )( )( badPaxPltaPsbPslPvtPsPvP
Eliminate variables in order
Initial factors
V S
LT
A B
X D
baltxsv
)|( )|( )|( )|( )|( )( )( badPaxPltaPsbPslPsPtfv)|( )|( )|( )( )( badPaxPltaPlbftf sv
)|( )( )( )( badPlafaflbf txs
a
xla badpafbafdbf )|()()()( )( dbfa
13
)|( )( )( badPafbaf xl
)|( )|( )( )( )( badPltaPaflbftf xsv
)|( )|( )|( )|( )|( )|( )( )( badPaxPltaPsbPslPvtPsPvP
Eliminate variables in order
Initial factors
V S
LT
A B
X D
baltxsv
)|( )|( )|( )|( )|( )( )( badPaxPltaPsbPslPsPtfv)|( )|( )|( )( )( badPaxPltaPlbftf sv
)|( )( )( )( badPlafaflbf txs
)( dbfa )()()( dfdbfdf bb
ab
14
Intermediate factorsIntermediate factors
baltxsv
ga (ltdbx)
gb (ltdxs)
gx (ltds)
gt (ltsv)
gv (lds)
gs(ld)
gl (d))()()()(
)()(
)(
dfdbfbaflaf
aflbf
tf
b
a
l
t
x
s
v
lsvtxba
In our previous example With a different ordering
V S
LT
A B
X D
Complexity is exponential in the size of these factors
15
Notes about variable Notes about variable eliminationelimination
bull Actual computation is done in the elimination steps
bull Computation depends on the order of elimination
bull For each query we need to compute everything againndash Many redundant calculations
16
The ideaThe idea
bull Compute joint over partitions of U
ndash 1048708 small subset of U (typically made of a variable and its parents) - clusters
ndash not necessary disjoint
ndash Calculate
ndash To compute P(X) need far less operations
iC
iiCU
XC
i
i
CPXP
)()(
iC)( iCP
17
Junction TreesJunction Trees
bull The junction tree algorithms generalize Variable Elimination to the efficient simultaneous execution of a large class of queries
bull Theoretical background was shown in the previous lecture
18
Constructing Junction TreesConstructing Junction Trees
1 Moralize the graph (if directed)
2 Choose a node ordering and find the cliques generated by variable elimination This gives a triangulation of the graph
3 Build a junction graph from the eliminated cliques
4 Find an appropriate spanning tree
19
Step 1 MoralizationStep 1 Moralization
a
b c
d e
f
g
h
a
b c
d e
f
g
h
a
b c
d e
f
g
h
1 For all w Vbull For all uvpa(w) add an edge e=u-v
2 Undirect all edges
GMG = ( V E )
20
Step 2 TriangulationStep 2 Triangulation
Add edges to GM such that there is no cyclewith length 4 that does not contain a chord
NO YES
a
b c
d e
f
g
h
a
b c
d e
f
g
h
GM GT
21
Step 2 Triangulation (cont)Step 2 Triangulation (cont)
bull Each elimination ordering triangulates the graph not necessarily in the same way
A
H
B
D
F
C
E
G
A
H
B
D
F
C
E
G
A
H
B
D
F
C
E
G
A
H
B
D
F
C
E
G
A
H
B
D
F
C
E
G
A
H
B
D
F
C
E
G
A
H
B
D
F
C
E
G
A
H
B
D
F
C
E
G
A
H
B
D
F
C
E
G
22
Step 2 Triangulation (cont)Step 2 Triangulation (cont)
bull Intuitively triangulations with as few fill-ins as possible are preferredndash Leaves us with small cliques (small probability tables)
bull A common heuristic Repeat until no nodes remain
ndash Find the node whose elimination would require the least number of fill-ins (may be zero)
ndash Eliminate that node and note the need for a fill-in edge between any two non-adjacent neighbors
bull Add the fill-in edges to the original graph
23
a
b c
d e
f
g
h
a
b c
d e
f
g
h
a
b c
d e
f
g
a
b c
d e
f
a
b c
d e
a
b
d e
aa
e
a
d e
vertex induced addedremoved clique edges
1 h egh -2 g ceg -3 f def -4 c ace a-e
vertex induced added removed clique edges
5 b abd a-d6 d ade -7 e ae -8 a a -
GT
GM
Eliminate the vertex that requires least number of edges to be added
24
Step 3 Junction GraphStep 3 Junction Graph
bull A junction graph for an undirected graph G is an undirected labeled graph
bull The nodes are the cliques in G
bull If two cliques intersect they are joined in the junction graph by an edge labeled with their intersection
25
a
b
d
a
c
e
d e
f
a
d e
e
g
h
c
e
g
a
b c
d e
f
g
h
Bayesian NetworkG = ( V E )
a
b c
d e
f
g
h
a
b c
d e
f
g
h
Moral graph GM Triangulated graph GT
abd
ade
ace
ceg
eghdef
ad ae ce
de eg
seperators
Junction graph GJ (not complete)eg ceg egh = eg Cliques
e
e
e
a
e
26
Step 4 Junction TreeStep 4 Junction Tree
bull A junction tree is a sub-graph of the junction graph that ndash Is a tree
ndash Contains all the cliques (spanning tree)
ndash Satisfies the running intersection property
for each pair of nodes U V all nodes on the path between U and V contain
(as seen in the previous part of the lecture)
VU
27
Step 4 Junction Tree (cont)Step 4 Junction Tree (cont)
bull Theorem An undirected graph is triangulated if and only if its junction graph has a junction tree
bull Definition The weight of a link in a junction graph is the number of variable in the label The weight of a junction tree is the sum of weights of the labels
bull Theorem A sub-tree of the junction graph of a triangulated graph is a junction tree if and only if it is a spanning of maximal weight
28
Junction graph GJ (not complete)
abd
ade
ace
ceg
eghdef
ad ae ce
de eg
e
e
e
a
e
abd
ade
ace
ceg
eghdef
ad ae ce
de eg
Junction tree GJT
There are several methods to find MST
Kruskalrsquos algorithm choose successively a link of
maximal weight unless it creates a cycle
29
Another exampleAnother example
bull Compute the elimination cliques(the order here is f d e c b a)
bull Form the complete junction graph over the maximal elimination cliques and find a maximum-weight spanning tree
30
Junction Trees and Elimination Junction Trees and Elimination OrderOrder
bull We can use different orderings in variable elimination - affects efficiency
bull Each ordering corresponds to a junction tree
bull Just as some elimination orderings are more efficient than others some junction trees are better than others (Recall our mention of heuristics for triangulation)
31
OK I have this tree now whatOK I have this tree now what
bull A separator S divides the remaining variables into two groupsndash Variables in each group appear
on one side in the cluster tree
bull Examplesndash AB L S T V amp D X
ndash AL T V amp BDSX
ndash BL S amp A DT V X
ndash A X amp BDL S T V
ndash T V amp A B D K S X
LT
A B
X
V S
D
TV
ALTBLS
XAALB
ABD
AAB
BL
T
AL
32
Elimination in Junction TreesElimination in Junction Trees
bull Let X and Y be the partition induced by S
Observation
bull Eliminating all variables in X results in a factor fX(S)
bull Proof Since S is a separator only variables in S are adjacentto variables in X
bull NoteThe same factor would result regardless of theelimination order
x
y
A BSfX(S)
fY(S)
33
Recursive Elimination in Recursive Elimination in Junction TreesJunction Trees
bull How do we compute fX(S)
bull By recursive decomposition alongcluster tree
ndash Let X1 and X2 be the disjoint partitioning of X C implied by theseparators S1 and S2
ndash Eliminate X1 to get fX1(S1)
ndash Eliminate X2 to get fX2(S2)
ndash Eliminate variables in C S toget fX(S)
C
S
S2S1
x1
x2
y
- Slide 6
- Slide 7
- Slide 8
- Slide 9
- Slide 10
- Slide 11
- Slide 12
- Slide 13
- Slide 23
- Slide 25
- Slide 28
-
12
)|( )( )( badPafbaf xl
)|( )|( )( )( )( badPltaPaflbftf xsv
)|( )|( )|( )|( )|( )|( )( )( badPaxPltaPsbPslPvtPsPvP
Eliminate variables in order
Initial factors
V S
LT
A B
X D
baltxsv
)|( )|( )|( )|( )|( )( )( badPaxPltaPsbPslPsPtfv)|( )|( )|( )( )( badPaxPltaPlbftf sv
)|( )( )( )( badPlafaflbf txs
a
xla badpafbafdbf )|()()()( )( dbfa
13
)|( )( )( badPafbaf xl
)|( )|( )( )( )( badPltaPaflbftf xsv
)|( )|( )|( )|( )|( )|( )( )( badPaxPltaPsbPslPvtPsPvP
Eliminate variables in order
Initial factors
V S
LT
A B
X D
baltxsv
)|( )|( )|( )|( )|( )( )( badPaxPltaPsbPslPsPtfv)|( )|( )|( )( )( badPaxPltaPlbftf sv
)|( )( )( )( badPlafaflbf txs
)( dbfa )()()( dfdbfdf bb
ab
14
Intermediate factorsIntermediate factors
baltxsv
ga (ltdbx)
gb (ltdxs)
gx (ltds)
gt (ltsv)
gv (lds)
gs(ld)
gl (d))()()()(
)()(
)(
dfdbfbaflaf
aflbf
tf
b
a
l
t
x
s
v
lsvtxba
In our previous example With a different ordering
V S
LT
A B
X D
Complexity is exponential in the size of these factors
15
Notes about variable Notes about variable eliminationelimination
bull Actual computation is done in the elimination steps
bull Computation depends on the order of elimination
bull For each query we need to compute everything againndash Many redundant calculations
16
The ideaThe idea
bull Compute joint over partitions of U
ndash 1048708 small subset of U (typically made of a variable and its parents) - clusters
ndash not necessary disjoint
ndash Calculate
ndash To compute P(X) need far less operations
iC
iiCU
XC
i
i
CPXP
)()(
iC)( iCP
17
Junction TreesJunction Trees
bull The junction tree algorithms generalize Variable Elimination to the efficient simultaneous execution of a large class of queries
bull Theoretical background was shown in the previous lecture
18
Constructing Junction TreesConstructing Junction Trees
1 Moralize the graph (if directed)
2 Choose a node ordering and find the cliques generated by variable elimination This gives a triangulation of the graph
3 Build a junction graph from the eliminated cliques
4 Find an appropriate spanning tree
19
Step 1 MoralizationStep 1 Moralization
a
b c
d e
f
g
h
a
b c
d e
f
g
h
a
b c
d e
f
g
h
1 For all w Vbull For all uvpa(w) add an edge e=u-v
2 Undirect all edges
GMG = ( V E )
20
Step 2 TriangulationStep 2 Triangulation
Add edges to GM such that there is no cyclewith length 4 that does not contain a chord
NO YES
a
b c
d e
f
g
h
a
b c
d e
f
g
h
GM GT
21
Step 2 Triangulation (cont)Step 2 Triangulation (cont)
bull Each elimination ordering triangulates the graph not necessarily in the same way
A
H
B
D
F
C
E
G
A
H
B
D
F
C
E
G
A
H
B
D
F
C
E
G
A
H
B
D
F
C
E
G
A
H
B
D
F
C
E
G
A
H
B
D
F
C
E
G
A
H
B
D
F
C
E
G
A
H
B
D
F
C
E
G
A
H
B
D
F
C
E
G
22
Step 2 Triangulation (cont)Step 2 Triangulation (cont)
bull Intuitively triangulations with as few fill-ins as possible are preferredndash Leaves us with small cliques (small probability tables)
bull A common heuristic Repeat until no nodes remain
ndash Find the node whose elimination would require the least number of fill-ins (may be zero)
ndash Eliminate that node and note the need for a fill-in edge between any two non-adjacent neighbors
bull Add the fill-in edges to the original graph
23
a
b c
d e
f
g
h
a
b c
d e
f
g
h
a
b c
d e
f
g
a
b c
d e
f
a
b c
d e
a
b
d e
aa
e
a
d e
vertex induced addedremoved clique edges
1 h egh -2 g ceg -3 f def -4 c ace a-e
vertex induced added removed clique edges
5 b abd a-d6 d ade -7 e ae -8 a a -
GT
GM
Eliminate the vertex that requires least number of edges to be added
24
Step 3 Junction GraphStep 3 Junction Graph
bull A junction graph for an undirected graph G is an undirected labeled graph
bull The nodes are the cliques in G
bull If two cliques intersect they are joined in the junction graph by an edge labeled with their intersection
25
a
b
d
a
c
e
d e
f
a
d e
e
g
h
c
e
g
a
b c
d e
f
g
h
Bayesian NetworkG = ( V E )
a
b c
d e
f
g
h
a
b c
d e
f
g
h
Moral graph GM Triangulated graph GT
abd
ade
ace
ceg
eghdef
ad ae ce
de eg
seperators
Junction graph GJ (not complete)eg ceg egh = eg Cliques
e
e
e
a
e
26
Step 4 Junction TreeStep 4 Junction Tree
bull A junction tree is a sub-graph of the junction graph that ndash Is a tree
ndash Contains all the cliques (spanning tree)
ndash Satisfies the running intersection property
for each pair of nodes U V all nodes on the path between U and V contain
(as seen in the previous part of the lecture)
VU
27
Step 4 Junction Tree (cont)Step 4 Junction Tree (cont)
bull Theorem An undirected graph is triangulated if and only if its junction graph has a junction tree
bull Definition The weight of a link in a junction graph is the number of variable in the label The weight of a junction tree is the sum of weights of the labels
bull Theorem A sub-tree of the junction graph of a triangulated graph is a junction tree if and only if it is a spanning of maximal weight
28
Junction graph GJ (not complete)
abd
ade
ace
ceg
eghdef
ad ae ce
de eg
e
e
e
a
e
abd
ade
ace
ceg
eghdef
ad ae ce
de eg
Junction tree GJT
There are several methods to find MST
Kruskalrsquos algorithm choose successively a link of
maximal weight unless it creates a cycle
29
Another exampleAnother example
bull Compute the elimination cliques(the order here is f d e c b a)
bull Form the complete junction graph over the maximal elimination cliques and find a maximum-weight spanning tree
30
Junction Trees and Elimination Junction Trees and Elimination OrderOrder
bull We can use different orderings in variable elimination - affects efficiency
bull Each ordering corresponds to a junction tree
bull Just as some elimination orderings are more efficient than others some junction trees are better than others (Recall our mention of heuristics for triangulation)
31
OK I have this tree now whatOK I have this tree now what
bull A separator S divides the remaining variables into two groupsndash Variables in each group appear
on one side in the cluster tree
bull Examplesndash AB L S T V amp D X
ndash AL T V amp BDSX
ndash BL S amp A DT V X
ndash A X amp BDL S T V
ndash T V amp A B D K S X
LT
A B
X
V S
D
TV
ALTBLS
XAALB
ABD
AAB
BL
T
AL
32
Elimination in Junction TreesElimination in Junction Trees
bull Let X and Y be the partition induced by S
Observation
bull Eliminating all variables in X results in a factor fX(S)
bull Proof Since S is a separator only variables in S are adjacentto variables in X
bull NoteThe same factor would result regardless of theelimination order
x
y
A BSfX(S)
fY(S)
33
Recursive Elimination in Recursive Elimination in Junction TreesJunction Trees
bull How do we compute fX(S)
bull By recursive decomposition alongcluster tree
ndash Let X1 and X2 be the disjoint partitioning of X C implied by theseparators S1 and S2
ndash Eliminate X1 to get fX1(S1)
ndash Eliminate X2 to get fX2(S2)
ndash Eliminate variables in C S toget fX(S)
C
S
S2S1
x1
x2
y
- Slide 6
- Slide 7
- Slide 8
- Slide 9
- Slide 10
- Slide 11
- Slide 12
- Slide 13
- Slide 23
- Slide 25
- Slide 28
-
13
)|( )( )( badPafbaf xl
)|( )|( )( )( )( badPltaPaflbftf xsv
)|( )|( )|( )|( )|( )|( )( )( badPaxPltaPsbPslPvtPsPvP
Eliminate variables in order
Initial factors
V S
LT
A B
X D
baltxsv
)|( )|( )|( )|( )|( )( )( badPaxPltaPsbPslPsPtfv)|( )|( )|( )( )( badPaxPltaPlbftf sv
)|( )( )( )( badPlafaflbf txs
)( dbfa )()()( dfdbfdf bb
ab
14
Intermediate factorsIntermediate factors
baltxsv
ga (ltdbx)
gb (ltdxs)
gx (ltds)
gt (ltsv)
gv (lds)
gs(ld)
gl (d))()()()(
)()(
)(
dfdbfbaflaf
aflbf
tf
b
a
l
t
x
s
v
lsvtxba
In our previous example With a different ordering
V S
LT
A B
X D
Complexity is exponential in the size of these factors
15
Notes about variable Notes about variable eliminationelimination
bull Actual computation is done in the elimination steps
bull Computation depends on the order of elimination
bull For each query we need to compute everything againndash Many redundant calculations
16
The ideaThe idea
bull Compute joint over partitions of U
ndash 1048708 small subset of U (typically made of a variable and its parents) - clusters
ndash not necessary disjoint
ndash Calculate
ndash To compute P(X) need far less operations
iC
iiCU
XC
i
i
CPXP
)()(
iC)( iCP
17
Junction TreesJunction Trees
bull The junction tree algorithms generalize Variable Elimination to the efficient simultaneous execution of a large class of queries
bull Theoretical background was shown in the previous lecture
18
Constructing Junction TreesConstructing Junction Trees
1 Moralize the graph (if directed)
2 Choose a node ordering and find the cliques generated by variable elimination This gives a triangulation of the graph
3 Build a junction graph from the eliminated cliques
4 Find an appropriate spanning tree
19
Step 1 MoralizationStep 1 Moralization
a
b c
d e
f
g
h
a
b c
d e
f
g
h
a
b c
d e
f
g
h
1 For all w Vbull For all uvpa(w) add an edge e=u-v
2 Undirect all edges
GMG = ( V E )
20
Step 2 TriangulationStep 2 Triangulation
Add edges to GM such that there is no cyclewith length 4 that does not contain a chord
NO YES
a
b c
d e
f
g
h
a
b c
d e
f
g
h
GM GT
21
Step 2 Triangulation (cont)Step 2 Triangulation (cont)
bull Each elimination ordering triangulates the graph not necessarily in the same way
A
H
B
D
F
C
E
G
A
H
B
D
F
C
E
G
A
H
B
D
F
C
E
G
A
H
B
D
F
C
E
G
A
H
B
D
F
C
E
G
A
H
B
D
F
C
E
G
A
H
B
D
F
C
E
G
A
H
B
D
F
C
E
G
A
H
B
D
F
C
E
G
22
Step 2 Triangulation (cont)Step 2 Triangulation (cont)
bull Intuitively triangulations with as few fill-ins as possible are preferredndash Leaves us with small cliques (small probability tables)
bull A common heuristic Repeat until no nodes remain
ndash Find the node whose elimination would require the least number of fill-ins (may be zero)
ndash Eliminate that node and note the need for a fill-in edge between any two non-adjacent neighbors
bull Add the fill-in edges to the original graph
23
a
b c
d e
f
g
h
a
b c
d e
f
g
h
a
b c
d e
f
g
a
b c
d e
f
a
b c
d e
a
b
d e
aa
e
a
d e
vertex induced addedremoved clique edges
1 h egh -2 g ceg -3 f def -4 c ace a-e
vertex induced added removed clique edges
5 b abd a-d6 d ade -7 e ae -8 a a -
GT
GM
Eliminate the vertex that requires least number of edges to be added
24
Step 3 Junction GraphStep 3 Junction Graph
bull A junction graph for an undirected graph G is an undirected labeled graph
bull The nodes are the cliques in G
bull If two cliques intersect they are joined in the junction graph by an edge labeled with their intersection
25
a
b
d
a
c
e
d e
f
a
d e
e
g
h
c
e
g
a
b c
d e
f
g
h
Bayesian NetworkG = ( V E )
a
b c
d e
f
g
h
a
b c
d e
f
g
h
Moral graph GM Triangulated graph GT
abd
ade
ace
ceg
eghdef
ad ae ce
de eg
seperators
Junction graph GJ (not complete)eg ceg egh = eg Cliques
e
e
e
a
e
26
Step 4 Junction TreeStep 4 Junction Tree
bull A junction tree is a sub-graph of the junction graph that ndash Is a tree
ndash Contains all the cliques (spanning tree)
ndash Satisfies the running intersection property
for each pair of nodes U V all nodes on the path between U and V contain
(as seen in the previous part of the lecture)
VU
27
Step 4 Junction Tree (cont)Step 4 Junction Tree (cont)
bull Theorem An undirected graph is triangulated if and only if its junction graph has a junction tree
bull Definition The weight of a link in a junction graph is the number of variable in the label The weight of a junction tree is the sum of weights of the labels
bull Theorem A sub-tree of the junction graph of a triangulated graph is a junction tree if and only if it is a spanning of maximal weight
28
Junction graph GJ (not complete)
abd
ade
ace
ceg
eghdef
ad ae ce
de eg
e
e
e
a
e
abd
ade
ace
ceg
eghdef
ad ae ce
de eg
Junction tree GJT
There are several methods to find MST
Kruskalrsquos algorithm choose successively a link of
maximal weight unless it creates a cycle
29
Another exampleAnother example
bull Compute the elimination cliques(the order here is f d e c b a)
bull Form the complete junction graph over the maximal elimination cliques and find a maximum-weight spanning tree
30
Junction Trees and Elimination Junction Trees and Elimination OrderOrder
bull We can use different orderings in variable elimination - affects efficiency
bull Each ordering corresponds to a junction tree
bull Just as some elimination orderings are more efficient than others some junction trees are better than others (Recall our mention of heuristics for triangulation)
31
OK I have this tree now whatOK I have this tree now what
bull A separator S divides the remaining variables into two groupsndash Variables in each group appear
on one side in the cluster tree
bull Examplesndash AB L S T V amp D X
ndash AL T V amp BDSX
ndash BL S amp A DT V X
ndash A X amp BDL S T V
ndash T V amp A B D K S X
LT
A B
X
V S
D
TV
ALTBLS
XAALB
ABD
AAB
BL
T
AL
32
Elimination in Junction TreesElimination in Junction Trees
bull Let X and Y be the partition induced by S
Observation
bull Eliminating all variables in X results in a factor fX(S)
bull Proof Since S is a separator only variables in S are adjacentto variables in X
bull NoteThe same factor would result regardless of theelimination order
x
y
A BSfX(S)
fY(S)
33
Recursive Elimination in Recursive Elimination in Junction TreesJunction Trees
bull How do we compute fX(S)
bull By recursive decomposition alongcluster tree
ndash Let X1 and X2 be the disjoint partitioning of X C implied by theseparators S1 and S2
ndash Eliminate X1 to get fX1(S1)
ndash Eliminate X2 to get fX2(S2)
ndash Eliminate variables in C S toget fX(S)
C
S
S2S1
x1
x2
y
- Slide 6
- Slide 7
- Slide 8
- Slide 9
- Slide 10
- Slide 11
- Slide 12
- Slide 13
- Slide 23
- Slide 25
- Slide 28
-
14
Intermediate factorsIntermediate factors
baltxsv
ga (ltdbx)
gb (ltdxs)
gx (ltds)
gt (ltsv)
gv (lds)
gs(ld)
gl (d))()()()(
)()(
)(
dfdbfbaflaf
aflbf
tf
b
a
l
t
x
s
v
lsvtxba
In our previous example With a different ordering
V S
LT
A B
X D
Complexity is exponential in the size of these factors
15
Notes about variable Notes about variable eliminationelimination
bull Actual computation is done in the elimination steps
bull Computation depends on the order of elimination
bull For each query we need to compute everything againndash Many redundant calculations
16
The ideaThe idea
bull Compute joint over partitions of U
ndash 1048708 small subset of U (typically made of a variable and its parents) - clusters
ndash not necessary disjoint
ndash Calculate
ndash To compute P(X) need far less operations
iC
iiCU
XC
i
i
CPXP
)()(
iC)( iCP
17
Junction TreesJunction Trees
bull The junction tree algorithms generalize Variable Elimination to the efficient simultaneous execution of a large class of queries
bull Theoretical background was shown in the previous lecture
18
Constructing Junction TreesConstructing Junction Trees
1 Moralize the graph (if directed)
2 Choose a node ordering and find the cliques generated by variable elimination This gives a triangulation of the graph
3 Build a junction graph from the eliminated cliques
4 Find an appropriate spanning tree
19
Step 1 MoralizationStep 1 Moralization
a
b c
d e
f
g
h
a
b c
d e
f
g
h
a
b c
d e
f
g
h
1 For all w Vbull For all uvpa(w) add an edge e=u-v
2 Undirect all edges
GMG = ( V E )
20
Step 2 TriangulationStep 2 Triangulation
Add edges to GM such that there is no cyclewith length 4 that does not contain a chord
NO YES
a
b c
d e
f
g
h
a
b c
d e
f
g
h
GM GT
21
Step 2 Triangulation (cont)Step 2 Triangulation (cont)
bull Each elimination ordering triangulates the graph not necessarily in the same way
A
H
B
D
F
C
E
G
A
H
B
D
F
C
E
G
A
H
B
D
F
C
E
G
A
H
B
D
F
C
E
G
A
H
B
D
F
C
E
G
A
H
B
D
F
C
E
G
A
H
B
D
F
C
E
G
A
H
B
D
F
C
E
G
A
H
B
D
F
C
E
G
22
Step 2 Triangulation (cont)Step 2 Triangulation (cont)
bull Intuitively triangulations with as few fill-ins as possible are preferredndash Leaves us with small cliques (small probability tables)
bull A common heuristic Repeat until no nodes remain
ndash Find the node whose elimination would require the least number of fill-ins (may be zero)
ndash Eliminate that node and note the need for a fill-in edge between any two non-adjacent neighbors
bull Add the fill-in edges to the original graph
23
a
b c
d e
f
g
h
a
b c
d e
f
g
h
a
b c
d e
f
g
a
b c
d e
f
a
b c
d e
a
b
d e
aa
e
a
d e
vertex induced addedremoved clique edges
1 h egh -2 g ceg -3 f def -4 c ace a-e
vertex induced added removed clique edges
5 b abd a-d6 d ade -7 e ae -8 a a -
GT
GM
Eliminate the vertex that requires least number of edges to be added
24
Step 3 Junction GraphStep 3 Junction Graph
bull A junction graph for an undirected graph G is an undirected labeled graph
bull The nodes are the cliques in G
bull If two cliques intersect they are joined in the junction graph by an edge labeled with their intersection
25
a
b
d
a
c
e
d e
f
a
d e
e
g
h
c
e
g
a
b c
d e
f
g
h
Bayesian NetworkG = ( V E )
a
b c
d e
f
g
h
a
b c
d e
f
g
h
Moral graph GM Triangulated graph GT
abd
ade
ace
ceg
eghdef
ad ae ce
de eg
seperators
Junction graph GJ (not complete)eg ceg egh = eg Cliques
e
e
e
a
e
26
Step 4 Junction TreeStep 4 Junction Tree
bull A junction tree is a sub-graph of the junction graph that ndash Is a tree
ndash Contains all the cliques (spanning tree)
ndash Satisfies the running intersection property
for each pair of nodes U V all nodes on the path between U and V contain
(as seen in the previous part of the lecture)
VU
27
Step 4 Junction Tree (cont)Step 4 Junction Tree (cont)
bull Theorem An undirected graph is triangulated if and only if its junction graph has a junction tree
bull Definition The weight of a link in a junction graph is the number of variable in the label The weight of a junction tree is the sum of weights of the labels
bull Theorem A sub-tree of the junction graph of a triangulated graph is a junction tree if and only if it is a spanning of maximal weight
28
Junction graph GJ (not complete)
abd
ade
ace
ceg
eghdef
ad ae ce
de eg
e
e
e
a
e
abd
ade
ace
ceg
eghdef
ad ae ce
de eg
Junction tree GJT
There are several methods to find MST
Kruskalrsquos algorithm choose successively a link of
maximal weight unless it creates a cycle
29
Another exampleAnother example
bull Compute the elimination cliques(the order here is f d e c b a)
bull Form the complete junction graph over the maximal elimination cliques and find a maximum-weight spanning tree
30
Junction Trees and Elimination Junction Trees and Elimination OrderOrder
bull We can use different orderings in variable elimination - affects efficiency
bull Each ordering corresponds to a junction tree
bull Just as some elimination orderings are more efficient than others some junction trees are better than others (Recall our mention of heuristics for triangulation)
31
OK I have this tree now whatOK I have this tree now what
bull A separator S divides the remaining variables into two groupsndash Variables in each group appear
on one side in the cluster tree
bull Examplesndash AB L S T V amp D X
ndash AL T V amp BDSX
ndash BL S amp A DT V X
ndash A X amp BDL S T V
ndash T V amp A B D K S X
LT
A B
X
V S
D
TV
ALTBLS
XAALB
ABD
AAB
BL
T
AL
32
Elimination in Junction TreesElimination in Junction Trees
bull Let X and Y be the partition induced by S
Observation
bull Eliminating all variables in X results in a factor fX(S)
bull Proof Since S is a separator only variables in S are adjacentto variables in X
bull NoteThe same factor would result regardless of theelimination order
x
y
A BSfX(S)
fY(S)
33
Recursive Elimination in Recursive Elimination in Junction TreesJunction Trees
bull How do we compute fX(S)
bull By recursive decomposition alongcluster tree
ndash Let X1 and X2 be the disjoint partitioning of X C implied by theseparators S1 and S2
ndash Eliminate X1 to get fX1(S1)
ndash Eliminate X2 to get fX2(S2)
ndash Eliminate variables in C S toget fX(S)
C
S
S2S1
x1
x2
y
- Slide 6
- Slide 7
- Slide 8
- Slide 9
- Slide 10
- Slide 11
- Slide 12
- Slide 13
- Slide 23
- Slide 25
- Slide 28
-
15
Notes about variable Notes about variable eliminationelimination
bull Actual computation is done in the elimination steps
bull Computation depends on the order of elimination
bull For each query we need to compute everything againndash Many redundant calculations
16
The ideaThe idea
bull Compute joint over partitions of U
ndash 1048708 small subset of U (typically made of a variable and its parents) - clusters
ndash not necessary disjoint
ndash Calculate
ndash To compute P(X) need far less operations
iC
iiCU
XC
i
i
CPXP
)()(
iC)( iCP
17
Junction TreesJunction Trees
bull The junction tree algorithms generalize Variable Elimination to the efficient simultaneous execution of a large class of queries
bull Theoretical background was shown in the previous lecture
18
Constructing Junction TreesConstructing Junction Trees
1 Moralize the graph (if directed)
2 Choose a node ordering and find the cliques generated by variable elimination This gives a triangulation of the graph
3 Build a junction graph from the eliminated cliques
4 Find an appropriate spanning tree
19
Step 1 MoralizationStep 1 Moralization
a
b c
d e
f
g
h
a
b c
d e
f
g
h
a
b c
d e
f
g
h
1 For all w Vbull For all uvpa(w) add an edge e=u-v
2 Undirect all edges
GMG = ( V E )
20
Step 2 TriangulationStep 2 Triangulation
Add edges to GM such that there is no cyclewith length 4 that does not contain a chord
NO YES
a
b c
d e
f
g
h
a
b c
d e
f
g
h
GM GT
21
Step 2 Triangulation (cont)Step 2 Triangulation (cont)
bull Each elimination ordering triangulates the graph not necessarily in the same way
A
H
B
D
F
C
E
G
A
H
B
D
F
C
E
G
A
H
B
D
F
C
E
G
A
H
B
D
F
C
E
G
A
H
B
D
F
C
E
G
A
H
B
D
F
C
E
G
A
H
B
D
F
C
E
G
A
H
B
D
F
C
E
G
A
H
B
D
F
C
E
G
22
Step 2 Triangulation (cont)Step 2 Triangulation (cont)
bull Intuitively triangulations with as few fill-ins as possible are preferredndash Leaves us with small cliques (small probability tables)
bull A common heuristic Repeat until no nodes remain
ndash Find the node whose elimination would require the least number of fill-ins (may be zero)
ndash Eliminate that node and note the need for a fill-in edge between any two non-adjacent neighbors
bull Add the fill-in edges to the original graph
23
a
b c
d e
f
g
h
a
b c
d e
f
g
h
a
b c
d e
f
g
a
b c
d e
f
a
b c
d e
a
b
d e
aa
e
a
d e
vertex induced addedremoved clique edges
1 h egh -2 g ceg -3 f def -4 c ace a-e
vertex induced added removed clique edges
5 b abd a-d6 d ade -7 e ae -8 a a -
GT
GM
Eliminate the vertex that requires least number of edges to be added
24
Step 3 Junction GraphStep 3 Junction Graph
bull A junction graph for an undirected graph G is an undirected labeled graph
bull The nodes are the cliques in G
bull If two cliques intersect they are joined in the junction graph by an edge labeled with their intersection
25
a
b
d
a
c
e
d e
f
a
d e
e
g
h
c
e
g
a
b c
d e
f
g
h
Bayesian NetworkG = ( V E )
a
b c
d e
f
g
h
a
b c
d e
f
g
h
Moral graph GM Triangulated graph GT
abd
ade
ace
ceg
eghdef
ad ae ce
de eg
seperators
Junction graph GJ (not complete)eg ceg egh = eg Cliques
e
e
e
a
e
26
Step 4 Junction TreeStep 4 Junction Tree
bull A junction tree is a sub-graph of the junction graph that ndash Is a tree
ndash Contains all the cliques (spanning tree)
ndash Satisfies the running intersection property
for each pair of nodes U V all nodes on the path between U and V contain
(as seen in the previous part of the lecture)
VU
27
Step 4 Junction Tree (cont)Step 4 Junction Tree (cont)
bull Theorem An undirected graph is triangulated if and only if its junction graph has a junction tree
bull Definition The weight of a link in a junction graph is the number of variable in the label The weight of a junction tree is the sum of weights of the labels
bull Theorem A sub-tree of the junction graph of a triangulated graph is a junction tree if and only if it is a spanning of maximal weight
28
Junction graph GJ (not complete)
abd
ade
ace
ceg
eghdef
ad ae ce
de eg
e
e
e
a
e
abd
ade
ace
ceg
eghdef
ad ae ce
de eg
Junction tree GJT
There are several methods to find MST
Kruskalrsquos algorithm choose successively a link of
maximal weight unless it creates a cycle
29
Another exampleAnother example
bull Compute the elimination cliques(the order here is f d e c b a)
bull Form the complete junction graph over the maximal elimination cliques and find a maximum-weight spanning tree
30
Junction Trees and Elimination Junction Trees and Elimination OrderOrder
bull We can use different orderings in variable elimination - affects efficiency
bull Each ordering corresponds to a junction tree
bull Just as some elimination orderings are more efficient than others some junction trees are better than others (Recall our mention of heuristics for triangulation)
31
OK I have this tree now whatOK I have this tree now what
bull A separator S divides the remaining variables into two groupsndash Variables in each group appear
on one side in the cluster tree
bull Examplesndash AB L S T V amp D X
ndash AL T V amp BDSX
ndash BL S amp A DT V X
ndash A X amp BDL S T V
ndash T V amp A B D K S X
LT
A B
X
V S
D
TV
ALTBLS
XAALB
ABD
AAB
BL
T
AL
32
Elimination in Junction TreesElimination in Junction Trees
bull Let X and Y be the partition induced by S
Observation
bull Eliminating all variables in X results in a factor fX(S)
bull Proof Since S is a separator only variables in S are adjacentto variables in X
bull NoteThe same factor would result regardless of theelimination order
x
y
A BSfX(S)
fY(S)
33
Recursive Elimination in Recursive Elimination in Junction TreesJunction Trees
bull How do we compute fX(S)
bull By recursive decomposition alongcluster tree
ndash Let X1 and X2 be the disjoint partitioning of X C implied by theseparators S1 and S2
ndash Eliminate X1 to get fX1(S1)
ndash Eliminate X2 to get fX2(S2)
ndash Eliminate variables in C S toget fX(S)
C
S
S2S1
x1
x2
y
- Slide 6
- Slide 7
- Slide 8
- Slide 9
- Slide 10
- Slide 11
- Slide 12
- Slide 13
- Slide 23
- Slide 25
- Slide 28
-
16
The ideaThe idea
bull Compute joint over partitions of U
ndash 1048708 small subset of U (typically made of a variable and its parents) - clusters
ndash not necessary disjoint
ndash Calculate
ndash To compute P(X) need far less operations
iC
iiCU
XC
i
i
CPXP
)()(
iC)( iCP
17
Junction TreesJunction Trees
bull The junction tree algorithms generalize Variable Elimination to the efficient simultaneous execution of a large class of queries
bull Theoretical background was shown in the previous lecture
18
Constructing Junction TreesConstructing Junction Trees
1 Moralize the graph (if directed)
2 Choose a node ordering and find the cliques generated by variable elimination This gives a triangulation of the graph
3 Build a junction graph from the eliminated cliques
4 Find an appropriate spanning tree
19
Step 1 MoralizationStep 1 Moralization
a
b c
d e
f
g
h
a
b c
d e
f
g
h
a
b c
d e
f
g
h
1 For all w Vbull For all uvpa(w) add an edge e=u-v
2 Undirect all edges
GMG = ( V E )
20
Step 2 TriangulationStep 2 Triangulation
Add edges to GM such that there is no cyclewith length 4 that does not contain a chord
NO YES
a
b c
d e
f
g
h
a
b c
d e
f
g
h
GM GT
21
Step 2 Triangulation (cont)Step 2 Triangulation (cont)
bull Each elimination ordering triangulates the graph not necessarily in the same way
A
H
B
D
F
C
E
G
A
H
B
D
F
C
E
G
A
H
B
D
F
C
E
G
A
H
B
D
F
C
E
G
A
H
B
D
F
C
E
G
A
H
B
D
F
C
E
G
A
H
B
D
F
C
E
G
A
H
B
D
F
C
E
G
A
H
B
D
F
C
E
G
22
Step 2 Triangulation (cont)Step 2 Triangulation (cont)
bull Intuitively triangulations with as few fill-ins as possible are preferredndash Leaves us with small cliques (small probability tables)
bull A common heuristic Repeat until no nodes remain
ndash Find the node whose elimination would require the least number of fill-ins (may be zero)
ndash Eliminate that node and note the need for a fill-in edge between any two non-adjacent neighbors
bull Add the fill-in edges to the original graph
23
a
b c
d e
f
g
h
a
b c
d e
f
g
h
a
b c
d e
f
g
a
b c
d e
f
a
b c
d e
a
b
d e
aa
e
a
d e
vertex induced addedremoved clique edges
1 h egh -2 g ceg -3 f def -4 c ace a-e
vertex induced added removed clique edges
5 b abd a-d6 d ade -7 e ae -8 a a -
GT
GM
Eliminate the vertex that requires least number of edges to be added
24
Step 3 Junction GraphStep 3 Junction Graph
bull A junction graph for an undirected graph G is an undirected labeled graph
bull The nodes are the cliques in G
bull If two cliques intersect they are joined in the junction graph by an edge labeled with their intersection
25
a
b
d
a
c
e
d e
f
a
d e
e
g
h
c
e
g
a
b c
d e
f
g
h
Bayesian NetworkG = ( V E )
a
b c
d e
f
g
h
a
b c
d e
f
g
h
Moral graph GM Triangulated graph GT
abd
ade
ace
ceg
eghdef
ad ae ce
de eg
seperators
Junction graph GJ (not complete)eg ceg egh = eg Cliques
e
e
e
a
e
26
Step 4 Junction TreeStep 4 Junction Tree
bull A junction tree is a sub-graph of the junction graph that ndash Is a tree
ndash Contains all the cliques (spanning tree)
ndash Satisfies the running intersection property
for each pair of nodes U V all nodes on the path between U and V contain
(as seen in the previous part of the lecture)
VU
27
Step 4 Junction Tree (cont)Step 4 Junction Tree (cont)
bull Theorem An undirected graph is triangulated if and only if its junction graph has a junction tree
bull Definition The weight of a link in a junction graph is the number of variable in the label The weight of a junction tree is the sum of weights of the labels
bull Theorem A sub-tree of the junction graph of a triangulated graph is a junction tree if and only if it is a spanning of maximal weight
28
Junction graph GJ (not complete)
abd
ade
ace
ceg
eghdef
ad ae ce
de eg
e
e
e
a
e
abd
ade
ace
ceg
eghdef
ad ae ce
de eg
Junction tree GJT
There are several methods to find MST
Kruskalrsquos algorithm choose successively a link of
maximal weight unless it creates a cycle
29
Another exampleAnother example
bull Compute the elimination cliques(the order here is f d e c b a)
bull Form the complete junction graph over the maximal elimination cliques and find a maximum-weight spanning tree
30
Junction Trees and Elimination Junction Trees and Elimination OrderOrder
bull We can use different orderings in variable elimination - affects efficiency
bull Each ordering corresponds to a junction tree
bull Just as some elimination orderings are more efficient than others some junction trees are better than others (Recall our mention of heuristics for triangulation)
31
OK I have this tree now whatOK I have this tree now what
bull A separator S divides the remaining variables into two groupsndash Variables in each group appear
on one side in the cluster tree
bull Examplesndash AB L S T V amp D X
ndash AL T V amp BDSX
ndash BL S amp A DT V X
ndash A X amp BDL S T V
ndash T V amp A B D K S X
LT
A B
X
V S
D
TV
ALTBLS
XAALB
ABD
AAB
BL
T
AL
32
Elimination in Junction TreesElimination in Junction Trees
bull Let X and Y be the partition induced by S
Observation
bull Eliminating all variables in X results in a factor fX(S)
bull Proof Since S is a separator only variables in S are adjacentto variables in X
bull NoteThe same factor would result regardless of theelimination order
x
y
A BSfX(S)
fY(S)
33
Recursive Elimination in Recursive Elimination in Junction TreesJunction Trees
bull How do we compute fX(S)
bull By recursive decomposition alongcluster tree
ndash Let X1 and X2 be the disjoint partitioning of X C implied by theseparators S1 and S2
ndash Eliminate X1 to get fX1(S1)
ndash Eliminate X2 to get fX2(S2)
ndash Eliminate variables in C S toget fX(S)
C
S
S2S1
x1
x2
y
- Slide 6
- Slide 7
- Slide 8
- Slide 9
- Slide 10
- Slide 11
- Slide 12
- Slide 13
- Slide 23
- Slide 25
- Slide 28
-
17
Junction TreesJunction Trees
bull The junction tree algorithms generalize Variable Elimination to the efficient simultaneous execution of a large class of queries
bull Theoretical background was shown in the previous lecture
18
Constructing Junction TreesConstructing Junction Trees
1 Moralize the graph (if directed)
2 Choose a node ordering and find the cliques generated by variable elimination This gives a triangulation of the graph
3 Build a junction graph from the eliminated cliques
4 Find an appropriate spanning tree
19
Step 1 MoralizationStep 1 Moralization
a
b c
d e
f
g
h
a
b c
d e
f
g
h
a
b c
d e
f
g
h
1 For all w Vbull For all uvpa(w) add an edge e=u-v
2 Undirect all edges
GMG = ( V E )
20
Step 2 TriangulationStep 2 Triangulation
Add edges to GM such that there is no cyclewith length 4 that does not contain a chord
NO YES
a
b c
d e
f
g
h
a
b c
d e
f
g
h
GM GT
21
Step 2 Triangulation (cont)Step 2 Triangulation (cont)
bull Each elimination ordering triangulates the graph not necessarily in the same way
A
H
B
D
F
C
E
G
A
H
B
D
F
C
E
G
A
H
B
D
F
C
E
G
A
H
B
D
F
C
E
G
A
H
B
D
F
C
E
G
A
H
B
D
F
C
E
G
A
H
B
D
F
C
E
G
A
H
B
D
F
C
E
G
A
H
B
D
F
C
E
G
22
Step 2 Triangulation (cont)Step 2 Triangulation (cont)
bull Intuitively triangulations with as few fill-ins as possible are preferredndash Leaves us with small cliques (small probability tables)
bull A common heuristic Repeat until no nodes remain
ndash Find the node whose elimination would require the least number of fill-ins (may be zero)
ndash Eliminate that node and note the need for a fill-in edge between any two non-adjacent neighbors
bull Add the fill-in edges to the original graph
23
a
b c
d e
f
g
h
a
b c
d e
f
g
h
a
b c
d e
f
g
a
b c
d e
f
a
b c
d e
a
b
d e
aa
e
a
d e
vertex induced addedremoved clique edges
1 h egh -2 g ceg -3 f def -4 c ace a-e
vertex induced added removed clique edges
5 b abd a-d6 d ade -7 e ae -8 a a -
GT
GM
Eliminate the vertex that requires least number of edges to be added
24
Step 3 Junction GraphStep 3 Junction Graph
bull A junction graph for an undirected graph G is an undirected labeled graph
bull The nodes are the cliques in G
bull If two cliques intersect they are joined in the junction graph by an edge labeled with their intersection
25
a
b
d
a
c
e
d e
f
a
d e
e
g
h
c
e
g
a
b c
d e
f
g
h
Bayesian NetworkG = ( V E )
a
b c
d e
f
g
h
a
b c
d e
f
g
h
Moral graph GM Triangulated graph GT
abd
ade
ace
ceg
eghdef
ad ae ce
de eg
seperators
Junction graph GJ (not complete)eg ceg egh = eg Cliques
e
e
e
a
e
26
Step 4 Junction TreeStep 4 Junction Tree
bull A junction tree is a sub-graph of the junction graph that ndash Is a tree
ndash Contains all the cliques (spanning tree)
ndash Satisfies the running intersection property
for each pair of nodes U V all nodes on the path between U and V contain
(as seen in the previous part of the lecture)
VU
27
Step 4 Junction Tree (cont)Step 4 Junction Tree (cont)
bull Theorem An undirected graph is triangulated if and only if its junction graph has a junction tree
bull Definition The weight of a link in a junction graph is the number of variable in the label The weight of a junction tree is the sum of weights of the labels
bull Theorem A sub-tree of the junction graph of a triangulated graph is a junction tree if and only if it is a spanning of maximal weight
28
Junction graph GJ (not complete)
abd
ade
ace
ceg
eghdef
ad ae ce
de eg
e
e
e
a
e
abd
ade
ace
ceg
eghdef
ad ae ce
de eg
Junction tree GJT
There are several methods to find MST
Kruskalrsquos algorithm choose successively a link of
maximal weight unless it creates a cycle
29
Another exampleAnother example
bull Compute the elimination cliques(the order here is f d e c b a)
bull Form the complete junction graph over the maximal elimination cliques and find a maximum-weight spanning tree
30
Junction Trees and Elimination Junction Trees and Elimination OrderOrder
bull We can use different orderings in variable elimination - affects efficiency
bull Each ordering corresponds to a junction tree
bull Just as some elimination orderings are more efficient than others some junction trees are better than others (Recall our mention of heuristics for triangulation)
31
OK I have this tree now whatOK I have this tree now what
bull A separator S divides the remaining variables into two groupsndash Variables in each group appear
on one side in the cluster tree
bull Examplesndash AB L S T V amp D X
ndash AL T V amp BDSX
ndash BL S amp A DT V X
ndash A X amp BDL S T V
ndash T V amp A B D K S X
LT
A B
X
V S
D
TV
ALTBLS
XAALB
ABD
AAB
BL
T
AL
32
Elimination in Junction TreesElimination in Junction Trees
bull Let X and Y be the partition induced by S
Observation
bull Eliminating all variables in X results in a factor fX(S)
bull Proof Since S is a separator only variables in S are adjacentto variables in X
bull NoteThe same factor would result regardless of theelimination order
x
y
A BSfX(S)
fY(S)
33
Recursive Elimination in Recursive Elimination in Junction TreesJunction Trees
bull How do we compute fX(S)
bull By recursive decomposition alongcluster tree
ndash Let X1 and X2 be the disjoint partitioning of X C implied by theseparators S1 and S2
ndash Eliminate X1 to get fX1(S1)
ndash Eliminate X2 to get fX2(S2)
ndash Eliminate variables in C S toget fX(S)
C
S
S2S1
x1
x2
y
- Slide 6
- Slide 7
- Slide 8
- Slide 9
- Slide 10
- Slide 11
- Slide 12
- Slide 13
- Slide 23
- Slide 25
- Slide 28
-
18
Constructing Junction TreesConstructing Junction Trees
1 Moralize the graph (if directed)
2 Choose a node ordering and find the cliques generated by variable elimination This gives a triangulation of the graph
3 Build a junction graph from the eliminated cliques
4 Find an appropriate spanning tree
19
Step 1 MoralizationStep 1 Moralization
a
b c
d e
f
g
h
a
b c
d e
f
g
h
a
b c
d e
f
g
h
1 For all w Vbull For all uvpa(w) add an edge e=u-v
2 Undirect all edges
GMG = ( V E )
20
Step 2 TriangulationStep 2 Triangulation
Add edges to GM such that there is no cyclewith length 4 that does not contain a chord
NO YES
a
b c
d e
f
g
h
a
b c
d e
f
g
h
GM GT
21
Step 2 Triangulation (cont)Step 2 Triangulation (cont)
bull Each elimination ordering triangulates the graph not necessarily in the same way
A
H
B
D
F
C
E
G
A
H
B
D
F
C
E
G
A
H
B
D
F
C
E
G
A
H
B
D
F
C
E
G
A
H
B
D
F
C
E
G
A
H
B
D
F
C
E
G
A
H
B
D
F
C
E
G
A
H
B
D
F
C
E
G
A
H
B
D
F
C
E
G
22
Step 2 Triangulation (cont)Step 2 Triangulation (cont)
bull Intuitively triangulations with as few fill-ins as possible are preferredndash Leaves us with small cliques (small probability tables)
bull A common heuristic Repeat until no nodes remain
ndash Find the node whose elimination would require the least number of fill-ins (may be zero)
ndash Eliminate that node and note the need for a fill-in edge between any two non-adjacent neighbors
bull Add the fill-in edges to the original graph
23
a
b c
d e
f
g
h
a
b c
d e
f
g
h
a
b c
d e
f
g
a
b c
d e
f
a
b c
d e
a
b
d e
aa
e
a
d e
vertex induced addedremoved clique edges
1 h egh -2 g ceg -3 f def -4 c ace a-e
vertex induced added removed clique edges
5 b abd a-d6 d ade -7 e ae -8 a a -
GT
GM
Eliminate the vertex that requires least number of edges to be added
24
Step 3 Junction GraphStep 3 Junction Graph
bull A junction graph for an undirected graph G is an undirected labeled graph
bull The nodes are the cliques in G
bull If two cliques intersect they are joined in the junction graph by an edge labeled with their intersection
25
a
b
d
a
c
e
d e
f
a
d e
e
g
h
c
e
g
a
b c
d e
f
g
h
Bayesian NetworkG = ( V E )
a
b c
d e
f
g
h
a
b c
d e
f
g
h
Moral graph GM Triangulated graph GT
abd
ade
ace
ceg
eghdef
ad ae ce
de eg
seperators
Junction graph GJ (not complete)eg ceg egh = eg Cliques
e
e
e
a
e
26
Step 4 Junction TreeStep 4 Junction Tree
bull A junction tree is a sub-graph of the junction graph that ndash Is a tree
ndash Contains all the cliques (spanning tree)
ndash Satisfies the running intersection property
for each pair of nodes U V all nodes on the path between U and V contain
(as seen in the previous part of the lecture)
VU
27
Step 4 Junction Tree (cont)Step 4 Junction Tree (cont)
bull Theorem An undirected graph is triangulated if and only if its junction graph has a junction tree
bull Definition The weight of a link in a junction graph is the number of variable in the label The weight of a junction tree is the sum of weights of the labels
bull Theorem A sub-tree of the junction graph of a triangulated graph is a junction tree if and only if it is a spanning of maximal weight
28
Junction graph GJ (not complete)
abd
ade
ace
ceg
eghdef
ad ae ce
de eg
e
e
e
a
e
abd
ade
ace
ceg
eghdef
ad ae ce
de eg
Junction tree GJT
There are several methods to find MST
Kruskalrsquos algorithm choose successively a link of
maximal weight unless it creates a cycle
29
Another exampleAnother example
bull Compute the elimination cliques(the order here is f d e c b a)
bull Form the complete junction graph over the maximal elimination cliques and find a maximum-weight spanning tree
30
Junction Trees and Elimination Junction Trees and Elimination OrderOrder
bull We can use different orderings in variable elimination - affects efficiency
bull Each ordering corresponds to a junction tree
bull Just as some elimination orderings are more efficient than others some junction trees are better than others (Recall our mention of heuristics for triangulation)
31
OK I have this tree now whatOK I have this tree now what
bull A separator S divides the remaining variables into two groupsndash Variables in each group appear
on one side in the cluster tree
bull Examplesndash AB L S T V amp D X
ndash AL T V amp BDSX
ndash BL S amp A DT V X
ndash A X amp BDL S T V
ndash T V amp A B D K S X
LT
A B
X
V S
D
TV
ALTBLS
XAALB
ABD
AAB
BL
T
AL
32
Elimination in Junction TreesElimination in Junction Trees
bull Let X and Y be the partition induced by S
Observation
bull Eliminating all variables in X results in a factor fX(S)
bull Proof Since S is a separator only variables in S are adjacentto variables in X
bull NoteThe same factor would result regardless of theelimination order
x
y
A BSfX(S)
fY(S)
33
Recursive Elimination in Recursive Elimination in Junction TreesJunction Trees
bull How do we compute fX(S)
bull By recursive decomposition alongcluster tree
ndash Let X1 and X2 be the disjoint partitioning of X C implied by theseparators S1 and S2
ndash Eliminate X1 to get fX1(S1)
ndash Eliminate X2 to get fX2(S2)
ndash Eliminate variables in C S toget fX(S)
C
S
S2S1
x1
x2
y
- Slide 6
- Slide 7
- Slide 8
- Slide 9
- Slide 10
- Slide 11
- Slide 12
- Slide 13
- Slide 23
- Slide 25
- Slide 28
-
19
Step 1 MoralizationStep 1 Moralization
a
b c
d e
f
g
h
a
b c
d e
f
g
h
a
b c
d e
f
g
h
1 For all w Vbull For all uvpa(w) add an edge e=u-v
2 Undirect all edges
GMG = ( V E )
20
Step 2 TriangulationStep 2 Triangulation
Add edges to GM such that there is no cyclewith length 4 that does not contain a chord
NO YES
a
b c
d e
f
g
h
a
b c
d e
f
g
h
GM GT
21
Step 2 Triangulation (cont)Step 2 Triangulation (cont)
bull Each elimination ordering triangulates the graph not necessarily in the same way
A
H
B
D
F
C
E
G
A
H
B
D
F
C
E
G
A
H
B
D
F
C
E
G
A
H
B
D
F
C
E
G
A
H
B
D
F
C
E
G
A
H
B
D
F
C
E
G
A
H
B
D
F
C
E
G
A
H
B
D
F
C
E
G
A
H
B
D
F
C
E
G
22
Step 2 Triangulation (cont)Step 2 Triangulation (cont)
bull Intuitively triangulations with as few fill-ins as possible are preferredndash Leaves us with small cliques (small probability tables)
bull A common heuristic Repeat until no nodes remain
ndash Find the node whose elimination would require the least number of fill-ins (may be zero)
ndash Eliminate that node and note the need for a fill-in edge between any two non-adjacent neighbors
bull Add the fill-in edges to the original graph
23
a
b c
d e
f
g
h
a
b c
d e
f
g
h
a
b c
d e
f
g
a
b c
d e
f
a
b c
d e
a
b
d e
aa
e
a
d e
vertex induced addedremoved clique edges
1 h egh -2 g ceg -3 f def -4 c ace a-e
vertex induced added removed clique edges
5 b abd a-d6 d ade -7 e ae -8 a a -
GT
GM
Eliminate the vertex that requires least number of edges to be added
24
Step 3 Junction GraphStep 3 Junction Graph
bull A junction graph for an undirected graph G is an undirected labeled graph
bull The nodes are the cliques in G
bull If two cliques intersect they are joined in the junction graph by an edge labeled with their intersection
25
a
b
d
a
c
e
d e
f
a
d e
e
g
h
c
e
g
a
b c
d e
f
g
h
Bayesian NetworkG = ( V E )
a
b c
d e
f
g
h
a
b c
d e
f
g
h
Moral graph GM Triangulated graph GT
abd
ade
ace
ceg
eghdef
ad ae ce
de eg
seperators
Junction graph GJ (not complete)eg ceg egh = eg Cliques
e
e
e
a
e
26
Step 4 Junction TreeStep 4 Junction Tree
bull A junction tree is a sub-graph of the junction graph that ndash Is a tree
ndash Contains all the cliques (spanning tree)
ndash Satisfies the running intersection property
for each pair of nodes U V all nodes on the path between U and V contain
(as seen in the previous part of the lecture)
VU
27
Step 4 Junction Tree (cont)Step 4 Junction Tree (cont)
bull Theorem An undirected graph is triangulated if and only if its junction graph has a junction tree
bull Definition The weight of a link in a junction graph is the number of variable in the label The weight of a junction tree is the sum of weights of the labels
bull Theorem A sub-tree of the junction graph of a triangulated graph is a junction tree if and only if it is a spanning of maximal weight
28
Junction graph GJ (not complete)
abd
ade
ace
ceg
eghdef
ad ae ce
de eg
e
e
e
a
e
abd
ade
ace
ceg
eghdef
ad ae ce
de eg
Junction tree GJT
There are several methods to find MST
Kruskalrsquos algorithm choose successively a link of
maximal weight unless it creates a cycle
29
Another exampleAnother example
bull Compute the elimination cliques(the order here is f d e c b a)
bull Form the complete junction graph over the maximal elimination cliques and find a maximum-weight spanning tree
30
Junction Trees and Elimination Junction Trees and Elimination OrderOrder
bull We can use different orderings in variable elimination - affects efficiency
bull Each ordering corresponds to a junction tree
bull Just as some elimination orderings are more efficient than others some junction trees are better than others (Recall our mention of heuristics for triangulation)
31
OK I have this tree now whatOK I have this tree now what
bull A separator S divides the remaining variables into two groupsndash Variables in each group appear
on one side in the cluster tree
bull Examplesndash AB L S T V amp D X
ndash AL T V amp BDSX
ndash BL S amp A DT V X
ndash A X amp BDL S T V
ndash T V amp A B D K S X
LT
A B
X
V S
D
TV
ALTBLS
XAALB
ABD
AAB
BL
T
AL
32
Elimination in Junction TreesElimination in Junction Trees
bull Let X and Y be the partition induced by S
Observation
bull Eliminating all variables in X results in a factor fX(S)
bull Proof Since S is a separator only variables in S are adjacentto variables in X
bull NoteThe same factor would result regardless of theelimination order
x
y
A BSfX(S)
fY(S)
33
Recursive Elimination in Recursive Elimination in Junction TreesJunction Trees
bull How do we compute fX(S)
bull By recursive decomposition alongcluster tree
ndash Let X1 and X2 be the disjoint partitioning of X C implied by theseparators S1 and S2
ndash Eliminate X1 to get fX1(S1)
ndash Eliminate X2 to get fX2(S2)
ndash Eliminate variables in C S toget fX(S)
C
S
S2S1
x1
x2
y
- Slide 6
- Slide 7
- Slide 8
- Slide 9
- Slide 10
- Slide 11
- Slide 12
- Slide 13
- Slide 23
- Slide 25
- Slide 28
-
20
Step 2 TriangulationStep 2 Triangulation
Add edges to GM such that there is no cyclewith length 4 that does not contain a chord
NO YES
a
b c
d e
f
g
h
a
b c
d e
f
g
h
GM GT
21
Step 2 Triangulation (cont)Step 2 Triangulation (cont)
bull Each elimination ordering triangulates the graph not necessarily in the same way
A
H
B
D
F
C
E
G
A
H
B
D
F
C
E
G
A
H
B
D
F
C
E
G
A
H
B
D
F
C
E
G
A
H
B
D
F
C
E
G
A
H
B
D
F
C
E
G
A
H
B
D
F
C
E
G
A
H
B
D
F
C
E
G
A
H
B
D
F
C
E
G
22
Step 2 Triangulation (cont)Step 2 Triangulation (cont)
bull Intuitively triangulations with as few fill-ins as possible are preferredndash Leaves us with small cliques (small probability tables)
bull A common heuristic Repeat until no nodes remain
ndash Find the node whose elimination would require the least number of fill-ins (may be zero)
ndash Eliminate that node and note the need for a fill-in edge between any two non-adjacent neighbors
bull Add the fill-in edges to the original graph
23
a
b c
d e
f
g
h
a
b c
d e
f
g
h
a
b c
d e
f
g
a
b c
d e
f
a
b c
d e
a
b
d e
aa
e
a
d e
vertex induced addedremoved clique edges
1 h egh -2 g ceg -3 f def -4 c ace a-e
vertex induced added removed clique edges
5 b abd a-d6 d ade -7 e ae -8 a a -
GT
GM
Eliminate the vertex that requires least number of edges to be added
24
Step 3 Junction GraphStep 3 Junction Graph
bull A junction graph for an undirected graph G is an undirected labeled graph
bull The nodes are the cliques in G
bull If two cliques intersect they are joined in the junction graph by an edge labeled with their intersection
25
a
b
d
a
c
e
d e
f
a
d e
e
g
h
c
e
g
a
b c
d e
f
g
h
Bayesian NetworkG = ( V E )
a
b c
d e
f
g
h
a
b c
d e
f
g
h
Moral graph GM Triangulated graph GT
abd
ade
ace
ceg
eghdef
ad ae ce
de eg
seperators
Junction graph GJ (not complete)eg ceg egh = eg Cliques
e
e
e
a
e
26
Step 4 Junction TreeStep 4 Junction Tree
bull A junction tree is a sub-graph of the junction graph that ndash Is a tree
ndash Contains all the cliques (spanning tree)
ndash Satisfies the running intersection property
for each pair of nodes U V all nodes on the path between U and V contain
(as seen in the previous part of the lecture)
VU
27
Step 4 Junction Tree (cont)Step 4 Junction Tree (cont)
bull Theorem An undirected graph is triangulated if and only if its junction graph has a junction tree
bull Definition The weight of a link in a junction graph is the number of variable in the label The weight of a junction tree is the sum of weights of the labels
bull Theorem A sub-tree of the junction graph of a triangulated graph is a junction tree if and only if it is a spanning of maximal weight
28
Junction graph GJ (not complete)
abd
ade
ace
ceg
eghdef
ad ae ce
de eg
e
e
e
a
e
abd
ade
ace
ceg
eghdef
ad ae ce
de eg
Junction tree GJT
There are several methods to find MST
Kruskalrsquos algorithm choose successively a link of
maximal weight unless it creates a cycle
29
Another exampleAnother example
bull Compute the elimination cliques(the order here is f d e c b a)
bull Form the complete junction graph over the maximal elimination cliques and find a maximum-weight spanning tree
30
Junction Trees and Elimination Junction Trees and Elimination OrderOrder
bull We can use different orderings in variable elimination - affects efficiency
bull Each ordering corresponds to a junction tree
bull Just as some elimination orderings are more efficient than others some junction trees are better than others (Recall our mention of heuristics for triangulation)
31
OK I have this tree now whatOK I have this tree now what
bull A separator S divides the remaining variables into two groupsndash Variables in each group appear
on one side in the cluster tree
bull Examplesndash AB L S T V amp D X
ndash AL T V amp BDSX
ndash BL S amp A DT V X
ndash A X amp BDL S T V
ndash T V amp A B D K S X
LT
A B
X
V S
D
TV
ALTBLS
XAALB
ABD
AAB
BL
T
AL
32
Elimination in Junction TreesElimination in Junction Trees
bull Let X and Y be the partition induced by S
Observation
bull Eliminating all variables in X results in a factor fX(S)
bull Proof Since S is a separator only variables in S are adjacentto variables in X
bull NoteThe same factor would result regardless of theelimination order
x
y
A BSfX(S)
fY(S)
33
Recursive Elimination in Recursive Elimination in Junction TreesJunction Trees
bull How do we compute fX(S)
bull By recursive decomposition alongcluster tree
ndash Let X1 and X2 be the disjoint partitioning of X C implied by theseparators S1 and S2
ndash Eliminate X1 to get fX1(S1)
ndash Eliminate X2 to get fX2(S2)
ndash Eliminate variables in C S toget fX(S)
C
S
S2S1
x1
x2
y
- Slide 6
- Slide 7
- Slide 8
- Slide 9
- Slide 10
- Slide 11
- Slide 12
- Slide 13
- Slide 23
- Slide 25
- Slide 28
-
21
Step 2 Triangulation (cont)Step 2 Triangulation (cont)
bull Each elimination ordering triangulates the graph not necessarily in the same way
A
H
B
D
F
C
E
G
A
H
B
D
F
C
E
G
A
H
B
D
F
C
E
G
A
H
B
D
F
C
E
G
A
H
B
D
F
C
E
G
A
H
B
D
F
C
E
G
A
H
B
D
F
C
E
G
A
H
B
D
F
C
E
G
A
H
B
D
F
C
E
G
22
Step 2 Triangulation (cont)Step 2 Triangulation (cont)
bull Intuitively triangulations with as few fill-ins as possible are preferredndash Leaves us with small cliques (small probability tables)
bull A common heuristic Repeat until no nodes remain
ndash Find the node whose elimination would require the least number of fill-ins (may be zero)
ndash Eliminate that node and note the need for a fill-in edge between any two non-adjacent neighbors
bull Add the fill-in edges to the original graph
23
a
b c
d e
f
g
h
a
b c
d e
f
g
h
a
b c
d e
f
g
a
b c
d e
f
a
b c
d e
a
b
d e
aa
e
a
d e
vertex induced addedremoved clique edges
1 h egh -2 g ceg -3 f def -4 c ace a-e
vertex induced added removed clique edges
5 b abd a-d6 d ade -7 e ae -8 a a -
GT
GM
Eliminate the vertex that requires least number of edges to be added
24
Step 3 Junction GraphStep 3 Junction Graph
bull A junction graph for an undirected graph G is an undirected labeled graph
bull The nodes are the cliques in G
bull If two cliques intersect they are joined in the junction graph by an edge labeled with their intersection
25
a
b
d
a
c
e
d e
f
a
d e
e
g
h
c
e
g
a
b c
d e
f
g
h
Bayesian NetworkG = ( V E )
a
b c
d e
f
g
h
a
b c
d e
f
g
h
Moral graph GM Triangulated graph GT
abd
ade
ace
ceg
eghdef
ad ae ce
de eg
seperators
Junction graph GJ (not complete)eg ceg egh = eg Cliques
e
e
e
a
e
26
Step 4 Junction TreeStep 4 Junction Tree
bull A junction tree is a sub-graph of the junction graph that ndash Is a tree
ndash Contains all the cliques (spanning tree)
ndash Satisfies the running intersection property
for each pair of nodes U V all nodes on the path between U and V contain
(as seen in the previous part of the lecture)
VU
27
Step 4 Junction Tree (cont)Step 4 Junction Tree (cont)
bull Theorem An undirected graph is triangulated if and only if its junction graph has a junction tree
bull Definition The weight of a link in a junction graph is the number of variable in the label The weight of a junction tree is the sum of weights of the labels
bull Theorem A sub-tree of the junction graph of a triangulated graph is a junction tree if and only if it is a spanning of maximal weight
28
Junction graph GJ (not complete)
abd
ade
ace
ceg
eghdef
ad ae ce
de eg
e
e
e
a
e
abd
ade
ace
ceg
eghdef
ad ae ce
de eg
Junction tree GJT
There are several methods to find MST
Kruskalrsquos algorithm choose successively a link of
maximal weight unless it creates a cycle
29
Another exampleAnother example
bull Compute the elimination cliques(the order here is f d e c b a)
bull Form the complete junction graph over the maximal elimination cliques and find a maximum-weight spanning tree
30
Junction Trees and Elimination Junction Trees and Elimination OrderOrder
bull We can use different orderings in variable elimination - affects efficiency
bull Each ordering corresponds to a junction tree
bull Just as some elimination orderings are more efficient than others some junction trees are better than others (Recall our mention of heuristics for triangulation)
31
OK I have this tree now whatOK I have this tree now what
bull A separator S divides the remaining variables into two groupsndash Variables in each group appear
on one side in the cluster tree
bull Examplesndash AB L S T V amp D X
ndash AL T V amp BDSX
ndash BL S amp A DT V X
ndash A X amp BDL S T V
ndash T V amp A B D K S X
LT
A B
X
V S
D
TV
ALTBLS
XAALB
ABD
AAB
BL
T
AL
32
Elimination in Junction TreesElimination in Junction Trees
bull Let X and Y be the partition induced by S
Observation
bull Eliminating all variables in X results in a factor fX(S)
bull Proof Since S is a separator only variables in S are adjacentto variables in X
bull NoteThe same factor would result regardless of theelimination order
x
y
A BSfX(S)
fY(S)
33
Recursive Elimination in Recursive Elimination in Junction TreesJunction Trees
bull How do we compute fX(S)
bull By recursive decomposition alongcluster tree
ndash Let X1 and X2 be the disjoint partitioning of X C implied by theseparators S1 and S2
ndash Eliminate X1 to get fX1(S1)
ndash Eliminate X2 to get fX2(S2)
ndash Eliminate variables in C S toget fX(S)
C
S
S2S1
x1
x2
y
- Slide 6
- Slide 7
- Slide 8
- Slide 9
- Slide 10
- Slide 11
- Slide 12
- Slide 13
- Slide 23
- Slide 25
- Slide 28
-
22
Step 2 Triangulation (cont)Step 2 Triangulation (cont)
bull Intuitively triangulations with as few fill-ins as possible are preferredndash Leaves us with small cliques (small probability tables)
bull A common heuristic Repeat until no nodes remain
ndash Find the node whose elimination would require the least number of fill-ins (may be zero)
ndash Eliminate that node and note the need for a fill-in edge between any two non-adjacent neighbors
bull Add the fill-in edges to the original graph
23
a
b c
d e
f
g
h
a
b c
d e
f
g
h
a
b c
d e
f
g
a
b c
d e
f
a
b c
d e
a
b
d e
aa
e
a
d e
vertex induced addedremoved clique edges
1 h egh -2 g ceg -3 f def -4 c ace a-e
vertex induced added removed clique edges
5 b abd a-d6 d ade -7 e ae -8 a a -
GT
GM
Eliminate the vertex that requires least number of edges to be added
24
Step 3 Junction GraphStep 3 Junction Graph
bull A junction graph for an undirected graph G is an undirected labeled graph
bull The nodes are the cliques in G
bull If two cliques intersect they are joined in the junction graph by an edge labeled with their intersection
25
a
b
d
a
c
e
d e
f
a
d e
e
g
h
c
e
g
a
b c
d e
f
g
h
Bayesian NetworkG = ( V E )
a
b c
d e
f
g
h
a
b c
d e
f
g
h
Moral graph GM Triangulated graph GT
abd
ade
ace
ceg
eghdef
ad ae ce
de eg
seperators
Junction graph GJ (not complete)eg ceg egh = eg Cliques
e
e
e
a
e
26
Step 4 Junction TreeStep 4 Junction Tree
bull A junction tree is a sub-graph of the junction graph that ndash Is a tree
ndash Contains all the cliques (spanning tree)
ndash Satisfies the running intersection property
for each pair of nodes U V all nodes on the path between U and V contain
(as seen in the previous part of the lecture)
VU
27
Step 4 Junction Tree (cont)Step 4 Junction Tree (cont)
bull Theorem An undirected graph is triangulated if and only if its junction graph has a junction tree
bull Definition The weight of a link in a junction graph is the number of variable in the label The weight of a junction tree is the sum of weights of the labels
bull Theorem A sub-tree of the junction graph of a triangulated graph is a junction tree if and only if it is a spanning of maximal weight
28
Junction graph GJ (not complete)
abd
ade
ace
ceg
eghdef
ad ae ce
de eg
e
e
e
a
e
abd
ade
ace
ceg
eghdef
ad ae ce
de eg
Junction tree GJT
There are several methods to find MST
Kruskalrsquos algorithm choose successively a link of
maximal weight unless it creates a cycle
29
Another exampleAnother example
bull Compute the elimination cliques(the order here is f d e c b a)
bull Form the complete junction graph over the maximal elimination cliques and find a maximum-weight spanning tree
30
Junction Trees and Elimination Junction Trees and Elimination OrderOrder
bull We can use different orderings in variable elimination - affects efficiency
bull Each ordering corresponds to a junction tree
bull Just as some elimination orderings are more efficient than others some junction trees are better than others (Recall our mention of heuristics for triangulation)
31
OK I have this tree now whatOK I have this tree now what
bull A separator S divides the remaining variables into two groupsndash Variables in each group appear
on one side in the cluster tree
bull Examplesndash AB L S T V amp D X
ndash AL T V amp BDSX
ndash BL S amp A DT V X
ndash A X amp BDL S T V
ndash T V amp A B D K S X
LT
A B
X
V S
D
TV
ALTBLS
XAALB
ABD
AAB
BL
T
AL
32
Elimination in Junction TreesElimination in Junction Trees
bull Let X and Y be the partition induced by S
Observation
bull Eliminating all variables in X results in a factor fX(S)
bull Proof Since S is a separator only variables in S are adjacentto variables in X
bull NoteThe same factor would result regardless of theelimination order
x
y
A BSfX(S)
fY(S)
33
Recursive Elimination in Recursive Elimination in Junction TreesJunction Trees
bull How do we compute fX(S)
bull By recursive decomposition alongcluster tree
ndash Let X1 and X2 be the disjoint partitioning of X C implied by theseparators S1 and S2
ndash Eliminate X1 to get fX1(S1)
ndash Eliminate X2 to get fX2(S2)
ndash Eliminate variables in C S toget fX(S)
C
S
S2S1
x1
x2
y
- Slide 6
- Slide 7
- Slide 8
- Slide 9
- Slide 10
- Slide 11
- Slide 12
- Slide 13
- Slide 23
- Slide 25
- Slide 28
-
23
a
b c
d e
f
g
h
a
b c
d e
f
g
h
a
b c
d e
f
g
a
b c
d e
f
a
b c
d e
a
b
d e
aa
e
a
d e
vertex induced addedremoved clique edges
1 h egh -2 g ceg -3 f def -4 c ace a-e
vertex induced added removed clique edges
5 b abd a-d6 d ade -7 e ae -8 a a -
GT
GM
Eliminate the vertex that requires least number of edges to be added
24
Step 3 Junction GraphStep 3 Junction Graph
bull A junction graph for an undirected graph G is an undirected labeled graph
bull The nodes are the cliques in G
bull If two cliques intersect they are joined in the junction graph by an edge labeled with their intersection
25
a
b
d
a
c
e
d e
f
a
d e
e
g
h
c
e
g
a
b c
d e
f
g
h
Bayesian NetworkG = ( V E )
a
b c
d e
f
g
h
a
b c
d e
f
g
h
Moral graph GM Triangulated graph GT
abd
ade
ace
ceg
eghdef
ad ae ce
de eg
seperators
Junction graph GJ (not complete)eg ceg egh = eg Cliques
e
e
e
a
e
26
Step 4 Junction TreeStep 4 Junction Tree
bull A junction tree is a sub-graph of the junction graph that ndash Is a tree
ndash Contains all the cliques (spanning tree)
ndash Satisfies the running intersection property
for each pair of nodes U V all nodes on the path between U and V contain
(as seen in the previous part of the lecture)
VU
27
Step 4 Junction Tree (cont)Step 4 Junction Tree (cont)
bull Theorem An undirected graph is triangulated if and only if its junction graph has a junction tree
bull Definition The weight of a link in a junction graph is the number of variable in the label The weight of a junction tree is the sum of weights of the labels
bull Theorem A sub-tree of the junction graph of a triangulated graph is a junction tree if and only if it is a spanning of maximal weight
28
Junction graph GJ (not complete)
abd
ade
ace
ceg
eghdef
ad ae ce
de eg
e
e
e
a
e
abd
ade
ace
ceg
eghdef
ad ae ce
de eg
Junction tree GJT
There are several methods to find MST
Kruskalrsquos algorithm choose successively a link of
maximal weight unless it creates a cycle
29
Another exampleAnother example
bull Compute the elimination cliques(the order here is f d e c b a)
bull Form the complete junction graph over the maximal elimination cliques and find a maximum-weight spanning tree
30
Junction Trees and Elimination Junction Trees and Elimination OrderOrder
bull We can use different orderings in variable elimination - affects efficiency
bull Each ordering corresponds to a junction tree
bull Just as some elimination orderings are more efficient than others some junction trees are better than others (Recall our mention of heuristics for triangulation)
31
OK I have this tree now whatOK I have this tree now what
bull A separator S divides the remaining variables into two groupsndash Variables in each group appear
on one side in the cluster tree
bull Examplesndash AB L S T V amp D X
ndash AL T V amp BDSX
ndash BL S amp A DT V X
ndash A X amp BDL S T V
ndash T V amp A B D K S X
LT
A B
X
V S
D
TV
ALTBLS
XAALB
ABD
AAB
BL
T
AL
32
Elimination in Junction TreesElimination in Junction Trees
bull Let X and Y be the partition induced by S
Observation
bull Eliminating all variables in X results in a factor fX(S)
bull Proof Since S is a separator only variables in S are adjacentto variables in X
bull NoteThe same factor would result regardless of theelimination order
x
y
A BSfX(S)
fY(S)
33
Recursive Elimination in Recursive Elimination in Junction TreesJunction Trees
bull How do we compute fX(S)
bull By recursive decomposition alongcluster tree
ndash Let X1 and X2 be the disjoint partitioning of X C implied by theseparators S1 and S2
ndash Eliminate X1 to get fX1(S1)
ndash Eliminate X2 to get fX2(S2)
ndash Eliminate variables in C S toget fX(S)
C
S
S2S1
x1
x2
y
- Slide 6
- Slide 7
- Slide 8
- Slide 9
- Slide 10
- Slide 11
- Slide 12
- Slide 13
- Slide 23
- Slide 25
- Slide 28
-
24
Step 3 Junction GraphStep 3 Junction Graph
bull A junction graph for an undirected graph G is an undirected labeled graph
bull The nodes are the cliques in G
bull If two cliques intersect they are joined in the junction graph by an edge labeled with their intersection
25
a
b
d
a
c
e
d e
f
a
d e
e
g
h
c
e
g
a
b c
d e
f
g
h
Bayesian NetworkG = ( V E )
a
b c
d e
f
g
h
a
b c
d e
f
g
h
Moral graph GM Triangulated graph GT
abd
ade
ace
ceg
eghdef
ad ae ce
de eg
seperators
Junction graph GJ (not complete)eg ceg egh = eg Cliques
e
e
e
a
e
26
Step 4 Junction TreeStep 4 Junction Tree
bull A junction tree is a sub-graph of the junction graph that ndash Is a tree
ndash Contains all the cliques (spanning tree)
ndash Satisfies the running intersection property
for each pair of nodes U V all nodes on the path between U and V contain
(as seen in the previous part of the lecture)
VU
27
Step 4 Junction Tree (cont)Step 4 Junction Tree (cont)
bull Theorem An undirected graph is triangulated if and only if its junction graph has a junction tree
bull Definition The weight of a link in a junction graph is the number of variable in the label The weight of a junction tree is the sum of weights of the labels
bull Theorem A sub-tree of the junction graph of a triangulated graph is a junction tree if and only if it is a spanning of maximal weight
28
Junction graph GJ (not complete)
abd
ade
ace
ceg
eghdef
ad ae ce
de eg
e
e
e
a
e
abd
ade
ace
ceg
eghdef
ad ae ce
de eg
Junction tree GJT
There are several methods to find MST
Kruskalrsquos algorithm choose successively a link of
maximal weight unless it creates a cycle
29
Another exampleAnother example
bull Compute the elimination cliques(the order here is f d e c b a)
bull Form the complete junction graph over the maximal elimination cliques and find a maximum-weight spanning tree
30
Junction Trees and Elimination Junction Trees and Elimination OrderOrder
bull We can use different orderings in variable elimination - affects efficiency
bull Each ordering corresponds to a junction tree
bull Just as some elimination orderings are more efficient than others some junction trees are better than others (Recall our mention of heuristics for triangulation)
31
OK I have this tree now whatOK I have this tree now what
bull A separator S divides the remaining variables into two groupsndash Variables in each group appear
on one side in the cluster tree
bull Examplesndash AB L S T V amp D X
ndash AL T V amp BDSX
ndash BL S amp A DT V X
ndash A X amp BDL S T V
ndash T V amp A B D K S X
LT
A B
X
V S
D
TV
ALTBLS
XAALB
ABD
AAB
BL
T
AL
32
Elimination in Junction TreesElimination in Junction Trees
bull Let X and Y be the partition induced by S
Observation
bull Eliminating all variables in X results in a factor fX(S)
bull Proof Since S is a separator only variables in S are adjacentto variables in X
bull NoteThe same factor would result regardless of theelimination order
x
y
A BSfX(S)
fY(S)
33
Recursive Elimination in Recursive Elimination in Junction TreesJunction Trees
bull How do we compute fX(S)
bull By recursive decomposition alongcluster tree
ndash Let X1 and X2 be the disjoint partitioning of X C implied by theseparators S1 and S2
ndash Eliminate X1 to get fX1(S1)
ndash Eliminate X2 to get fX2(S2)
ndash Eliminate variables in C S toget fX(S)
C
S
S2S1
x1
x2
y
- Slide 6
- Slide 7
- Slide 8
- Slide 9
- Slide 10
- Slide 11
- Slide 12
- Slide 13
- Slide 23
- Slide 25
- Slide 28
-
25
a
b
d
a
c
e
d e
f
a
d e
e
g
h
c
e
g
a
b c
d e
f
g
h
Bayesian NetworkG = ( V E )
a
b c
d e
f
g
h
a
b c
d e
f
g
h
Moral graph GM Triangulated graph GT
abd
ade
ace
ceg
eghdef
ad ae ce
de eg
seperators
Junction graph GJ (not complete)eg ceg egh = eg Cliques
e
e
e
a
e
26
Step 4 Junction TreeStep 4 Junction Tree
bull A junction tree is a sub-graph of the junction graph that ndash Is a tree
ndash Contains all the cliques (spanning tree)
ndash Satisfies the running intersection property
for each pair of nodes U V all nodes on the path between U and V contain
(as seen in the previous part of the lecture)
VU
27
Step 4 Junction Tree (cont)Step 4 Junction Tree (cont)
bull Theorem An undirected graph is triangulated if and only if its junction graph has a junction tree
bull Definition The weight of a link in a junction graph is the number of variable in the label The weight of a junction tree is the sum of weights of the labels
bull Theorem A sub-tree of the junction graph of a triangulated graph is a junction tree if and only if it is a spanning of maximal weight
28
Junction graph GJ (not complete)
abd
ade
ace
ceg
eghdef
ad ae ce
de eg
e
e
e
a
e
abd
ade
ace
ceg
eghdef
ad ae ce
de eg
Junction tree GJT
There are several methods to find MST
Kruskalrsquos algorithm choose successively a link of
maximal weight unless it creates a cycle
29
Another exampleAnother example
bull Compute the elimination cliques(the order here is f d e c b a)
bull Form the complete junction graph over the maximal elimination cliques and find a maximum-weight spanning tree
30
Junction Trees and Elimination Junction Trees and Elimination OrderOrder
bull We can use different orderings in variable elimination - affects efficiency
bull Each ordering corresponds to a junction tree
bull Just as some elimination orderings are more efficient than others some junction trees are better than others (Recall our mention of heuristics for triangulation)
31
OK I have this tree now whatOK I have this tree now what
bull A separator S divides the remaining variables into two groupsndash Variables in each group appear
on one side in the cluster tree
bull Examplesndash AB L S T V amp D X
ndash AL T V amp BDSX
ndash BL S amp A DT V X
ndash A X amp BDL S T V
ndash T V amp A B D K S X
LT
A B
X
V S
D
TV
ALTBLS
XAALB
ABD
AAB
BL
T
AL
32
Elimination in Junction TreesElimination in Junction Trees
bull Let X and Y be the partition induced by S
Observation
bull Eliminating all variables in X results in a factor fX(S)
bull Proof Since S is a separator only variables in S are adjacentto variables in X
bull NoteThe same factor would result regardless of theelimination order
x
y
A BSfX(S)
fY(S)
33
Recursive Elimination in Recursive Elimination in Junction TreesJunction Trees
bull How do we compute fX(S)
bull By recursive decomposition alongcluster tree
ndash Let X1 and X2 be the disjoint partitioning of X C implied by theseparators S1 and S2
ndash Eliminate X1 to get fX1(S1)
ndash Eliminate X2 to get fX2(S2)
ndash Eliminate variables in C S toget fX(S)
C
S
S2S1
x1
x2
y
- Slide 6
- Slide 7
- Slide 8
- Slide 9
- Slide 10
- Slide 11
- Slide 12
- Slide 13
- Slide 23
- Slide 25
- Slide 28
-
26
Step 4 Junction TreeStep 4 Junction Tree
bull A junction tree is a sub-graph of the junction graph that ndash Is a tree
ndash Contains all the cliques (spanning tree)
ndash Satisfies the running intersection property
for each pair of nodes U V all nodes on the path between U and V contain
(as seen in the previous part of the lecture)
VU
27
Step 4 Junction Tree (cont)Step 4 Junction Tree (cont)
bull Theorem An undirected graph is triangulated if and only if its junction graph has a junction tree
bull Definition The weight of a link in a junction graph is the number of variable in the label The weight of a junction tree is the sum of weights of the labels
bull Theorem A sub-tree of the junction graph of a triangulated graph is a junction tree if and only if it is a spanning of maximal weight
28
Junction graph GJ (not complete)
abd
ade
ace
ceg
eghdef
ad ae ce
de eg
e
e
e
a
e
abd
ade
ace
ceg
eghdef
ad ae ce
de eg
Junction tree GJT
There are several methods to find MST
Kruskalrsquos algorithm choose successively a link of
maximal weight unless it creates a cycle
29
Another exampleAnother example
bull Compute the elimination cliques(the order here is f d e c b a)
bull Form the complete junction graph over the maximal elimination cliques and find a maximum-weight spanning tree
30
Junction Trees and Elimination Junction Trees and Elimination OrderOrder
bull We can use different orderings in variable elimination - affects efficiency
bull Each ordering corresponds to a junction tree
bull Just as some elimination orderings are more efficient than others some junction trees are better than others (Recall our mention of heuristics for triangulation)
31
OK I have this tree now whatOK I have this tree now what
bull A separator S divides the remaining variables into two groupsndash Variables in each group appear
on one side in the cluster tree
bull Examplesndash AB L S T V amp D X
ndash AL T V amp BDSX
ndash BL S amp A DT V X
ndash A X amp BDL S T V
ndash T V amp A B D K S X
LT
A B
X
V S
D
TV
ALTBLS
XAALB
ABD
AAB
BL
T
AL
32
Elimination in Junction TreesElimination in Junction Trees
bull Let X and Y be the partition induced by S
Observation
bull Eliminating all variables in X results in a factor fX(S)
bull Proof Since S is a separator only variables in S are adjacentto variables in X
bull NoteThe same factor would result regardless of theelimination order
x
y
A BSfX(S)
fY(S)
33
Recursive Elimination in Recursive Elimination in Junction TreesJunction Trees
bull How do we compute fX(S)
bull By recursive decomposition alongcluster tree
ndash Let X1 and X2 be the disjoint partitioning of X C implied by theseparators S1 and S2
ndash Eliminate X1 to get fX1(S1)
ndash Eliminate X2 to get fX2(S2)
ndash Eliminate variables in C S toget fX(S)
C
S
S2S1
x1
x2
y
- Slide 6
- Slide 7
- Slide 8
- Slide 9
- Slide 10
- Slide 11
- Slide 12
- Slide 13
- Slide 23
- Slide 25
- Slide 28
-
27
Step 4 Junction Tree (cont)Step 4 Junction Tree (cont)
bull Theorem An undirected graph is triangulated if and only if its junction graph has a junction tree
bull Definition The weight of a link in a junction graph is the number of variable in the label The weight of a junction tree is the sum of weights of the labels
bull Theorem A sub-tree of the junction graph of a triangulated graph is a junction tree if and only if it is a spanning of maximal weight
28
Junction graph GJ (not complete)
abd
ade
ace
ceg
eghdef
ad ae ce
de eg
e
e
e
a
e
abd
ade
ace
ceg
eghdef
ad ae ce
de eg
Junction tree GJT
There are several methods to find MST
Kruskalrsquos algorithm choose successively a link of
maximal weight unless it creates a cycle
29
Another exampleAnother example
bull Compute the elimination cliques(the order here is f d e c b a)
bull Form the complete junction graph over the maximal elimination cliques and find a maximum-weight spanning tree
30
Junction Trees and Elimination Junction Trees and Elimination OrderOrder
bull We can use different orderings in variable elimination - affects efficiency
bull Each ordering corresponds to a junction tree
bull Just as some elimination orderings are more efficient than others some junction trees are better than others (Recall our mention of heuristics for triangulation)
31
OK I have this tree now whatOK I have this tree now what
bull A separator S divides the remaining variables into two groupsndash Variables in each group appear
on one side in the cluster tree
bull Examplesndash AB L S T V amp D X
ndash AL T V amp BDSX
ndash BL S amp A DT V X
ndash A X amp BDL S T V
ndash T V amp A B D K S X
LT
A B
X
V S
D
TV
ALTBLS
XAALB
ABD
AAB
BL
T
AL
32
Elimination in Junction TreesElimination in Junction Trees
bull Let X and Y be the partition induced by S
Observation
bull Eliminating all variables in X results in a factor fX(S)
bull Proof Since S is a separator only variables in S are adjacentto variables in X
bull NoteThe same factor would result regardless of theelimination order
x
y
A BSfX(S)
fY(S)
33
Recursive Elimination in Recursive Elimination in Junction TreesJunction Trees
bull How do we compute fX(S)
bull By recursive decomposition alongcluster tree
ndash Let X1 and X2 be the disjoint partitioning of X C implied by theseparators S1 and S2
ndash Eliminate X1 to get fX1(S1)
ndash Eliminate X2 to get fX2(S2)
ndash Eliminate variables in C S toget fX(S)
C
S
S2S1
x1
x2
y
- Slide 6
- Slide 7
- Slide 8
- Slide 9
- Slide 10
- Slide 11
- Slide 12
- Slide 13
- Slide 23
- Slide 25
- Slide 28
-
28
Junction graph GJ (not complete)
abd
ade
ace
ceg
eghdef
ad ae ce
de eg
e
e
e
a
e
abd
ade
ace
ceg
eghdef
ad ae ce
de eg
Junction tree GJT
There are several methods to find MST
Kruskalrsquos algorithm choose successively a link of
maximal weight unless it creates a cycle
29
Another exampleAnother example
bull Compute the elimination cliques(the order here is f d e c b a)
bull Form the complete junction graph over the maximal elimination cliques and find a maximum-weight spanning tree
30
Junction Trees and Elimination Junction Trees and Elimination OrderOrder
bull We can use different orderings in variable elimination - affects efficiency
bull Each ordering corresponds to a junction tree
bull Just as some elimination orderings are more efficient than others some junction trees are better than others (Recall our mention of heuristics for triangulation)
31
OK I have this tree now whatOK I have this tree now what
bull A separator S divides the remaining variables into two groupsndash Variables in each group appear
on one side in the cluster tree
bull Examplesndash AB L S T V amp D X
ndash AL T V amp BDSX
ndash BL S amp A DT V X
ndash A X amp BDL S T V
ndash T V amp A B D K S X
LT
A B
X
V S
D
TV
ALTBLS
XAALB
ABD
AAB
BL
T
AL
32
Elimination in Junction TreesElimination in Junction Trees
bull Let X and Y be the partition induced by S
Observation
bull Eliminating all variables in X results in a factor fX(S)
bull Proof Since S is a separator only variables in S are adjacentto variables in X
bull NoteThe same factor would result regardless of theelimination order
x
y
A BSfX(S)
fY(S)
33
Recursive Elimination in Recursive Elimination in Junction TreesJunction Trees
bull How do we compute fX(S)
bull By recursive decomposition alongcluster tree
ndash Let X1 and X2 be the disjoint partitioning of X C implied by theseparators S1 and S2
ndash Eliminate X1 to get fX1(S1)
ndash Eliminate X2 to get fX2(S2)
ndash Eliminate variables in C S toget fX(S)
C
S
S2S1
x1
x2
y
- Slide 6
- Slide 7
- Slide 8
- Slide 9
- Slide 10
- Slide 11
- Slide 12
- Slide 13
- Slide 23
- Slide 25
- Slide 28
-
29
Another exampleAnother example
bull Compute the elimination cliques(the order here is f d e c b a)
bull Form the complete junction graph over the maximal elimination cliques and find a maximum-weight spanning tree
30
Junction Trees and Elimination Junction Trees and Elimination OrderOrder
bull We can use different orderings in variable elimination - affects efficiency
bull Each ordering corresponds to a junction tree
bull Just as some elimination orderings are more efficient than others some junction trees are better than others (Recall our mention of heuristics for triangulation)
31
OK I have this tree now whatOK I have this tree now what
bull A separator S divides the remaining variables into two groupsndash Variables in each group appear
on one side in the cluster tree
bull Examplesndash AB L S T V amp D X
ndash AL T V amp BDSX
ndash BL S amp A DT V X
ndash A X amp BDL S T V
ndash T V amp A B D K S X
LT
A B
X
V S
D
TV
ALTBLS
XAALB
ABD
AAB
BL
T
AL
32
Elimination in Junction TreesElimination in Junction Trees
bull Let X and Y be the partition induced by S
Observation
bull Eliminating all variables in X results in a factor fX(S)
bull Proof Since S is a separator only variables in S are adjacentto variables in X
bull NoteThe same factor would result regardless of theelimination order
x
y
A BSfX(S)
fY(S)
33
Recursive Elimination in Recursive Elimination in Junction TreesJunction Trees
bull How do we compute fX(S)
bull By recursive decomposition alongcluster tree
ndash Let X1 and X2 be the disjoint partitioning of X C implied by theseparators S1 and S2
ndash Eliminate X1 to get fX1(S1)
ndash Eliminate X2 to get fX2(S2)
ndash Eliminate variables in C S toget fX(S)
C
S
S2S1
x1
x2
y
- Slide 6
- Slide 7
- Slide 8
- Slide 9
- Slide 10
- Slide 11
- Slide 12
- Slide 13
- Slide 23
- Slide 25
- Slide 28
-
30
Junction Trees and Elimination Junction Trees and Elimination OrderOrder
bull We can use different orderings in variable elimination - affects efficiency
bull Each ordering corresponds to a junction tree
bull Just as some elimination orderings are more efficient than others some junction trees are better than others (Recall our mention of heuristics for triangulation)
31
OK I have this tree now whatOK I have this tree now what
bull A separator S divides the remaining variables into two groupsndash Variables in each group appear
on one side in the cluster tree
bull Examplesndash AB L S T V amp D X
ndash AL T V amp BDSX
ndash BL S amp A DT V X
ndash A X amp BDL S T V
ndash T V amp A B D K S X
LT
A B
X
V S
D
TV
ALTBLS
XAALB
ABD
AAB
BL
T
AL
32
Elimination in Junction TreesElimination in Junction Trees
bull Let X and Y be the partition induced by S
Observation
bull Eliminating all variables in X results in a factor fX(S)
bull Proof Since S is a separator only variables in S are adjacentto variables in X
bull NoteThe same factor would result regardless of theelimination order
x
y
A BSfX(S)
fY(S)
33
Recursive Elimination in Recursive Elimination in Junction TreesJunction Trees
bull How do we compute fX(S)
bull By recursive decomposition alongcluster tree
ndash Let X1 and X2 be the disjoint partitioning of X C implied by theseparators S1 and S2
ndash Eliminate X1 to get fX1(S1)
ndash Eliminate X2 to get fX2(S2)
ndash Eliminate variables in C S toget fX(S)
C
S
S2S1
x1
x2
y
- Slide 6
- Slide 7
- Slide 8
- Slide 9
- Slide 10
- Slide 11
- Slide 12
- Slide 13
- Slide 23
- Slide 25
- Slide 28
-
31
OK I have this tree now whatOK I have this tree now what
bull A separator S divides the remaining variables into two groupsndash Variables in each group appear
on one side in the cluster tree
bull Examplesndash AB L S T V amp D X
ndash AL T V amp BDSX
ndash BL S amp A DT V X
ndash A X amp BDL S T V
ndash T V amp A B D K S X
LT
A B
X
V S
D
TV
ALTBLS
XAALB
ABD
AAB
BL
T
AL
32
Elimination in Junction TreesElimination in Junction Trees
bull Let X and Y be the partition induced by S
Observation
bull Eliminating all variables in X results in a factor fX(S)
bull Proof Since S is a separator only variables in S are adjacentto variables in X
bull NoteThe same factor would result regardless of theelimination order
x
y
A BSfX(S)
fY(S)
33
Recursive Elimination in Recursive Elimination in Junction TreesJunction Trees
bull How do we compute fX(S)
bull By recursive decomposition alongcluster tree
ndash Let X1 and X2 be the disjoint partitioning of X C implied by theseparators S1 and S2
ndash Eliminate X1 to get fX1(S1)
ndash Eliminate X2 to get fX2(S2)
ndash Eliminate variables in C S toget fX(S)
C
S
S2S1
x1
x2
y
- Slide 6
- Slide 7
- Slide 8
- Slide 9
- Slide 10
- Slide 11
- Slide 12
- Slide 13
- Slide 23
- Slide 25
- Slide 28
-
32
Elimination in Junction TreesElimination in Junction Trees
bull Let X and Y be the partition induced by S
Observation
bull Eliminating all variables in X results in a factor fX(S)
bull Proof Since S is a separator only variables in S are adjacentto variables in X
bull NoteThe same factor would result regardless of theelimination order
x
y
A BSfX(S)
fY(S)
33
Recursive Elimination in Recursive Elimination in Junction TreesJunction Trees
bull How do we compute fX(S)
bull By recursive decomposition alongcluster tree
ndash Let X1 and X2 be the disjoint partitioning of X C implied by theseparators S1 and S2
ndash Eliminate X1 to get fX1(S1)
ndash Eliminate X2 to get fX2(S2)
ndash Eliminate variables in C S toget fX(S)
C
S
S2S1
x1
x2
y
- Slide 6
- Slide 7
- Slide 8
- Slide 9
- Slide 10
- Slide 11
- Slide 12
- Slide 13
- Slide 23
- Slide 25
- Slide 28
-
33
Recursive Elimination in Recursive Elimination in Junction TreesJunction Trees
bull How do we compute fX(S)
bull By recursive decomposition alongcluster tree
ndash Let X1 and X2 be the disjoint partitioning of X C implied by theseparators S1 and S2
ndash Eliminate X1 to get fX1(S1)
ndash Eliminate X2 to get fX2(S2)
ndash Eliminate variables in C S toget fX(S)
C
S
S2S1
x1
x2
y
- Slide 6
- Slide 7
- Slide 8
- Slide 9
- Slide 10
- Slide 11
- Slide 12
- Slide 13
- Slide 23
- Slide 25
- Slide 28
-