Download - 03 Uncertainty inference(discrete)
Unit 3 Uncertainty Inference:Discrete
Wang, Yuan-Kai, 王元凱[email protected]
http://www.ykwang.tw
Department of Electrical Engineering, Fu Jen Univ.輔仁大學電機工程系
2006~2011
Bayesian Networks
Fu Jen University Department of Electrical Engineering Wang, Yuan-Kai Copyright
Reference this document as: Wang, Yuan-Kai, “Uncertainty Inference - Discrete,"
Lecture Notes of Wang, Yuan-Kai, Fu Jen University, Taiwan, 2011.
1
Fu Jen University Department of Electrical Engineering Wang, Yuan-Kai Copyright
王元凱 Unit - Uncertainty Inference (Discrete) p.
Goal of this Unit• Review advanced concepts of statistics
– Statistical Inference – Pattern recognition
2
Fu Jen University Department of Electrical Engineering Wang, Yuan-Kai Copyright
王元凱 Unit - Uncertainty Inference (Discrete) p.
Related Units• Previous unit(s)
– Probability Review– Statistics Review
• Next units– Uncertainty Inference (Continuous)
3
Fu Jen University Department of Electrical Engineering Wang, Yuan-Kai Copyright
王元凱 Unit - Uncertainty Inference (Discrete) p.
Self-Study• Artificial Intelligence: a modern approach
– Russell & Norvig, 2nd, Prentice Hall, 2003.pp.462~474,
– Chapter 13, Sec. 13.1~13.3• 統計學的世界
– 墨爾著,鄭惟厚譯, 天下文化,2002• 深入淺出統計學
– D. Grifiths, 楊仁和譯,2009, O’ Reilly
4
Fu Jen University Department of Electrical Engineering Wang, Yuan-Kai Copyright
王元凱 Unit - Uncertainty Inference (Discrete) p.
5
5
Contents1. Acting Under Uncertainty …………………. 62. Basic Probability ..................………..……. 153. Marginal Probability ..…….......................... 274. Inference Using Full Joint Distribution ... 305. Independence ............................................ 436. Bayes' Rule and Its Use ............................ 477. Summary ……………………………………. 62
Fu Jen University Department of Electrical Engineering Wang, Yuan-Kai Copyright
王元凱 Unit - Uncertainty Inference (Discrete) p. 6
1. Acting Under Uncertainty
environmentagent
?
sensors
actuators
?
?
?
model
Fu Jen University Department of Electrical Engineering Wang, Yuan-Kai Copyright
王元凱 Unit - Uncertainty Inference (Discrete) p. 7
Example 1-Localization (1/3)• Where is it
– It is a robot–Sensor: camera, laser range finder, sonar–State: (x, y, orientation), Prob.
Fu Jen University Department of Electrical Engineering Wang, Yuan-Kai Copyright
王元凱 Unit - Uncertainty Inference (Discrete) p. 8
Example 1-Localization (2/3)• Where is it
– It is a mobile station/robot–Sensor: Wireless LAN–State: (x, y), Prob.
Fu Jen University Department of Electrical Engineering Wang, Yuan-Kai Copyright
王元凱 Unit - Uncertainty Inference (Discrete) p. 9
Example 1-Localization (3/3)• Where is it
– It is a moving text–Sensor: computer vision techniques–State: (x, y, moving direction), Prob.
t-3 t-2 t-1 t
Output t
Fu Jen University Department of Electrical Engineering Wang, Yuan-Kai Copyright
王元凱 Unit - Uncertainty Inference (Discrete) p. 10
Example 2-Correlation of Features and Words of Color
• Word of color
– Red
– Light red
...
• Feature of color (Average)RGB=(255,0,0)
RGB=(220,10,10)
RGB=(180,20,20)
RGB=(150,30,30)
RGB=(223,0,0)
RGB=(147,25,25)
Uncertainty
Fu Jen University Department of Electrical Engineering Wang, Yuan-Kai Copyright
王元凱 Unit - Uncertainty Inference (Discrete) p. 11
Example 3-Target Tracking for Robot
targetrobot
• The robot must keepthe target in view
• The target’s trajectoryis not known in advance
• The environment mayor may not be known
Fu Jen University Department of Electrical Engineering Wang, Yuan-Kai Copyright
王元凱 Unit - Uncertainty Inference (Discrete) p. 12
Inaccuracy & Uncertainty
• Movement Inaccuracy
Sensor Inaccuracy
EnvironmentalUncertainty
Fu Jen University Department of Electrical Engineering Wang, Yuan-Kai Copyright
王元凱 Unit - Uncertainty Inference (Discrete) p. 13
Degree of Belief• Probability theory
–Assigns a numerical degree of beliefbetween 0 and 1 to an evidence
–Provides a way of summarizing the uncertainty
Fu Jen University Department of Electrical Engineering Wang, Yuan-Kai Copyright
王元凱 Unit - Uncertainty Inference (Discrete) p. 14
Techniques for Uncertainty
• Bayes rule/Bayesian network with probability theory
• Certainty factor in expert system• Fuzzy theory with possibility theory• Dempster-Shafer theory
Fu Jen University Department of Electrical Engineering Wang, Yuan-Kai Copyright
王元凱 Unit - Uncertainty Inference (Discrete) p. 15
2. Basic Probability• Terms
–Random variables–Full joint distribution (FJD)–Conditional probability table (CPT)
Fu Jen University Department of Electrical Engineering Wang, Yuan-Kai Copyright
王元凱 Unit - Uncertainty Inference (Discrete) p. 16
Random Variable• Boolean random variable
–Rain : true, false• Discrete random variable
–Rain: cloudy, sunny, drizzle, drench• Continuous random variable
–Rain: rainfall in millimeterWe will focus on Boolean & discrete cases in most examples of this book
Fu Jen University Department of Electrical Engineering Wang, Yuan-Kai Copyright
王元凱 Unit - Uncertainty Inference (Discrete) p. 17
For Boolean & Discrete R.V.• P(X) is a vector• Boolean R.V.
–Rain: true, false–P(Rain) = <0.72, 0.28>
• Discrete R.V.–Rain: cloudy, sunny, drizzle, drench–P(Rain) = <0.72, 0.1, 0.08, 0.1>–Normalized, i.e. sums to 1
Fu Jen University Department of Electrical Engineering Wang, Yuan-Kai Copyright
王元凱 Unit - Uncertainty Inference (Discrete) p. 18
Full Joint Distribution (1/3)• For a set of random variables
{X1, X2 , , Xn}• X1 X2 Xn are atomic events• P(X1 X2 Xn) is
–A full joint probability distribution–A table of all joint prob. of all atomic
events, if {X1, , Xn} are discrete• All questions about probability of
joint events can be answered by the table
Fu Jen University Department of Electrical Engineering Wang, Yuan-Kai Copyright
王元凱 Unit - Uncertainty Inference (Discrete) p. 19
Full Joint Distribution (2/3)• Ex: X1: Rain, X2: Wind
–X1: drizzle, drench, cloudy, –X2: strong, weak–X1 X2 are atomic events–P(X1 X2) is a 3x2 matrix of values
RainWind Drizzle Drench Cloudy
Strong 0.15 0.12 0.06Weak 0.55 0.08 0.04P(X1=drizzle X2=strong) = 0.15, ...
Fu Jen University Department of Electrical Engineering Wang, Yuan-Kai Copyright
王元凱 Unit - Uncertainty Inference (Discrete) p. 20
Full Joint Distribution (3/3)• All questions about probability of
joint events can be answered by the table–P(Wind=Strong),
P(Rain=Drizzle Wind=Strong), ...
Fu Jen University Department of Electrical Engineering Wang, Yuan-Kai Copyright
王元凱 Unit - Uncertainty Inference (Discrete) p. 21
Posterior v.s. Prior Probabilities
• P(Cavity|Toothache)–Conditional probability–Posterior probability
(after the fact/evidence)• P(Cavity)
–Prior probability (the fact)
Fu Jen University Department of Electrical Engineering Wang, Yuan-Kai Copyright
王元凱 Unit - Uncertainty Inference (Discrete) p. 22
An Example (1/2)• For a dental diagnosis
–Let {Cavity,Toothache} be a set of Boolean random variables
• Denotations for Boolean R.V.–P(Cavity=true) = P(cavity)–P(Cavity=false) = P(cavity)
Fu Jen University Department of Electrical Engineering Wang, Yuan-Kai Copyright
王元凱 Unit - Uncertainty Inference (Discrete) p. 23
An Example (2/2)• The full joint probability
distribution P(Toothache Cavity)
cavity 0.04 0.06cavity 0.01 0.89
toothache toothache
11.006.001.004.0)( toothachecavityP
80.001.004.0
04.0)(
)()|(
toothacheP
toothachecavityPtoothachecavityP
Fu Jen University Department of Electrical Engineering Wang, Yuan-Kai Copyright
王元凱 Unit - Uncertainty Inference (Discrete) p. 24
Conditional Probability (Math)• P(X1=x1i| X2 =x2j) is a conditional
probability• P(X1 | X2) is a conditional
distribution function–All P(X1=x1i| X2 =x2j) for all possible i, j
• For Boolean & discrete R.V.s, conditional distribution function is a table–Conditional distribution of continuous
R.V. will not used in our discussions
Fu Jen University Department of Electrical Engineering Wang, Yuan-Kai Copyright
王元凱 Unit - Uncertainty Inference (Discrete) p. 25
Conditional Probability Table (CPT)
• For the dental diagnosis problem,–Toothache & Cavity are Boolean R.V.s–P(Toothache|Cavity) is a CPT
Cavity P(toothache|Cavity)T 0.90F 0.05
Cavity P(toothache|Cavity) P(toothache|Cavity)T 0.90 0.1F 0.05 0.95
Fu Jen University Department of Electrical Engineering Wang, Yuan-Kai Copyright
王元凱 Unit - Uncertainty Inference (Discrete) p. 26
CPT v.s. FJD
Cavity P(toothache|Cavity) P(toothache|Cavity)T 0.90 0.1F 0.05 0.95
cavity 0.04 0.06cavity 0.01 0.89
toothache toothache
Sum of all atomic events = 1
Sum of a row = 1
Fu Jen University Department of Electrical Engineering Wang, Yuan-Kai Copyright
王元凱 Unit - Uncertainty Inference (Discrete) p. 27
3. Marginal Probability
• Probability of a random variable is the sum of the probabilities of the atomic events containing the random variable
• Marginalization (summing out)
cavity 0.04 0.06cavity 0.01 0.89
toothache toothache
)(
)()(ij XEe
ji ePXP
P(toothache)=P(toothache cavity)+
P(toothache cavity)=0.04 + 0.01 = 0.05
Marginal probability
Fu Jen University Department of Electrical Engineering Wang, Yuan-Kai Copyright
王元凱 Unit - Uncertainty Inference (Discrete) p. 28
Marginal Probability (1/2)• Suppose a problem of a world
contains only 3 random variables {X1, X2, X3}
33
)()( 332121Xx
xXXXPXXP
1)( 321 XXXP
33
)()(
332211
2211
XxxXxXxXP
xXxXP
Fu Jen University Department of Electrical Engineering Wang, Yuan-Kai Copyright
王元凱 Unit - Uncertainty Inference (Discrete) p. 29
Marginal Probability (2/2)• Using higher order joint probability
to calculate marginal and other lower order joint probability
)( 11 xXP
22
)( 2211Xx
xXxXP
22 33
)( 332211X Xx x
xXxXxXP
Fu Jen University Department of Electrical Engineering Wang, Yuan-Kai Copyright
王元凱 Unit - Uncertainty Inference (Discrete) p. 30
4. Inference Using Full Joint Distributions
• Probabilistic inference–Uses the full joint distribution as the
"knowledge base"– Is the computation from observed
evidence of posterior probabilities for query• Compute conditional probability
• The most simple inference method:Inference by enumeration
Fu Jen University Department of Electrical Engineering Wang, Yuan-Kai Copyright
王元凱 Unit - Uncertainty Inference (Discrete) p. 31
The Dental Diagnosis Example• The set of random variables:
Toothache, Cavity, and Catch–All are Boolean random variables–Note: P(toothache) P(Toothache=true)
• The full joint distribution is a 2x2x2 table
Fu Jen University Department of Electrical Engineering Wang, Yuan-Kai Copyright
王元凱 Unit - Uncertainty Inference (Discrete) p. 32
The Full Joint Distribution
• 8 atomic events (sum=1)– P(toothachecatch cavity)=0.108– P(toothachecatch cavity)=0.16– P(toothachecatch cavity)=0.012– P(toothachecatch cavity)=0.064– ...
Fu Jen University Department of Electrical Engineering Wang, Yuan-Kai Copyright
王元凱 Unit - Uncertainty Inference (Discrete) p. 33
Inference by Enumeration• P(cavity|toothache)
• We can answer the query by–Enumerating P(cavitytoothache)
from the full joint distribution–Enumerating P(toothache) from the full
joint distribution
)()(
toothachePtoothachecavityP
Fu Jen University Department of Electrical Engineering Wang, Yuan-Kai Copyright
王元凱 Unit - Uncertainty Inference (Discrete) p. 34
Joint Probability• P(cavity toothache)
= 0.016+0.064 = 0.08
Marginal probability
Order-2 joint probability
Fu Jen University Department of Electrical Engineering Wang, Yuan-Kai Copyright
王元凱 Unit - Uncertainty Inference (Discrete) p. 35
Marginal Probability• P(toothache)
= 0.108+0.012+0.016+0.064 = 0.2
Marginal probability of Toothache
Fu Jen University Department of Electrical Engineering Wang, Yuan-Kai Copyright
王元凱 Unit - Uncertainty Inference (Discrete) p. 36
Conditional Probability• P(cavity|toothache)
)()(
toothachePtoothachecavityP
4.02.008.0
Fu Jen University Department of Electrical Engineering Wang, Yuan-Kai Copyright
王元凱 Unit - Uncertainty Inference (Discrete) p. 37
An Exercise• P(cavity toothache)
P(Cavity=true Toothache=true)= 0.108+0.012+0.072+0.008+0.016+0.064= 0.28
Fu Jen University Department of Electrical Engineering Wang, Yuan-Kai Copyright
王元凱 Unit - Uncertainty Inference (Discrete) p. 38
Normalization (1/2)• P(cavity|toothache) and
P(cavity|toothache) have the same denominator P(toothache)
)()()|(
toothachePtoothachecavityPtootheachecavityP
)()()|(
toothachePtoothachecavityPtootheachecavityP
• 1/P(toothache) can be viewed as a normalization constant for probability calculation and derivation
Fu Jen University Department of Electrical Engineering Wang, Yuan-Kai Copyright
王元凱 Unit - Uncertainty Inference (Discrete) p. 39
Normalization (2/2)• P(cavity|toothache)
= P(cavitytoothache)• P(cavity|toothache)
= P(cavitytoothache)= [ P(cavitytoothache catch)
+ P(cavitytoothache catch) ]= [ 0.108 + 0.012 ] = 0.12 = 0.6
Fu Jen University Department of Electrical Engineering Wang, Yuan-Kai Copyright
王元凱 Unit - Uncertainty Inference (Discrete) p. 40
A General Inference Procedure (1/2)
• Let P(X|E=e) be the query– X be the query variable– E be the set of evidence variables– e be the observed values of E– H be the remaining unobserved variables
(Hidden variables)• Inference of the query P(X|E=e) is
HhhHeEXPeEXPeEXP)()()|(
1
2
Fu Jen University Department of Electrical Engineering Wang, Yuan-Kai Copyright
王元凱 Unit - Uncertainty Inference (Discrete) p. 41
A General Inference Procedure (2/2)
• E.x.: Query P(cavity|toothache)–X: Cavity, E/e: Toothache/true,
H: Catch)|( toothachecavityP )|( eEXP
)( toothachecavityP
)]( )([
catchtoothachecavityPcatchtoothachecavityP
)( eEXP
Hh
hHeEXP )(
Fu Jen University Department of Electrical Engineering Wang, Yuan-Kai Copyright
王元凱 Unit - Uncertainty Inference (Discrete) p. 42
Complexity of the Enumeration-based Inference
Algorithm• For n Boolean R.V.s
– Space complexity: O(2n)(Store the full joint distribution)
– Time complexity: O(2n) (worst case)• For n discrete R.V.s, all have d discrete
values– Time complexity: O(dn) (worst case)– Space complexity: O(dn)
• It is not a practical algorithm, • More efficient algorithm is needed
Fu Jen University Department of Electrical Engineering Wang, Yuan-Kai Copyright
王元凱 Unit - Uncertainty Inference (Discrete) p. 43
5. Independence• A and B are independent iff
–P(A|B) = P(A), or–P(B|A) = P(B), or–P(AB) = P(A)P(B)
A B
Fu Jen University Department of Electrical Engineering Wang, Yuan-Kai Copyright
王元凱 Unit - Uncertainty Inference (Discrete) p. 44
The Dental Diagnosis Example (1/2)
• Original example: 3 random variables• Add a fourth variable: Weather
– The Weather variable has 4 values– The full joint distribution:
P(Toothache,Catch,Cavity,Weather)has 32 elements (2x2x2x4)
• By commonsense, Weather is independentof the original 3 variables– P(ToothacheCatch Cavity Weather)
= P(Toothache Catch Cavity)P(Weather)
Fu Jen University Department of Electrical Engineering Wang, Yuan-Kai Copyright
王元凱 Unit - Uncertainty Inference (Discrete) p. 45
The Dental Diagnosis Example (2/2)
• P(Toothache,Catch,Cavity,Weather)= P(Toothache,Catch,Cavity)P(Weather)– The 32-element table for 4 variables can be
constructed from • One 8-element table, and• One 4-element table
–Reduced from 32 elements to 12 elements
Fu Jen University Department of Electrical Engineering Wang, Yuan-Kai Copyright
王元凱 Unit - Uncertainty Inference (Discrete) p. 46
Advantages of Independence• Independence can dramatically
reduce the amount of elements of the full distribution table– i.e., independence can help in reducing
• The size of the domain representation, and • The complexity of the inference problem
• Independence are usually based on knowledge of the domain
Fu Jen University Department of Electrical Engineering Wang, Yuan-Kai Copyright
王元凱 Unit - Uncertainty Inference (Discrete) p. 47
6. Bayes' Rule and Its Use• Given two random variables X, Y• By product rule
)()|()( YPYXPYXP
)()()|()|(
XPYPYXPXYP
)()|()( XPXYPYXP
• Bayes' rule underlies all modern AI systems for probabilistic inference
Fu Jen University Department of Electrical Engineering Wang, Yuan-Kai Copyright
王元凱 Unit - Uncertainty Inference (Discrete) p. 48
Bayes’ Theorem
)()()/()/(
ePhPhePehP
Posterior
Prior
Probability of Evidence
Likelihood
Probability of an hypothesis, h, can be updated when evidence, e, has been obtained.
Note: it is usually not necessary to calculate P(e) directly as it can be obtained by normalizing the posterior probabilities, P(hi | e).
Fu Jen University Department of Electrical Engineering Wang, Yuan-Kai Copyright
王元凱 Unit - Uncertainty Inference (Discrete) p. 49
A Simple ExampleConsider two related variables:1. Drug (D) with values y or n2. Test (T) with values +ve or –ve
And suppose we have the following probabilities:P(D = y) = 0.001P(T = +ve | D = y) = 0.8P(T = +ve | D = n) = 0.01
These probabilities are sufficient to define a joint probability distribution.
Suppose an athlete tests positive. What is the probability that he has taken the drug?
074.09990010001080
001080)()|()()|(
)()|(
......
nDPnDveTPyDPyDveTPyDPyDveTPve)y|TP(D
Fu Jen University Department of Electrical Engineering Wang, Yuan-Kai Copyright
王元凱 Unit - Uncertainty Inference (Discrete) p. 50
A More Complex CaseSuppose now that there is a similar link between Lung Cancer (L) and a chest X-ray (X) and that we also have the following relationships:
History of smoking (S) has a direct influence on bronchitis (B) and lung cancer (L);
L and B have a direct influence on fatigue (F).
What is the probability that someone has bronchitis given that they smoke, have fatigue and have received a positive X-ray result?
lb
l
lxfsbP
lxfsbP
xfsPxfsbPxfsbP
,),,,,(
),,,(
),,(),,,(),,|(
111
1111
111
11111111
where, for example, the variable B takes on values b1 (has bronchitis) and b2(does not have bronchitis).R.E. Neapolitan, Learning Bayesian Networks (2004)
Fu Jen University Department of Electrical Engineering Wang, Yuan-Kai Copyright
王元凱 Unit - Uncertainty Inference (Discrete) p. 51
An Example• Prior knowledge
– The probability of a patient has meningitis is 1/50,000: P(m)=1/50000
– The probability of a patient has stiff neck is 1/20: P(s)=1/20
– The meningitis causes the patient to have a stiff neck 50% of the time: P(s|m)=0.5
• The probability of a stiff-neck patient has meningitis
0002.020/150000/15.0
)()()|()|(
sPmPmsPsmP
Fu Jen University Department of Electrical Engineering Wang, Yuan-Kai Copyright
王元凱 Unit - Uncertainty Inference (Discrete) p. 52
Generalization of Bayes’ Rule• Conditionalized on more evidences, say
e
)|()|(),|(),|(
eXPeYPeYXPeXYP
• P(Cavity|toothachecatch)= (P(toothachecatch|Cavity)P(Cavity))
/ P(toothachecatch)=> P(toothache catch Cavity)
/ P(toothachecatch) Better way?
Fu Jen University Department of Electrical Engineering Wang, Yuan-Kai Copyright
王元凱 Unit - Uncertainty Inference (Discrete) p. 53
Use Independence in Conditional (1/2)
• Probe Catch and Toothache are independent, given the presence or the absence of a Cavity– Both Catch and Toothache are caused by
Cavity, but neither has a direct effect on the other
• P(toothachecatch|Cavity)= P(toothache|Cavity) P(catch|Cavity)
• Conditional independence
Fu Jen University Department of Electrical Engineering Wang, Yuan-Kai Copyright
王元凱 Unit - Uncertainty Inference (Discrete) p. 54
Use Independence in Conditional (2/2)
• If Catch is conditionally independentof Toothache given Cavity
• Equivalent statements–P(Catch Toothache|Cavity)
=P(Catch|Cavity) P(Toothache|Cavity)–P(Catch|ToothacheCavity)
=P(Catch|Cavity)–P(Toothache|CatchCavity)
=P(Toothache|Cavity)
Fu Jen University Department of Electrical Engineering Wang, Yuan-Kai Copyright
王元凱 Unit - Uncertainty Inference (Discrete) p. 55
Conditional Independency for the Dentist Diagnosis Problem (1/2)
• The conditional independence, like independence, can also decompose the full joint distribution into smaller pieces
• P(Toothache Catch Cavity)• = P(Toothache Catch|Cavity)P(Cavity)
(by product rule)• = P(Toothache|Cavity) P(Catch|Cavity)
P(Cavity)(by definition of conditional independence)
Fu Jen University Department of Electrical Engineering Wang, Yuan-Kai Copyright
王元凱 Unit - Uncertainty Inference (Discrete) p. 56
Conditional Independency for the Dentist Diagnosis Problem (2/2)
• P(Toothache Catch Cavity) has 23-1 elements
• P(Toothache|Cavity)P(Catch|Cavity)P(Cavity) has 2+2+1=5 elements
cavity 0.9 0.1cavity 0.05 0.95
P(toothache|Cavity) P(toothache|Cavity)
Conditional Probability Table (CPT)• Conditional independence can reduce the
amount of probabilities
Fu Jen University Department of Electrical Engineering Wang, Yuan-Kai Copyright
王元凱 Unit - Uncertainty Inference (Discrete) p. 57
Probabilistic Modeling of Problems (1/2)
• Usually random variables have two semantics–Cause–Effect
• Conditional probability P(Y|X) can be rewritten as P(Cause|Effect) or P(Effect|Cause)
Fu Jen University Department of Electrical Engineering Wang, Yuan-Kai Copyright
王元凱 Unit - Uncertainty Inference (Discrete) p. 58
Probabilistic Modeling of Problems (2/2)
• We usually assume some effects are conditionally independent given a cause–P(effect1 effect2 | cause)
= P(effect1 | cause) P(effect2 | cause)The conditional independence can be drawn as a graphic model
Fu Jen University Department of Electrical Engineering Wang, Yuan-Kai Copyright
王元凱 Unit - Uncertainty Inference (Discrete) p. 59
Advantages of Conditional Independence
• The use of conditional independence reduces the size of the joint distribution from O(2n) to O(n)
• Conditional independence is our most basic and robust form of knowledge about uncertain environments
Fu Jen University Department of Electrical Engineering Wang, Yuan-Kai Copyright
王元凱 Unit - Uncertainty Inference (Discrete) p. 60
Conditional Independence (Math)
• If P(Xi|Xj,X)=P(Xi|X), we say that variable Xi is conditional independent of Xj, given X– Denoted as I(Xi,Xj|X)– It means for Xi, if we know X, we can ignore
Xj• If I(Xi,Xj|X), P(Xi,Xj|X) = P(Xi|X) P(Xj|X)
– By chain rule, P(Xi,Xj|X) = P(Xi|Xj,X) p(Xj|X)– Since p(Xi|Xj,X)=p(Xi|X), – Then p(Xi,Xj|X)= p(Xi|X) p(Xj|X)
Fu Jen University Department of Electrical Engineering Wang, Yuan-Kai Copyright
王元凱 Unit - Uncertainty Inference (Discrete) p. 61
Pairwise Independence• A generalization of pairwise independence
– We say that the variables X1, …, Xn are mutually conditionally independent, given a set X
n
iin XXpXXXXp
121 )|()|,,(
When X is empty, we have p(X1,X2, …,Xn)=p(X1)p(X2)…p(Xn)
Why?),,|(),,,( 11
121 XXXPXXXP ii
n
in
By chain rule
),,,|()|,,,( 111
21 XXXXPXXXXP ii
n
in
Since Xi is conditional independent of the others given X
n
iin XXpXXXXp
121 )|()|,,(
Fu Jen University Department of Electrical Engineering Wang, Yuan-Kai Copyright
王元凱 Unit - Uncertainty Inference (Discrete) p. 62
7. Summary• The full joint distribution can
answer any query of the domain–However, it is intractable
• Independence and conditional independence is important for the reduction of the full joint distribution
• We can now move on to Bayesian networks