cs b 553: a lgorithms for o ptimization and l earning structure learning
TRANSCRIPT
CS B553: ALGORITHMS FOR OPTIMIZATION AND LEARNINGStructure Learning
AGENDA
Learning probability distributions from example data
To what extent can Bayes net structure be learned?
Constraint methods (inferring conditional independence)
Scoring methods (learning => optimization)
BASIC QUESTION
Given examples drawn from a distribution P* with independence relations given by the Bayesian structure G*, can we recover G*?
BASIC QUESTION
Given examples drawn from a distribution P* with independence relations given by the Bayesian structure G*, can we recover G*
construct a network that encodes the same independence relations as G*?
G* G1
G2
LEARNING IN THE FACE OF NOISY DATA
Ex: flip two independent coins Dataset of 20 flips: 3 HH, 6 HT, 5 TH, 6 TT
X Y
Model 1
X Y
Model 2
LEARNING IN THE FACE OF NOISY DATA
Ex: flip two independent coins Dataset of 20 flips: 3 HH, 6 HT, 5 TH, 6 TT
X Y
Model 1
X Y
Model 2
ML parameters
P(X=H) = 9/20P(Y=H) = 8/20
P(X=H) = 9/20P(Y=H|X=H) = 3/9P(Y=H|X=T) = 5/11
LEARNING IN THE FACE OF NOISY DATA
Ex: flip two independent coins Dataset of 20 flips: 3 HH, 6 HT, 5 TH, 6 TT
X Y
Model 1
X Y
Model 2
ML parameters
P(X=H) = 9/20P(Y=H) = 8/20
P(X=H) = 9/20P(Y=H|X=H) = 3/9P(Y=H|X=T) = 5/11 Errors are
likely to be larger!
PRINCIPLE
Learning structure must trade off fit of data vs. complexity of network
Complex networks More parameters to learn More data fragmentation = greater sensitivity to
noise
APPROACH #1: CONSTRAINT-BASED LEARNING
First, identify an undirected skeleton of edges in G*
If an edge X-Y is in G*, then no subset of evidence variables can make X and Y independent
If X-Y is not in G*, then we can find evidence variables to make X and Y independent
Then, assign directionality to preserve independences
BUILD-SKELETON ALGORITHM
Given X={X1,…,Xn}, query Independent?(X,Y,U)
H = complete graph over X For all pairs Xi, Xj, test separation as follows:
Enumerate all possible separating sets U If Independent?(Xi,Xj,U) then remove Xi—Xj from
H
In practice:• Must restrict to bounded size subsets |U|d (i.e.,
assume G* has bounded degree). O(n2(n-2)d) tests• Independence can’t be tested exactly
ASSIGNING DIRECTIONALITY
Note that V-structures XYZ introduce a dependency between X and Z given Y In structures XYZ, XYZ, and XYZ, X and Z are
independent given Y In fact Y must be given for X and Z to be
independent Idea: look at separating sets for all triples X-Y-
Z in the skeleton without edge X-Z
Y
X Z
Triangle
Directionality is irrelevant
ASSIGNING DIRECTIONALITY
Note that V-structures XYZ introduce a dependency between X and Z given Y In structures XYZ, XYZ, and XYZ, X and Z are
independent given Y In fact Y must be given for X and Z to be
independent Idea: look at separating sets for all triples X-Y-
Z in the skeleton without edge X-Z
Y
X Z
Triangle
Y
X Z
Y separates X, Z
Not a v-structureDirectionality is irrelevant
ASSIGNING DIRECTIONALITY
Note that V-structures XYZ introduce a dependency between X and Z given Y In structures XYZ, XYZ, and XYZ, X and Z are
independent given Y In fact Y must be given for X and Z to be
independent Idea: look at separating sets for all triples X-Y-
Z in the skeleton without edge X-Z
Y
X Z
Triangle
Y
X Z
Y separates X, Z
Not a v-structure
Y
X Z
YU separates X, Z
A v-structureDirectionality is irrelevant
ASSIGNING DIRECTIONALITY
Note that V-structures XYZ introduce a dependency between X and Z given Y In structures XYZ, XYZ, and XYZ, X and Z are
independent given Y In fact Y must be given for X and Z to be
independent Idea: look at separating sets for all triples X-Y-
Z in the skeleton without edge X-Z
Y
X Z
Triangle
Y
X Z
Y separates X, Z
Not a v-structure
Y
X Z
YU separates X, Z
A v-structureDirectionality is irrelevant
STATISTICAL INDEPENDENCE TESTING
Question: are X and Y independent? Null hypothesis H0: X and Y are independent
Alternative hypothesis HA: X and Y are not independent
STATISTICAL INDEPENDENCE TESTING
Question: are X and Y independent? Null hypothesis H0: X and Y are independent
Alternative hypothesis HA: X and Y are not independent
2 test: use the statistic
withthe empirical probability of X Can compute (table lookup) the probability of
getting a value at least this extreme if H0 is true (p-value)
If p < some threshold, e.g. 1-0.95, H0 is rejected
APPROACH #2: SCORE-BASED METHODS
Learning => optimization Define scoring function Score(G;D) that
evaluates quality of structure G, and optimize it Combinatorial optimization problem
Issues: Choice of scoring function: maximum likelihood
score, Bayesian score Efficient optimization techniques
MAXIMUM-LIKELIHOOD SCORES
ScoreL(G;D) = likelihood of the BN with the most likely parameter settings under structure G Let L(G,G;D) be the likelihood of data using
parameters G with structure G Let G
* = arg max L(,G;D) as described in last lecture
Then ScoreL(G;D) = L(G*,G;D)
ISSUE WITH ML SCORE
ISSUE WITH ML SCORE
Independent coin example
X Y
G1
X Y
G2
ML parameters
P(X=H) = 9/20P(Y=H) = 8/20
P(X=H) = 9/20P(Y=H|X=H) = 3/9P(Y=H|X=T) = 5/11
Likelihood score
log L(G1*,G1;D)= 9 log(9/20) + 11 log(11/20) + 8 log (8/20) + 12 log (12/20)
log L(G2*,G2;D)= 9 log(9/20) + 11 log(11/20) + 3 log (3/9) + 6 log (6/9) + 5 log (5/11) + 6 log(6/11)
ISSUE WITH ML SCORE
X Y
G1
X Y
G2
Likelihood score
log L(G1*,G1;D)-log L(G2*,G2;D) = 8 log (8/20) + 12 log (12/20) – [3 log (3/9) + 6 log (6/9) + 5 log (5/11) + 6 log(6/11)]
ISSUE WITH ML SCORE
X Y
G1
X Y
G2
Likelihood score
log L(G1*,G1;D)-log L(G2*,G2;D) = 8 log (8/20) + 12 log (12/20) – 8 [3/8 log (3/9) + 5/8 log (5/11) ] – 12 [6/12 log (6/9) + 6/12 log(6/11)]
ISSUE WITH ML SCORE
X Y
G1
X Y
G2
Likelihood score
log L(G1*,G1;D)-log L(G2*,G2;D) = 8 log (8/20) + 12 log (12/20) – 8 [3/8 log (3/9) + 5/8 log (5/11) ] – 12 [6/12 log (6/9) + 6/12 log(6/11)] = 8 [log (8/20) - 3/8 log (3/9) + 5/8 log (5/11) ] + 12 [log (12/20) - 6/12 log (6/9) + 6/12 log(6/11)]
ISSUE WITH ML SCORE
X Y
G1
X Y
G2
Likelihood score
log L(G1*,G1;D)-log L(G2*,G2;D) = 8 log (8/20) + 12 log (12/20) – 8 [3/8 log (3/9) + 5/8 log (5/11) ] – 12 [6/12 log (6/9) + 6/12 log(6/11)] = 8 [log (8/20) - 3/8 log (3/9) + 5/8 log (5/11) ] + 12 [log (12/20) - 6/12 log (6/9) + 6/12 log(6/11)]=
ISSUE WITH ML SCORE
X Y
G1
X Y
G2
Likelihood score
log L(G1*,G1;D)-log L(G2*,G2;D) = 8 log (8/20) + 12 log (12/20) – 8 [3/8 log (3/9) + 5/8 log (5/11) ] – 12 [6/12 log (6/9) + 6/12 log(6/11)] = 8 [log (8/20) - 3/8 log (3/9) + 5/8 log (5/11) ] + 12 [log (12/20) - 6/12 log (6/9) + 6/12 log(6/11)]= =
ISSUE WITH ML SCORE
X Y
G1
X Y
G2
Likelihood score
log L(G1*,G1;D)-log L(G2*,G2;D) = 8 log (8/20) + 12 log (12/20) – 8 [3/8 log (3/9) + 5/8 log (5/11) ] – 12 [6/12 log (6/9) + 6/12 log(6/11)] = 8 [log (8/20) - 3/8 log (3/9) + 5/8 log (5/11) ] + 12 [log (12/20) - 6/12 log (6/9) + 6/12 log(6/11)]= =
MUTUAL INFORMATION PROPERTIES
(the mutual information between X and Y)
with Q(x,y) = P(x)P(y)
0 by nonnegativity of KL divergence
Implication: ML scores do not decrease for more connected
graphs=> Overfitting to data!
POSSIBLE SOLUTIONS
Fix complexity of graphs (e.g., bounded in-degree) See HW7
Penalize complex graphs Bayesian scores
IDEA OF BAYESIAN SCORING
Note that parameters are uncertain Bayesian approach: put a prior on parameter
values and marginalize them out P(D|G) =
For example, use Beta/Dirichlet priors => marginal is manageable to compute E.g., uniform hyperparameter over network Set virtual counts to 2^-|PaXi|
LARGE SAMPLE APPROXIMATION
log P(D|G) =log L(G
*;D) – ½ log M Dim[G] + O(1) With M the number of samples, Dim[G] the
number of free parameters of G Bayesian Information Criterion (BIC) score:
ScoreBIC(G;D) = log L (G*;D) – ½ log M Dim[G]
LARGE SAMPLE APPROXIMATION
log P(D|G) =log L(G
*;D) – ½ log M Dim[G] + O(1) With M the number of samples, Dim[G] the
number of free parameters of G Bayesian Information Criterion (BIC) score:
ScoreBIC(G;D) = log L (G*;D) – ½ log M Dim[G]
Fit data set Prefer simple models
STRUCTURE OPTIMIZATION, GIVEN A SCORE…
The problem is well-defined, but combinatorially complex! Superexponential in # of variables
Idea: search locally through the space of graphs using graph operators Add edge Delete edge Reverse edge
SEARCH STRATEGIES
Greedy Pick operator that leads to greatest score Local minima? Plateaux?
Overcoming plateaux Search with basin flooding Tabu search Perturbation methods (similar to simulated
annealing, except on data weighting) Implementation details:
Evaluate ’s between structures quickly (local decomposibility)
RECAP
Bayes net structure learning: from equivalence class of networks that encode the same conditional independences
Constraint-based methods Statistical independence tests
Score-based methods Learning => optimization