f ixing m ax -p roduct : a u nified l ook at m essage p assing a lgorithms nicholas ruozzi and...
Post on 19-Dec-2015
221 views
TRANSCRIPT
FIXING MAX-PRODUCT: A UNIFIED LOOK AT MESSAGE PASSING ALGORITHMS
Nicholas Ruozzi and Sekhar Tatikonda
Yale University
PREVIOUS WORK
Recent work related to max-product has focused on convergent and correct message passing schemes:
TRMP [Wainwright et al. 2005]
MPLP [Globerson and Jaakkola 2007]
Max-Sum Diffusion [Werner 2007]
“Splitting” Max-product [Ruozzi and Tatikonda 2010]
Typical approach: focus on a dual formulation of the MAP LP
Message passing scheme is derived as a coordinate ascent scheme on a concave dual:
MPLP TRMP Max-Sum Diffusion
PREVIOUS WORK
MAP MAP LPConcave
Dual
Many of these algorithms can be seen as maximizing a specific lower bound [Sontag and Jaakkola 2009]
The maximization is performed over reparameterizations of the objective function that satisfy specific constraints
Different constraints correspond to different dual formulations
PREVIOUS WORK
THIS WORK
Focus on the primal problem:
Choose a reparameterization of the objective function
Reparameterization in terms of messages
Construct concave lower bounds from this reparameterization by exploiting concavity of min
Perform coordinate ascent on these lower bounds
MAPReparamet-erization
Concave Lower Bound
THIS WORK
Many of the common message passing schemes can be captured by the “splitting” family of reparameterizations
Many possible lower bounds of interest
Produces an unconstrained concave optimization problem
MAPReparamet-erization
Concave Lower Bound
OUTLINE
Background Min-sum Reparameterizations
“Splitting” Reparameterization Lower bounds
Message Updates
MIN-SUM
Minimize an objective function that factorizes as a sum of potentials (assume f is bounded from below)
(some multiset whose elements are subsets of the variables)
REPARAMETERIZATIONS
We can rewrite the objective function as
This does not change the objective function as long as the messages are finite valued at each x
The objective function is reparameterized in terms of the messages
No dependence on messages passed from i to ®
BELIEFS
Typically, we express the reparameterization in terms of beliefs (meant to represent min-marginals):
With this definition, we have:
MIN-SUM
The min-sum algorithm updates ensure that, after updating m®i,
In other words,
Can estimate an assignment from a collection of messages by choosing
Upon convergence,
CORRECTNESS GUARANTEES
The min-sum algorithm does not guarantee the correctness of this estimate upon convergence
Assignments that minimize bi need not minimize f:
Notable exceptions: trees, single cycles, singly connected graphs
LOWER BOUNDS
Can derive lower bounds that are concave in the messages from reparameterizations:
Lower bound is a concave function of the messages (and beliefs)
We want to find the choice of messages that maximizes the lower bound
This lower bound may not be tight
OUTLINE
Background Min-sum Reparameterizations
“Splitting” Reparameterization Lower bounds
Message Updates
“GOOD” REPARAMETERIZATIONS
Many possible reparameterizations
How do we choose reparameterizations that produce “nice” lower bounds?
Want estimates corresponding to the optimal choice of messages to minimize the objective function
Want the bound to be concave in the messages
Want the coordinate ascent scheme to remain local
“SPLITTING” REPARAMETERIZATION
TRMP:
is a collection of spanning trees in the factor graph and ¹ is a probability distribution on spanning trees
Choose c® = ¹®
Can extend this to a collection of singly connected subgraphs [Ruozzi and Tatikonda 2010]
“SPLITTING” REPARAMETERIZATION
Min-sum, TRMP, MPLP, and Max-Sum Diffusion can all be characterized as splitting reparameterizations
One possible lower bound:
We could choose c such that f can be written as a nonnegative combination of the beliefs
OUTLINE
Background Min-sum Reparameterizations
“Splitting” Reparameterization Lower bounds
Message Updates
FROM LOWER BOUNDS TO MESSAGE UPDATES
We can construct the message updates by ensuring that we perform coordinate ascent on our lower bounds
Can perform block updates over trees [Meltzer et al. 2009] [Kolmogorov 2009] [Sontag and Jaakkola 2009]
Key observation:
Equality iff there is an x that simultaneously minimizes both functions
SPLITTING UPDATE
Suppose all coefficients are positive and ci > 0
We want
Solving for m®i gives
Do this for all ®2 i
CONCLUSION MPLP, TRMP, and Max-Sum Diffusion are all instances of
the splitting reparameterization for specific choices of the constants and lower bound
Different lower bounds produce different unconstrained concave optimization problems
Choice of lower bound corresponds to choosing different dual formulations
Maximization is performed with respect to the messages, not the beliefs
Many more reparameterizations and lower bounds are possible
Is there a reparameterization in which the lower bounds are strictly concave?