Network Lasso: Clustering and

Optimization in Large GraphsDavid Hallac, Jure Leskovec, Stephen Boyd

Stanford University

Presented by Yu Zhao

What is this paper about

Lasso problem

The lasso solution is unique when rank(X) = p, because the criterion is strictly convex.

What is this paper about

Network lasso problem

The variables are , where . (The total number of scalar variables is mp.) Here is the variable at node i, is the cost function at node i, and is the cost function associated with edge .

Convex problem definition Proposed solution(ADMM) Non-convex extension Experiments

Outline

Convex problem definition

(1)

(2)

Convex problem definition

A distributed and scalable method was developed for solving the network lasso problem, in which each vertex variable xi is controlled by one “agent”, and the agents exchange (small) messages over the graph to solve the problem iteratively.

Convex problem definition

General settings for different applications

e.g. Control system: Nodes: possible states xi: actions to take when state i Graph: state transitions Weights: how much we care about the

actions in neighboring states differing

Convex problem definition

General settings for different applications

The sum-of-norms regularization that we use is like group lasso, which encourages not just , for edge , but , consensus across the edge.

Convex problem definition

Regularization Path simply a minimizer of local

computations ():

Convex problem definition

Network lasso and clustering -norms penalty defines network lasso. Cluster size:

Convex problem definition

Inference on New Nodes we can interpolate the solution to estimate

the value of on a new node .

Proposed solution(ADMM)

Alternating Direction Method of Multipliers(ADMM)

S. Boyd, N. Parikh, E. Chu, B. Peleato, and J. Eckstein. Distributed optimization and statistical learning via the alternating direction method of multipliers. Foundations and Trends in Machine Learning, 3:1–122, 2011.

Proposed solution(ADMM)

ADMM in network lasso 1). Introduce a copy of , called , at each

edge .

Proposed solution(ADMM)

ADMM in network lasso 2). Augmented Lagrangian

M. R. Hestenes. Multiplier and gradient methods. Journal of Optimization Theory and Applications, 4:302–320, 1969.

Proposed solution(ADMM)

ADMM in network lasso 3). ADMM updates

Proposed solution(ADMM)

Regularization Path compute the regularization path as a

function of to gain insight into the network structure

Start at update: stop:

Non-convex extension

replace the group lasso penalty with a monotonically nondecreasing concave function , where , and whose domain is u,

ADMM is not guaranteed to converge, and even if it does, it need not be to a global optimum

Non-convex extension

Heuristic solution: to keep track of the iteration which yields the minimum objective, and to return that as the solution instead of the most recent step.

Non-convex extension

Non-convex z-Update Compared to the convex case, the only

difference in the ADMM solution is the z-update, which is now

Experiments

1. Network-Enhanced Classification We first analyze a synthetic network in which

each node has a support vector machine (SVM) classifier,

but does not have enough training data to accurately estimate it

Idea: “borrow” training examples from their relevant

neighbors to improve their own results neighbors with different underlying models has

non-zero lasso penalties

Experiments

1. Network-Enhanced Classification Dataset:

randomly generate a dataset containing 1000 nodes, each with its own classifier, a support vector machine in R50. Each node tries to predict , where

Network: The 1000 nodes are split into 20 equally-sized

groups. Each group has a common underlying classifier while different groups have independent models.

Experiments

1. Network-Enhanced Classification Objective function:

Experiments

1. Network-Enhanced Classification Results(regularization path):

Experiments

1. Network-Enhanced Classification Results(prediction accuracy):

Experiments

1. Network-Enhanced Classification Results(timing):

Convergence comparison between centralized and ADMM methods for SVM problem

Experiments

1. Network-Enhanced Classification Results(timing):

Convergence time for large-scale 3-regular graph solved at a single (constant) value of

Experiments

2. spatial clustering and regressors Attempt to estimate the price of homes

based on latitude/longitude data and a set of features.

Experiments

2. spatial clustering and regressors Dataset:

a list of real estate transactions over a oneweek period in May 2008 in the Greater Sacramento area.

Network: build the graph by using the latitude/longitude

coordinates of each house connect every remaining house to the five nearest

homes with an edge weight inversely proportional to the distance between the houses

785 nodes, 2447 edges, and has a diameter of 61.

Experiments

2. spatial clustering and regressors Optimization Parameter and Objective

Function: At each nodes, solve for

Objective function:

Experiments

2. spatial clustering and regressors Results:

Experiments

2. spatial clustering and regressors Results:

Conclusion

The network lasso is a useful way of representing convex optimization problems, and the magnitude of the improvements in the experiments show that this approach is worth exploring further, as there are many potential ideas to build on.

The non-convex method gave comparable performance to the convex approach, and we leave for future work the analysis of different non-convex functions

we could attempt to iteratively reweigh the edge weights to attain some desired outcome

Within the ADMM algorithm, there are many ways to improve speed, performance, and robustness

Questions?