Answering Distance Queries in directed

graphs using fast matrix multiplication

Seminar in AlgorithmsProf. Haim Kaplan

Lecture by Lior Eldar1/07/2007

Structure of Lecture

• Introduction & History

• Alg1 – APSP

• Alg2 – preprocess & query

• Alg3 – Hybrid

• Summary

Problem Definition• Given a weighted directed graph,

we are requested to find:– APSP - All pairs shortest paths – find for any pair– SSSP - Single Source shortest paths – find all

distances from s.

• A hybrid problem comes to mind:– Preprocess the graph faster than APSP– Answer ANY two-node distance query faster than

SSSP.– What’s it good for?

),,( wEVG

Previously known results – APSP

• Undirected graphs– Approximated algorithm by Thorup and Zwick:

• Preprocess undirected weighted graph in

expected time.

• Generate data structure of size

• Answer any query in O(1)

• BUT: answer is approximate with a factor of 2k-1.

– For non-negative integer weights at most M – Shoshan and Zwick developed an algorithm of run time

kmnO /1

knO /11

376.2, MnO

• Directed graphs – Zwick - runs in 58.268.0 nMO

Previously known results - SSSP

• Positive weights:– Directed graphs with positive weights – Dijkstra

with – Undirected graphs with positive integer edge

weights – Thorup with

• Negative weights – much harder:– Bellman-Ford– Goldberg and Tarjan – assumes edge weight

values are at least – N.

nnmO log

mO

mnO

NnmO log

New Algorithm by Yuster / Zwick

• Solves the hybrid pre-processing-query problem for:– Directed graphs

– Integer weights from –M to M

• Achieves the following performance:– Pre-processing

– Query answering – O(n)

• Faster than previously known APSP (Zwick) so long as the number of queries is

• Better than SSSP performance (Goldberg&Tarjan) for dense graphs with small alphabet – gap of

376.2, MnO

158.2 MnO

13.0nO

Beyond the numbers…• An extension of this algorithm allows complete

freedom in optimization of the pre-processing - query problem.

• to optimize an algorithm for an arbitrary number of queries q, we want: preprocessing time + q * query time to be minimal.

• This defines the ratio between query time and pre-processing time - completely controlled by the algorithm inputs.

• Meaning: if we know in advance the number of queries we can fine-tune the algorithm as we wish.

Before we begin - scope

• Assumptions:– No negative cycles

• Inputs:– Directed Weighted Graph G=(V,E,w)– Weights are –M,…0,…,M

• Outputs:– Data structure such that – given any two nodes –

produces the shortest distance between them (and not the path itself) – with high probability.

Matrix Multiplication• The matrix product C=AB, where A is an

matrix, B is , and C is matrix, is defined as follows:

• Define: the minimal number of algebraic operations for computing the matrix product.

• Define as the smallest exponent such that

• Theorem by Coppersmith and Winograd:

m

kjkkiji baC

1,,,

lxmmxn lxn

),,( nmlM

),,( tsr )(),,( nOnnnM tsr

376.2

Distance Products• The distance product , where A is an

matrix, B is , and C is matrix, is defined as follows:

• Recall: if W is an n x n matrix of the edge weights of a graph then is the distance matrix of the graph.

• Lemma by Alon: can be computed almost as fast as “regular” matrix multiplication:

jkkimkji baC ,,1, min

lxmmxn lxn

nW

BAC

BAC

),,( tsrMnO

State-of-the-art APSP• Randomized algorithm by Zwick that runs in

time• Intuition:

– Computation of all short paths is intensive.– BUT: long paths are made up of short paths: once

we pay the initial price we can leverage this work to compute longer paths with less effort.

• Strategy: Giving up on certainty - with a small number of distance updates we can be almost sure that any long-enough path has at least one representative that is updated.

58.268.0 nMO

Basic Operations• Truncation

– Replace any entry larger than t with

• Selection– Extract from D the elements whose row indices are

in A, and column indices are in B.

• Min-Assignment– Assign to each element the smallest between the

two corresponding elements of D and D‘.

tD

],[ BAD

],[],[ 'min BADBAD

Pseudo-code• Simply sample nodes and multiply decimated

matrices…

endfor

VBDBVD

nnVsampleB

ntolfor

WD

nV

W

sMsM

l

nxn

],[*],[D

)2/3/()ln(9,

log 1

,...,2,1

M),shortpath( algorithm

min

2/3

On matrices and nodes…

• Column-decimated matrix ],[ BVD

D

],[ BVD

Distance between any two nodes

Shortest directed path from any node to any

node in B

On matrices and nodes…(2)

• Row-decimated matrix ],[ VBD

],[ VBD

Distance between any two nodes

Shortest directed path from any node in B to

any node

What do we prove?

• Lemma: if there is a shortest path between nodes i and j in G that uses at most edges, then after the -th iteration of the algorithm, with high probability we have

• Meaning: at each iteration we update with high probability all the paths in the graph of a certain length. This serves as a basis for the next iteration.

jiji d ,, l

l2/3

Proof Outline

• By Induction:– Base case: easy – the input W contains all paths of

length – Induction step:

• Suppose that the claim holds for and show that it also holds for

• Take any two nodes that their shortest distance is at least . The -th iteration matrix product will (almost certainly) plug in their shortest distance at location (i,j) of D.

12/3 0

1ll

12/3 l l

Why?

• Set

• The path p from i to j is at least 2s/3.

• This divides p into three subsections:– Left – at most s/3– Right – at most s/3– Middle – exactly s/3

ls 2/3

i i’ k j’ j

3/s3/s 3/s

The Details

• The left and right “thirds” - help attain the induction step.– The path p(i,k) and p(k,j) are short enough – at most 2s/3

good for previous step:

• The middle “third” – ensures the fault probability is low enough.– Prob(no k is selected) =

– Probability still goes to 0 (as n tends to infinity) after computation of

• entries• iterations

1l

3

3/1ln9

1ns

ns

2n nO log

So…• Assuming all previous steps were good enough:

– With high probability each long-enough path has a representative in B

– The update of the D using the product

plugs in the correct result.

• Note that:– Each element is first limited to s*M– This is necessary for the fast-matrix-multiplication

algorithm

],[*],[ VBDBVD

Complexity

• Where does the trick hide?– The matrix alphabet increases linearly with iteration

number

– The product size decreases with iteration number

• For each iteration :– Alphabet size: s*M

– Product complexity: , where

– Total:

• Disregarding the log function, and optimizing between fast and naïve matrix products we get:

lsl 2/3,

1,,1 r snnr /

)( 58.268.0 nMO

1,,111,,1 rrr MnnOsMnO

Fast Product versus Naive

)( 2 rnO

)( )1,,1(1 rrnO

*assuming small M

58.2

Complexity Behavior• For a given matrix alphabet M, we find the

cross-over point between the matrix algorithms.

• For high r (>M-dependent threshold) we use FMM– Complexity dependent on M

• For low r (<threshold) we use naïve multiplication– Complexity not dependent on M

• Q: How does complexity change over the iteration number?

Pre-processing algorithm

• Motivation:– We rarely query all node-pairs

• Strategy:– Replace the costly matrix product

with 2 smaller products:1.

2.

– Generate data structure such that each query costs only

)( 2nO

],[*],[ VBDBVD

],[*],[ BBDBVD

)( 2nO)(nO

],[*],[ VBDBBD

Starting with the query…

• Pseudo-code:

• What is a sufficient trait of D, such that the returned value will be, with high probability

• Answer: with high probability, a node k on the path from i to j should have:

jVDViD

Dnxn,*,return

j)i,,dist( algorithm

),( ji

),(, kid ki

),(, jkd jk

Preprocessing algorithm

endfor

VBDBBD

VBDBVD

nnBsampleB

ntolfor

WDVB

nV

W

sMsM

sMsM

l

nxn

],[*],[V]D[B,

],[*],[B]D[V,

)2/3/()ln(9,

log 1

,

,...,2,1

M),shortpath( algorithm

min

min

2/3

New matrix type• Row&Column-decimated matrix ],[ BBD

D

],[ BBD

Query data structure for any two nodes

Query data-structure for any 2 nodes in B

What do we prove?

• Lemma 4.1: If or , and there is a shortest path from i to j in G that uses at most edges, then after the -th iteration of the preprocessing algorithm, with high probability we have .

• Meaning: D has the necessary trait: for any path p, if we iterate long enough, then with high probability, for at least one node k (in p(i,j)) the entries d(i,k), d(k,j) will contain shortest paths. Hence, “query” will return the correct result.

lBi lBj l2/3

l

jijid ,,

Proof Outline - preprocess

• By Induction:– Base case: easy – B=V, and the input W contains

all paths of length . – Induction step:

• Suppose that the claim holds for and show that it also holds for

• Take any two nodes that their shortest distance is at most . The l-th iteration matrix products (2) will (almost certainly) plug in their shortest distance at location (i,j) of D provided that EITHER or

.

1)2/3( 0

1ll

l2/3

lBilBj

Why?

• Set

• The path p from i to j is at least 2s/3.

• This divides p into three subsections:– Left – at most s/3– Right – at most s/3– Middle – exactly s/3

ls 2/3

i i’ k j’ j

3/s3/s 3/s

The Details• Assume that .• With high probability ( ) there will be k in

p(i,j), such that (remember why?)• Both are also in ,since • We therefore attain the induction step:

– The path p(i,k) and p(k,j) are short enough – at most 2s/3 good for previous step.

– The end-points of these paths (k) are in – Therefore their shortest distance is in D– The second product then updates correctly.

(assumption critical here)

lBi

lBk

1lB VBBB ll 01 ...

)( 3nO

1lB

Where’s the catch?

• In APSP, we assure that:– At every iteration l we compute the shortest path

of length at most .– BUT: we had to update all pairs each time

• In the preprocess algorithm, we assure:– At every iteration l, we compute the shortest path

of length at most only for a selected subset.– BUT: this subset covers all possible subsequent

queries, with high probability.

l)2/3(

l)2/3(

2n

Complexity• Matrix product: instead of operations

we only get

• As before, for each iteration , the alphabet size is s*M.

• Total complexity:

• No matrix-product switch here!

lsl 2/3,

1,,1 r

MnOnO

MnOMnnOsMnOr

rrrrrrr

22

)1111,,11,,

1,,2 rr

Performance

• For small M, as long as the number of queries is less than we get better results than APSP.

• For small M:– The algorithm overtakes Goldberg’s algorithm, if

the graph is dense– For a dense-enough graph , we can run

many SSSP queries and still be faster:

58.1158.2 nn

88.138.2~ nOmnOnmO 88.1nm

nmnnm //

The larger picture• We saw:

– Alg1: heavy pre-processing, light query– Alg2: light pre-processing, heavy query– Alg3: ?

Query-oriented (APSP)

Preprocess- oriented (pre-process)

The Third Way• Suppose we know in advance the we require

no more than queries.• We use the following:

1. Perform iterations of the APSP algorithm

2. Perform iterations of the pre-process algorithm

3. Take the matrix B from the last step of step 1. The product returns in any shortest-distance query.

rnq

nr 5.1log1

nr 5.1log

rnO }{,},{ jBDBiD

Huh?

• After the first stage D holds all the shortest path of all “short” paths, of lengths at most with high probability.

• When the second starts stage it can be sure that the induction holds for all

• The second stage takes care of the “long” paths, with respect to querying. Meaning:– If the path is long it will have a representative in

one of the second-phase iterations– If it is too-short – it will fall under the jurisdiction

of the first stage.

rn 1

nrl 5.1log1

Complexity

• The first stage ( updates) costs at most

• The second stage costs only

• The query costs

• For example – if want to answer a distance query in , we can pre-process in time

2n 1,,11 rrMnO

1,,1 rrrMnO

rnO

4/3nO 5.2MnO

Q&A (I ask - you answer)

• Q: Why couldn’t we sample B in the query step of Alg2 – the one that initially costs O(n)?

• A: Because if the path is too short – we will have no guarantee that it will have a representative in B. Alg3 solves this because short distances are computed rigorously.

• Conclusion: the less we sample out of V when we query, the more steps we need to run APSP to begin with.

Final Procedure

• Given q queries, determine the query complexity using .

• This assumes M is small enough so that we use fast product. Otherwise compare to

• Execute alg3 using steps of APSP and steps of pre-process

• Query all q queries.

1,,11 rrr Mnqn

qnr rn 2

nr 5.1log1 nr 5.1log

Summary

• For the problem we defined: directed graph, with integer weights, whose absolute value is at most M, we have seen:– Alg1: State-of-the-art APSP in– Alg2: State-of-the-art SSSP in– Alg3: A method to calibrate between the two, for a

known number of queries.

58.268.0 nMO

38.2MnO

Thank You!