Quantifying AlgorithmicImprovements over Time

Lars KotthoffUniversity of Wyoming

larsko@uwyo.edu1

IJCAI, 17 July 2018

1joint work with Alexandre Fréchette, Tomasz Michalak, Talal Rahwan,Holger H. Hoos, Kevin Leyton-Brown

Contributions – Standalone Performance798602199798501630798470169798466233798461169798360514794178118784476788

671833

dual pivot (2009)

median 9 (1993)

median 9 random (1993)

mid (1978)

median 3 random (1978)

random (1961)

median 3 (1978)

first (1961)

insertion (1946)

dual pivot (2009)

median 9 (1993)

median 9 random (1993)

mid (1978)

median 3 random (1978)

random (1961)

median 3 (1978)

first (1961)

insertion (1946)

Standalone Performance 1

Contributions – Marginal Performance

How much does an algorithm contribute to the state of the art(defined by a coalition of all other algorithms)?

ϕi = v(Ci ∪ {i})− v(Ci)

Xu, Hutter, Hoos, Leyton-Brown. “Evaluating Component SolverContributions to Portfolio-Based Algorithm Selectors.” SAT 2012

2

Contributions – Marginal Performance798602199798501630798470169798466233798461169798360514794178118784476788

671833

98900185531000

dual pivot (2009)

median 9 (1993)

median 9 random (1993)

mid (1978)

median 3 random (1978)

random (1961)

median 3 (1978)

first (1961)

insertion (1946)

dual pivot (2009)

median 9 (1993)

median 3 random (1978)

median 9 random (1993)

mid (1978)

median 3 (1978)

first (1961)

insertion (1946)

random (1961)

Standalone Performance Marginal Performance

3

Contributions – Temporal Marginal Performance

How much does an algorithm contribute to the state of the art(defined by a coalition of all other algorithms available at thetime)?

ϕ≻i = v≻(Ci ∪ {i})− v≻(Ci)

where ≻ is a relation that encodes temporal precedence.

4

Contributions – Temporal Marginal Performance798602199798501630798470169798466233798461169798360514794178118784476788

671833

132120306718339890020497157036907552541137

dual pivot (2009)

median 9 (1993)

median 9 random (1993)

mid (1978)

median 3 random (1978)

random (1961)

median 3 (1978)

first (1961)

insertion (1946)

random (1961)

insertion (1946)

dual pivot (2009)

median 9 (1993)

mid (1978)

median 3 random (1978)

median 3 (1978)

median 9 random (1993)

first (1961)

Standalone Performance Temporal Marginal Performance 5

Shapley Value

How much does an algorithm contribute to all possible coalitionsof other algorithms?

ϕi =1

|Π|∑π∈ΠN

v(Cπi ∪ {i})− v(Cπ

i )

We can compute this in polynomial time.

Shapley. “A Value for n-person Games.” In Contributions to the Theory ofGames, 1953.Fréchette, Alexandre, Lars Kotthoff, Talal Rahwan, Holger H. Hoos, KevinLeyton-Brown, and Tomasz P. Michalak. “Using the Shapley Value to AnalyzeAlgorithm Portfolios.” In 30th AAAI Conference on Artificial Intelligence, 2016.

6

Shapley Value

How much does an algorithm contribute to all possible coalitionsof other algorithms?

ϕi =1

|Π|∑π∈ΠN

v(Cπi ∪ {i})− v(Cπ

i )

We can compute this in polynomial time.

Shapley. “A Value for n-person Games.” In Contributions to the Theory ofGames, 1953.Fréchette, Alexandre, Lars Kotthoff, Talal Rahwan, Holger H. Hoos, KevinLeyton-Brown, and Tomasz P. Michalak. “Using the Shapley Value to AnalyzeAlgorithm Portfolios.” In 30th AAAI Conference on Artificial Intelligence, 2016.

6

Contributions – Shapley Value798602199798501630798470169798466233798461169798360514794178118784476788

671833

1002670581001674121001537151001533841001510971001311869943466298059604

84173

98900185531000

dual pivot (2009)

median 9 (1993)

median 9 random (1993)

mid (1978)

median 3 random (1978)

random (1961)

median 3 (1978)

first (1961)

insertion (1946)

dual pivot (2009)

median 9 (1993)

median 3 random (1978)

median 9 random (1993)

mid (1978)

median 3 (1978)

first (1961)

insertion (1946)

random (1961)

Standalone Performance Shapley Value Marginal Performance 7

Temporal Shapley Value

How much does an algorithm contribute to all possible coalitionsof other algorithms, taking temporal precedence into account?

ϕ≻i =

1

|Π≻|∑π∈Π≻

v≻(Cπi ∪ {i})− v≻(Cπ

i )

We can compute this in polynomial time as well.

Kotthoff, Lars, Alexandre Fréchette, Tomasz P. Michalak, Talal Rahwan,Holger H. Hoos, and Kevin Leyton-Brown. “Quantifying AlgorithmicImprovements over Time.” In 27th International Joint Conference on ArtificialIntelligence (IJCAI) Special Track on the Evolution of the Contours of AI,2018.

8

Contributions – Temporal Shapley Value798602199798501630798470169798466233798461169798360514794178118784476788

671833

100267058100167412100153715100153384100151097100131186

9943466298059604

84173

405450356392238462

67183398900571985041122506100742550

132120306718339890020497157036907552541137

dual pivot (2009)

median 9 (1993)

median 9 random (1993)

mid (1978)

median 3 random (1978)

random (1961)

median 3 (1978)

first (1961)

insertion (1946)

random (1961)

insertion (1946)

dual pivot (2009)

median 9 (1993)

mid (1978)

median 3 random (1978)

median 3 (1978)

median 9 random (1993)

first (1961)

Standalone PerformanceShapley Value

Temporal Shapley ValueTemporal Marginal Performance

9

Quicksort Over Time

1e+02

1e+05

1e+08

1946 1961 1978 1993 2009

year

Sum

of t

empo

ral S

hapl

ey v

alue

s

10

SAT Competition●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

144.25268

144.08601

139.91915

57.41773

56.33454

51.75097

44.43418

43.43408

35.08748

28.8338

27.08395

19.16712

19.13952

16.3337

12.43352

10.19402

9.69765

9.0321

8.30224

8.0001

7.75017

7.03675

6.64388

4.87001

4.50018

4.26672

4.26672

3.91678

2.00002

2.00001

1.83337

1.16668

0.83336

4e−05

0

0

0

0

78.42398

63.90765

55.75744

55.15192

52.43065

50.72959

45.33413

44.61692

44.50427

36.26344

31.60638

30.53198

30.41135

28.4814

25.76449

21.82523

20.7125

20.49854

20.15654

19.71357

18.71084

17.82205

16.86642

16.31361

15.36641

14.86641

14.18306

13.18303

9.83857

8.74145

4.81501

3.56499

1.90815

1.88997

0.53389

0.45055

0.14286

0

gnovelty+_2007ranov_2007

adaptnovelty_2007TNM_2009

sparrow2011_2011sapsrt_2007

March−KS_2007KCNFS_2007

dimetheus_2.100_2014hybridGM3_2009

iPAWS_2009sattime2011_2011

BalancedZ_2014adaptg2wsat2011_2011

DEWSATZ−1A_2007CSCCSat2014_SC2014_2014

CCgscore_2014probSAT_sc14_2014

Ncca+_v1.05_2014CSHCrandMC_2013

gnovelty+2_2009YalSAT_03l_2014

CCA2014_2.0_2014sattime_2014

MPhaseSAT_M−2011−02−16_2011minisat−SAT_2007

MXC_2007gNovelty+−T_2009

csls−pnorm−8cores_2011march_br_sat+unsat_2013

march_rw−2011−03−02_2011MiraXT−v3_2007

march_hi_2011gNovelty+GCwa_1.0_2013

minipure_1.0.1_2013Solver43a_a_2013Solver43b_b_2013

strangenight_satcomp11−st_2013

dimetheus_2.100_2014BalancedZ_2014CCgscore_2014CSCCSat2014_SC2014_2014probSAT_sc14_2014Ncca+_v1.05_2014CCA2014_2.0_2014sattime_2014YalSAT_03l_2014sparrow2011_2011TNM_2009sattime2011_2011adaptg2wsat2011_2011MPhaseSAT_M−2011−02−16_2011CSHCrandMC_2013ranov_2007iPAWS_2009gnovelty+_2007adaptnovelty_2007hybridGM3_2009gnovelty+2_2009gNovelty+−T_2009march_br_sat+unsat_2013gNovelty+GCwa_1.0_2013march_rw−2011−03−02_2011march_hi_2011March−KS_2007KCNFS_2007csls−pnorm−8cores_2011sapsrt_2007DEWSATZ−1A_2007minipure_1.0.1_2013MXC_2007minisat−SAT_2007MiraXT−v3_2007Solver43a_a_2013Solver43b_b_2013strangenight_satcomp11−st_2013

Temporal Shapley Value Shapley Value11

SAT Competition Over Time

0

200

400

600

2007 2009 2011 2013 2014

year

Sum

of t

empo

ral S

hapl

ey V

alue

s

12

MiniZinc (CP) Competition

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

9987.86155

360.7717

1261.4996

29828.71058

4376.2985

922.81764

49647.2861

0

170.56476

21435.6316

5163.9305

197.036

16878.17865

12376.60633

193.93191

4775.12936

17831.07372

14097.37673

8886.77925

91361.24195

150.02742

20393.03799

192.78811

3267.92072

12513.1466

9828.08541

32189.3621534970.99061

7059.47499

15552.43907

6190.81406

7087.25379

6756.47091

13073.5682114389.56267

5262.55503

15095.99322

16977.5178

36090.70992

6071.40394

18324.39121

15746.3306815810.6008

52.195710

6853.37461

11324.41791

Choco_2014

Choco_2016

Choco3_2015

Chuffed_2015

Chuffed_2016

Concrete_2016

G12Chuffed_2014

G12FD_2015

G12FD_2016

Gecode_2014

Gecode_2015

Gecode_2016

JaCoP_2014

JaCoP_2015

JaCoP_2016

LCG−Glucose_2016

OpturionCPX_2014

OpturionCPX_2015

OR−Tools_2015

ORTools_2014

PicatCP_2014

PicatCP_2016

SICStus_2014

SICStus_2016

Choco_2014

Choco_2016

Choco3_2015

Chuffed_2015Chuffed_2016

Concrete_2016

G12Chuffed_2014

G12FD_2015

G12FD_2016

Gecode_2014

Gecode_2015Gecode_2016

JaCoP_2014

JaCoP_2015

JaCoP_2016

LCG−Glucose_2016

OpturionCPX_2014

OpturionCPX_2015

OR−Tools_2015ORTools_2014

PicatCP_2014PicatCP_2016

SICStus_2014

SICStus_2016

Temporal Shapley Value Shapley Value13

MiniZinc Competition Over Time

0

50000

100000

150000

200000

2014 2015 2016

year

Sum

of t

empo

ral S

hapl

ey V

alue

s

14

Summary

▷ standalone performance does not indicate how algorithmscomplement each other

▷ marginal performance is not fair▷ Shapley Value

▷ provides better characterization of algorithms’ performance▷ rewards algorithms that introduce novel and complementary

concepts▷ enables better analysis of algorithms’ performance

▷ Temporal Shapley Value▷ takes when an algorithm was conceived into account▷ all desirable properties of Shapley Value▷ rewards earlier algorithms, which may have inspired later

algorithms

15

I’m hiring!

Several funded graduate positions available.

16