J SchedDOI 10.1007/s10951-014-0402-0

The Multi-Mode Resource-Constrained Multi-Project SchedulingProblemThe MISTA 2013 challenge

Tony Wauters · Joris Kinable · Pieter Smet ·Wim Vancroonenburg · Greet Vanden Berghe ·Jannes Verstichel

Received: 30 November 2013 / Accepted: 28 October 2014© Springer Science+Business Media New York 2014

Abstract Scheduling projects is a difficult and time con-suming process, and has far-reaching implications for anyorganization’s operations. By generalizing various aspectsof project scheduling, decision makers are enabled to cap-ture reality and act accordingly. In the context of the MISTA2013 conference, the first MISTA challenge, organized bythe authors, introduced such a general problem model: theMulti-Mode Resource-Constrained Multi-Project Schedul-ing Problem (MRCMPSP). The present paper reports on thecompetition and provides a discussion on its results. Further-more, it provides an analysis of the submitted algorithms,and a study of their common elements. By making all bench-mark datasets and results publicly available, further researchon the MRCMPSP is stimulated.

Keywords Project scheduling · Multi-mode · Multi-project · Heuristics · Competition

1 Introduction

Project scheduling is well known in operations research andmanagement, and has been the subject of intensive researchsince the late fifties. It has seen many practical applications,ranging from high-level scheduling of software projects andconstruction projects to fine-grained scheduling of machineoperations on a production floor [e.g. scheduling orders in afood production facility (Wauters et al. 2012)]. However, the

T. Wauters (B) · J. Kinable · P. Smet · W. Vancroonenburg ·G. Vanden Berghe · J. VerstichelDepartment of Computer Science, CODeS-iMinds-ITEC,KU Leuven, Gebroeders De Smetstraat 1, 9000 Gent, Belgiume-mail: tony.wauters@cs.kuleuven.be

classical Resource-Constrained Project Scheduling Problem(RCPSP) is not general enough to model all aspects of a real-world problem. Therefore, many extensions to the RCPSPhave been presented in the literature, such as min/max timelags, multi-mode, multi-skill, etc.

The MISTA 2013 challenge, organized along with theMISTA 2013 conference, tries to encourage research on anew general project scheduling problem: the Multi-ModeResource-Constrained Multi-Project Scheduling Problem(MRCMPSP). Multiple projects have to be scheduled simul-taneously, while taking into account the availability of localand global resources. The MRCMPSP has a considerablepractical relevance, especially in the construction and pro-duction sectors. The challenge results in a new benchmarkfor the MRCMPSP by releasing public instances, a solutionvalidator, an instance generator and state-of-the-art results.The MRCMPSP is attractive for its simple problem descrip-tion and the lack of complicated constraints to express andevaluate. It therefore requires relatively limited modellingeffort, which enables competitors and future researchers tofocus on developing powerful algorithms and on the intrin-sic scheduling complexity. The MISTA 2013 challenge is, asfar as the authors know, the first academic project schedulingcompetition. A large international audience has been reached(Fig. 1): 21 teams from 14 countries and 3 continents sub-mitted algorithms. Teams were compared on a benchmarkPC with a fixed time limit.

Challenges/competitions are important for stimulatingresearch on particular problems or problem domains. Exam-ples of such challenges are the international timetabling com-petitions (ITC 2002, ITC 2007, ITC 2011) (McCollum etal. 2007; Post et al. 2013), the nurse rostering competition(Haspeslagh et al. 2012), the ROADEF challenge series (Sol-non et al. 2008) and the cross-domain heuristic search chal-lenge (CHeSC 2011) (Burke et al. 2011).

123

J Sched

111 713713713

Fig. 1 World map of MISTA challenge website visits by country (image courtesy of Google Analytics)

The present paper is structured as follows: Section 2 intro-duces the MRCPSP and gives a formal definition of the prob-lem, its constraints and objectives. Section 3 describes thecompetition format, the different phases, test setup and finalranking. Section 4 provides a detailed analysis and discus-sion of the competition results and submitted algorithms. Aconclusion and future perspectives are given in Sect. 5.

2 Problem description

The MRCMPSP is a generalization of the (single-mode)RCPSP in two ways (Fig. 2). First, jobs can be executed

Fig. 2 RCPSP generalization hierarchy

in multiple modes (MRCPSP), and second, multiple projectshave to be scheduled simultaneously while sharing resources(RCMPSP). These modes allow including time/cost, time/re-source and resource/resource trade-offs. The RCPSP wasproven to be NP-hard (Blazewicz et al. 1983), consequentlythe MRCMPSP is also NP-hard. Moreover, the problem offinding a feasible mode assignment subject to more than onenon-renewable resource constraint is NP-complete (Kolischand Drexl 1997).

2.1 Projects and activities

A set P of n projects has to be scheduled, under the restric-tion of several time and resource constraints. The projects areidentified by their index i ∈ P , with P = {0, . . . , n − 1}.Each project i ∈ P consists of a set of non-preemptive jobsor activities Ji . For each activity j ∈ {1, . . . , |Ji |} of projecti , a start time si j ≥ 0 has to be determined. The first andlast activities of the projects ( j = 0, resp. j = |Ji | + 1) aredummy activities, which have a zero duration and no resourcerequirements. Each project i ∈ P has a release date ri , i.e.the earliest time at which the activities of project i can start(si0 ≥ ri ).

123

J Sched

2.2 Local and global resources

A set Li = Lρi ∪ Lν

i of resources is associated with eachproject i ∈ P , where the subset Lρ

i = {1, . . . , |Lρi |}

denotes the local renewable resources and Lνi = {|Lρ

i | +1, . . . , |Lρ

i | + |Lνi |} the non-renewable resources. Renew-

able resources have a fixed capacity per time unit, while thenon-renewable resources have a fixed capacity over the entireproject duration. Each resource l ∈ Li has a fixed capacityof cil .

Finally, a set Gρ of global renewable resources is sharedamong the projects. Similar to the local resources, the avail-ability of the global resources is limited by cρ

g , g ∈ Gρ .There are no global non-renewable resources.

2.3 Execution modes

For each activity j ∈ Ji , i ∈ P , one or more executionmodes are available. The execution mode in which an activityis performed determines both the time required to completethe activity, as well as the specific resource requirements.Let Mi j = {1, . . . , |Mi j |} be the set of possible modes inwhich activity j ∈ Ji , i ∈ P , can be performed, and di jm

the processing time of activity j ∈ Ji in mode m ∈ Mi j .The constants rρ

i jml , rνi jml and rρ

i jmg , respectively, define theconsumption of local renewable, non-renewable and globalrenewable resources when activity j ∈ Ji is processed inmode m ∈ Mi j .

Note that a feasible schedule must always adhere to thefollowing resource constraints:

∑

j∈Ji

∑

m∈Mi j

xi j t yi jm rρi jml ≤ cil ∀i ∈ P, l ∈ Lρ

i ,

t ∈ [0, T ](1)

∑

j∈Ji

∑

m∈Mi j

yi jm rνi jml ≤ cil ∀i ∈ P, l ∈ Lν

i

(2)∑

i∈P

∑

j∈Ji

∑

m∈Mi j

xi j t yi jm rρi jmg ≤ cg ∀g ∈ Gρ, t ∈ [0, T ]

(3)∑

m∈Mi j

yi jm = 1 ∀i ∈ P, j ∈ Ji

(4)

xi j t ∈ {0, 1} ∀i ∈ P, j ∈ Ji ,

m ∈ Mi j , t ∈ [0, T ](5)

yi jm ∈ {0, 1} ∀i ∈ P, j ∈ Ji ,

m ∈ Mi j

(6)

where binary variables xi j t indicate whether a particularactivity j ∈ Ji is performed at time t ∈ [0, T ], while binaryvariables yi jm denote whether activity j ∈ Ji runs in modem ∈ Mi j . T is an upper bound on the time horizon of thescheduling problem.

2.4 Precedence constraints

Certain activities may require the completion of other tasksprior to their own start. In such a case, if activity j ∈ Ji

must precede activity j ′ ∈ Ji , a precedence relation j ≺j ′ is defined. Feasible schedules must obey all precedencerelations:

si j +∑

m∈Mi j

yi jmdi jm ≤ si j ′ i f j ≺ j ′ (7)

pred( j) denotes the predecessor set of activity j , i.e.pred( j) = { j ′| j ′ ≺ j}.

2.5 Objectives

The objective is to find a feasible schedule while minimiz-ing the total project delay (TPD) (Sect. 2.5.1) and the totalmakespan (TMS) (Sect. 2.5.2). The two objectives are con-sidered in lexicographical order, where TPD is the primaryobjective, while TMS is used as a tie breaker.

2.5.1 Total project delay (TPD)

The project delay is defined as the difference between thecritical path duration (CPD), a theoretical lower bound onthe earliest finish time of the project, and the actual projectduration (makespan). More formally, the project delay (PDi)for a project i ∈ P is defined as:

PDi = MSi − CPDi

where MSi , the makespan of project i , is:

MSi = fi − ri with fi = si(|Ji |+1).

and CPDi , the critical path duration1 of project i , is:

CPDi = EFi(|Ji |+1),

where EFi j is the earliest finish time of activity j ∈ Ji , whichis recursively defined as:

EFi0 = 0 ∀i ∈ P

EFi j = maxj ′∈pred( j)

(E Fi j ′) + minm

(di jm) ∀i ∈ P, j > 0

1 Values for the critical path duration of the projects are provided in theinput file.

123

J Sched

Finally, the TPD is expressed as:

TPD =n∑

i=1

PDi

2.5.2 Total makespan (TMS)

The TMS is the duration of the whole multi-project schedule.

T M S = maxi∈P

( fi ) − mini∈P

(ri )

3 Competition

The initial announcement of the MISTA 2013 challengeattracted 21 teams to register for the competition. The major-ity of participants were affiliated with academic institutions,whereas only three teams were affiliated with industry. Thecompetition’s rules did not stipulate any restrictions on howan algorithm could obtain a solution. The only requirementsfor the approach were that the code was executable by theorganizers, and that it was either completely free or free underacademic licencing.

The competition consisted of three phases. The teams wereprovided with a first dataset to guide the initial developmentof their algorithms (A instances). Problem size in the firstdataset ranged from 2 projects and 10 activities per projectto 10 projects and 30 activities per project. Three monthsafter the competition’s announcement, the teams were askedto submit their approach in order to determine the qualifyingteams for the next phase. The qualification criteria consistedof executable software that was able to produce feasible solu-tions for the initial dataset within a time limit of 5 min. 16out of the 21 registered teams qualified for the second phase.

After the announcement of the qualification results, asecond set, containing larger instances, was published (Binstances). The problem size varied between instances with10 projects and 10 activities per project and instances with20 projects and 30 activities per project. Again, the partici-pants had 3 months to fine-tune their approaches, given thenew dataset and the best obtained solutions from the quali-fications phase. At the end of this second phase, 9 out of 16teams submitted an algorithm for the final phase.

In the last phase, the performance of the final submissionswas evaluated. The final ranking was determined by run-ning each participant’s algorithm ten times on all instancesfrom the second dataset and an additional hidden dataset (Xinstances), with instances of similar size as the B instances.These experiments were executed by the organizers on a

Table 1 Final ranking

Rank Team ID Members Mean rank

1 11 Asta et al. (2013) 1.10

2 8 Geiger (2013) 2.55

3 1 Toffolo et al. (2013) 3.05

4 20 Artigues and Hebrard (2013) 3.60

5 15 Alonso-Pecina et al. (2013) 6.75

5 13 Borba et al. (2013) 6.75

5 17 López-Ibáñez et al. (2013) 6.75

8 14 Bouly et al. (2013) 6.85

9 21 Schnell (2013) 7.60

recent desktop pc2, with a time limit3 of 5 min per instance.For each instance, the algorithms were ranked based on theaverage of all runs. Finally, the ranks were averaged over allinstances per team. Table 1 shows the final results.

Additional details on the competition rules, the datasetsand general information regarding the competition can befound at the competition web page (Wauters et al. 2013).

3.1 Instance generation

Instances of this new MRCMPSP have been generatedby combining multiple MRCPSP instances from PSPLIB.Release dates and global resources had to be added. Therelease dates are generated using a poisson process, i.e. theproject inter arrival times are exponentially distributed withλ = 0.2. The release date of the first project is always r0 = 0.One of the two renewable resources is a global resourcewith a 2/3 probability. The capacity of the global renew-able resources has been set to a value between 50 and 100%of the local renewable resource capacity. To guarantee fea-sibility the capacity should be at least 10, i.e. in the PSPLibfiles, there are jobs with a resource consumption of 10. Thefollowing formula is applied:

cρg = max

(C APmax

5 + rand(6)

10, 10

),

where C APmax is the maximum capacity of the correspond-ing local renewable resources and rand(6) is a uniform ran-dom number between 0 (inclusive) and 6 (exclusive). Theinstance generator is available for download on the compe-tition web page.

2 Intel Core i7-2600 at 3.4 Ghz and 8 GB RAM with hyper-threading,turbo boost and energy saver disabled. ITC2011 benchmark tool score:619 s.3 The time limit was checked by measuring the wall clock time of thealgorithms. The algorithm should stop autonomously within the timelimit. Small deviations of less than 0.1 s were allowed.

123

J Sched

4 Discussion

4.1 Detailed results

Table 2 gives an overview of the detailed results per team andper instance, averaged over 10 runs. This produced the finalranking (by mean rank over all instances) shown in Table 3. Itis clear that the competition’s final winner, team 11, producedthe best average results over all instances, except for instancesB-2 and X-1. The second and third ranking positions weremore contested, with team 1, team 8 and team 20 alternatinglyproducing second and third best results (Table 4).

Figure 3 shows box plots of each team’s results perinstance, giving an indication of the spread of the results.It is clear that teams 13, 14, 15, 17 and 21 in general have alarger spread of results per instance, indicating their methodsare either more susceptible to random events or possibly notyet converged after the time limit is reached. Team 1, 8, 11and 20 generally have a low spread of the results; in partic-ular team 20 has a spread of 0 for all instances because thesubmission always produces the same result.

The final ranking method, by mean rank over all instances,was decided as the scoring system at the onset of the compe-tition. Any scoring system is biased in one way or another,e.g. the ranking transformation discards information on therelative performance among submissions on each instance.In order to investigate how the scoring system influenced thefinal results, different scoring systems were tested. In addi-tion to ranking by mean rank, the following systems werecompared:

– Ranking by mean: ranks according to the mean of theaveraged results over all instances. This system is biasedby the difference in magnitude of the objective functionvalue between instances: larger instances will typicallyhave a higher magnitude of objective function value, andthus will contribute more to the mean. Scoring well onlarge instances is thus important. However, the relativeperformance among submissions per instance is not lost.

– Ranking by geometric mean: ranks according to the geo-metric mean of the averaged results over all instances.This system is less biased by the difference in magni-tude of the instance objective function value, and alsomaintains information on relative performance betweensubmissions per instance.

– Ranking by total F1-score: this scoring system is adoptedfrom Formula 1 racing point system (before 2010),and was used by the CHeSC 2011 competition (Burkeet al. 2011). For each instance, the submissions areattributed points from 10 to 0. The highest average resultis attributed 10 points, the second highest 8, the third 6,and finally 5, 4, 3, 2 and 1 to the remaining results inthat order. The total number of points over all instances

is then used to rank the submissions. Since the attribut-ion of points is essentially a ranking transformation, rel-ative performance between submissions per instance islost. Furthermore, the method is biased towards the bestresults per instance.

– Ranking by mean best rank: this method differs fromthe competition ranking method by computing the ranksper instance based on the best result produced by eachsubmission, rather than their averaged result. Therefore,this method considers the ‘best’ results for any submis-sion, but may be biased by ‘lucky results’, and also losesinformation on relative performance between teams perinstance. A summary of the best results found for allinstances and teams can be found in Table 5.

– Ranking by mean rank over all runs: this methodattributes a rank per instance to each run of all teams(i.e. a rank between 1 and 90 as there were 9 fully fea-sible teams and 10 runs with different random seed perteam). Next, a mean rank is computed per team by com-puting the average rank over all instances and all runs ofthis team.

These 6 different scoring systems produce the final ranks4

shown in Table 4. It is clear that the order of the best foursubmissions (Teams 1, 8, 11, and 20) is stable to the consid-ered scoring mechanisms. However, there are some changesbetween positions 5 through 8, differentiating the teams rank-ing 5th according to the mean rank. It becomes apparent thatteam 17 produced better results than the other two teams atrank 5.

4.2 Properties of submitted algorithms

A variety of methods have been submitted to the MISTA2013 challenge.

Toffolo et al. (2013) (Team 1) use a decomposition-basedmatheuristic. The approach considers resolution of severalinteger programming (IP) models, in three different stages.A feasible set of execution modes for the jobs is obtained,in the first stage. The second stage decomposes the probleminto time windows. Each time window is solved with an IPmodel, considering the execution modes obtained from thefirst stage. The result of the second stage is a feasible solution,which is improved in the final stage by a hybrid local searchbased on forward-backward improvement (Lova et al. 2009;Valls et al. 2005) and an IP model to perform controlledchanges to the solution.

4 Details on how all these ranks were calculated can be reviewedin the detailed Google Docs Sheet, containing all results, athttps://docs.google.com/spreadsheet/ccc?key=0Ar3CEQ-QxKb6dHZqbG9xNC0tUlN5UV95aGRsZFYyZHcq&usp=sharing.

123

J Sched

Tabl

e2

Ave

rage

resu

ltspe

rin

stan

cefo

rsu

bmis

sion

sto

the

MIS

TAco

mpe

titio

n

Inst

ance

Bes

tfo

und

(Tea

m)

Team

1Te

am2

Team

8Te

am11

Team

13Te

am14

Team

15Te

am17

Team

18Te

am20

Team

21

B-1

3490

0127

(11)

3665

0131

.646

2601

36.0

3595

0130

.235

1801

28.0

3839

0132

.245

2501

38.4

3978

0131

.145

4501

34.2

3685

0127

.343

2001

28.0

4455

0144

.9

B-2

4340

0160

(1)

4448

0161

.559

6101

60.9

5260

0176

.745

4301

68.1

7752

0169

.256

8601

72.8

8082

0172

.960

4401

67.5

–52

6001

53.0

5342

0171

.0

B-3

5450

0210

(11)

6652

0208

.172

8602

1361

8802

16.5

5540

0211

.071

2002

20.5

7353

0226

.176

4202

22.8

7353

0221

.364

7402

09.5

6380

0205

7472

0228

.5

B-4

1274

0028

9(8

)15

7360

300.

5–

1316

5028

6.8

1305

4028

4.4

1822

6029

3.7

1670

4030

4.0

1771

0029

3.1

1695

6031

0.8

1609

7028

5.3

1469

0029

7.0

1777

7032

7.3

B-5

8200

0254

(11)

9309

0254

.712

3160

257.

388

9002

62.7

8326

0254

.311

3860

266.

911

0460

282

1111

0026

1.2

1165

0026

8.7

1038

6025

6.6

1075

0024

9.0

1280

6030

5.2

B-6

9120

0227

(11)

1150

8023

3.6

1136

9023

5.1

1103

7024

6.9

9527

0231

.712

9010

257.

814

5570

288.

713

6300

267.

913

2000

247.

011

5480

233.

810

8300

217.

012

8680

266.

6

B-7

7920

0228

(11)

9314

0244

.1–

8513

0236

.080

1102

32.1

1027

9024

2.5

1134

1026

2.2

9982

0237

.611

2880

268.

692

5802

32.3

9050

0231

.014

0970

289.

7

B-8

3176

0053

3(1

1)33

5130

533.

763

4320

579.

237

3480

573.

633

1440

547.

747

9160

566.

143

0380

588.

451

8640

593.

139

2940

583.

448

9190

549.

336

6200

528.

039

0210

610.

2

B-9

4192

0074

6(1

1)54

3180

771.

0–

4793

5080

7.2

4263

5075

5.4

8090

1084

9.1

9124

7116

7.6

7275

5081

7.3

6037

6085

4.2

6200

1078

3.2

5465

0074

675

0791

055.

6

B-1

032

4900

456

(11)

3404

2043

9.5

4080

5047

3.9

3682

3047

6.8

3337

8045

9.6

4599

7048

2.2

3823

4050

7.2

4832

6049

1.5

4366

9048

6.2

4210

2045

2.8

4034

3042

7.0

3621

3046

9.4

X-1

3920

0142

(1)

4029

0143

.970

9901

49.9

4171

0143

.740

4901

42.4

6181

0148

.549

8501

54.8

5417

0151

.150

2701

48.9

4392

0148

.147

8001

44.0

5095

0160

.9

X-2

3490

0163

(11)

4374

0169

.845

1801

64.9

3828

0168

.435

7001

64.1

4333

0169

.345

1501

75.1

4515

0167

.552

0001

79.0

4153

0164

.342

3001

59.0

6674

0203

.3

X-3

3240

0192

(11)

3394

0189

.541

5601

87.4

3924

0195

.533

0401

93.2

4391

0192

.648

3102

05.5

4552

0187

.946

0301

96.0

4113

0183

.139

1001

86.0

5191

0205

.9

X-4

9550

0213

(11)

1013

1020

8.0

1346

3021

1.2

1010

8021

2.7

9706

0212

.311

6900

219.

612

0890

220.

312

3080

218.

910

8620

210.

510

2820

207.

510

5400

198.

012

1490

229.

5

X-5

1768

0037

4(1

1)20

8560

378.

827

2290

390.

619

6940

389.

717

8450

373.

023

2370

397.

424

0170

430.

623

0450

380.

824

5070

401.

922

1130

378.

420

7650

367.

026

8910

482.

6

X-6

7190

0232

(11)

9356

0246

.810

4840

244.

989

3102

55.1

7384

0240

.913

7740

266.

410

7710

284.

311

1760

248.

812

3840

278.

395

3802

31.0

8725

0214

.017

8060

328.

1

X-7

8610

0237

(11)

9701

0236

.714

0710

242.

289

6902

36.6

8675

0235

.511

8030

243.

212

3960

253.

311

5720

241.

112

6470

268.

698

1302

31.3

9930

0229

.013

5930

286.

3

X-8

1233

0028

3(1

1)15

0420

291.

720

5300

295.

614

6040

305.

812

5720

289.

318

4310

313.

818

5130

356.

318

3070

302.

317

3170

312.

614

8130

286.

716

6050

270.

026

4200

376.

3

X-9

3268

0064

3(1

1)42

2520

660.

067

2350

739.

437

8050

691.

933

0320

647.

159

2750

739.

667

0700

995.

651

6180

699.

452

8240

754.

049

2920

681.

751

3000

635.

062

0020

908.

0

X-1

016

0000

381

(11)

1870

4038

5.5

–17

4720

399.

516

1440

382.

326

0000

414.

020

7090

441.

024

1140

400.

225

9690

429.

121

9410

395.

419

7400

376.

037

8300

597.

7

Mea

n13

6060

304.

516

0172

309.

5–

1541

3032

0.6

1389

7880

7.6

2127

1632

9.2

2118

1387

2.7

2058

5182

4.3

1928

5783

6.0

–17

1709

298.

021

9390

882.

4

Geo

m.

9892

0555

.111

4261

313.

6–

1110

7724

4.1

1010

4331

9.3

1443

0758

3.4

1408

0157

8.2

1420

3210

6.4

1390

8662

1.6

–12

1054

656.

915

5392

199.

6

mea

n

F1-s

core

126

–14

219

645

4345

45–

110

28

Res

ults

are

aver

aged

over

10ru

nsw

ithdi

ffer

ents

eeds

for

each

inst

ance

.Bol

dhi

ghlig

htin

gin

dica

tes

the

best

aver

age

resu

ltpe

rin

stan

ce.D

ashe

sin

dica

tein

feas

ible

resu

ltsge

nera

ted

for

atle

asto

neru

n

123

J Sched

Table 3 Ranking per instance based on the averaged result per submission of the final phase

Instance Team 1 Team 8 Team 11 Team 13 Team 14 Team 15 Team 17 Team 20 Team 21

B-1 3 2 1 4 8 5 9 6 7

B-2 1 4 2 8 6 9 7 3 5

B-3 4 2 1 5 7 9 6 3 8

B-4 4 2 1 9 5 7 6 3 8

B-5 3 2 1 7 5 6 8 4 9

B-6 4 3 1 6 9 8 7 2 5

B-7 4 2 1 6 8 5 7 3 9

B-8 2 4 1 8 7 9 6 3 5

B-9 3 2 1 8 9 6 5 4 7

B-10 2 4 1 8 5 9 7 6 3

X-1 1 3 2 9 5 8 6 4 7

X-2 5 2 1 4 7 6 8 3 9

X-3 2 4 1 5 8 6 7 3 9

X-4 3 2 1 6 7 9 5 4 8

X-5 4 2 1 6 7 5 8 3 9

X-6 4 3 1 8 5 6 7 2 9

X-7 3 2 1 6 7 5 8 4 9

X-8 3 2 1 7 8 6 5 4 9

X-9 3 2 1 7 9 5 6 4 8

X-10 3 2 1 8 5 6 7 4 9

Average rank 3.05 2.55 1.10 6.75 6.85 6.75 6.75 3.60 7.60

Final rank 3 2 1 5 8 5 5 4 9

Ties are resolved by the average rank. Bold highlighting indicates best rank. Note that team 2 and team 18 are not ranked due to infeasible results

Table 4 Ranking results of final submissions by different ranking methods

Ranking method. Team

1 8 11 13 14 15 17 20 21

Mean 3 2 1 8 7 6 5 4 9

Geom. Mean 3 2 1 8 6 7 5 4 9

Mean ranka 3 2 1 6 8 6 6 4 9

F1 score 3 2 1 6 8 6 6 4 9

Best rank 3 2 1 7 5.5 5.5 8 4 9

Mean over all 3 2 1 8 6 7 5 4 9

runs rank

Considered methods: ranking by mean over all instances, ranking by geometric mean over all instances, ranking by mean rank over all instances,ranking by F1-score, ranking by mean best rank, ranking by mean over all runs rank. Bold highlights indicate best rankeda Competition ranking method

Geiger (2013) (Team 8) proposes an iterated variableneighbourhood search with four different neighbourhoods:exchange, inversion, single mode change and double modechange. Random initial solutions are generated with an addi-tional mode repair procedure. A serial schedule generationscheme (SSGS) constructs schedules with a given activitylist and mode assignment.

Asta et al. (2013) (Team 11) apply a combination ofMonte-Carlo Tree Search (MCTS) and hyper-heuristics.MCTS first partitions the projects from which their order-ing is then determined. An initial solution is constructedby randomly scheduling the activities in each project, whileonly taking into account the precedence constraints. A multi-threaded hyper-heuristic with threshold acceptance is used

123

J Sched

B1 B2 B3 B4

B5 B6 B7 B8

B9 B10 X1 X2

X3 X4 X5 X6

X7 X8 X9 X10

3.5e+07

4.0e+07

4.5e+07

5e+07

6e+07

7e+07

8e+07

6e+07

7e+07

8e+07

1.4e+08

1.6e+08

1.8e+08

8.0e+07

1.0e+08

1.2e+08

1.4e+08

1.0e+08

1.2e+08

1.4e+08

1.6e+08

8.0e+07

1.0e+08

1.2e+08

1.4e+08

1.6e+08

3.5e+08

4.0e+08

4.5e+08

5.0e+08

5.5e+08

4e+08

6e+08

8e+08

1e+09

3.5e+08

4.0e+08

4.5e+08

5.0e+08

4.0e+07

4.5e+07

5.0e+07

5.5e+07

6.0e+07

4e+07

5e+07

6e+07

7e+07

3.5e+07

4.0e+07

4.5e+07

5.0e+07

5.5e+07

1.0e+08

1.1e+08

1.2e+08

1.3e+08

1.75e+08

2.00e+08

2.25e+08

2.50e+08

2.75e+08

1.0e+08

1.5e+08

1.0e+08

1.2e+08

1.4e+08

1.5e+08

2.0e+08

2.5e+08

3.0e+08

4e+08

5e+08

6e+08

7e+08

8e+08

1.5e+08

2.0e+08

2.5e+08

3.0e+08

3.5e+08

4.0e+08

1 8 11 13 14 15 17 20 21 1 8 11 13 14 15 17 20 21 1 8 11 13 14 15 17 20 21 1 8 11 13 14 15 17 20 21

1 8 11 13 14 15 17 20 21 1 8 11 13 14 15 17 20 21 1 8 11 13 14 15 17 20 21 1 8 11 13 14 15 17 20 21

1 8 11 13 14 15 17 20 21 1 8 11 13 14 15 17 20 21 1 8 11 13 14 15 17 20 21 1 8 11 13 14 15 17 20 21

1 8 11 13 14 15 17 20 21 1 8 11 13 14 15 17 20 21 1 8 11 13 14 15 17 20 21 1 8 11 13 14 15 17 20 21

1 8 11 13 14 15 17 20 21 1 8 11 13 14 15 17 20 21 1 8 11 13 14 15 17 20 21 1 8 11 13 14 15 17 20 21Team

Res

ult

Fig. 3 Summarizing boxplots of participants’ results per instance

123

J Sched

Tabl

e5

Bes

tres

ults

per

inst

ance

and

per

team

for

subm

issi

ons

toth

eM

ISTA

com

petit

ion

Inst

ance

Bes

tfo

und

(Tea

m)

Team

1Te

am2

Team

8Te

am11

Team

13Te

am14

Team

15Te

am17

Team

18Te

am20

Team

21

B-1

3490

0127

(11)

3630

0132

4380

0133

3530

0125

3490

0127

3810

0133

4200

0139

3710

0127

4250

0133

3610

0130

4320

0128

4150

0144

B-2

4340

0160

(1)

4340

0160

5540

0164

4900

0176

4430

0167

7560

0165

5280

0170

7900

0174

5840

0160

5370

0161

5260

0153

5160

0168

B-3

5450

0210

(11)

6600

0207

6940

0218

5980

0215

5450

0210

6760

0221

6980

0225

6930

0214

6840

0225

6210

0211

6380

0205

7270

0222

B-4

1274

0028

9(8

)15

4800

295

1986

0030

012

7400

289

1292

0028

717

6600

294

1582

0030

816

7100

287

1657

0030

815

6800

283

1469

0029

716

8000

315

B-5

8200

0254

(11)

9190

0254

1054

0025

586

6002

5482

0002

5410

7300

263

1059

0029

910

6100

255

1106

0027

010

0000

254

1075

0024

911

9700

305

B-6

9120

0227

(11)

1128

0023

210

9700

235

1044

0024

291

2002

2712

3900

257

1360

0030

912

8600

259

1287

0024

811

1900

230

1083

0021

712

2100

253

B-7

7920

0228

(11)

9080

0246

1348

0024

283

4002

3479

2002

2894

4002

3910

5100

254

9160

0228

1106

0027

090

3002

3390

5002

3112

5500

267

B-8

3176

0053

3(1

1)32

7600

529

6010

0056

635

8500

568

3176

0053

345

9100

544

4025

0054

949

5900

584

3755

0057

847

2800

533

3662

0052

835

7400

588

B-9

4192

0074

6(1

1)53

7300

769

7875

0085

146

7400

796

4192

0074

675

7700

837

7914

0117

669

9600

812

5979

0085

161

3000

779

5465

0074

670

4201

024

B-1

032

4900

456

(11)

3325

0044

738

5000

472

3518

0046

932

4900

456

4516

0047

236

9700

510

4550

0048

343

2400

487

4133

0045

240

3300

427

3420

0047

4

X-1

3920

0142

(1)

3920

0142

6370

0150

3940

0142

3980

0142

5880

0152

4520

0148

4860

0150

4680

0149

4320

0149

4780

0144

4960

0156

X-2

3490

0163

(11)

4180

0165

4020

0163

3680

0165

3490

0163

4070

0170

4150

0166

4310

0162

4990

0177

4010

0163

4230

0159

5960

0189

X-3

3240

0192

(11)

3260

0188

3940

0187

3720

0195

3240

0192

4140

0189

4180

0202

4180

0182

4440

0195

3880

0183

3910

0186

4920

0204

X-4

9550

0213

(11)

9860

0207

1226

0021

197

0002

1595

5002

1311

3000

213

1126

0021

111

3300

210

1064

0020

898

8002

0410

5400

198

1136

0023

5

X-5

1768

0037

4(1

1)20

4300

375

2528

0038

919

3800

386

1768

0037

421

9800

378

2208

0042

322

3600

376

2337

0040

621

7600

373

2076

0036

725

3900

464

X-6

7190

0232

(11)

8800

0240

9340

0240

8440

0253

7190

0232

1151

0025

310

1200

282

1008

0024

511

7000

269

9190

0226

8720

0214

1613

0031

7

X-7

8610

0237

(11)

9440

0234

1327

0024

287

9002

3186

1002

3711

5500

238

1106

0024

510

2900

235

1217

0027

195

3002

3099

3002

2912

4500

276

X-8

1233

0028

3(1

1)14

7800

289

1891

0029

913

8000

296

1233

0028

317

5600

308

1733

0035

715

7800

298

1687

0031

114

3500

282

1656

0027

023

3700

358

X-9

3268

0064

3(1

1)41

6900

662

6486

0074

136

4500

688

3268

0064

356

1400

721

6129

0097

047

0800

696

5185

0074

745

6200

682

5130

0063

556

9200

866

X-1

016

0000

381

(11)

1851

0038

526

1100

394

1669

0040

216

0000

381

2499

0041

219

8900

431

2248

0039

325

2100

418

2050

0039

619

7400

376

3430

0058

6

Bol

dhi

ghlig

htin

gin

dica

tes

the

best

resu

ltpe

rin

stan

ce.N

ote

that

the

tabl

ein

clud

esbe

stre

sults

for

team

s2

and

18,a

sth

eydi

dpr

oduc

ere

sults

for

all

inst

ance

s(b

utso

me

infe

asib

le).

The

yar

eho

wev

erno

tcon

side

red

for

any

rank

ing

met

hod

123

J Sched

Table 6 Properties of the nine algorithms in the competition final

Team Search strategy NN Population SGS FB Solver Parallel

1 ILS/LNS 1 No Serial Yes Gurobi 5.5 Yes

8 VNS/ILS 4 No Serial No – Yes

11 HH 13 Yes Serial No – Yes

13 FILS 2 No Serial No – –

14 EA + ILS 3 Yes Serial Yes – –

15 SA + TS 2–3 No ? No – –

17 ILS 2 No Serial Yes – Yes

20 LNS 1 No None No ILOG Cplex/CPOptimizer –

21 SD 4 No Serial No SCIP –

to control 13 neighbourhood moves, which change modeassignments, activity ordering, and project ordering. Fur-thermore, hill climbing heuristics are used within the hyper-heuristic to intensify the search, similar to the local searchpart of memetic algorithms.

Borba et al. (2013) (Team 13) propose a stochastic localsearch procedure with two neighbourhoods (single modechange, swap). Dynamic programming determines the ini-tial mode selection.

Bouly et al. (2013) (Team 14) make use of an evolutionaryalgorithm and local search. A triplet encoding including thesequence of jobs and the chosen modes combined with aSSGS and a repair procedure to produce feasible schedules.The local search applies three different neighbourhoods andis used as a post-processing procedure for the evolutionaryalgorithm.

Alonso-Pecina et al. (2013) (Team 15) propose a three-phase approach. First, feasible solutions are constructed foreach project separately, which are afterwards combined intoan initial feasible solution for the complete problem. Thesecond phase improves the solution with a simulated anneal-ing metaheuristic. It applies two neighbourhoods that changemode assignments and activity ordering. The solution is fur-ther improved in the third phase with a tabu search algorithmexploring the two previous neighbourhoods, and one addi-tional neighbourhood that changes two mode assignmentssimultaneously.

López-Ibáñez et al. (2013) (Team 17) used an automaticdesign method to develop an algorithm for the MRCMPSP.The design method uses ‘irace’ to find the best combinationof algorithm parameters out of a set. The proposed algorithmis an iterated local search with two neighbourhoods (singlemode change, swap). Several initial solutions are generatedusing different construction heuristics.

Artigues and Hebrard (2013) (Team 20) introduce ahybrid approach, which focuses mainly on mode selection.It includes a MIP relaxation model for producing an ini-tial mode assignment. The improvement method alternates

between mode selection, conducted with large neighbour-hood search, and fixed-mode optimization based on a stan-dard Constraint Programming model.

Schnell extends SCIP (Schnell 2013) (Team 21) with amulti-mode cumulative constraint, enabling generation ofoptimal solutions for 2-project instances in less than 35 sec.For the larger instances, a steepest descent local searchheuristic with four neighbourhoods is applied. An initialsolution is constructed by solving all projects separately forminimal makespan, and sequencing the projects for a mini-mal total project delay. Sorting all activities in this sequenceby increasing starting times results in the initial activity listto which a SSGS is applied. Two neighbourhoods relax upto two projects, enabling generation of a different solution.One randomly selects the projects, while the other selectsprojects based on the lower bound of the relaxed solution. Athird neighbourhood relaxes some of the activities that arescheduled at the end of the current solution. A fourth relaxesthe mode assignments in an effort to minimize the completerenewable resource energy. The resulting partially relaxedproblem is solved with SCIP.

Table 6 shows the presence of different components in thenine feasible submissions. The following components andproperties are shown:

– Search strategy: The reported search strategy: iter-ated local search (ILS), variable neighbourhood search(VNS), hyper-heuristic (HH), first improving local search(FILS), evolutionary algorithm (EA), simulated anneal-ing (SA), tabu search (TS), large neighbourhood search(LNS) and steepest descent (SD).

– NN: number of neighbourhoods.– Population: whether or not the approach is population

based.– SGS: if and which schedule generation scheme is used.– FB: whether or not the forward-backward or double jus-

tification procedure is used (Lova et al. 2009; Valls et al.2005).

123

J Sched

– Solver: if and which (general purpose) solver is used.– Parallel: parallel execution.

All teams submitted a heuristic procedure in which someincluded exact components. Some model elements are com-mon to most approaches, namely an activity list and a modeassignment representation combined with a SSGS. The singlemode change neighbourhood, where the mode assignment ofa single job is changed, is part of most teams’ approaches.It is noteworthy that every team has a different approach togenerate feasible initial solution(s), and especially to finda feasible mode assignment, e.g. random, monte-carlo treesearch, multi-dimensional knapsack,. . . The teams do not allexplicitly apply parallel execution. Some teams use a CP orMIP for solving subproblems to optimality or for conductinga large neighbourhood search.

5 Conclusion

This paper introduced the MRCMPSP, subject of the MISTA2013 challenge. The submitted approaches and the resultshave been thoroughly analysed and discussed. The problemshows to be very challenging, and a large variety of algo-rithms and results can be observed. The challenge serves asa new benchmark platform for the MRCMPSP and stimu-lates research for this and related problems. Many instancesand results are available for comparison and facilitate futureresearch on the problem.

Future research can take advantage of the collected knowl-edge, for example by (1) exploring synergy by combining thekey components of the feasible algorithms, (2) applying thecollected set of algorithms in a portfolio approach.

Acknowledgments We would like to thank the challenge sponsors:iMinds, CONUNDRA and OM Partners. We would also like to thankall participating teams. This research is partially supported by a Ph. D.Grant of the agency for Innovation by Science and Technology (IWT).

Appendix 1: File formats

Instance input file format

All problem inputs are integer valued, and comply with thefollowing file format. Each problem instance is defined byn + 1 files, one global file and n project files.

Global file

The global file contains the number of projects, the releasedates, the path to the project files, and the global resource

capacities. Values are space separated and should respect thefollowing order:

Number of projectsRelease date project 0Critical path duration project 0Path to project file of project 0Release date project 1Critical path duration project 1Path to project file of project 1...Release date project n-1Critical path duration project n

-1Path to project file of project n

-1Number of resource typesGlobal resource capacities (-1 is

not global)

Project files

Each project is defined in a separate file. The relative path tothis file is given in the global file (e.g. j20.mm/j2010_1.mm).A project file contains the number of activities (includingthe dummy activities), the precedence relations between theactivities, the execution modes, and the local resource capac-ities. The individual projects are represented in the PSPLIB5

MRCPSP file format.Important note: In order to preserve compatibility with

PSPLIB, the global resources always overwrite the localresources and their capacities. For example, consider thefollowing global resource capacities (16,−1,−1,−1) andlocal resource capacities (RR1 = 14, RR2 = 18, NR1 = 60,NR2 = 68), then there exists one global renewable resourcewith capacity 16, one local renewable resource with capacity18 and two non-renewable resources with capacities 60 and68. The local renewable resource with capacity 14 can beignored.

Solution output format

A solution for the MRCMPSP is defined as follows:For each activity j of project i ,

– the selected mode mi j ,– and the start time si j ,

must be given.An example is given in Table 7.

5 PSPLIB benchmark website: http://www.om-db.wi.tum.de/psplib/main.html.

123

J Sched

Table 7 Solution output format example

Project i Activity j Mode mi j Start time si j

0 0 2 0

0 1 1 5

. . . . . . . . . . . .

1 0 0 4

1 1 0 8

. . . . . . . . . . . .

Solution validator

The validator checks feasibility of a solution, and computesthe total objective cost. It is available for download at theMISTA challenge website (Wauters et al. 2013). The valida-tor requires Java. It can be run as follows:

java -jar MRCMPSP -validator.jarglobal_problem_fileschedule_file

The validator was used for evaluating the quality of solu-tions produced by the algorithms submitted to the MISTA2013 challenge. Note that the validator does not report valid-ity of the instances.

Example

An example MRCMPSP instance and a corresponding fea-sible solution can be found on the challenge website. Theinstance has 2 projects, 10 jobs per project and 1 globalrenewable resource with capacity 12. The global file cor-responding to this example is

2017j10.mm/j1010_1.mm622j10.mm/j1010_2.mm412 -1 -1 -1

References

Alonso-Pecina, F., Pecero, J. E., & Romero, D. (2013). A three-phasesbased algorithm for the multi-mode resource-constrained multi-project scheduling problem. In G. Kendall, G. Vanden Berghe,& B. McCollum (Eds.), Proceedings of the 6th MultidisciplinaryInternational Scheduling Conference (pp. 812–814). : MISTA.

Artigues, C., & Hebrard, E. (2013). MIP relaxation and large neighbor-hood search for a multi-mode resource-constrained multi-projectscheduling problem. In G. Kendall, G. Vanden Berghe, & B.McCollum (Eds.), Proceedings of the 6th Multidisciplinary Inter-national Scheduling Conference (pp. 815–819). : MISTA.

Asta, S., Karapetyan, D., Kheiri, A., Özcan, E., & Parkes, A. (2013).Combining monte-carlo and hyper-heuristic methods for the multi-mode resource-constrained multi-project scheduling problem. InG. Kendall, G. Vanden Berghe, & B. McCollum (Eds.), Proceed-ings of the 6th Multidisciplinary International Scheduling Confer-ence (pp. 836–839). : MISTA.

Blazewicz, J., Lenstra, J., & Rinnooy Kan, A. (1983). Schedul-ing subject to resource constraints: Classification and complex-ity. Discrete Applied Mathematics, 5(1), 11–24. doi:10.1016/0166-218X(83)90012-4.

Borba, L.M., Benavides, A.J., Zubaran, T., Carniel, G.C., & Ritt, M.(2013). A simple stochastic local search for multi-mode resource-constrained multi-project scheduling. In G. Kendall, G. VandenBerghe, & B. McCollum (Eds.), Proceedings of the 6th Multidis-ciplinary International Scheduling Conference (pp. 826–830). :MISTA.

Bouly, H., Dang, D.C., Moukrim, A., & Xu, H. (2013). Evolutionaryalgorithm and post-processing to minimize the total delay in multi-project scheduling. In G. Kendall, G. Vanden Berghe, & B. McCol-lum (Eds.), Proceedings of the 6th Multidisciplinary InternationalScheduling Conference (pp. 831–835). : MISTA.

Burke, E. K., Gendreau, M., Hyde, M., Kendall, G., McCollum, B.,Ochoa, G., et al. (2011). The cross-domain heuristic search chal-lenge - an international research competition. In C. A. CoelloCoello (Ed.), Learning and Intelligent Optimization, Lecture Notesin Computer Science (Vol. 6683, pp. 631–634). Berlin Heidelberg:Springer. doi:10.1007/978-3-642-25566-3_49.

Geiger, M.J. (2013). Iterated variable neighborhood search for theresource constrained multi-mode multi-project scheduling prob-lem. In G. Kendall, G. Vanden Berghe, & B. McCollum (Eds.),Proceedings of the 6th Multidisciplinary International SchedulingConference (pp. 807–811). : MISTA.

Haspeslagh, S., De Causmaecker, P., Schaerf, A., & Stølevik,M., (2012). The first international nurse rostering competition2010. Annals of Operations Research, pp 1–16, doi:10.1007/s10479-012-1062-0.

Kolisch, R., & Drexl, A. (1997). Local search for nonpreemptive multi-mode resource-constrained project scheduling. IIE Transactions,29, 987–999.

López-Ibáñez, M., Mascia, F., Marmion, M.E., & Stützle, T. (2013).Automatic design of a hybrid iterated local search for the multi-mode resource-constrained multi-project scheduling problem. InG. Kendall, G. Vanden Berghe, & B. McCollum (Eds.), Proceed-ings of the 6th Multidisciplinary International Scheduling Confer-ence (pp. 820–825). : MISTA.

Lova, A., Tormos, P., Cervantes, M., & Barber, F. (2009). An efficienthybrid genetic algorithm for scheduling projects with resourceconstraints and multiple execution modes. International Journal ofProduction Economics, 117(2), 302–316. doi:10.1016/j.ijpe.2008.11.002.

McCollum, B., Mcmullan, P., Burke, E. K., Parkes, A. J., & Qu, R.(2007). The second international timetabling competition: Exam-ination timetabling track.

Post, G., Di Gaspero, L., Kingston, J., McCollum, B., Schaerf, A.(2013). The third international timetabling competition. Annalsof Operations Research pp 1–7, doi:10.1007/s10479-013-1340-5,http://dx.doi.org/10.1007/s10479-013-1340-5

Schnell, A. (2013). A hybrid local search algorithm integrating an exactcp-sat approach to solve the multi-mode resource-constrainedmulti-project scheduling problem. In G. Kendall, G. VandenBerghe, & B. McCollum (Eds.), Proceedings of the 6th Multi-

123

J Sched

disciplinary International Scheduling Conference (pp. 802–806).: MISTA.

Solnon, C., Cung, V. D., Nguyen, A., & Artigues, C. (2008). The carsequencing problem: Overview of state-of-the-art methods andindustrial case-study of the roadef2005 challenge problem. Euro-pean Journal of Operational Research, 191(3), 912–927. doi:10.1016/j.ejor.2007.04.033.

Toffolo, T. A. M., Santos, H. G., Carvalho, M., & Soares, J. (2013).An integer programming approach for the multi-mode resource-constrained multi-project scheduling problem. In G. Kendall, G.Vanden Berghe, & B. McCollum (Eds.), Proceedings of the 6thMultidisciplinary International Scheduling Conference (pp. 840–847). : MISTA.

Valls, V., Ballestin, F., & Quintanilla, S. (2005). Justification and rcpsp:A technique that pays. European Journal of Operational Research,165(2), 375–386. doi:10.1016/j.ejor.2004.04.008.

Wauters, T., Verbeeck, K., Verstraete, P., & De Causmaecker, P. (2012).Real-world production scheduling for the food industry: An inte-grated approach. Engineering Applications of Artificial Intelli-gence, 25(2), 222–228. doi:10.1016/j.engappai.2011.05.002.

Wauters, T., Kinable, J., Smet, P., Vancroonenburg, W., Verstichel(2013). The MISTA 2013 challenge. http://allserv.kahosl.be/mista2013challenge/, [Online]

123