Outline Introduction Evaluation criteria & Methodology Presenting the tools Summary and Recommendation

Comparing two Queuing Network Solvers:JMT vs. PDQ

A presentation for the report of the Course CSI 5112 (W11)

Adnan Faisal (CU100841800)Mostafa Khaghani Milani (CU100836314)

University of Ottawa

25 March 2011

Faisal & Milani, UOttawa, 25 March 2011 Carleton University

Comparing two Queuing Network Solvers: JMT vs. PDQ

Outline Introduction Evaluation criteria & Methodology Presenting the tools Summary and Recommendation

IntroductionThe company & the problemQueuing Networks

Evaluation criteria & MethodologyGeneral criteriaQN theory related criteriaMethodology

Presenting the toolsJMT vs PDQGRL evaluation

Summary and Recommendation

Faisal & Milani, UOttawa, 25 March 2011 Carleton University

Comparing two Queuing Network Solvers: JMT vs. PDQ

Outline Introduction Evaluation criteria & Methodology Presenting the tools Summary and Recommendation

The company & the problem

Who are PerfTiger Inc.?

I A company for Performance evaluation and optimizationof Computer Systems

I use Queuing Network techniques

I small size (10 people)

I 2 Managers: Strategic decisions

I 6 Junior Researchers : Data collection, analysis, reportmaking, presentation

I 2 Senior Researchers: Project supervision, analysis, new ideas. . .

Faisal & Milani, UOttawa, 25 March 2011 Carleton University

Comparing two Queuing Network Solvers: JMT vs. PDQ

Outline Introduction Evaluation criteria & Methodology Presenting the tools Summary and Recommendation

The company & the problem

What do they need?

I A queuing network solver is needed

I QNAP21: obsolete, lack of user-friendliness and functionalitiesI Two options:

1. JMT (Java Modeling Tools)2

2. PDQ (Pretty Damn Quick)3

1http://hal.inria.fr/inria-00076243/en/2http://jmt.sourceforge.net3http://www.perfdynamics.com/Tools/PDQ.html

Faisal & Milani, UOttawa, 25 March 2011 Carleton University

Comparing two Queuing Network Solvers: JMT vs. PDQ

Outline Introduction Evaluation criteria & Methodology Presenting the tools Summary and Recommendation

The company & the problem

Problems targeted

I Bottleneck analysis for single and multi-class traffic

I Scalability analysis for web application, network etc.

I Capacity planning for any information system

I What-if analysis for change prediction of existing systems

Faisal & Milani, UOttawa, 25 March 2011 Carleton University

Comparing two Queuing Network Solvers: JMT vs. PDQ

Outline Introduction Evaluation criteria & Methodology Presenting the tools Summary and Recommendation

Queuing Networks

A Queuing Station

Figure: The arrival rate and service time can take any statisticaldistribution (e.g., Exponential) and different queue policy (e.g, FCFS,Processor Sharing etc.)

Faisal & Milani, UOttawa, 25 March 2011 Carleton University

Comparing two Queuing Network Solvers: JMT vs. PDQ

Outline Introduction Evaluation criteria & Methodology Presenting the tools Summary and Recommendation

Queuing Networks

Network of queues

Figure: An open Queuing network. We can also have closed,open-closed queuing networks.

I These kinds of queuing networks can be solved by both JMT

and PDQ

Faisal & Milani, UOttawa, 25 March 2011 Carleton University

Comparing two Queuing Network Solvers: JMT vs. PDQ

Outline Introduction Evaluation criteria & Methodology Presenting the tools Summary and Recommendation

General criteria

Criteria

I Deployment: Must run on both Windows and Linux in thecomputers4 the company has.

I Learning curve: Our researcher must be able to learn thetool by 2-4 weeks.

I Usability: It is good if the tool has GUI.

I Cost: The cost of the software must be less than 1000 dollarsI Accuracy:

I the analytic results must be accurate up to 4 decimal points.I the simulated results must give Confidence Interval and

Maximum Relative Error.

4Average PerfTiger machines have quad core processor with 4 GB RAMFaisal & Milani, UOttawa, 25 March 2011 Carleton University

Comparing two Queuing Network Solvers: JMT vs. PDQ

Outline Introduction Evaluation criteria & Methodology Presenting the tools Summary and Recommendation

General criteria

Criteria

I Consistency: the simulated results must fall into the sameConfidence Interval every time we run a model.

I Compatibility: the results of the model should be compatibleto be exported by MS Excel or any other spreadsheet /statistical computing program.

I Documentation: the tool must be well documented.

I Efficiency: for a model with 4 classes and 10 stations,analytic and simulated solvers must solve the model by 1second and 3 minutes respectively.

I Ease of profiling: for simulation, the software should allowthe logging of the jobs moving in the model.

Faisal & Milani, UOttawa, 25 March 2011 Carleton University

Comparing two Queuing Network Solvers: JMT vs. PDQ

Outline Introduction Evaluation criteria & Methodology Presenting the tools Summary and Recommendation

QN theory related criteria

Criteria

I Arrival rate distributions: must support Exponential, Paretoand Constant distribution.

I Service time distributions: must support Exponentialdistribution.

I Queue policy: FCFS must be supported. Good to have otherqueue policies (e.g., LCFS).

I Priority class: having priority class (with and withoutpreemption) is good.

I Class types: Both open and closed classes must besupported.

Faisal & Milani, UOttawa, 25 March 2011 Carleton University

Comparing two Queuing Network Solvers: JMT vs. PDQ

Outline Introduction Evaluation criteria & Methodology Presenting the tools Summary and Recommendation

QN theory related criteria

Criteria

I BCMP models: must be able to solve BCMP modelsanalytically.

I Non-BCMP models: must be able solve models with Fork,Join etc. using approximation or simulation.

I Load-dependent stations: must be able to solve modelswith load-dependent service time.

I What-if analysis: A built-in what-if analyzer would be highlyappreciable.

I Bottleneck identification: Bottleneck analyzer formulti-class model will be a big advantage.

Faisal & Milani, UOttawa, 25 March 2011 Carleton University

Comparing two Queuing Network Solvers: JMT vs. PDQ

Outline Introduction Evaluation criteria & Methodology Presenting the tools Summary and Recommendation

Methodology

Methodology

I Identify 3 groups of stakeholders (Managers, JuniorResearchers, Senior Researchers)

I Put weight on each of the criteria for each stakeholder basedon previous experience

I Make 1 GRL model with 2 scenarios for the two tools

I Find out the tool that satisfies more stakeholders andrecommend that tool.

Faisal & Milani, UOttawa, 25 March 2011 Carleton University

Comparing two Queuing Network Solvers: JMT vs. PDQ

Outline Introduction Evaluation criteria & Methodology Presenting the tools Summary and Recommendation

JMT vs PDQ

Introducing JMT

Figure: JMT (developed by Politecnico di Milano) start-up screenshows 5 available tools to choose from.

Faisal & Milani, UOttawa, 25 March 2011 Carleton University

Comparing two Queuing Network Solvers: JMT vs. PDQ

Outline Introduction Evaluation criteria & Methodology Presenting the tools Summary and Recommendation

JMT vs PDQ

Introducing PDQ

Figure: PDQ (developed by Performance Dynamics Company) has noGUI, is not a stand-alone application but a library of functions (writtenin C) for solving queuing network models.

Faisal & Milani, UOttawa, 25 March 2011 Carleton University

Comparing two Queuing Network Solvers: JMT vs. PDQ

Outline Introduction Evaluation criteria & Methodology Presenting the tools Summary and Recommendation

JMT vs PDQ

Comparison

Criteria JMT PDQ

DeploymentRuns both in Windowsand Linux since devel-oped in Java.

Runs easily in Linux. Re-quires Cygwin or Virtual-box to run on Windows.

Learning curveVery easy and intuitive. An scripting langauge

(e.g., Perl) and PDQ syn-tax must be learnt.

UsabilityGUI and XML input. No GUI, script-based in-

put.

CostFree and open-source. Free and open-source.

SolverAnalytic and simulation. Only analytic.

Table: PDQ vs JMT

Faisal & Milani, UOttawa, 25 March 2011 Carleton University

Comparing two Queuing Network Solvers: JMT vs. PDQ

Outline Introduction Evaluation criteria & Methodology Presenting the tools Summary and Recommendation

JMT vs PDQ

Comparison

Criteria JMT PDQ

Accuracyresults are correct bothfor simulation and ana-lytic solvers.

Gives correct results.

Output com-patibility

Output can be exportedto MS Excel via XML.

Output can be exportedto R package.

DocumentationUser manual, book andonline help available.

User manual, book andonline help available.

EfficiencyBoth analytic and simu-lation based solvers meettime requirement.

The analytic solver meetstime requirement.

Table: PDQ vs JMTFaisal & Milani, UOttawa, 25 March 2011 Carleton University

Comparing two Queuing Network Solvers: JMT vs. PDQ

Outline Introduction Evaluation criteria & Methodology Presenting the tools Summary and Recommendation

JMT vs PDQ

Comparison

Criteria JMT PDQ

Profilingallowed during simula-tion

N/A

Supported distri-butions

Exponential, Constant,Normal and 8 more.

Only Exponential.

Queue policyLCFS and FCFS. LCFS and FCFS.

Priority trafficSupported in simula-tion.

Supported using ap-proximation.

Traffic typesopen, closed, mixed. open, closed, mixed.

Faisal & Milani, UOttawa, 25 March 2011 Carleton University

Comparing two Queuing Network Solvers: JMT vs. PDQ

Outline Introduction Evaluation criteria & Methodology Presenting the tools Summary and Recommendation

JMT vs PDQ

Comparison

Criteria JMT PDQ

non-BCMP mod-els

solved by simulation analytically solved withapproximation.

Load-dependentstations

yes and very easy to do! yes, but the coding isnot intuitive.

What-if analysisyes yes

Bottleneck iden-tification

Easy to do for both sin-gle class and multiclassmodels (using JABA)

Tricky for multiclass.

Faisal & Milani, UOttawa, 25 March 2011 Carleton University

Comparing two Queuing Network Solvers: JMT vs. PDQ

Outline Introduction Evaluation criteria & Methodology Presenting the tools Summary and Recommendation

GRL evaluation

Case: Use PDQ

Figure: GRL Scenario for choosing PDQ

Faisal & Milani, UOttawa, 25 March 2011 Carleton University

Comparing two Queuing Network Solvers: JMT vs. PDQ

Outline Introduction Evaluation criteria & Methodology Presenting the tools Summary and Recommendation

GRL evaluation

Case: Use JMT

Figure: GRL Scenario for choosing JMT

Faisal & Milani, UOttawa, 25 March 2011 Carleton University

Comparing two Queuing Network Solvers: JMT vs. PDQ

Outline Introduction Evaluation criteria & Methodology Presenting the tools Summary and Recommendation

Conclusion

I Based on our evaluation we suggest PerfTiger to chooseJMT as their queuing network solver.

I Our personal experience was used to give weights to thedifferent softgoals.

I Details of the evaluation will be given on the final report.

Faisal & Milani, UOttawa, 25 March 2011 Carleton University

Comparing two Queuing Network Solvers: JMT vs. PDQ

Outline Introduction Evaluation criteria & Methodology Presenting the tools Summary and Recommendation

Conclusion

I Based on our evaluation we suggest PerfTiger to chooseJMT as their queuing network solver.

I Our personal experience was used to give weights to thedifferent softgoals.

I Details of the evaluation will be given on the final report.

Faisal & Milani, UOttawa, 25 March 2011 Carleton University

Comparing two Queuing Network Solvers: JMT vs. PDQ