![Page 1: Algorithm and Experiment Design with HeuristicLab › trac.fcgi › export › HEAD... · Create Batch Runs and Experiments • Batch runs –execute the same optimizer (e.g. algorithm,](https://reader036.vdocuments.mx/reader036/viewer/2022070715/5ed831010fa3e705ec0e017c/html5/thumbnails/1.jpg)
Algorithm and Experiment Designwith HeuristicLab
An Open Source Optimization Environment forResearch and Education
S. Wagner, G. KronbergerHeuristic and Evolutionary Algorithms Laboratory (HEAL)
School of Informatics/Communications/Media, Campus HagenbergUniversity of Applied Sciences Upper Austria
![Page 2: Algorithm and Experiment Design with HeuristicLab › trac.fcgi › export › HEAD... · Create Batch Runs and Experiments • Batch runs –execute the same optimizer (e.g. algorithm,](https://reader036.vdocuments.mx/reader036/viewer/2022070715/5ed831010fa3e705ec0e017c/html5/thumbnails/2.jpg)
Instructor Biographies
• Stefan Wagner– Full professor for complex software systems (since 2009)
University of Applied Sciences Upper Austria– Co-founder of the HEAL research group– Project manager and chief architect of HeuristicLab– PhD in technical sciences (2009)
Johannes Kepler University Linz, Austria– Associate professor (2005 – 2009)
University of Applied Sciences Upper Austria– http://heal.heuristiclab.com/team/wagner
• Gabriel Kronberger– Full professor for business intelligence (since 2011)
University of Applied Sciences Upper Austria– Member of the HEAL research group– Architect of HeuristicLab– PhD in technical sciences (2010)
Johannes Kepler University Linz, Austria– Research assistant (2005 – 2011)
University of Applied Sciences Upper Austria– http://heal.heuristiclab.com/team/kronberger
HeuristicLab Tutorial http://dev.heuristiclab.com 2
![Page 3: Algorithm and Experiment Design with HeuristicLab › trac.fcgi › export › HEAD... · Create Batch Runs and Experiments • Batch runs –execute the same optimizer (e.g. algorithm,](https://reader036.vdocuments.mx/reader036/viewer/2022070715/5ed831010fa3e705ec0e017c/html5/thumbnails/3.jpg)
Agenda
• Objectives of the Tutorial• Introduction• Where to get HeuristicLab?• Plugin Infrastructure• Graphical User Interface• Available Algorithms & Problems
• Demonstration Part I: Working with HeuristicLab• Demonstration Part II: Data-based Modeling
• Some Additional Features• Planned Features• Team• Suggested Readings• Bibliography• Questions & Answers
HeuristicLab Tutorial http://dev.heuristiclab.com 3
![Page 4: Algorithm and Experiment Design with HeuristicLab › trac.fcgi › export › HEAD... · Create Batch Runs and Experiments • Batch runs –execute the same optimizer (e.g. algorithm,](https://reader036.vdocuments.mx/reader036/viewer/2022070715/5ed831010fa3e705ec0e017c/html5/thumbnails/4.jpg)
Objectives of the Tutorial
• Introduce general motivation and design principles of HeuristicLab
• Show where to get HeuristicLab
• Explain basic GUI usability concepts
• Demonstrate basic features
• Demonstrate editing and analysis of optimization experiments
• Demonstrate custom algorithms and graphical algorithm designer
• Demonstrate data-based modeling features
• Outline some additional features
HeuristicLab Tutorial http://dev.heuristiclab.com 4
![Page 5: Algorithm and Experiment Design with HeuristicLab › trac.fcgi › export › HEAD... · Create Batch Runs and Experiments • Batch runs –execute the same optimizer (e.g. algorithm,](https://reader036.vdocuments.mx/reader036/viewer/2022070715/5ed831010fa3e705ec0e017c/html5/thumbnails/5.jpg)
Introduction
• Motivation and Goals– graphical user interface– paradigm independence– multiple algorithms and problems– large scale experiments and analyses– parallelization– extensibility, flexibility and reusability– visual and interactive algorithm development– multiple layers of abstraction
• Facts– development of HeuristicLab started in 2002– based on Microsoft .NET and C#– used in research and education– second place at the Microsoft Innovation Award 2009– open source (GNU General Public License)– version 3.3.0 released on May 18th, 2010– latest version 3.3.16 "Prague"
HeuristicLab Tutorial http://dev.heuristiclab.com 5
![Page 6: Algorithm and Experiment Design with HeuristicLab › trac.fcgi › export › HEAD... · Create Batch Runs and Experiments • Batch runs –execute the same optimizer (e.g. algorithm,](https://reader036.vdocuments.mx/reader036/viewer/2022070715/5ed831010fa3e705ec0e017c/html5/thumbnails/6.jpg)
Where to get HeuristicLab?
• Download binaries– deployed as ZIP archives– latest stable version 3.3.16 "Prague"
• released on July 31st, 2019
– daily trunk builds– http://dev.heuristiclab.com/download
• Check out sources– SVN repository– HeuristicLab 3.3.16 tag
• https://src.heuristiclab.com/svn/core/tags/3.3.16
– Stable development version• https://src.heuristiclab.com/svn/core/stable
• License– GNU General Public License (Version 3)
• System requirements– Microsoft .NET Framework 4.5– enough RAM and CPU power ;-)
HeuristicLab Tutorial http://dev.heuristiclab.com 6
![Page 7: Algorithm and Experiment Design with HeuristicLab › trac.fcgi › export › HEAD... · Create Batch Runs and Experiments • Batch runs –execute the same optimizer (e.g. algorithm,](https://reader036.vdocuments.mx/reader036/viewer/2022070715/5ed831010fa3e705ec0e017c/html5/thumbnails/7.jpg)
Plugin Infrastructure
• HeuristicLab consists of many assemblies– 160+ plugins in HeuristicLab 3.3.16– plugins can be loaded or unloaded at runtime– plugins can be updated via internet– application plugins provide GUI frontends
• Extensibility– developing and deploying new plugins is easy– dependencies are explicitly defined,
automatically checked and resolved– automatic discovery of interface
implementations (service locator pattern)
• Plugin Manager– GUI to check, install, update or delete plugins
HeuristicLab Tutorial http://dev.heuristiclab.com 7
![Page 8: Algorithm and Experiment Design with HeuristicLab › trac.fcgi › export › HEAD... · Create Batch Runs and Experiments • Batch runs –execute the same optimizer (e.g. algorithm,](https://reader036.vdocuments.mx/reader036/viewer/2022070715/5ed831010fa3e705ec0e017c/html5/thumbnails/8.jpg)
Plugin Architecture
HeuristicLab Tutorial http://dev.heuristiclab.com 8
![Page 9: Algorithm and Experiment Design with HeuristicLab › trac.fcgi › export › HEAD... · Create Batch Runs and Experiments • Batch runs –execute the same optimizer (e.g. algorithm,](https://reader036.vdocuments.mx/reader036/viewer/2022070715/5ed831010fa3e705ec0e017c/html5/thumbnails/9.jpg)
Graphical User Interface
• HeuristicLab GUI is made up of views– views are visual representations of content objects
– views are composed in the same way as their content
– views and content objects are loosely coupled
– multiple different views may exist for the same content
• Drag & Drop– views support drag & drop operations
– content objects can be copied or moved (shift key)
– enabled for collection items and content objects
HeuristicLab Tutorial http://dev.heuristiclab.com 9
![Page 10: Algorithm and Experiment Design with HeuristicLab › trac.fcgi › export › HEAD... · Create Batch Runs and Experiments • Batch runs –execute the same optimizer (e.g. algorithm,](https://reader036.vdocuments.mx/reader036/viewer/2022070715/5ed831010fa3e705ec0e017c/html5/thumbnails/10.jpg)
Graphical User Interface
HeuristicLab Tutorial http://dev.heuristiclab.com 10
Algorithm View
Problem View
ParameterCollectionView
Parameter View
Double Value View
![Page 11: Algorithm and Experiment Design with HeuristicLab › trac.fcgi › export › HEAD... · Create Batch Runs and Experiments • Batch runs –execute the same optimizer (e.g. algorithm,](https://reader036.vdocuments.mx/reader036/viewer/2022070715/5ed831010fa3e705ec0e017c/html5/thumbnails/11.jpg)
Graphical User Interface
• ViewHost– control which hosts views– right-click on windows icon to switch views– double-click on windows icon to open another view– drag & drop windows icon to copy contents
HeuristicLab Tutorial http://dev.heuristiclab.com 11
![Page 12: Algorithm and Experiment Design with HeuristicLab › trac.fcgi › export › HEAD... · Create Batch Runs and Experiments • Batch runs –execute the same optimizer (e.g. algorithm,](https://reader036.vdocuments.mx/reader036/viewer/2022070715/5ed831010fa3e705ec0e017c/html5/thumbnails/12.jpg)
Available AlgorithmsPopulation-based
• CMA-ES
• Evolution Strategy
• Genetic Algorithm
• Offspring Selection Genetic Algorithm (OSGA)
• Island Genetic Algorithm
• Island Offspring Selection Genetic Algorithm
• Parameter-less Population Pyramid (P3)
• SASEGASA
• Relevant Alleles Preserving GA (RAPGA)
• Age-Layered Population Structure (ALPS)
• Genetic Programming
• NSGA-II
• Scatter Search
• Particle Swarm Optimization
Trajectory-based
• Local Search
• Tabu Search
• Robust Taboo Search
• Variable Neighborhood Search
• Simulated Annealing
HeuristicLab Tutorial http://dev.heuristiclab.com 12
Data Analysis
• Linear Discriminant Analysis
• Linear Regression
• Multinomial Logit Classification
• k-Nearest Neighbor
• k-Means
• Neighbourhood Component Analysis
• Artificial Neural Networks
• Random Forests
• Support Vector Machines
• Gaussian Processes
• Gradient Boosted Trees
• Gradient Boosted Regression
• Barnes-Hut t-SNE
• Kernel Ridge Regression
• Elastic-net Regression
Additional Algorithms
• Performance Benchmarks
• Hungarian Algorithm
• Cross Validation
• LM-BFGS
![Page 13: Algorithm and Experiment Design with HeuristicLab › trac.fcgi › export › HEAD... · Create Batch Runs and Experiments • Batch runs –execute the same optimizer (e.g. algorithm,](https://reader036.vdocuments.mx/reader036/viewer/2022070715/5ed831010fa3e705ec0e017c/html5/thumbnails/13.jpg)
Available ProblemsCombinatorial Problems
• Test Problems (1-max, NK, HIFF, Deceptive Trap)
• Traveling Salesman
• Probabilistic Traveling Salesman
• Vehicle Routing (MDCVRPTW, PDPTW, …)
• Knapsack
• Bin Packing
• Graph Coloring
• Job Shop Scheduling
• Linear Assignment
• Quadratic Assignment
• Orienteering
Genetic Programming Problems
• Test Problems (Even Parity, MUX)
• Symbolic Classification
• Symbolic Regression
• Symbolic Time-Series Prognosis
• Artificial Ant
• Lawn Mower
• Robocode
• Grammatical Evolution
Additional Problems
• Single-/Multi-Objective Test Function– Ackley, Griewank, Rastrigin, Sphere, etc.
• Programmable Problem
• External Evaluation Problem (generic TCP/IP, Scilab, MATLAB)
• Regression, Classification, Clustering
• Trading
HeuristicLab Tutorial http://dev.heuristiclab.com 13
![Page 14: Algorithm and Experiment Design with HeuristicLab › trac.fcgi › export › HEAD... · Create Batch Runs and Experiments • Batch runs –execute the same optimizer (e.g. algorithm,](https://reader036.vdocuments.mx/reader036/viewer/2022070715/5ed831010fa3e705ec0e017c/html5/thumbnails/14.jpg)
Agenda
• Objectives of the Tutorial• Introduction• Where to get HeuristicLab?• Plugin Infrastructure• Graphical User Interface• Available Algorithms & Problems
• Demonstration Part I: Working with HeuristicLab• Demonstration Part II: Data-based Modeling
• Some Additional Features• Planned Features• Team• Suggested Readings• Bibliography• Questions & Answers
HeuristicLab Tutorial http://dev.heuristiclab.com 14
![Page 15: Algorithm and Experiment Design with HeuristicLab › trac.fcgi › export › HEAD... · Create Batch Runs and Experiments • Batch runs –execute the same optimizer (e.g. algorithm,](https://reader036.vdocuments.mx/reader036/viewer/2022070715/5ed831010fa3e705ec0e017c/html5/thumbnails/15.jpg)
Demonstration Part I:Working with HeuristicLab
• Create, Parameterize and Execute Algorithms
• Save and Load Items
• Create Batch Runs and Experiments
• Multi-core CPUs and Parallelization
• Analyze Runs
• Analyzers
• Building User-Defined Algorithms
HeuristicLab Tutorial http://dev.heuristiclab.com 15
![Page 16: Algorithm and Experiment Design with HeuristicLab › trac.fcgi › export › HEAD... · Create Batch Runs and Experiments • Batch runs –execute the same optimizer (e.g. algorithm,](https://reader036.vdocuments.mx/reader036/viewer/2022070715/5ed831010fa3e705ec0e017c/html5/thumbnails/16.jpg)
HeuristicLab Optimizer
HeuristicLab Tutorial http://dev.heuristiclab.com 16
double-click to opensample algorithms
and problems
![Page 17: Algorithm and Experiment Design with HeuristicLab › trac.fcgi › export › HEAD... · Create Batch Runs and Experiments • Batch runs –execute the same optimizer (e.g. algorithm,](https://reader036.vdocuments.mx/reader036/viewer/2022070715/5ed831010fa3e705ec0e017c/html5/thumbnails/17.jpg)
Create Algorithm
HeuristicLab Tutorial http://dev.heuristiclab.com 17
![Page 18: Algorithm and Experiment Design with HeuristicLab › trac.fcgi › export › HEAD... · Create Batch Runs and Experiments • Batch runs –execute the same optimizer (e.g. algorithm,](https://reader036.vdocuments.mx/reader036/viewer/2022070715/5ed831010fa3e705ec0e017c/html5/thumbnails/18.jpg)
Create or Load Problem
HeuristicLab Tutorial http://dev.heuristiclab.com 18
![Page 19: Algorithm and Experiment Design with HeuristicLab › trac.fcgi › export › HEAD... · Create Batch Runs and Experiments • Batch runs –execute the same optimizer (e.g. algorithm,](https://reader036.vdocuments.mx/reader036/viewer/2022070715/5ed831010fa3e705ec0e017c/html5/thumbnails/19.jpg)
Import or Parameterize Problem Data
HeuristicLab Tutorial http://dev.heuristiclab.com 19
![Page 20: Algorithm and Experiment Design with HeuristicLab › trac.fcgi › export › HEAD... · Create Batch Runs and Experiments • Batch runs –execute the same optimizer (e.g. algorithm,](https://reader036.vdocuments.mx/reader036/viewer/2022070715/5ed831010fa3e705ec0e017c/html5/thumbnails/20.jpg)
Parameterize Algorithm
HeuristicLab Tutorial http://dev.heuristiclab.com 20
![Page 21: Algorithm and Experiment Design with HeuristicLab › trac.fcgi › export › HEAD... · Create Batch Runs and Experiments • Batch runs –execute the same optimizer (e.g. algorithm,](https://reader036.vdocuments.mx/reader036/viewer/2022070715/5ed831010fa3e705ec0e017c/html5/thumbnails/21.jpg)
Start, Pause, Resume, Stop and Reset
HeuristicLab Tutorial http://dev.heuristiclab.com 21
![Page 22: Algorithm and Experiment Design with HeuristicLab › trac.fcgi › export › HEAD... · Create Batch Runs and Experiments • Batch runs –execute the same optimizer (e.g. algorithm,](https://reader036.vdocuments.mx/reader036/viewer/2022070715/5ed831010fa3e705ec0e017c/html5/thumbnails/22.jpg)
Inspect Results
HeuristicLab Tutorial http://dev.heuristiclab.com 22
![Page 23: Algorithm and Experiment Design with HeuristicLab › trac.fcgi › export › HEAD... · Create Batch Runs and Experiments • Batch runs –execute the same optimizer (e.g. algorithm,](https://reader036.vdocuments.mx/reader036/viewer/2022070715/5ed831010fa3e705ec0e017c/html5/thumbnails/23.jpg)
Compare Runs
• A run is created each time when the algorithm is stopped– runs contain all results and parameter settings– previous results are not forgotten and can be compared
HeuristicLab Tutorial http://dev.heuristiclab.com 23
![Page 24: Algorithm and Experiment Design with HeuristicLab › trac.fcgi › export › HEAD... · Create Batch Runs and Experiments • Batch runs –execute the same optimizer (e.g. algorithm,](https://reader036.vdocuments.mx/reader036/viewer/2022070715/5ed831010fa3e705ec0e017c/html5/thumbnails/24.jpg)
Save and Load
• Save to and load from disk– HeuristicLab items (i.e., algorithms, problems, experiments, …)
can be saved to and loaded from a file– algorithms can be paused, saved, loaded and resumed– data format is custom compressed XML– saving and loading files might take several minutes– saving and loading large experiments requires some memory
HeuristicLab Tutorial http://dev.heuristiclab.com 24
![Page 25: Algorithm and Experiment Design with HeuristicLab › trac.fcgi › export › HEAD... · Create Batch Runs and Experiments • Batch runs –execute the same optimizer (e.g. algorithm,](https://reader036.vdocuments.mx/reader036/viewer/2022070715/5ed831010fa3e705ec0e017c/html5/thumbnails/25.jpg)
Create Batch Runs and Experiments
• Batch runs– execute the same optimizer (e.g. algorithm, batch run, experiment)
several times
• Experiments– execute different optimizers– suitable for large scale algorithm comparison and analysis
• Experiments and batch runs can be nested
• Generated runs can be compared afterwards
HeuristicLab Tutorial http://dev.heuristiclab.com 25
![Page 26: Algorithm and Experiment Design with HeuristicLab › trac.fcgi › export › HEAD... · Create Batch Runs and Experiments • Batch runs –execute the same optimizer (e.g. algorithm,](https://reader036.vdocuments.mx/reader036/viewer/2022070715/5ed831010fa3e705ec0e017c/html5/thumbnails/26.jpg)
Create Batch Runs and Experiments
HeuristicLab Tutorial http://dev.heuristiclab.com 26
drag & drop here
to add additional
algorithms,
batch runs,
experiments, etc.
![Page 27: Algorithm and Experiment Design with HeuristicLab › trac.fcgi › export › HEAD... · Create Batch Runs and Experiments • Batch runs –execute the same optimizer (e.g. algorithm,](https://reader036.vdocuments.mx/reader036/viewer/2022070715/5ed831010fa3e705ec0e017c/html5/thumbnails/27.jpg)
Clipboard
• Store items– click on the buttons to add or remove items– drag & drop items on the clipboard– use the menu to add a copy of a shown item to the clipboard
• Show items– double-click on an item in the clipboard to show its view
• Save and restore clipboard content– click on the save button to write the clipboard content to disk– clipboard is automatically restored when HeuristicLab is started the next time
HeuristicLab Tutorial http://dev.heuristiclab.com 27
![Page 28: Algorithm and Experiment Design with HeuristicLab › trac.fcgi › export › HEAD... · Create Batch Runs and Experiments • Batch runs –execute the same optimizer (e.g. algorithm,](https://reader036.vdocuments.mx/reader036/viewer/2022070715/5ed831010fa3e705ec0e017c/html5/thumbnails/28.jpg)
Clipboard
HeuristicLab Tutorial http://dev.heuristiclab.com 28
drag & drop
here to add
algorithms,
problems,
batch runs,
experiments,
etc.
![Page 29: Algorithm and Experiment Design with HeuristicLab › trac.fcgi › export › HEAD... · Create Batch Runs and Experiments • Batch runs –execute the same optimizer (e.g. algorithm,](https://reader036.vdocuments.mx/reader036/viewer/2022070715/5ed831010fa3e705ec0e017c/html5/thumbnails/29.jpg)
Start, Pause, Resume, Stop, Reset
HeuristicLab Tutorial http://dev.heuristiclab.com 29
![Page 30: Algorithm and Experiment Design with HeuristicLab › trac.fcgi › export › HEAD... · Create Batch Runs and Experiments • Batch runs –execute the same optimizer (e.g. algorithm,](https://reader036.vdocuments.mx/reader036/viewer/2022070715/5ed831010fa3e705ec0e017c/html5/thumbnails/30.jpg)
Compare Runs
HeuristicLab Tutorial http://dev.heuristiclab.com 30
![Page 31: Algorithm and Experiment Design with HeuristicLab › trac.fcgi › export › HEAD... · Create Batch Runs and Experiments • Batch runs –execute the same optimizer (e.g. algorithm,](https://reader036.vdocuments.mx/reader036/viewer/2022070715/5ed831010fa3e705ec0e017c/html5/thumbnails/31.jpg)
Analyze Runs
• HeuristicLab provides interactive views to analyze and compare all runs of a run collection– textual analysis
• RunCollection Tabular View
– graphical analysis• RunCollection BubbleChart• RunCollection BoxPlots
• Filtering is automatically applied to all open run collection views
HeuristicLab Tutorial http://dev.heuristiclab.com 31
![Page 32: Algorithm and Experiment Design with HeuristicLab › trac.fcgi › export › HEAD... · Create Batch Runs and Experiments • Batch runs –execute the same optimizer (e.g. algorithm,](https://reader036.vdocuments.mx/reader036/viewer/2022070715/5ed831010fa3e705ec0e017c/html5/thumbnails/32.jpg)
Runs – Tabular View
HeuristicLab Tutorial http://dev.heuristiclab.com 32
![Page 33: Algorithm and Experiment Design with HeuristicLab › trac.fcgi › export › HEAD... · Create Batch Runs and Experiments • Batch runs –execute the same optimizer (e.g. algorithm,](https://reader036.vdocuments.mx/reader036/viewer/2022070715/5ed831010fa3e705ec0e017c/html5/thumbnails/33.jpg)
Runs – Tabular View
• Sort columns– click on column header to sort column– Ctrl-click on column header to sort multiple columns
• Show or hide columns– right-click on table to open dialog to show or hide columns
• Compute statistical values– select multiple numerical values to see count, sum, minimum,
maximum, average and standard deviation
• Select, copy and paste into other applications
HeuristicLab Tutorial http://dev.heuristiclab.com 33
![Page 34: Algorithm and Experiment Design with HeuristicLab › trac.fcgi › export › HEAD... · Create Batch Runs and Experiments • Batch runs –execute the same optimizer (e.g. algorithm,](https://reader036.vdocuments.mx/reader036/viewer/2022070715/5ed831010fa3e705ec0e017c/html5/thumbnails/34.jpg)
Runs – BubbleChart
HeuristicLab Tutorial http://dev.heuristiclab.com 34
![Page 35: Algorithm and Experiment Design with HeuristicLab › trac.fcgi › export › HEAD... · Create Batch Runs and Experiments • Batch runs –execute the same optimizer (e.g. algorithm,](https://reader036.vdocuments.mx/reader036/viewer/2022070715/5ed831010fa3e705ec0e017c/html5/thumbnails/35.jpg)
Runs – BubbleChart
• Choose values to plot– choose which values to show on the x-axis, the y-axis and as bubble size– possible values are all parameter settings and results
• Add jitter– add jitter to separate overlapping bubbles
• Zoom in and out– click on Zoom and click and drag in the chart area to zoom in– double click on the chart area background or on the circle buttons beside the scroll bars to zoom out
• Color bubbles– click on Select, choose a color and click and drag in the chart area to select and color bubbles– apply coloring automatically by clicking on the axis coloring buttons
• Show runs– double click on a bubble to open its run
• Export image– right-click to open context menu to copy or save image– save image as pixel (BMP, JPG, PNG, GIF, TIF) or vector graphics (EMF)
• Show box plots– right-click to open context menu to show box plots view
HeuristicLab Tutorial http://dev.heuristiclab.com 35
![Page 36: Algorithm and Experiment Design with HeuristicLab › trac.fcgi › export › HEAD... · Create Batch Runs and Experiments • Batch runs –execute the same optimizer (e.g. algorithm,](https://reader036.vdocuments.mx/reader036/viewer/2022070715/5ed831010fa3e705ec0e017c/html5/thumbnails/36.jpg)
Runs – BoxPlots
HeuristicLab Tutorial http://dev.heuristiclab.com 36
![Page 37: Algorithm and Experiment Design with HeuristicLab › trac.fcgi › export › HEAD... · Create Batch Runs and Experiments • Batch runs –execute the same optimizer (e.g. algorithm,](https://reader036.vdocuments.mx/reader036/viewer/2022070715/5ed831010fa3e705ec0e017c/html5/thumbnails/37.jpg)
Runs – BoxPlots
• Choose values to plot– choose which values to show on the x-axis and y-axis– possible values are all parameter settings and results
• Zoom in and out– click on Zoom and click and drag in the chart area to zoom in– double click on the chart area background or on the circle buttons
beside the scroll bars to zoom out
• Show or hide statistical values– click on the lower left button to show or hide statistical values
• Export image– right-click to open context menu to copy or save image– save image as pixel (BMP, JPG, PNG, GIF, TIF) or vector graphics (EMF)
HeuristicLab Tutorial http://dev.heuristiclab.com 37
![Page 38: Algorithm and Experiment Design with HeuristicLab › trac.fcgi › export › HEAD... · Create Batch Runs and Experiments • Batch runs –execute the same optimizer (e.g. algorithm,](https://reader036.vdocuments.mx/reader036/viewer/2022070715/5ed831010fa3e705ec0e017c/html5/thumbnails/38.jpg)
Runs – Multi-Line Chart
HeuristicLab Tutorial http://dev.heuristiclab.com 38
![Page 39: Algorithm and Experiment Design with HeuristicLab › trac.fcgi › export › HEAD... · Create Batch Runs and Experiments • Batch runs –execute the same optimizer (e.g. algorithm,](https://reader036.vdocuments.mx/reader036/viewer/2022070715/5ed831010fa3e705ec0e017c/html5/thumbnails/39.jpg)
Filter Runs
HeuristicLab Tutorial http://dev.heuristiclab.com 39
![Page 40: Algorithm and Experiment Design with HeuristicLab › trac.fcgi › export › HEAD... · Create Batch Runs and Experiments • Batch runs –execute the same optimizer (e.g. algorithm,](https://reader036.vdocuments.mx/reader036/viewer/2022070715/5ed831010fa3e705ec0e017c/html5/thumbnails/40.jpg)
Multi-core CPUs and Parallelization
• Parallel execution of optimizers in experiments– optimizers in an experiment are executed sequentially from top to bottom per
default– experiments support parallel execution of their optimizers– select a not yet executed optimizer and start it manually to utilize another
core– execution of one of the next optimizers is started automatically after an
optimizer is finished
• Parallel execution of algorithms– HeuristicLab provides special operators for parallelization– engines decide how to execute parallel operations– sequential engine executes everything sequentially– parallel engine executes parallel operations on multiple cores– Hive engine (under development) executes parallel operations on multiple
computers– all implemented algorithms support parallel solution evaluation
HeuristicLab Tutorial http://dev.heuristiclab.com 40
![Page 41: Algorithm and Experiment Design with HeuristicLab › trac.fcgi › export › HEAD... · Create Batch Runs and Experiments • Batch runs –execute the same optimizer (e.g. algorithm,](https://reader036.vdocuments.mx/reader036/viewer/2022070715/5ed831010fa3e705ec0e017c/html5/thumbnails/41.jpg)
Parallel Execution of Experiments
HeuristicLab Tutorial http://dev.heuristiclab.com 41
1. start experiment
2. start other optimizers
![Page 42: Algorithm and Experiment Design with HeuristicLab › trac.fcgi › export › HEAD... · Create Batch Runs and Experiments • Batch runs –execute the same optimizer (e.g. algorithm,](https://reader036.vdocuments.mx/reader036/viewer/2022070715/5ed831010fa3e705ec0e017c/html5/thumbnails/42.jpg)
Parallel Execution of Algorithms
HeuristicLab Tutorial http://dev.heuristiclab.com 42
![Page 43: Algorithm and Experiment Design with HeuristicLab › trac.fcgi › export › HEAD... · Create Batch Runs and Experiments • Batch runs –execute the same optimizer (e.g. algorithm,](https://reader036.vdocuments.mx/reader036/viewer/2022070715/5ed831010fa3e705ec0e017c/html5/thumbnails/43.jpg)
Analyzers
• Special operators for analysis purposes– are executed after each iteration– serve as general purpose extension points of algorithms– can be selected and parameterized in the algorithm– perform algorithm-specific and/or problem-specific tasks– some analyzers are quite costly regarding runtime and memory– implementing and adding custom analyzers is easy
• Examples– TSPAlleleFrequencyAnalyzer– TSPPopulationDiversityAnalyzer– SuccessfulOffspringAnalyzer– SymbolicDataAnalysisVariableFrequencyAnalyzer– SymbolicRegressionSingleObjectiveTrainingBestSolutionAnalyzer– …
HeuristicLab Tutorial http://dev.heuristiclab.com 43
![Page 44: Algorithm and Experiment Design with HeuristicLab › trac.fcgi › export › HEAD... · Create Batch Runs and Experiments • Batch runs –execute the same optimizer (e.g. algorithm,](https://reader036.vdocuments.mx/reader036/viewer/2022070715/5ed831010fa3e705ec0e017c/html5/thumbnails/44.jpg)
Analyzers
HeuristicLab Tutorial http://dev.heuristiclab.com 44
![Page 45: Algorithm and Experiment Design with HeuristicLab › trac.fcgi › export › HEAD... · Create Batch Runs and Experiments • Batch runs –execute the same optimizer (e.g. algorithm,](https://reader036.vdocuments.mx/reader036/viewer/2022070715/5ed831010fa3e705ec0e017c/html5/thumbnails/45.jpg)
TSPAlleleFrequencyAnalyzer
HeuristicLab Tutorial http://dev.heuristiclab.com 45
![Page 46: Algorithm and Experiment Design with HeuristicLab › trac.fcgi › export › HEAD... · Create Batch Runs and Experiments • Batch runs –execute the same optimizer (e.g. algorithm,](https://reader036.vdocuments.mx/reader036/viewer/2022070715/5ed831010fa3e705ec0e017c/html5/thumbnails/46.jpg)
TSPPopulationDiversityAnalyzer
HeuristicLab Tutorial http://dev.heuristiclab.com 46
![Page 47: Algorithm and Experiment Design with HeuristicLab › trac.fcgi › export › HEAD... · Create Batch Runs and Experiments • Batch runs –execute the same optimizer (e.g. algorithm,](https://reader036.vdocuments.mx/reader036/viewer/2022070715/5ed831010fa3e705ec0e017c/html5/thumbnails/47.jpg)
Building User-Defined Algorithms
• Operator graphs– algorithms are represented as operator graphs– operator graphs of user-defined algorithms can be changed– algorithms can be defined in the graphical algorithm designer– use the menu to convert a standard algorithm into a user-defined algorithm
• Operators sidebar– drag & drop operators into an operator graph
• Programmable operators– add programmable operators in order to implement custom logic in an algorithm– no additional development environment needed
• Debug algorithms– use the debug engine to obtain detailed information during algorithm execution
HeuristicLab Tutorial http://dev.heuristiclab.com 47
![Page 48: Algorithm and Experiment Design with HeuristicLab › trac.fcgi › export › HEAD... · Create Batch Runs and Experiments • Batch runs –execute the same optimizer (e.g. algorithm,](https://reader036.vdocuments.mx/reader036/viewer/2022070715/5ed831010fa3e705ec0e017c/html5/thumbnails/48.jpg)
Building User-Defined Algorithms
HeuristicLab Tutorial http://dev.heuristiclab.com 48
![Page 49: Algorithm and Experiment Design with HeuristicLab › trac.fcgi › export › HEAD... · Create Batch Runs and Experiments • Batch runs –execute the same optimizer (e.g. algorithm,](https://reader036.vdocuments.mx/reader036/viewer/2022070715/5ed831010fa3e705ec0e017c/html5/thumbnails/49.jpg)
Building User-Defined Algorithms
HeuristicLab Tutorial http://dev.heuristiclab.com 49
![Page 50: Algorithm and Experiment Design with HeuristicLab › trac.fcgi › export › HEAD... · Create Batch Runs and Experiments • Batch runs –execute the same optimizer (e.g. algorithm,](https://reader036.vdocuments.mx/reader036/viewer/2022070715/5ed831010fa3e705ec0e017c/html5/thumbnails/50.jpg)
Programmable Operators
HeuristicLab Tutorial http://dev.heuristiclab.com 50
![Page 51: Algorithm and Experiment Design with HeuristicLab › trac.fcgi › export › HEAD... · Create Batch Runs and Experiments • Batch runs –execute the same optimizer (e.g. algorithm,](https://reader036.vdocuments.mx/reader036/viewer/2022070715/5ed831010fa3e705ec0e017c/html5/thumbnails/51.jpg)
Scripts
HeuristicLab Tutorial http://dev.heuristiclab.com 51
![Page 52: Algorithm and Experiment Design with HeuristicLab › trac.fcgi › export › HEAD... · Create Batch Runs and Experiments • Batch runs –execute the same optimizer (e.g. algorithm,](https://reader036.vdocuments.mx/reader036/viewer/2022070715/5ed831010fa3e705ec0e017c/html5/thumbnails/52.jpg)
Debugging Algorithms
HeuristicLab Tutorial http://dev.heuristiclab.com 52
![Page 53: Algorithm and Experiment Design with HeuristicLab › trac.fcgi › export › HEAD... · Create Batch Runs and Experiments • Batch runs –execute the same optimizer (e.g. algorithm,](https://reader036.vdocuments.mx/reader036/viewer/2022070715/5ed831010fa3e705ec0e017c/html5/thumbnails/53.jpg)
Agenda
• Objectives of the Tutorial• Introduction• Where to get HeuristicLab?• Plugin Infrastructure• Graphical User Interface• Available Algorithms & Problems
• Demonstration Part I: Working with HeuristicLab• Demonstration Part II: Data-based Modeling
• Some Additional Features• Planned Features• Team• Suggested Readings• Bibliography• Questions & Answers
HeuristicLab Tutorial http://dev.heuristiclab.com 53
![Page 54: Algorithm and Experiment Design with HeuristicLab › trac.fcgi › export › HEAD... · Create Batch Runs and Experiments • Batch runs –execute the same optimizer (e.g. algorithm,](https://reader036.vdocuments.mx/reader036/viewer/2022070715/5ed831010fa3e705ec0e017c/html5/thumbnails/54.jpg)
Demonstration Part II:Data-based Modeling
• Introduction
• Regression with HeuristicLab
• Model simplification and export
• Variable relevance analysis
• Classification with HeuristicLab
HeuristicLab Tutorial http://dev.heuristiclab.com 54
![Page 55: Algorithm and Experiment Design with HeuristicLab › trac.fcgi › export › HEAD... · Create Batch Runs and Experiments • Batch runs –execute the same optimizer (e.g. algorithm,](https://reader036.vdocuments.mx/reader036/viewer/2022070715/5ed831010fa3e705ec0e017c/html5/thumbnails/55.jpg)
Introduction to Data-based Modeling
• Dataset: Matrix (xi,j) i=1..N, j= 1..K
– N observations of K input variables
– xi,j = i-th observation of j-th variable
– Additionally: Vector of labels (y1…yN)T
• Goal: learn association of input variable valuesto labels
HeuristicLab Tutorial http://dev.heuristiclab.com 55
![Page 56: Algorithm and Experiment Design with HeuristicLab › trac.fcgi › export › HEAD... · Create Batch Runs and Experiments • Batch runs –execute the same optimizer (e.g. algorithm,](https://reader036.vdocuments.mx/reader036/viewer/2022070715/5ed831010fa3e705ec0e017c/html5/thumbnails/56.jpg)
Data Analysis in HeuristicLab
• Symbolic regression and classification using genetic programming
• External Libraries:
– Linear Regression, Logistic Regression,
– k-Nearest Neighbours, k-Means,
– Random Forest, Support Vector Machines, Neural Networks, Gaussian Processes
HeuristicLab Tutorial http://dev.heuristiclab.com 56
![Page 57: Algorithm and Experiment Design with HeuristicLab › trac.fcgi › export › HEAD... · Create Batch Runs and Experiments • Batch runs –execute the same optimizer (e.g. algorithm,](https://reader036.vdocuments.mx/reader036/viewer/2022070715/5ed831010fa3e705ec0e017c/html5/thumbnails/57.jpg)
Case Study: Regression
• Poly-10 benchmark problem dataset– 10 input variables x1 … x10
– y = x1x2 + x3x4 + x5x6 + x1x7x9 + x3x6x10
– non-linear modeling approach necessary
– frequently used in GP literature
– available as benchmark problem instance in HeuristicLab
HeuristicLab Tutorial http://dev.heuristiclab.com 57
![Page 58: Algorithm and Experiment Design with HeuristicLab › trac.fcgi › export › HEAD... · Create Batch Runs and Experiments • Batch runs –execute the same optimizer (e.g. algorithm,](https://reader036.vdocuments.mx/reader036/viewer/2022070715/5ed831010fa3e705ec0e017c/html5/thumbnails/58.jpg)
Demonstration
• problem configuration– data import– target and input variables– data partitions (training and test)
• algorithm configuration
• analysis of results– accuracy metrics– visualization of model output
HeuristicLab Tutorial http://dev.heuristiclab.com 58
![Page 59: Algorithm and Experiment Design with HeuristicLab › trac.fcgi › export › HEAD... · Create Batch Runs and Experiments • Batch runs –execute the same optimizer (e.g. algorithm,](https://reader036.vdocuments.mx/reader036/viewer/2022070715/5ed831010fa3e705ec0e017c/html5/thumbnails/59.jpg)
Linear Regression
• Create new algorithm
HeuristicLab Tutorial http://dev.heuristiclab.com 59
![Page 60: Algorithm and Experiment Design with HeuristicLab › trac.fcgi › export › HEAD... · Create Batch Runs and Experiments • Batch runs –execute the same optimizer (e.g. algorithm,](https://reader036.vdocuments.mx/reader036/viewer/2022070715/5ed831010fa3e705ec0e017c/html5/thumbnails/60.jpg)
Import Data from CSV-File
HeuristicLab Tutorial http://dev.heuristiclab.com 60
![Page 61: Algorithm and Experiment Design with HeuristicLab › trac.fcgi › export › HEAD... · Create Batch Runs and Experiments • Batch runs –execute the same optimizer (e.g. algorithm,](https://reader036.vdocuments.mx/reader036/viewer/2022070715/5ed831010fa3e705ec0e017c/html5/thumbnails/61.jpg)
Inspect and Configure Dataset
HeuristicLab Tutorial http://dev.heuristiclab.com 61
![Page 62: Algorithm and Experiment Design with HeuristicLab › trac.fcgi › export › HEAD... · Create Batch Runs and Experiments • Batch runs –execute the same optimizer (e.g. algorithm,](https://reader036.vdocuments.mx/reader036/viewer/2022070715/5ed831010fa3e705ec0e017c/html5/thumbnails/62.jpg)
Inspect Imported Data
HeuristicLab Tutorial http://dev.heuristiclab.com 62
![Page 63: Algorithm and Experiment Design with HeuristicLab › trac.fcgi › export › HEAD... · Create Batch Runs and Experiments • Batch runs –execute the same optimizer (e.g. algorithm,](https://reader036.vdocuments.mx/reader036/viewer/2022070715/5ed831010fa3e705ec0e017c/html5/thumbnails/63.jpg)
Set Target Variable
HeuristicLab Tutorial http://dev.heuristiclab.com 63
![Page 64: Algorithm and Experiment Design with HeuristicLab › trac.fcgi › export › HEAD... · Create Batch Runs and Experiments • Batch runs –execute the same optimizer (e.g. algorithm,](https://reader036.vdocuments.mx/reader036/viewer/2022070715/5ed831010fa3e705ec0e017c/html5/thumbnails/64.jpg)
Select Input Variables
HeuristicLab Tutorial http://dev.heuristiclab.com 64
![Page 65: Algorithm and Experiment Design with HeuristicLab › trac.fcgi › export › HEAD... · Create Batch Runs and Experiments • Batch runs –execute the same optimizer (e.g. algorithm,](https://reader036.vdocuments.mx/reader036/viewer/2022070715/5ed831010fa3e705ec0e017c/html5/thumbnails/65.jpg)
Configure Training and Test Partitions
HeuristicLab Tutorial http://dev.heuristiclab.com 65
![Page 66: Algorithm and Experiment Design with HeuristicLab › trac.fcgi › export › HEAD... · Create Batch Runs and Experiments • Batch runs –execute the same optimizer (e.g. algorithm,](https://reader036.vdocuments.mx/reader036/viewer/2022070715/5ed831010fa3e705ec0e017c/html5/thumbnails/66.jpg)
Run Linear Regression
HeuristicLab Tutorial http://dev.heuristiclab.com 66
![Page 67: Algorithm and Experiment Design with HeuristicLab › trac.fcgi › export › HEAD... · Create Batch Runs and Experiments • Batch runs –execute the same optimizer (e.g. algorithm,](https://reader036.vdocuments.mx/reader036/viewer/2022070715/5ed831010fa3e705ec0e017c/html5/thumbnails/67.jpg)
Inspect Results
HeuristicLab Tutorial http://dev.heuristiclab.com 67
![Page 68: Algorithm and Experiment Design with HeuristicLab › trac.fcgi › export › HEAD... · Create Batch Runs and Experiments • Batch runs –execute the same optimizer (e.g. algorithm,](https://reader036.vdocuments.mx/reader036/viewer/2022070715/5ed831010fa3e705ec0e017c/html5/thumbnails/68.jpg)
Inspect Scatterplot of Predicted and Target Values
HeuristicLab Tutorial http://dev.heuristiclab.com 68
![Page 69: Algorithm and Experiment Design with HeuristicLab › trac.fcgi › export › HEAD... · Create Batch Runs and Experiments • Batch runs –execute the same optimizer (e.g. algorithm,](https://reader036.vdocuments.mx/reader036/viewer/2022070715/5ed831010fa3e705ec0e017c/html5/thumbnails/69.jpg)
Inspect Linechart
HeuristicLab Tutorial http://dev.heuristiclab.com 69
![Page 70: Algorithm and Experiment Design with HeuristicLab › trac.fcgi › export › HEAD... · Create Batch Runs and Experiments • Batch runs –execute the same optimizer (e.g. algorithm,](https://reader036.vdocuments.mx/reader036/viewer/2022070715/5ed831010fa3e705ec0e017c/html5/thumbnails/70.jpg)
Inspect the Model
HeuristicLab Tutorial http://dev.heuristiclab.com 70
![Page 71: Algorithm and Experiment Design with HeuristicLab › trac.fcgi › export › HEAD... · Create Batch Runs and Experiments • Batch runs –execute the same optimizer (e.g. algorithm,](https://reader036.vdocuments.mx/reader036/viewer/2022070715/5ed831010fa3e705ec0e017c/html5/thumbnails/71.jpg)
Symbolic Regression with HeuristicLab
• Linear regression produced an inaccurate model.• Next: produce a nonlinear symbolic regression model using
genetic programming
• Genetic programming– evolve variable-length models– model representation: symbolic expression tree– structure and model parameters are evolved side-by-side– white-box models
HeuristicLab Tutorial http://dev.heuristiclab.com 71
-
x3+
x1 *
x2x1
![Page 72: Algorithm and Experiment Design with HeuristicLab › trac.fcgi › export › HEAD... · Create Batch Runs and Experiments • Batch runs –execute the same optimizer (e.g. algorithm,](https://reader036.vdocuments.mx/reader036/viewer/2022070715/5ed831010fa3e705ec0e017c/html5/thumbnails/72.jpg)
Symbolic Regression withHeuristicLab• Demonstration
– problem configuration– function set and terminal set– model size constraints– evaluation
• Algorithm configuration– selection– mutation
• Analysis of results– model accuracy– model structure and parameters
HeuristicLab Tutorial http://dev.heuristiclab.com 72
![Page 73: Algorithm and Experiment Design with HeuristicLab › trac.fcgi › export › HEAD... · Create Batch Runs and Experiments • Batch runs –execute the same optimizer (e.g. algorithm,](https://reader036.vdocuments.mx/reader036/viewer/2022070715/5ed831010fa3e705ec0e017c/html5/thumbnails/73.jpg)
Create New Genetic Algorithm
HeuristicLab Tutorial http://dev.heuristiclab.com 73
![Page 74: Algorithm and Experiment Design with HeuristicLab › trac.fcgi › export › HEAD... · Create Batch Runs and Experiments • Batch runs –execute the same optimizer (e.g. algorithm,](https://reader036.vdocuments.mx/reader036/viewer/2022070715/5ed831010fa3e705ec0e017c/html5/thumbnails/74.jpg)
Create New Symbolic Regression Problem
HeuristicLab Tutorial http://dev.heuristiclab.com 74
![Page 75: Algorithm and Experiment Design with HeuristicLab › trac.fcgi › export › HEAD... · Create Batch Runs and Experiments • Batch runs –execute the same optimizer (e.g. algorithm,](https://reader036.vdocuments.mx/reader036/viewer/2022070715/5ed831010fa3e705ec0e017c/html5/thumbnails/75.jpg)
Import Data
HeuristicLab Tutorial http://dev.heuristiclab.com 75
![Page 76: Algorithm and Experiment Design with HeuristicLab › trac.fcgi › export › HEAD... · Create Batch Runs and Experiments • Batch runs –execute the same optimizer (e.g. algorithm,](https://reader036.vdocuments.mx/reader036/viewer/2022070715/5ed831010fa3e705ec0e017c/html5/thumbnails/76.jpg)
Inspect Data and Configure Dataset
HeuristicLab Tutorial http://dev.heuristiclab.com 76
![Page 77: Algorithm and Experiment Design with HeuristicLab › trac.fcgi › export › HEAD... · Create Batch Runs and Experiments • Batch runs –execute the same optimizer (e.g. algorithm,](https://reader036.vdocuments.mx/reader036/viewer/2022070715/5ed831010fa3e705ec0e017c/html5/thumbnails/77.jpg)
Set Target and Input Variables
HeuristicLab Tutorial http://dev.heuristiclab.com 77
![Page 78: Algorithm and Experiment Design with HeuristicLab › trac.fcgi › export › HEAD... · Create Batch Runs and Experiments • Batch runs –execute the same optimizer (e.g. algorithm,](https://reader036.vdocuments.mx/reader036/viewer/2022070715/5ed831010fa3e705ec0e017c/html5/thumbnails/78.jpg)
Configure Maximal Model Depth and Length
HeuristicLab Tutorial http://dev.heuristiclab.com 78
![Page 79: Algorithm and Experiment Design with HeuristicLab › trac.fcgi › export › HEAD... · Create Batch Runs and Experiments • Batch runs –execute the same optimizer (e.g. algorithm,](https://reader036.vdocuments.mx/reader036/viewer/2022070715/5ed831010fa3e705ec0e017c/html5/thumbnails/79.jpg)
Configure Function Set (Grammar)
HeuristicLab Tutorial http://dev.heuristiclab.com 79
![Page 80: Algorithm and Experiment Design with HeuristicLab › trac.fcgi › export › HEAD... · Create Batch Runs and Experiments • Batch runs –execute the same optimizer (e.g. algorithm,](https://reader036.vdocuments.mx/reader036/viewer/2022070715/5ed831010fa3e705ec0e017c/html5/thumbnails/80.jpg)
Configure Function Set (Grammar)
HeuristicLab Tutorial http://dev.heuristiclab.com 80
![Page 81: Algorithm and Experiment Design with HeuristicLab › trac.fcgi › export › HEAD... · Create Batch Runs and Experiments • Batch runs –execute the same optimizer (e.g. algorithm,](https://reader036.vdocuments.mx/reader036/viewer/2022070715/5ed831010fa3e705ec0e017c/html5/thumbnails/81.jpg)
Configure Algorithm Parameters
HeuristicLab Tutorial http://dev.heuristiclab.com 81
![Page 82: Algorithm and Experiment Design with HeuristicLab › trac.fcgi › export › HEAD... · Create Batch Runs and Experiments • Batch runs –execute the same optimizer (e.g. algorithm,](https://reader036.vdocuments.mx/reader036/viewer/2022070715/5ed831010fa3e705ec0e017c/html5/thumbnails/82.jpg)
Configure Mutation Operator
HeuristicLab Tutorial http://dev.heuristiclab.com 82
![Page 83: Algorithm and Experiment Design with HeuristicLab › trac.fcgi › export › HEAD... · Create Batch Runs and Experiments • Batch runs –execute the same optimizer (e.g. algorithm,](https://reader036.vdocuments.mx/reader036/viewer/2022070715/5ed831010fa3e705ec0e017c/html5/thumbnails/83.jpg)
Configure Selection Operator
HeuristicLab Tutorial http://dev.heuristiclab.com 83
![Page 84: Algorithm and Experiment Design with HeuristicLab › trac.fcgi › export › HEAD... · Create Batch Runs and Experiments • Batch runs –execute the same optimizer (e.g. algorithm,](https://reader036.vdocuments.mx/reader036/viewer/2022070715/5ed831010fa3e705ec0e017c/html5/thumbnails/84.jpg)
Configure Tournament Group Size
HeuristicLab Tutorial http://dev.heuristiclab.com 84
![Page 85: Algorithm and Experiment Design with HeuristicLab › trac.fcgi › export › HEAD... · Create Batch Runs and Experiments • Batch runs –execute the same optimizer (e.g. algorithm,](https://reader036.vdocuments.mx/reader036/viewer/2022070715/5ed831010fa3e705ec0e017c/html5/thumbnails/85.jpg)
Start Algorithm and Inspect Results
HeuristicLab Tutorial http://dev.heuristiclab.com 85
![Page 86: Algorithm and Experiment Design with HeuristicLab › trac.fcgi › export › HEAD... · Create Batch Runs and Experiments • Batch runs –execute the same optimizer (e.g. algorithm,](https://reader036.vdocuments.mx/reader036/viewer/2022070715/5ed831010fa3e705ec0e017c/html5/thumbnails/86.jpg)
Inspect Quality Chart
HeuristicLab Tutorial http://dev.heuristiclab.com 86
![Page 87: Algorithm and Experiment Design with HeuristicLab › trac.fcgi › export › HEAD... · Create Batch Runs and Experiments • Batch runs –execute the same optimizer (e.g. algorithm,](https://reader036.vdocuments.mx/reader036/viewer/2022070715/5ed831010fa3e705ec0e017c/html5/thumbnails/87.jpg)
Inspect Best Model on Training Partition
HeuristicLab Tutorial http://dev.heuristiclab.com 87
![Page 88: Algorithm and Experiment Design with HeuristicLab › trac.fcgi › export › HEAD... · Create Batch Runs and Experiments • Batch runs –execute the same optimizer (e.g. algorithm,](https://reader036.vdocuments.mx/reader036/viewer/2022070715/5ed831010fa3e705ec0e017c/html5/thumbnails/88.jpg)
Inspect Linechart of Best Model on Training Partition
HeuristicLab Tutorial http://dev.heuristiclab.com 88
![Page 89: Algorithm and Experiment Design with HeuristicLab › trac.fcgi › export › HEAD... · Create Batch Runs and Experiments • Batch runs –execute the same optimizer (e.g. algorithm,](https://reader036.vdocuments.mx/reader036/viewer/2022070715/5ed831010fa3e705ec0e017c/html5/thumbnails/89.jpg)
Inspect Structure of Best Model on Training Partition
HeuristicLab Tutorial http://dev.heuristiclab.com 89
![Page 90: Algorithm and Experiment Design with HeuristicLab › trac.fcgi › export › HEAD... · Create Batch Runs and Experiments • Batch runs –execute the same optimizer (e.g. algorithm,](https://reader036.vdocuments.mx/reader036/viewer/2022070715/5ed831010fa3e705ec0e017c/html5/thumbnails/90.jpg)
Model Simplification and Export
• Demonstration– automatic simplification
– visualization of node impacts
– manual simplification• online update of results
– model export• Excel• MATLAB• LaTeX
HeuristicLab Tutorial http://dev.heuristiclab.com 90
![Page 91: Algorithm and Experiment Design with HeuristicLab › trac.fcgi › export › HEAD... · Create Batch Runs and Experiments • Batch runs –execute the same optimizer (e.g. algorithm,](https://reader036.vdocuments.mx/reader036/viewer/2022070715/5ed831010fa3e705ec0e017c/html5/thumbnails/91.jpg)
Detailed Model Analysis andSimplification
HeuristicLab Tutorial http://dev.heuristiclab.com 91
![Page 92: Algorithm and Experiment Design with HeuristicLab › trac.fcgi › export › HEAD... · Create Batch Runs and Experiments • Batch runs –execute the same optimizer (e.g. algorithm,](https://reader036.vdocuments.mx/reader036/viewer/2022070715/5ed831010fa3e705ec0e017c/html5/thumbnails/92.jpg)
Symbolic Simplification and NodeImpacts
HeuristicLab Tutorial http://dev.heuristiclab.com 92
![Page 93: Algorithm and Experiment Design with HeuristicLab › trac.fcgi › export › HEAD... · Create Batch Runs and Experiments • Batch runs –execute the same optimizer (e.g. algorithm,](https://reader036.vdocuments.mx/reader036/viewer/2022070715/5ed831010fa3e705ec0e017c/html5/thumbnails/93.jpg)
Manual Simplification
HeuristicLab Tutorial http://dev.heuristiclab.com 93
double-click nodes
![Page 94: Algorithm and Experiment Design with HeuristicLab › trac.fcgi › export › HEAD... · Create Batch Runs and Experiments • Batch runs –execute the same optimizer (e.g. algorithm,](https://reader036.vdocuments.mx/reader036/viewer/2022070715/5ed831010fa3e705ec0e017c/html5/thumbnails/94.jpg)
Automatic Symbolic Simplification
HeuristicLab Tutorial http://dev.heuristiclab.com 94
![Page 95: Algorithm and Experiment Design with HeuristicLab › trac.fcgi › export › HEAD... · Create Batch Runs and Experiments • Batch runs –execute the same optimizer (e.g. algorithm,](https://reader036.vdocuments.mx/reader036/viewer/2022070715/5ed831010fa3e705ec0e017c/html5/thumbnails/95.jpg)
Textual Representations Are Also Available• Use ViewHost to switch to textual representation view.
HeuristicLab Tutorial http://dev.heuristiclab.com 95
![Page 96: Algorithm and Experiment Design with HeuristicLab › trac.fcgi › export › HEAD... · Create Batch Runs and Experiments • Batch runs –execute the same optimizer (e.g. algorithm,](https://reader036.vdocuments.mx/reader036/viewer/2022070715/5ed831010fa3e705ec0e017c/html5/thumbnails/96.jpg)
Default Textual Representation forModel Export
HeuristicLab Tutorial http://dev.heuristiclab.com 96
![Page 97: Algorithm and Experiment Design with HeuristicLab › trac.fcgi › export › HEAD... · Create Batch Runs and Experiments • Batch runs –execute the same optimizer (e.g. algorithm,](https://reader036.vdocuments.mx/reader036/viewer/2022070715/5ed831010fa3e705ec0e017c/html5/thumbnails/97.jpg)
Textual Representation for Export to LaTeX
HeuristicLab Tutorial http://dev.heuristiclab.com 97
![Page 98: Algorithm and Experiment Design with HeuristicLab › trac.fcgi › export › HEAD... · Create Batch Runs and Experiments • Batch runs –execute the same optimizer (e.g. algorithm,](https://reader036.vdocuments.mx/reader036/viewer/2022070715/5ed831010fa3e705ec0e017c/html5/thumbnails/98.jpg)
LaTeX Export
HeuristicLab Tutorial http://dev.heuristiclab.com 98
![Page 99: Algorithm and Experiment Design with HeuristicLab › trac.fcgi › export › HEAD... · Create Batch Runs and Experiments • Batch runs –execute the same optimizer (e.g. algorithm,](https://reader036.vdocuments.mx/reader036/viewer/2022070715/5ed831010fa3e705ec0e017c/html5/thumbnails/99.jpg)
Variable Relevance Analysis
• Which variables are important for correct predictions?
• Demonstration– Variable frequency analyzer
– symbol frequency analyzer
– variable impacts
HeuristicLab Tutorial http://dev.heuristiclab.com 99
![Page 100: Algorithm and Experiment Design with HeuristicLab › trac.fcgi › export › HEAD... · Create Batch Runs and Experiments • Batch runs –execute the same optimizer (e.g. algorithm,](https://reader036.vdocuments.mx/reader036/viewer/2022070715/5ed831010fa3e705ec0e017c/html5/thumbnails/100.jpg)
Inspect Variable Frequency Chart
HeuristicLab Tutorial http://dev.heuristiclab.com 100
![Page 101: Algorithm and Experiment Design with HeuristicLab › trac.fcgi › export › HEAD... · Create Batch Runs and Experiments • Batch runs –execute the same optimizer (e.g. algorithm,](https://reader036.vdocuments.mx/reader036/viewer/2022070715/5ed831010fa3e705ec0e017c/html5/thumbnails/101.jpg)
Inspect Variable Impacts
HeuristicLab Tutorial http://dev.heuristiclab.com 101
![Page 102: Algorithm and Experiment Design with HeuristicLab › trac.fcgi › export › HEAD... · Create Batch Runs and Experiments • Batch runs –execute the same optimizer (e.g. algorithm,](https://reader036.vdocuments.mx/reader036/viewer/2022070715/5ed831010fa3e705ec0e017c/html5/thumbnails/102.jpg)
Inspect Symbol Frequencies
HeuristicLab Tutorial http://dev.heuristiclab.com 102
![Page 103: Algorithm and Experiment Design with HeuristicLab › trac.fcgi › export › HEAD... · Create Batch Runs and Experiments • Batch runs –execute the same optimizer (e.g. algorithm,](https://reader036.vdocuments.mx/reader036/viewer/2022070715/5ed831010fa3e705ec0e017c/html5/thumbnails/103.jpg)
Classification with HeuristicLab
• Symbolic classification– evolve discriminating function using GP
– find thresholds to assign classes
• Demonstration– real world medical application
– model accuracy
– visualization of model output• discriminating function output• ROC-curve• confusion matrix
HeuristicLab Tutorial http://dev.heuristiclab.com 103
![Page 104: Algorithm and Experiment Design with HeuristicLab › trac.fcgi › export › HEAD... · Create Batch Runs and Experiments • Batch runs –execute the same optimizer (e.g. algorithm,](https://reader036.vdocuments.mx/reader036/viewer/2022070715/5ed831010fa3e705ec0e017c/html5/thumbnails/104.jpg)
Case Study: Classification
• Real world medical dataset (Mammographic Mass) from UCI Machine Learning Repository– data from non-invasive mammography screening– variables:
• patient age• visual features of inspected mass lesions: shape, margin, density
– target variable: severity (malignant, benign)
– available as a benchmark problem instance in HeuristicLab
HeuristicLab Tutorial http://dev.heuristiclab.com 104
![Page 105: Algorithm and Experiment Design with HeuristicLab › trac.fcgi › export › HEAD... · Create Batch Runs and Experiments • Batch runs –execute the same optimizer (e.g. algorithm,](https://reader036.vdocuments.mx/reader036/viewer/2022070715/5ed831010fa3e705ec0e017c/html5/thumbnails/105.jpg)
Open Sample
HeuristicLab Tutorial http://dev.heuristiclab.com 105
![Page 106: Algorithm and Experiment Design with HeuristicLab › trac.fcgi › export › HEAD... · Create Batch Runs and Experiments • Batch runs –execute the same optimizer (e.g. algorithm,](https://reader036.vdocuments.mx/reader036/viewer/2022070715/5ed831010fa3e705ec0e017c/html5/thumbnails/106.jpg)
Configure and Run Algorithm
HeuristicLab Tutorial http://dev.heuristiclab.com 106
![Page 107: Algorithm and Experiment Design with HeuristicLab › trac.fcgi › export › HEAD... · Create Batch Runs and Experiments • Batch runs –execute the same optimizer (e.g. algorithm,](https://reader036.vdocuments.mx/reader036/viewer/2022070715/5ed831010fa3e705ec0e017c/html5/thumbnails/107.jpg)
Inspect Quality Linechart
HeuristicLab Tutorial http://dev.heuristiclab.com 107
![Page 108: Algorithm and Experiment Design with HeuristicLab › trac.fcgi › export › HEAD... · Create Batch Runs and Experiments • Batch runs –execute the same optimizer (e.g. algorithm,](https://reader036.vdocuments.mx/reader036/viewer/2022070715/5ed831010fa3e705ec0e017c/html5/thumbnails/108.jpg)
Inspect Best Training Solution
HeuristicLab Tutorial http://dev.heuristiclab.com 108
![Page 109: Algorithm and Experiment Design with HeuristicLab › trac.fcgi › export › HEAD... · Create Batch Runs and Experiments • Batch runs –execute the same optimizer (e.g. algorithm,](https://reader036.vdocuments.mx/reader036/viewer/2022070715/5ed831010fa3e705ec0e017c/html5/thumbnails/109.jpg)
Inspect Model Output and Thresholds
HeuristicLab Tutorial http://dev.heuristiclab.com 109
![Page 110: Algorithm and Experiment Design with HeuristicLab › trac.fcgi › export › HEAD... · Create Batch Runs and Experiments • Batch runs –execute the same optimizer (e.g. algorithm,](https://reader036.vdocuments.mx/reader036/viewer/2022070715/5ed831010fa3e705ec0e017c/html5/thumbnails/110.jpg)
Inspect Confusion Matrix
HeuristicLab Tutorial http://dev.heuristiclab.com 110
![Page 111: Algorithm and Experiment Design with HeuristicLab › trac.fcgi › export › HEAD... · Create Batch Runs and Experiments • Batch runs –execute the same optimizer (e.g. algorithm,](https://reader036.vdocuments.mx/reader036/viewer/2022070715/5ed831010fa3e705ec0e017c/html5/thumbnails/111.jpg)
Inspect ROC Curve
HeuristicLab Tutorial http://dev.heuristiclab.com 111
![Page 112: Algorithm and Experiment Design with HeuristicLab › trac.fcgi › export › HEAD... · Create Batch Runs and Experiments • Batch runs –execute the same optimizer (e.g. algorithm,](https://reader036.vdocuments.mx/reader036/viewer/2022070715/5ed831010fa3e705ec0e017c/html5/thumbnails/112.jpg)
Validation of Results
• Overfitting = memorizing data
• Strategies to reduce overfitting
– validation partition
– cross-validation
HeuristicLab Tutorial http://dev.heuristiclab.com 112
Training Validation Test
Test
Test
Test
Test
Test
Training
Training
Training
Training
Training
![Page 113: Algorithm and Experiment Design with HeuristicLab › trac.fcgi › export › HEAD... · Create Batch Runs and Experiments • Batch runs –execute the same optimizer (e.g. algorithm,](https://reader036.vdocuments.mx/reader036/viewer/2022070715/5ed831010fa3e705ec0e017c/html5/thumbnails/113.jpg)
Validation of Results• Demonstration
– Configuration of a validation set
– Inspection of best solution on validation set
– Analysis of training- and validation fitness correlation
– Cross-validation• Configuration• Analysis of results
HeuristicLab Tutorial http://dev.heuristiclab.com 113
![Page 114: Algorithm and Experiment Design with HeuristicLab › trac.fcgi › export › HEAD... · Create Batch Runs and Experiments • Batch runs –execute the same optimizer (e.g. algorithm,](https://reader036.vdocuments.mx/reader036/viewer/2022070715/5ed831010fa3e705ec0e017c/html5/thumbnails/114.jpg)
Configuration of Validation Partition
HeuristicLab Tutorial http://dev.heuristiclab.com 114
![Page 115: Algorithm and Experiment Design with HeuristicLab › trac.fcgi › export › HEAD... · Create Batch Runs and Experiments • Batch runs –execute the same optimizer (e.g. algorithm,](https://reader036.vdocuments.mx/reader036/viewer/2022070715/5ed831010fa3e705ec0e017c/html5/thumbnails/115.jpg)
Inspect Best Model on Validation Partition
HeuristicLab Tutorial http://dev.heuristiclab.com 115
![Page 116: Algorithm and Experiment Design with HeuristicLab › trac.fcgi › export › HEAD... · Create Batch Runs and Experiments • Batch runs –execute the same optimizer (e.g. algorithm,](https://reader036.vdocuments.mx/reader036/viewer/2022070715/5ed831010fa3e705ec0e017c/html5/thumbnails/116.jpg)
Inspect Linechart of Correlation ofTraining and Validation Fitness
HeuristicLab Tutorial http://dev.heuristiclab.com 116
![Page 117: Algorithm and Experiment Design with HeuristicLab › trac.fcgi › export › HEAD... · Create Batch Runs and Experiments • Batch runs –execute the same optimizer (e.g. algorithm,](https://reader036.vdocuments.mx/reader036/viewer/2022070715/5ed831010fa3e705ec0e017c/html5/thumbnails/117.jpg)
Agenda
• Objectives of the Tutorial• Introduction• Where to get HeuristicLab?• Plugin Infrastructure• Graphical User Interface• Available Algorithms & Problems
• Demonstration Part I: Working with HeuristicLab• Demonstration Part II: Data-based Modeling
• Some Additional Features• Planned Features• Team• Suggested Readings• Bibliography• Questions & Answers
HeuristicLab Tutorial http://dev.heuristiclab.com 117
![Page 118: Algorithm and Experiment Design with HeuristicLab › trac.fcgi › export › HEAD... · Create Batch Runs and Experiments • Batch runs –execute the same optimizer (e.g. algorithm,](https://reader036.vdocuments.mx/reader036/viewer/2022070715/5ed831010fa3e705ec0e017c/html5/thumbnails/118.jpg)
Some Additional Features
• HeuristicLab Hive– parallel and distributed execution of algorithms
and experiments on many computers in a network
• Optimization Knowledge Base (OKB)– database to store algorithms, problems, parameters and results– open to the public– open for other frameworks– analyze and store characteristics of problem instances and problem classes
• External solution evaluation and simulation-based optimization– interface to couple HeuristicLab with other applications
(MATLAB, Simulink, SciLab, AnyLogic, …)– supports different protocols (command line parameters, TCP, …)
• Parameter grid tests and meta-optimization– automatically create experiments to test large ranges of parameters– apply heuristic optimization algorithms to find optimal parameter settings for heuristic
optimization algorithms
HeuristicLab Tutorial http://dev.heuristiclab.com 118
![Page 119: Algorithm and Experiment Design with HeuristicLab › trac.fcgi › export › HEAD... · Create Batch Runs and Experiments • Batch runs –execute the same optimizer (e.g. algorithm,](https://reader036.vdocuments.mx/reader036/viewer/2022070715/5ed831010fa3e705ec0e017c/html5/thumbnails/119.jpg)
Planned Features
• Algorithms & Problems– steady-state genetic algorithm– unified tabu search for vehicle routing– estimation of distribution algorithms– evolution of arbitrary code (controller, etc.)– …
• Cloud Computing– port HeuristicLab Hive to Windows Azure
• Have a look at the HeuristicLab roadmap– http://dev.heuristiclab.com/trac.fcgi/roadmap
• Any other ideas, requests or recommendations?– join our HeuristicLab Google group [email protected]– write an e-mail to [email protected]
HeuristicLab Tutorial http://dev.heuristiclab.com 119
![Page 120: Algorithm and Experiment Design with HeuristicLab › trac.fcgi › export › HEAD... · Create Batch Runs and Experiments • Batch runs –execute the same optimizer (e.g. algorithm,](https://reader036.vdocuments.mx/reader036/viewer/2022070715/5ed831010fa3e705ec0e017c/html5/thumbnails/120.jpg)
HeuristicLab Team
HeuristicLab Tutorial http://dev.heuristiclab.com 120
Heuristic and Evolutionary Algorithms Laboratory (HEAL)School of Informatics, Communications and MediaUniversity of Applied Sciences Upper Austria
Softwarepark 11A-4232 HagenbergAUSTRIA
WWW: http://heal.heuristiclab.com
![Page 121: Algorithm and Experiment Design with HeuristicLab › trac.fcgi › export › HEAD... · Create Batch Runs and Experiments • Batch runs –execute the same optimizer (e.g. algorithm,](https://reader036.vdocuments.mx/reader036/viewer/2022070715/5ed831010fa3e705ec0e017c/html5/thumbnails/121.jpg)
Suggested Readings
• S. Voß, D. Woodruff (Edts.)Optimization Software Class LibrariesKluwer Academic Publishers, 2002
• M. Affenzeller, S. Winkler, S. Wagner, A. BehamGenetic Algorithms and Genetic ProgrammingModern Concepts and Practical ApplicationsCRC Press, 2009
HeuristicLab Tutorial http://dev.heuristiclab.com 121
![Page 122: Algorithm and Experiment Design with HeuristicLab › trac.fcgi › export › HEAD... · Create Batch Runs and Experiments • Batch runs –execute the same optimizer (e.g. algorithm,](https://reader036.vdocuments.mx/reader036/viewer/2022070715/5ed831010fa3e705ec0e017c/html5/thumbnails/122.jpg)
Bibliography• S. Wagner, M. Affenzeller
HeuristicLab: A generic and extensible optimization environmentAdaptive and Natural Computing Algorithms, pp. 538-541Springer, 2005
• S. Wagner, S. Winkler, R. Braune, G. Kronberger, A. Beham, M. AffenzellerBenefits of plugin-based heuristic optimization software systemsComputer Aided Systems Theory - EUROCAST 2007, Lecture Notes in Computer Science, vol. 4739, pp. 747-754Springer, 2007
• S. Wagner, G. Kronberger, A. Beham, S. Winkler, M. AffenzellerModeling of heuristic optimization algorithmsProceedings of the 20th European Modeling and Simulation Symposium, pp. 106-111DIPTEM University of Genova, 2008
• S. Wagner, G. Kronberger, A. Beham, S. Winkler, M. AffenzellerModel driven rapid prototyping of heuristic optimization algorithmsComputer Aided Systems Theory - EUROCAST 2009, Lecture Notes in Computer Science, vol. 5717, pp. 729-736Springer, 2009
• S. WagnerHeuristic optimization software systems - Modeling of heuristic optimization algorithms in the HeuristicLab software environmentPh.D. thesis, Johannes Kepler University Linz, Austria, 2009.
• S. Wagner, A. Beham, G. Kronberger, M. Kommenda, E. Pitzer, M. Kofler, S. Vonolfen, S. Winkler, V. Dorfer, M. AffenzellerHeuristicLab 3.3: A unified approach to metaheuristic optimizationActas del séptimo congreso español sobre Metaheurísticas, Algoritmos Evolutivos y Bioinspirados (MAEB'2010), 2010
• S. Wagner, G. Kronberger, A. Beham, M. Kommenda, A. Scheibenpflug, E. Pitzer, S. Vonolfen, M. Kofler, S. Winkler, V. Dorfer, M. AffenzellerArchitecture and Design of the HeuristicLab Optimization EnvironmentAdvanced Methods and Applications in Computational Intelligence, vol. 6, pp. 197-261, Springer, 2014
• Detailed list of all publications of the HEAL research group: http://research.fh-ooe.at/de/orgunit/356#showpublications
HeuristicLab Tutorial http://dev.heuristiclab.com 122
![Page 123: Algorithm and Experiment Design with HeuristicLab › trac.fcgi › export › HEAD... · Create Batch Runs and Experiments • Batch runs –execute the same optimizer (e.g. algorithm,](https://reader036.vdocuments.mx/reader036/viewer/2022070715/5ed831010fa3e705ec0e017c/html5/thumbnails/123.jpg)
Questions & Answers
HeuristicLab Tutorial http://dev.heuristiclab.com 123
http://dev.heuristiclab.com
http://www.youtube.com/heuristiclab
http://www.facebook.com/heuristiclab