a framework for selecting hyper-parameters · a framework for selecting hyper-parameters jim o’...
TRANSCRIPT
A Framework for Selecting Hyper-Parameters
Jim O’ Donoghue
In-MINDD is funded under the European Union Seventh Framework Programme, Grant Agreement Number 304979
British International Conference on Databases
7th July 2015
In-MINDD is funded under the European Union Seventh Framework Programme, Grant Agreement Number 304979
Background + Motivation
Algorithms + The CDN
Experiments + Results
Future Work
Conclusions
NEED TO FIX NUMBERS
In-MINDD is funded under the European Union Seventh Framework Programme, Grant Agreement Number 304979 3
1
In-MINDD is funded under the European Union Seventh Framework Programme, Grant Agreement Number 304979 4
1
In-MINDD is funded under the European Union Seventh Framework Programme, Grant Agreement Number 304979 5
1
Dementia Awareness + Prevention
In-MINDD is funded under the European Union Seventh Framework Programme, Grant Agreement Number 304979 6
1
Dementia Awareness + Prevention
Online Environment
In-MINDD is funded under the European Union Seventh Framework Programme, Grant Agreement Number 304979 7
1
Dementia Awareness + Prevention Online Environment
Risk Prediction Algorithm
In-MINDD is funded under the European Union Seventh Framework Programme, Grant Agreement Number 304979 8
1
Dementia Awareness + Prevention Online Environment
Risk Prediction Algorithm - Validation
In-MINDD is funded under the European Union Seventh Framework Programme, Grant Agreement Number 304979 9
3
In-MINDD is funded under the European Union Seventh Framework Programme, Grant Agreement Number 304979 10
3
In-MINDD is funded under the European Union Seventh Framework Programme, Grant Agreement Number 304979 11
3
High-Dimensional
In-MINDD is funded under the European Union Seventh Framework Programme, Grant Agreement Number 304979 12
3
High-Dimensional
Variable Interactions
In-MINDD is funded under the European Union Seventh Framework Programme, Grant Agreement Number 304979 13
3
High-Dimensional
Variable Interactions
In-MINDD is funded under the European Union Seventh Framework Programme, Grant Agreement Number 304979 14
3
High-Dimensional
Variable Interactions
Hyper-Parameter Selection
In-MINDD is funded under the European Union Seventh Framework Programme, Grant Agreement Number 304979 15
3
High-Dimensional
Variable Interactions
Hyper-Parameter Selection
In-MINDD is funded under the European Union Seventh Framework Programme, Grant Agreement Number 304979 16
4
In-MINDD is funded under the European Union Seventh Framework Programme, Grant Agreement Number 304979 17
4
Class Class
Input Features
In-MINDD is funded under the European Union Seventh Framework Programme, Grant Agreement Number 304979 18
4
Class Class
Input Features
Learned Features
In-MINDD is funded under the European Union Seventh Framework Programme, Grant Agreement Number 304979 19
4
Class
Connection Weights
Class
Input Features
Learned Features
In-MINDD is funded under the European Union Seventh Framework Programme, Grant Agreement Number 304979 20
4
Class
Connection Weights
Class
Input Features
Learned Features
In-MINDD is funded under the European Union Seventh Framework Programme, Grant Agreement Number 304979 21
4
Class
Connection Weights
Class
Input Features
Learned Features
In-MINDD is funded under the European Union Seventh Framework Programme, Grant Agreement Number 304979 22
4
Class
Connection Weights
Class
Input Features
Learned Features
In-MINDD is funded under the European Union Seventh Framework Programme, Grant Agreement Number 304979 23
4
Class
Connection Weights
Class
Input Features
Learned Features
In-MINDD is funded under the European Union Seventh Framework Programme, Grant Agreement Number 304979 24
4
Class
Connection Weights
Class
Input Features
Learned Features
In-MINDD is funded under the European Union Seventh Framework Programme, Grant Agreement Number 304979 25
4
Class
Connection Weights
Class
Input Features
Learned Features
In-MINDD is funded under the European Union Seventh Framework Programme, Grant Agreement Number 304979 26
4
Class
Connection Weights
Class
Input Features
Learned Features
In-MINDD is funded under the European Union Seventh Framework Programme, Grant Agreement Number 304979 27
4
Connection Weights
Class
Input Features
Learned Features
In-MINDD is funded under the European Union Seventh Framework Programme, Grant Agreement Number 304979 28
5
Class
Connection Weights
Class
Input Features
In-MINDD is funded under the European Union Seventh Framework Programme, Grant Agreement Number 304979 29
6
Connection Weights
Class
Input Features
In-MINDD is funded under the European Union Seventh Framework Programme, Grant Agreement Number 304979 30
7
Connection Weights
Input Features
Learned Features
In-MINDD is funded under the European Union Seventh Framework Programme, Grant Agreement Number 304979 31
8
Class
Connection Weights
Input Features
Learned Features
In-MINDD is funded under the European Union Seventh Framework Programme, Grant Agreement Number 304979 32
9
In-MINDD is funded under the European Union Seventh Framework Programme, Grant Agreement Number 304979 33
9
MySql
File System
Configurable Deep Network Framework
In-MINDD is funded under the European Union Seventh Framework Programme, Grant Agreement Number 304979 34
9
MySql
File System
Grid
Algorithm
Configurable Deep Network Framework
In-MINDD is funded under the European Union Seventh Framework Programme, Grant Agreement Number 304979 35
9
MySql
File System
Grid
Algorithm
Configurable Deep Network Framework
e1, e2, … , en
In-MINDD is funded under the European Union Seventh Framework Programme, Grant Agreement Number 304979 36
9
MySql
File System
Grid
Algorithm
Configurable Deep Network Framework
Query
e1, e2, … , en
In-MINDD is funded under the European Union Seventh Framework Programme, Grant Agreement Number 304979 37
9
MySql
File System
Grid
Algorithm
Configurable Deep Network Framework
Final Model
Query
e1, e2, … , en
In-MINDD is funded under the European Union Seventh Framework Programme, Grant Agreement Number 304979 38
10
In-MINDD is funded under the European Union Seventh Framework Programme, Grant Agreement Number 304979 39
11
CDN
In-MINDD is funded under the European Union Seventh Framework Programme, Grant Agreement Number 304979 40
12
Subset of the Data – dimensions What the variables are What the predictor is Purpose
In-MINDD is funded under the European Union Seventh Framework Programme, Grant Agreement Number 304979 41
13
To Choose:
In-MINDD is funded under the European Union Seventh Framework Programme, Grant Agreement Number 304979 42
13
To Choose:
learning rate α
In-MINDD is funded under the European Union Seventh Framework Programme, Grant Agreement Number 304979 43
13
To Choose:
learning rate α weight decay term λ
In-MINDD is funded under the European Union Seventh Framework Programme, Grant Agreement Number 304979 44
13
To Choose:
learning rate α weight decay term λ training iterations t
In-MINDD is funded under the European Union Seventh Framework Programme, Grant Agreement Number 304979 45
13
The Grid:
In-MINDD is funded under the European Union Seventh Framework Programme, Grant Agreement Number 304979 46
13
The Grid:
α, λ: [0.001, 0.003, 0.009, … , 0.1, 0.3, 0.9]
In-MINDD is funded under the European Union Seventh Framework Programme, Grant Agreement Number 304979 47
13
The Grid:
α, λ: [0.001, 0.003, 0.009, … , 0.1, 0.3, 0.9]
t: [100, 1000, 10000]
0
5
10
15
20
25
30
35
40
45
50
Valid. Cost
In-MINDD is funded under the European Union Seventh Framework Programme, Grant Agreement Number 304979 48
13
Categorical Continuous
Alpha 0.9 0.3 0.09 0.003
0
5
10
15
20
25
30
35
40
45
50
Valid. Cost
In-MINDD is funded under the European Union Seventh Framework Programme, Grant Agreement Number 304979 49
13
0
5
10
15
20
25
30
35
40
45
50
Valid. Cost
In-MINDD is funded under the European Union Seventh Framework Programme, Grant Agreement Number 304979 50
13 Alpha 0.9 0.3 0.09 0.003
Lambda 0.009 0.003 0.001
Lambda 0.009 0.003 0.001
0
5
10
15
20
25
30
35
40
45
50
Valid. Cost
In-MINDD is funded under the European Union Seventh Framework Programme, Grant Agreement Number 304979 51
13
100 1,000
10,000
Training Iterations
Alpha 0.9 0.3 0.09 0.003
0
5
10
15
20
25
30
35
40
45
50
Valid. Cost
Categorical Continuous Lambda 0.009 0.003 0.001
In-MINDD is funded under the European Union Seventh Framework Programme, Grant Agreement Number 304979 52
13
100 1,000
10,000
Training Iterations
Alpha 0.9 0.3 0.09 0.003
0
5
10
15
20
25
30
35
40
45
50
Valid. Cost
In-MINDD is funded under the European Union Seventh Framework Programme, Grant Agreement Number 304979 53
13
100 1,000
10,000
Training Iterations
Categorical Continuous Lambda 0.009 0.003 0.001
Alpha 0.9 0.3 0.09 0.003
0
5
10
15
20
25
30
35
40
45
50
Valid. Cost
In-MINDD is funded under the European Union Seventh Framework Programme, Grant Agreement Number 304979 54
13
100 1,000
10,000
Training Iterations
Categorical Continuous Lambda 0.009 0.003 0.001
Alpha 0.9 0.3 0.09 0.003
0
5
10
15
20
25
30
35
40
45
50
Valid. Cost
In-MINDD is funded under the European Union Seventh Framework Programme, Grant Agreement Number 304979 55
13
100 1,000
10,000
0.3046
Training Iterations
Categorical Continuous Lambda 0.009 0.003 0.001
Alpha 0.9 0.3 0.09 0.003
0
5
10
15
20
25
30
35
40
45
50
Valid. Cost
In-MINDD is funded under the European Union Seventh Framework Programme, Grant Agreement Number 304979 56
13
100 1,000
10,000
0.3046
Training Iterations
Categorical Continuous Lambda 0.009 0.003 0.001
Alpha 0.9 0.3 0.09 0.003
0
5
10
15
20
25
30
35
40
45
50
Valid. Cost
In-MINDD is funded under the European Union Seventh Framework Programme, Grant Agreement Number 304979 57
13
100 1,000
10,000
0.3046
Training Iterations
Categorical Continuous Lambda 0.009 0.003 0.001
Alpha 0.9 0.3 0.09 0.003
0
5
10
15
20
25
30
35
40
45
50
Valid. Cost
In-MINDD is funded under the European Union Seventh Framework Programme, Grant Agreement Number 304979 58
13
100 1,000
10,000
0.3046 0.2815
Training Iterations
Categorical Continuous Lambda 0.009 0.003 0.001
Alpha 0.9 0.3 0.09 0.003
0
5
10
15
20
25
30
35
40
45
50
Valid. Cost
In-MINDD is funded under the European Union Seventh Framework Programme, Grant Agreement Number 304979 59
13
100 1,000
10,000
0.3046 0.2815
Training Iterations
Categorical Continuous Lambda 0.009 0.003 0.001
Alpha 0.9 0.3 0.09 0.003
In-MINDD is funded under the European Union Seventh Framework Programme, Grant Agreement Number 304979 60
15
To Choose:
In-MINDD is funded under the European Union Seventh Framework Programme, Grant Agreement Number 304979 61
15
To Choose:
layer 1 nodes h(1)n
In-MINDD is funded under the European Union Seventh Framework Programme, Grant Agreement Number 304979 62
15
To Choose:
layer 1 nodes h(1)n
pre-training epochs e
In-MINDD is funded under the European Union Seventh Framework Programme, Grant Agreement Number 304979 63
15
The Grid:
h(1)n:
[10, 30, 337, 900, 1300, 2000]
In-MINDD is funded under the European Union Seventh Framework Programme, Grant Agreement Number 304979 64
15
The Grid:
h(1)n:
[10, 30, 337, 900, 1300, 2000]
e [1, 5, 10, 15, 20]
In-MINDD is funded under the European Union Seventh Framework Programme, Grant Agreement Number 304979 65
15
Parameter Initialisation:
− 4 6
𝑓𝑎𝑛_𝑖𝑛 + 𝑓𝑎𝑛_𝑜𝑢𝑡, + 4
6
𝑓𝑎𝑛_𝑖𝑛 + 𝑓𝑎𝑛_𝑜𝑢𝑡
Valid. Cost
-25
-20
-15
-10
-5
0
In-MINDD is funded under the European Union Seventh Framework Programme, Grant Agreement Number 304979 66
15
Epochs 1 5
Valid. Cost
-25
-20
-15
-10
-5
0
In-MINDD is funded under the European Union Seventh Framework Programme, Grant Agreement Number 304979 67
15
Nodes 10 30 100 337 900 1 300 2000
Valid. Cost
-25
-20
-15
-10
-5
0
In-MINDD is funded under the European Union Seventh Framework Programme, Grant Agreement Number 304979 68
15
Epochs 1 5
Nodes 10 30 100 337 900 1300 2000
Valid. Cost
-25
-20
-15
-10
-5
0
In-MINDD is funded under the European Union Seventh Framework Programme, Grant Agreement Number 304979 69
15
Epochs 1 5
Nodes 10 30 100 337 900 1300 2000
Valid. Cost
-25
-20
-15
-10
-5
0
In-MINDD is funded under the European Union Seventh Framework Programme, Grant Agreement Number 304979 70
15
Epochs 1 5
Nodes 10 30 100 337 900 1300 2000
Valid. Cost
-25
-20
-15
-10
-5
0
In-MINDD is funded under the European Union Seventh Framework Programme, Grant Agreement Number 304979 71
15
Epochs 1 5
Nodes 10 30 100 337 900 1300 2000
-19.580
In-MINDD is funded under the European Union Seventh Framework Programme, Grant Agreement Number 304979 72
14
To Choose:
Last layer nodes h(1)n
In-MINDD is funded under the European Union Seventh Framework Programme, Grant Agreement Number 304979 73
14
To Choose:
Last layer nodes h(1)n
The Grid:
[10, 30, 337, 900, 1300, 2000]
In-MINDD is funded under the European Union Seventh Framework Programme, Grant Agreement Number 304979 74
14
0
10
20
30
40
50
60
70
80
90
Valid. Cost
Categorical Continuous
In-MINDD is funded under the European Union Seventh Framework Programme, Grant Agreement Number 304979 75
14
0
10
20
30
40
50
60
70
80
90
Valid. Cost
Nodes 10 30 100 337 900 1300 2000
In-MINDD is funded under the European Union Seventh Framework Programme, Grant Agreement Number 304979 76
14
0
10
20
30
40
50
60
70
80
90
Valid. Cost
Categorical Continuous
Nodes 10 30 100 337 900 1300 2000
In-MINDD is funded under the European Union Seventh Framework Programme, Grant Agreement Number 304979 77
14
0
10
20
30
40
50
60
70
80
90
Valid. Cost
Categorical Continuous
Nodes 10 30 100 337 900 1300 2000
In-MINDD is funded under the European Union Seventh Framework Programme, Grant Agreement Number 304979 78
14
0
10
20
30
40
50
60
70
80
90
Valid. Cost
Categorical Continuous
Nodes 10 30 100 337 900 1300 2000
In-MINDD is funded under the European Union Seventh Framework Programme, Grant Agreement Number 304979 79
14
0
10
20
30
40
50
60
70
80
90
Valid. Cost
Categorical Continuous
Nodes 10 30 100 337 900 1300 2000
In-MINDD is funded under the European Union Seventh Framework Programme, Grant Agreement Number 304979 80
14
0
10
20
30
40
50
60
70
80
90
Valid. Cost
Categorical Continuous
Nodes 10 30 100 337 900 1300 2000
0.232
In-MINDD is funded under the European Union Seventh Framework Programme, Grant Agreement Number 304979 81
14
0
10
20
30
40
50
60
70
80
90
Valid. Cost
Categorical Continuous
Nodes 10 30 100 337 900 1300 2000
0.232
In-MINDD is funded under the European Union Seventh Framework Programme, Grant Agreement Number 304979 82
14
0
10
20
30
40
50
60
70
80
90
Valid. Cost
Categorical Continuous
Nodes 10 30 100 337 900 1 1300 2000
0.232
In-MINDD is funded under the European Union Seventh Framework Programme, Grant Agreement Number 304979 83
14
0
10
20
30
40
50
60
70
80
90
Valid. Cost
Categorical Continuous
Nodes 10 30 100 337 900 1 1300 2000
0.232
In-MINDD is funded under the European Union Seventh Framework Programme, Grant Agreement Number 304979 84
14
0
10
20
30
40
50
60
70
80
90
Valid. Cost
Categorical Continuous
Nodes 10 30 100 337 900 1 1300 2000
0.232 0.291
In-MINDD is funded under the European Union Seventh Framework Programme, Grant Agreement Number 304979 85
14
0
10
20
30
40
50
60
70
80
90
Valid. Cost
Categorical Continuous
Nodes 10 30 100 337 900 1 1300 2000
0.232 0.291
In-MINDD is funded under the European Union Seventh Framework Programme, Grant Agreement Number 304979 86
16
Lambda @ 0.03
10 2000 200 337
10 200 3567
10 200 3567
10 200 3567
10 200 3567
10 30
10 337
10 100 337
0
1
2
3
4
5
6
7
Alpha 0.001 0.01 0.9
In-MINDD is funded under the European Union Seventh Framework Programme, Grant Agreement Number 304979 87
16
Lambda @ 0.03
10 2000 200 337
10 200 3567
10 200 3567
10 200 3567
10 200 3567
10 30
10 337
10 100 337
0
1
2
3
4
5
6
7
Step 3000 1000 100
In-MINDD is funded under the European Union Seventh Framework Programme, Grant Agreement Number 304979 88
16
Lambda @ 0.03
10 2000 200 337
10 200 3567
10 200 3567
10 200 3567
10 200 3567
10 30
10 337
10 100 337
0
1
2
3
4
5
6
7
0.265 0.245
Alpha 0.001 0.01 0.9 Steps 3000 1000 100
0.272
In-MINDD is funded under the European Union Seventh Framework Programme, Grant Agreement Number 304979 89
16
Lambda @ 0.03
10 2000 200 337
10 200 3567
10 200 3567
10 200 3567
10 200 3567
10 30
10 337
10 100 337
Alpha 0.001 0.01 0.9 Steps 3000 1000 100
0
0.1
0.2
0.3
0.4
0.5
0.6
0.7
In-MINDD is funded under the European Union Seventh Framework Programme, Grant Agreement Number 304979 90
18
Activation functions Algorithms Inference Framework – to Mongo and input from Visualising learning Implementing Early Stopping Mini-batch Stochastic Gradient Descent
In-MINDD is funded under the European Union Seventh Framework Programme, Grant Agreement Number 304979 91
19
Much easier to model when you have one extensible network that can handle many type of data Constituent models can be used to select a starting point for deep learning configurations
In-MINDD is funded under the European Union Seventh Framework Programme, Grant Agreement Number 304979 92
In-MINDD is funded under the European Union Seventh Framework Programme, Grant Agreement Number 304979 93
16
Lambda @ 0.03
10 2000 200 337
10 200 3567
10 200 3567
10 200 3567
10 200 3567
10 30
10 337
10 100 337
0
1
2
3
4
5
6
7
0.265 0.245
Alpha 0.001 0.01 0.9 Steps 3000 1000 100
0.272