geometric margin domain description with instance-specific margins adam gripton thursday, 5 th may,...
TRANSCRIPT
![Page 1: Geometric margin domain description with instance-specific margins Adam Gripton Thursday, 5 th May, 2011](https://reader035.vdocuments.mx/reader035/viewer/2022062803/56649f3b5503460f94c59a69/html5/thumbnails/1.jpg)
Geometric margin domain description with instance-specific
margins
Adam GriptonThursday, 5th May, 2011
![Page 2: Geometric margin domain description with instance-specific margins Adam Gripton Thursday, 5 th May, 2011](https://reader035.vdocuments.mx/reader035/viewer/2022062803/56649f3b5503460f94c59a69/html5/thumbnails/2.jpg)
Presentation Contents
• High-level motivation• Development of system• Exact-centre method• Dual-optimisation method• Experimental data collection• Conclusions
![Page 3: Geometric margin domain description with instance-specific margins Adam Gripton Thursday, 5 th May, 2011](https://reader035.vdocuments.mx/reader035/viewer/2022062803/56649f3b5503460f94c59a69/html5/thumbnails/3.jpg)
High-level motivation
Task as originally stated:
• Expert replacement system to deal with Non-Destructive Assay (NDA) data provided by a sponsor for analysis and classification
• Involves automatic feature extraction and inference via classification step
![Page 4: Geometric margin domain description with instance-specific margins Adam Gripton Thursday, 5 th May, 2011](https://reader035.vdocuments.mx/reader035/viewer/2022062803/56649f3b5503460f94c59a69/html5/thumbnails/4.jpg)
High-level motivation
Data consignment:Fissile elements• Californium-252• Highly Enriched Uranium• Weapons-Grade Plutonium
Shielding methods• Aluminium• Steel ball• Steel planar to det• Lead• CHON (HE sim.)
Detectors• Sodium Iodide scintillator (NaI)• High-Resolution Germanium (semiconductor) spectrometer (HRGS)• Neutron array counter (N50R)
NaI
Neutron
HRGS
Source
Shield
NaI
Source
![Page 5: Geometric margin domain description with instance-specific margins Adam Gripton Thursday, 5 th May, 2011](https://reader035.vdocuments.mx/reader035/viewer/2022062803/56649f3b5503460f94c59a69/html5/thumbnails/5.jpg)
High-level motivation
Data consignment:
Spectroscopy experiments
![Page 6: Geometric margin domain description with instance-specific margins Adam Gripton Thursday, 5 th May, 2011](https://reader035.vdocuments.mx/reader035/viewer/2022062803/56649f3b5503460f94c59a69/html5/thumbnails/6.jpg)
High-level motivation
Data consignment:
τ0 2 * τ0 3 * τ0 etc
BX 0 279384403 138774738 91909165 to
BX 1 1805235 1785515 1770553 16
BX 2 49548 58784 65688
..etc … … …
Neutron multiplicity arrays
![Page 7: Geometric margin domain description with instance-specific margins Adam Gripton Thursday, 5 th May, 2011](https://reader035.vdocuments.mx/reader035/viewer/2022062803/56649f3b5503460f94c59a69/html5/thumbnails/7.jpg)
High-level motivationf1 f2 f3 f4 Class
0.1 0.2 -0.3 0.4 1
0.15 x -0.2 x 1
0.05 0.22 x x 2
x x -0.4 0.401 2
0.08 0.24 -0.5 0.399 3
• Features (columns) based on physically relevant projections of raw experimental data
• Class vector: refers to fissile material or shielding method• Some data absent: either not measured or not applicable
(structurally missing)
![Page 8: Geometric margin domain description with instance-specific margins Adam Gripton Thursday, 5 th May, 2011](https://reader035.vdocuments.mx/reader035/viewer/2022062803/56649f3b5503460f94c59a69/html5/thumbnails/8.jpg)
High-level motivation
Two principal aims:
1.Devise a novel contribution to existing literature on classification methods
2.Provide system of classification of abstract data that is applicable to provided dataset
![Page 9: Geometric margin domain description with instance-specific margins Adam Gripton Thursday, 5 th May, 2011](https://reader035.vdocuments.mx/reader035/viewer/2022062803/56649f3b5503460f94c59a69/html5/thumbnails/9.jpg)
Presentation Contents
• High-level motivation• Development of system• Exact-centre method• Dual-optimisation method• Experimental data collection• Conclusions
![Page 10: Geometric margin domain description with instance-specific margins Adam Gripton Thursday, 5 th May, 2011](https://reader035.vdocuments.mx/reader035/viewer/2022062803/56649f3b5503460f94c59a69/html5/thumbnails/10.jpg)
Development of system
Overview
Aim 2
Applicability To Dataset
Aim 1
Novel Contribution
Multi-Class
KernelMethods
Missing Data
![Page 11: Geometric margin domain description with instance-specific margins Adam Gripton Thursday, 5 th May, 2011](https://reader035.vdocuments.mx/reader035/viewer/2022062803/56649f3b5503460f94c59a69/html5/thumbnails/11.jpg)
Development of system
Overview
Multi-Class
KernelMethods
Missing Data• SVDD (Tax, Duin) and
Multi-Class Hybrid (Lee)• Geometric SVM (Chechik)
![Page 12: Geometric margin domain description with instance-specific margins Adam Gripton Thursday, 5 th May, 2011](https://reader035.vdocuments.mx/reader035/viewer/2022062803/56649f3b5503460f94c59a69/html5/thumbnails/12.jpg)
Development of system
• “Kernel trick” : ML algorithms that only query data values implicitly via the dot product
Working with Kernels
• Replace <x,y>←k(x,y) to imitate a mapping {x→φ(x)} such that k(x,y)=<φ(x), φ(y)>
• Valid if Mercer condition holds ({k(xi,xj)} p.semid.)• Allows analysis in complex superspace without
need to directly address its Cartesian form
![Page 13: Geometric margin domain description with instance-specific margins Adam Gripton Thursday, 5 th May, 2011](https://reader035.vdocuments.mx/reader035/viewer/2022062803/56649f3b5503460f94c59a69/html5/thumbnails/13.jpg)
Development of system
Support Vector Domain Description
• “One-class classification”• Fits sphere around cluster
of data, allowing errors {ξi}
• Extends in kernel space to more complex boundary
• Hybrid methods: multi-class classification
![Page 14: Geometric margin domain description with instance-specific margins Adam Gripton Thursday, 5 th May, 2011](https://reader035.vdocuments.mx/reader035/viewer/2022062803/56649f3b5503460f94c59a69/html5/thumbnails/14.jpg)
Development of system
Support Vector Domain Description
Dual formulation allows centre to be described in kernel space via weighting factors αi:
![Page 15: Geometric margin domain description with instance-specific margins Adam Gripton Thursday, 5 th May, 2011](https://reader035.vdocuments.mx/reader035/viewer/2022062803/56649f3b5503460f94c59a69/html5/thumbnails/15.jpg)
Development of system
Support Vector Domain Description
Values of αi form partition:• αi =0 inside• αi =1 outside• αi =C (support vectors)
Only support vectors determine size and position of sphere
![Page 16: Geometric margin domain description with instance-specific margins Adam Gripton Thursday, 5 th May, 2011](https://reader035.vdocuments.mx/reader035/viewer/2022062803/56649f3b5503460f94c59a69/html5/thumbnails/16.jpg)
Development of system
• Cannot use kernel methods directly with missing features
• Must impute (fill in) or assume probability distribution of missing values: pre-processing
• Missing features describe complex parametric curves in kernel space
• Seek a method which can address incomplete data directly: minimise point-to-line distances
Dealing with Missing Data
![Page 17: Geometric margin domain description with instance-specific margins Adam Gripton Thursday, 5 th May, 2011](https://reader035.vdocuments.mx/reader035/viewer/2022062803/56649f3b5503460f94c59a69/html5/thumbnails/17.jpg)
Development of system
• Chechik’s GM-SVM method provides analogue of binary SVM for structurally missing data
• Uses two loops of optimisation to replace instance-specific norms with scalings of full norm
• Questionable applicability to kernel spaces – difficult to choose proper scaling terms and ultimately equivalent to zero imputation
Dealing with Missing Data
![Page 18: Geometric margin domain description with instance-specific margins Adam Gripton Thursday, 5 th May, 2011](https://reader035.vdocuments.mx/reader035/viewer/2022062803/56649f3b5503460f94c59a69/html5/thumbnails/18.jpg)
Development of system
Synopsis for Novel System
Structurally missing features
Abstract, context-free
Domain description (one-class)
Avoid imputation / prob. models
Kernel Extension
Applicable to provided
data
![Page 19: Geometric margin domain description with instance-specific margins Adam Gripton Thursday, 5 th May, 2011](https://reader035.vdocuments.mx/reader035/viewer/2022062803/56649f3b5503460f94c59a69/html5/thumbnails/19.jpg)
Presentation Contents
• High-level motivation• Development of system• Exact-centre method• Dual-optimisation method• Experimental data collection• Conclusions
![Page 20: Geometric margin domain description with instance-specific margins Adam Gripton Thursday, 5 th May, 2011](https://reader035.vdocuments.mx/reader035/viewer/2022062803/56649f3b5503460f94c59a69/html5/thumbnails/20.jpg)
Exact-centre method
• Seeks solution in input space only• Demonstrates concept of optimisation-based
distance metric
![Page 21: Geometric margin domain description with instance-specific margins Adam Gripton Thursday, 5 th May, 2011](https://reader035.vdocuments.mx/reader035/viewer/2022062803/56649f3b5503460f94c59a69/html5/thumbnails/21.jpg)
Exact-centre method
• Cannot sample from entire feature space!• Selects centre point a such that φ(a) is optimal
centre (hence solves a slightly different problem)• Tricky (but possible) to optimise for soft margins
![Page 22: Geometric margin domain description with instance-specific margins Adam Gripton Thursday, 5 th May, 2011](https://reader035.vdocuments.mx/reader035/viewer/2022062803/56649f3b5503460f94c59a69/html5/thumbnails/22.jpg)
Exact-centre method
• Always performs at least as well as imputation in linear space w.r.t. sphere volume
• Often underperforms in quadratic space (which is expected, as domain restricted)
![Page 23: Geometric margin domain description with instance-specific margins Adam Gripton Thursday, 5 th May, 2011](https://reader035.vdocuments.mx/reader035/viewer/2022062803/56649f3b5503460f94c59a69/html5/thumbnails/23.jpg)
Presentation Contents
• High-level motivation• Development of system• Exact-centre method• Dual-optimisation method• Experimental data collection• Conclusions
![Page 24: Geometric margin domain description with instance-specific margins Adam Gripton Thursday, 5 th May, 2011](https://reader035.vdocuments.mx/reader035/viewer/2022062803/56649f3b5503460f94c59a69/html5/thumbnails/24.jpg)
Dual-optimisation method
• Motivated by desire to search over entire kernel feature space, to match imputation methods for non-trivial kernel maps
• Takes lead from dual formulation of SVDD where weighting factors αi are appended to dataset and implicitly describe centre a
![Page 25: Geometric margin domain description with instance-specific margins Adam Gripton Thursday, 5 th May, 2011](https://reader035.vdocuments.mx/reader035/viewer/2022062803/56649f3b5503460f94c59a69/html5/thumbnails/25.jpg)
Dual-optimisation method
• a must itself have full features, and therefore so must the “xi” in the sum
• Must therefore provide auxiliary dataset X* with full features to perform this computation
• Choice is largely arbitrary, but must span in FS• Weighting factors no longer “tied” to dataset
![Page 26: Geometric margin domain description with instance-specific margins Adam Gripton Thursday, 5 th May, 2011](https://reader035.vdocuments.mx/reader035/viewer/2022062803/56649f3b5503460f94c59a69/html5/thumbnails/26.jpg)
Dual-optimisation methodGiven an initial guess α:• Need to first produce full dataset Xa optimally
aligned to a, by optimisation over all possible imputations of incomplete dataset
• Then need to perform minimax optimisation step on vector of point-to-centre distances:
New candidate α at each optimisation step
![Page 27: Geometric margin domain description with instance-specific margins Adam Gripton Thursday, 5 th May, 2011](https://reader035.vdocuments.mx/reader035/viewer/2022062803/56649f3b5503460f94c59a69/html5/thumbnails/27.jpg)
Presentation Contents
• High-level motivation• Development of system• Exact-centre method• Dual-optimisation method• Experimental data collection• Conclusions
![Page 28: Geometric margin domain description with instance-specific margins Adam Gripton Thursday, 5 th May, 2011](https://reader035.vdocuments.mx/reader035/viewer/2022062803/56649f3b5503460f94c59a69/html5/thumbnails/28.jpg)
Experimental data collection
Preparatory trials of datasets constructed to exhibit degree of “structural missingness”:
• 2-D cluster of data with censoring applied to all values |x| > 1
• Two disjoint clusters –in [f1,f2], and in [f3,f4]
• One common dimension and three other dimensions each common to one part of set
Synthetic Data
![Page 29: Geometric margin domain description with instance-specific margins Adam Gripton Thursday, 5 th May, 2011](https://reader035.vdocuments.mx/reader035/viewer/2022062803/56649f3b5503460f94c59a69/html5/thumbnails/29.jpg)
Experimental data collection
Structure of comparisons:
Synthetic Data
Linear KernelK(x,y)=<x,y>
Quadratic KernelK(x,y)=(1+<x,y>)2
Hard Margin(all within sphere)
Soft Margin(50% outwith
sphere)
Imputation with [zeros, feature
means, 3 nearest neighbours]
vs.
Our XC and DO methods
![Page 30: Geometric margin domain description with instance-specific margins Adam Gripton Thursday, 5 th May, 2011](https://reader035.vdocuments.mx/reader035/viewer/2022062803/56649f3b5503460f94c59a69/html5/thumbnails/30.jpg)
Experimental data collection
Structure of comparisons:
Synthetic Data
Linear KernelK(x,y)=<x,y>
Quadratic KernelK(x,y)=(1+<x,y>)2
Hard Margin(all within sphere)
Soft Margin(50% outwith
sphere)
• Dual-optimisation method on hard margins only
• Particle-Swarm Optimisation also used to provide cross-validated classification study
•Main study is into effect on sphere size
![Page 31: Geometric margin domain description with instance-specific margins Adam Gripton Thursday, 5 th May, 2011](https://reader035.vdocuments.mx/reader035/viewer/2022062803/56649f3b5503460f94c59a69/html5/thumbnails/31.jpg)
Experimental data collection
Four main features selected for analysis:• Compton edge position (6 features)• Area under graph up to Compton edge (6)• Mean multiplicity of neutron data (1)• Poisson fit on neutron data (9) and chi-
squared goodness-of-fit (3)Total 25 features
Feature Extraction
![Page 32: Geometric margin domain description with instance-specific margins Adam Gripton Thursday, 5 th May, 2011](https://reader035.vdocuments.mx/reader035/viewer/2022062803/56649f3b5503460f94c59a69/html5/thumbnails/32.jpg)
Experimental data collection
Feature Extraction
PCA used on groups of features with identical presence flags to reduce dataset to 10 principal components missingness pattern intact
![Page 33: Geometric margin domain description with instance-specific margins Adam Gripton Thursday, 5 th May, 2011](https://reader035.vdocuments.mx/reader035/viewer/2022062803/56649f3b5503460f94c59a69/html5/thumbnails/33.jpg)
Presentation Contents
• High-level motivation• Development of system• Exact-centre method• Dual-optimisation method• Experimental data collection• Conclusions
![Page 34: Geometric margin domain description with instance-specific margins Adam Gripton Thursday, 5 th May, 2011](https://reader035.vdocuments.mx/reader035/viewer/2022062803/56649f3b5503460f94c59a69/html5/thumbnails/34.jpg)
Conclusions
• Dual-Opt method generally equalled or surpassed imputation methods in hard margin cases; XC method, predictably, did not operate as well in quadratic case
• Unreasonably small spheres start appearing with a soft-margin classifier as datapoints with few features start holding too much weight
• Cross-validation study using a joint optimiser shows improvement with quadratic kernel
![Page 35: Geometric margin domain description with instance-specific margins Adam Gripton Thursday, 5 th May, 2011](https://reader035.vdocuments.mx/reader035/viewer/2022062803/56649f3b5503460f94c59a69/html5/thumbnails/35.jpg)
Conclusions
• Insight provided into the behaviour of a kernel method with missing data – not much literature deals with this issue
• Link exists with the Randomised Maximum Likelihood (RML) sampling technique
• Deliberate concentration for now on entirely uninformed methods; scope exists for incorporation of this information where known to improve efficiency
![Page 36: Geometric margin domain description with instance-specific margins Adam Gripton Thursday, 5 th May, 2011](https://reader035.vdocuments.mx/reader035/viewer/2022062803/56649f3b5503460f94c59a69/html5/thumbnails/36.jpg)
Conclusions
• Sphere size ≠ Overall classification accuracy (c.f. a delta-function Parzen window) but this is arguably not we set out to achieve
• Divergent remit – not a catch-all procedure for handling all types of data, but gives insight into how structural missingness can be analysed
Caveats
![Page 37: Geometric margin domain description with instance-specific margins Adam Gripton Thursday, 5 th May, 2011](https://reader035.vdocuments.mx/reader035/viewer/2022062803/56649f3b5503460f94c59a69/html5/thumbnails/37.jpg)
Conclusions
• Fuller exploration into PSJO technique to provide alternative to auxiliary dataset
• Heavily reliant on optimisation procedures: could make more efficient than nested loop
• Extension to popular radial-basis (RBF) kernel• A more concrete application to sponsor
dataset
Room for Improvement
![Page 38: Geometric margin domain description with instance-specific margins Adam Gripton Thursday, 5 th May, 2011](https://reader035.vdocuments.mx/reader035/viewer/2022062803/56649f3b5503460f94c59a69/html5/thumbnails/38.jpg)
Thank you for listening…
Figure 1.1 (a)