PowerPoint Presentation

Machine Learning & BioinformaticsMachine Learning and Bioinformatics1Tien-Hao Chang (Darby Chang)Using kernel density estimation in enhancing conventional instance-based classifierKernel approximationKernel density estimationKernel approximation in O(n)Multi-dimensional kernel approximationApplication in data classificationMachine Learning and Bioinformatics2Identifying boundary of different classes of objects

Machine Learning and Bioinformatics3Boundary identified

Machine Learning and Bioinformatics4Problem definition of kernel approximationMachine Learning and Bioinformatics5Kernel approximationSpherical Gaussian functionsHartman et al. showed that a linear combination of spherical Gaussian functions can approximate any function with arbitrarily small errorLayered neural networks with Gaussian hidden units as universal approximations, Neural Computation, Vol. 2, No. 2, 1990

Machine Learning and Bioinformatics6Kernel approximationSpherical Gaussian functionsMachine Learning and Bioinformatics7Problem definition of kernel density estimationAssume that we are given a set of samples taken from a probability distribution in a d-dimensional vector spaceThe problem definition of kernel density estimation is to find a linear combination of kernel functions that provides a good approximation of the probability density functionMachine Learning and Bioinformatics8Probability density estimationMachine Learning and Bioinformatics9Probability density estimationMachine Learning and Bioinformatics10The Gamma functionMachine Learning and Bioinformatics11Machine Learning and Bioinformatics12Machine Learning and Bioinformatics13Kernel approximationAn 1-D example

Kernel approximation in O(n) Machine Learning and Bioinformatics14Machine Learning and Bioinformatics15A 2-D example of uniform samplingThe O(n) kernel approximation algorithmThe basic ideaMachine Learning and Bioinformatics16Machine Learning and Bioinformatics17The O(n) kernel approximation algorithmAn 1-D example

The O(n) kernel approximation algorithmMachine Learning and Bioinformatics18The O(n) kernel approximation algorithmMachine Learning and Bioinformatics19Machine Learning and Bioinformatics20

The O(n) kernel approximation algorithmMachine Learning and Bioinformatics21The O(n) kernel approximation algorithmMachine Learning and Bioinformatics22The O(n) kernel approximation algorithmGeneralizationMachine Learning and Bioinformatics23The O(n) kernel approximation algorithmGeneralizationMachine Learning and Bioinformatics24Machine Learning and Bioinformatics25An example of the effect of different setting of beta

The smoothing effectMachine Learning and Bioinformatics26Machine Learning and Bioinformatics27An example of the smoothing effect

Multi-dimensional kernel approximationMachine Learning and Bioinformatics28Multi-dimensional kernel approximationMachine Learning and Bioinformatics29Multi-dimensional kernel approximationMachine Learning and Bioinformatics30Machine Learning and Bioinformatics31Multi-dimensional kernel approximationMachine Learning and Bioinformatics32Machine Learning and Bioinformatics33Multi-dimensional kernel approximationMachine Learning and Bioinformatics34Multi-dimensional kernel approximationMachine Learning and Bioinformatics35Machine Learning and Bioinformatics36Multi-dimensional kernel approximationMachine Learning and Bioinformatics37Machine Learning and Bioinformatics38Any Questions?Lack something?Machine Learning and Bioinformatics39Application in data classificationRecent development in data classification focuses on the support vector machines (SVM), due to accuracy concernIn this lecture, we will describe a kernel density estimation based data classification algorithm that can delivers the same level of accuracy as the SVM and enjoys some advantagesMachine Learning and Bioinformatics40The KDE-based classifierThe learning algorithm constructs one approximate probability density function for one class of objectsMachine Learning and Bioinformatics41Machine Learning and Bioinformatics42The KDE-based classifierMachine Learning and Bioinformatics43The KDE-based classifierMachine Learning and Bioinformatics44Machine Learning and Bioinformatics45The KDE-based classifierMachine Learning and Bioinformatics46Machine Learning and Bioinformatics47Parameter tuningThe discussions so far are based on the assumption that the sampling density is sufficiently high, which may not hold for some real data setsMachine Learning and Bioinformatics48Parameter tuningMachine Learning and Bioinformatics49Parameter tuningMachine Learning and Bioinformatics50Time complexityMachine Learning and Bioinformatics51Comparison of classification accuracyMachine Learning and Bioinformatics52Comparison of classification accuracy3 larger data setsMachine Learning and Bioinformatics53Data reductionAs the learning algorithm is instance-based, removal of redundant training samples will lower the complexity of the prediction phaseThe effect of a nave data reduction mechanism was studiedThe nave mechanism removes a training sample, if all of its 10 nearest samples belong to the same class as this particular sampleMachine Learning and Bioinformatics54With the KDE based statistical learning algorithm, each training sample is associated with a kernel function with typically a varying widthMachine Learning and Bioinformatics55

In comparison with the decision function of SVM in which only support vectors are associated with a kernel function with a fixed widthMachine Learning and Bioinformatics56

Effect of data reductionsatimagelettershuttle# of training samples in the original data set44351500043500# of training samples after data reduction is applied18157794627% of training samples remaining40.92%51.96%1.44%Classification accuracy after data reduction is applied92.15%96.18%99.32%Degradation of accuracy due to data reduction-0.15%-0.94%-0.62%Machine Learning and Bioinformatics57Effect of data reduction# of training samples after data reduction is applied# of support vectors identified by LIBSVMsatimage18151689letter77948931shuttle627287Machine Learning and Bioinformatics58Execution times (in seconds)Proposed algorithm without data reductionProposed algorithm with data reductionSVMCross validationsatimage67026564622letter28251724386814shuttle9679559.9467825Make classifiersatimage5.910.8521.66letter17.056.48282.05shuttle17450.69129.84Testsatimage21.37.411.53letter128.651.7494.91shuttle996.15.582.13Machine Learning and Bioinformatics59