dimensionality reduction and interactive visualization of ... · systematic feature space reduction...

Systematic Feature Space Reduction and Interactive Analysis © Andreas König 2001

Andreas KönigDresden University of Technology

Chair of Electronic Devices and Integrated Circuits01062 Dresden

Dimensionality Reduction and Interactive Visualization ofMultivariate Data - Methods, Tools, Applications

Outline of the Tutorial:

• Introduction to Dimensionality Reduction• Feature Space Assessment Measures • Taxonomy and Presentation of Selected Methods• Tools and Applications• Summary and Perspective


Introduction to Dimensionality Reduction

Medical Database

Financial Database

Sensor Measurements

Features for Classification

v1v2....

vM

Computational complexity and Curse of Dimensionality promote efficientdimensionality reduction, visualization can provide transparence & insightDimensionality reduction is an ubiquituous problem in many disciplines !

Manufacturing Data

Technical problems are often characterized by sets of high dimensional dataSignificance, correlations, redundancy, or irrelevancy of the variables viwith regard to the given application a priori unknown



Methods for classification may be salient for visualization and vice versa can be a linear or a nonlinear mapping

Projection of multivariate data by DR mappings & ensuing visualization andinteractive analysis is a research topic of interest for more than 3 decadesRecently, data mining/warehouse & knowledge discovery applications giverenewed strong incentive & drive to the fieldFor classification, feature extraction & selection for vectorsare typically defined as (see, e.g., [Kittler 86])

Tovvvv ],...,,[ 21=

r

)(vAy rr=

)(max)(:))((max)(:

χχ JXJSelectionFeaturevAJAJExtractionFeature A

==

r

A mapping is optimized with regard toassessment criterion J and with d<o and

do RR →Φ :T

dyyyy ],...,,[ 21=r



The dimensionality reducing mapping can be unsupervised or employsupervised information

Classificationresult (J)

Raw featurecomputation y = A(x) Classifier

Train/Test

Modification of A

Sensor

Optimization criteria can be, e.g., signal, topology, distance preservation ordiscriminance gainDimensionality reduction (DR) methods can be salient both for multivariatedata classification and visualizationClassification: discriminance optimization with DR constraint for lean andwell performing recognition systemVisualization: DR fixed to dimension two or three with, e.g., structurepreservation constraint for data analysis, advanced MMI & system design


Introduction to Dimensionality ReductionMotivation of Data Visualization

This representation is not well adapted to human structure perception capabilities


Introduction to Dimensionality ReductionMotivation of Data Visualization

Violine Database

Table representation

Feature space visualization


Introduction to Dimensionality ReductionBenchmark Data for Method Demonstration

Iris data: Well known Iris flower data, 4 dimensions, training & test set with75 vectors each, 3 classes virginica, setosa, and versicolorMechx data: Mechatronic data from turbine jet engine compressor monitoring.Five data sets with 375 24-D vectors each, 4 classes corresponding tooperating regions and compressor stabilityX-Ray data: X-ray inspection of ball grid array packages in electronics manu-facturing. Two sets, 40 10-D vectors each, 3 classes for ok & defect type

Cube data: Artificial data, 3dimensions, 400 vectors, 8 classes.Data points on 8 edges of two oppositesides of a cube rotated by 45o


Feature Space Assessment MeasuresSupervised Assessment Options of Feature Spaces

Over1Over2

Over3

BimodalBanana

Measure discriminance byseparability, overlap,compactness, ...., of classregions in feature space


Feature Space Assessment MeasuresTaxonomy of Evaluation and Assessment Measures

Raw featurecomputation y = Asx

ClassifierTrain/Test

Classificationresult

Evaluation and Assessment Measures

unsupervised supervised

signaldistancetopology

SNRMSEENLMqm

Classifier output

Rate qk

Feature space

separabilitycompactnessoverlap

ENN qoi qciJs qsi


Feature Space Assessment MeasuresSimple Parametric Overlap Measure

Class specific distributions modelled by Gaussian functionsPairwise overlap can be computed from mean values µi, µj and standarddeviations σi, σj by

The feature merit for separating one class from all others is given by

The merit of a single feature to distinguish all classes could be computed by

Global summation can be misleading in some casesSimple & efficient measure for fast parametric first order feature selection

jjii

jiv NN

qijl σσ

µµ)1()1( −+−

−=

∑≠−

=L

ijvv ijlil

qL

q1

1

∑=

=L

ivv ill

qL

q1

1


The Kronecker Delta assures, that only vectors of class l are regarded+ Nonparametric, multivariate compactness measure+ The measure is free of user-definable parameters– Measure has O(N2) complexity– Normalization properties only allow observation of relative changes

Feature Space Assessment MeasuresNonparametric Compactness Measure

∑ ∑

∑ ∑ ∑

= +=

= = +=

∂−

∂∂−

= N

i

N

ijXjiB

L

l

N

i

N

ijXiji

llci

ij

ij

dN

dlNNL

q

1 1

1 1 1

*)),(1(1

*),(*),()1(

21

ωω

ωωω

Inspired by the work of Fukunaga et al. on scatter matrices & nonlinearmappings, a compactness measure can be derived based on intra/inter classdistances

≠=

=∂−= ∑= ji

jiji

M

qjqiqX if

ifandxxdwith

ij ωωωω

ωω01

),()(1

2

),( liω∂


Feature Space Assessment MeasuresNonparametric Overlap Measure

Inspired by probability estimation in Edited-Nearest-Neighbor method (ENN) orLeave-One-Out (LOO) classificationFor each vector, a number of k nearest-neighbors are computed & according tothe neighbors´ affiliation to same/different class an overlap measure is computed

∑∑

∑ ∑∑

=

=

= =

=

+=

c jiN

jk

ii

k

i

k

iiNNL

c coi

n

nq

NLq

1

1

1 1

1 *2

11

≠−=

=−=iji

ijiNN

NN

NNi ifn

ifnqand

d

dnwith

ji

jk

ji

ωωωω

1

Simplification of the measure can be achieved by only computing the rank orjust the number of nearest neighbors in the measure instead of the distances


Feature Space Assessment MeasuresNonparametric Overlap Measure

Assessment of the nonparametric overlap measure:+ qoi is well normalized in [0,1]; 1.0 indicates no overlap of class regions+ qoi is fine grained and thus well suited for optimization– The method has one required parameter k (k typically set to 5-10)– qoi has O(N2) complexity

Application example of qo for synthetic & application data:

Data set qo qo´ qo´´Over1 1,000000 1,000000 1,000000Over2 0,991100 0,993100 0,992800Over3 0,975300 0,979400 0,978200

Banana 1,000000 1,000000 1,000000Bimodal 0,990900 0,989800 0,990000Iris train 0,915600

Pins 0,952700 Mech1 0,978800


Feature Space Assessment MeasuresNonparametric Separability Measure

∑=

−−=

L

i i

RNNisi N

TNL

q i

1

)1(1

Here, Ni denotes the number of patterns per class & L the number of classes+ Fast due to O(N) complexity, no required user-definable parameters+ normed response [0,1], where 1.0 means linear separability– Coarse grained & thus less well suited for optimization

Exploits the fast training run of a Reduced-Nearest-Neighbor-Classifier (RNN)Separability is proportional to the number of chosen reference vectors TRNNi perclass during dynamic RNN configuration


Feature Space Assessment MeasuresNonparametric Separability Measure

Application example of qsi to synthetic and application data in comparison withFukunaga´s measure J from scatter matrices

Synthetic data asssessment results show the sensitivity to underlying &gradually decreasing separabilityIdentical vectors in the data set with different class affiliations return anassessment value of qsi = 0.0

Data set qs Js

Over1 1,000000 0,030863Over2 0,985612 0,025812Over3 0,960884 0,026084

Banana 0,993421 0,002372Bimodal 0,983333 0,007937Iris train 0,906667 2,659621

Pins 0,908108 3,461134Mech1 0,989333 4,824150


Taxonomy of Dimensionality Reduction Methods

Dimensionality Reduction Methods

Linear Methods Nonlinear Methods

Signal preserving Discriminance Topologypreserving

Distancepreserving

Discriminance

PCA FA Scattermatrices

FS & FW NLM LSB Visor TOPAS Koontz &Fukunaga

SOM BP (Discr.Analyis)

BP (Auto-Associative)

BP (NetPruning)

Signalpreserving

CCAM-PCA


Sideviews of hyper-cubeViews grow with

Limited applicabilityGets intractable fordata with M >> 1A priori knowledgerequiredBetter: Automaticfeature selection

Selected Dimensionality Reduction MethodsManual Selection

Dimensionality reduction and visualization by manual feature selection Visualization by multiple (linked) pairwise plots, six for Iris data:

2)1( −MM


Dimensionality reduction method from communication scienceObjective: Find a linear transformation with and so that the reconstruction error is minimimum for given mVisualization employs the first two (three) principal componentsCompute covariance matrix of the database:

Selected Dimensionality Reduction MethodsProjection by Principal Component Analysis

TN

ijiji vv

NK ))((1

1∑

=

−−= µµ rrrr

Compute Eigenvalues λi and Eigenvectors ψi of matrix KData is decorrelated by applyingLargest Eigenvalue corresponds to component with largest variance in the dataEigenvalues are sorted and the m Eigenvectors corresponding to the largestEigenvalues are selected for projection:

with ∑=

=N

iiv

N 1

1 rrµ

imi vym

rr..1.1

Ψ=

ii vy rrΨ=

imi vWy rr..1=

2

1

)ˆ(1i

N

ii vv

NE rr

−= ∑=

iT

mi yWv rr..1

ˆ =


For compression in channel coding or classification, m can be chosenaccording to Scree-plot or achieved classification rate RPCA is a linear, signal-preserving approach based on the assumption ofunimodal Gaussian data. Example for Mech1 data:

Selected Dimensionality Reduction MethodsProjection by Principal Component Analysis

Most variance is embodied in the first two principal componentsR grows with M2 for M-D data (Numerical & accuracy problems)Bad visualization for high intrinsic dimensionality or nonlinear components


Selected Dimensionality Reduction MethodsNonlinear Signal Preserving Mappings

For nonlinear data, the assumptions of PCA do not hold:

Principal componentsCurvi-linear components

Remedy: Curvi-linear components (CCA)Piecewise linear approximation in anhierarchical approach using clustering &local PCA computation (M-PCA)

µr

µr

2µr

1µr


Selected Dimensionality Reduction Methods Nonlinear Signal Preserving Mappings

Application of backpropagation networks in autoassociative mode:

Network learning tries to preserve thefeature values over the bottleneck layer3L networks perform similar to PCANumber of bottleneck layer neuronsdefine projection dimension dExample with 4-2-4 BP network5L networks performs nonlinearcompression, extracting principal curvesTopology must be specified by userHard to train and to interpretExample with 4-9-2-9-4 BP network



Visualization of Iristrain data by3L-BP in autoassociative mode:

Visualization of Iristest data by3L-BP in autoassociative mode:



Visualization of Iristrain data by5L-BP in autoassociative mode:

Visualization of Iristest data by3L-BP in autoassociative mode:


Selected Dimensionality Reduction Methods Topology Preserving Mapping: Kohonen´s SOM

The Self-Organizing feature Map (SOM), introduced by Teuvo Kohonen isthe probably most well-known and applied neural networkThe SOM was derived from physiological evidence observed in the somatosensory cortex

Input vector ivr

Weight vector jwr

∑=

= −=M

iiji

Njc wvd SOM

1

21 )(min

WTA

Winning neuron Nc12

M

Componentplanes

Neigborhood function(Gaussian, pyramid, box)

M

)(tα

)(tr


The SOM features the properties of data quantization, probability densityapproximation, topology preserving dimensionality reducing mappingTypically, 1D- or 2D-SOM neuron grids are employed (3D in Robotics)SOM learning in its common technical implementation:

Random initialization of neuron weight vectorsIterative presentation of stimuli vectors and computation of thewinner neuron Nc

Adaptation of the winning neuron and the neigbors

Reduce and r(t); Terminate learning by max. steps/error


jwr

ivr

∑=

= −=M

iiji

Njc wvd SOM

1

21 )(min

∉∈−+

=+))(()())(())())((()()(

)1(trNjfortwtrNjfortwvtrNttw

twcij

cijkicij

ij

α

)(tα


During the training process, the SOM unfolds in the multivariate patternspace and creates a topology preserving mapping to the 2D neuron gridExample of SOM visualization for Cube-data:



SOM component planes for Cube-data:


Plane 1

Plane 2

Plane 3


SOM only indicates the presence of cluster, yet intra/inter cluster distancesremain obscureThe Unified-Distance-Matrix complements SOM with distance information

Selected Dimensionality Reduction Methods Enhancement of SOM Projection and Visualization

)(*5.0

),(

),(

),(

),(

11

11

1

1

ijijij

jiijij

jiijij

jiijij

ijijij

dyxdxydz

uuddyx

uuddxy

uuddy

uuddx

+=

=

=

=

=

++

++

+

+

Distances from neuronuij to neighbors:

U-Matrix U-Matrix U-Matrix U-Matrix 2j+12i-1 ... dzi-1,j-1 dyi-1,j dzi-1,j

2i ... dxi,j-1 dui,j dxi,j

2i+1 ... dzi,j-1 dyi,j dzi,j

du(i,j) arbitrarily, set tomean of 8 neighbors


SOM quantizes the data, i.e., SOM weight vectors are representatives ofclusters of database vectors, which themselves are not directly visibleSOM interpolation properties cause the placement of weight vectors infeature space regions actually void of data samplesFor high intrinsic dimension larger than SOM dimension the SOM tends tofold and twist in the attempt to establish a mapping to the 2D neural grid

Selected Dimensionality Reduction Methods Assessment of SOM for Projection and Visualization

SOM only indicatesthe presence ofclusters. Distanceinformation requires,e.g., U-Matrix

Alternative: Sammon´s Nonlinear distance preserving mapping (NLM)


After SOM training, thecomputed weight vectorsare subject to NLMComparison of SOM &SOM/NLM mapping forIristrain dataIntra/inter cluster distancesbecome overtSOM unfolding & effect ofinterpolation vectors clearerThe idea can be extended toa generalization of thecomponent planes

Selected Dimensionality Reduction Methods Enhancing SOM Visualization with NLM

Interpolation vectors


Sammon´s stress assesses distortion of distances while mapping the datavectors from X space to corresponding pivot points in Y space by his NLM:

Selected Dimensionality Reduction Methods Distance Preserving Nonlinear Mapping: Sammon´s NLM

∑∑∑∑= == =

=−

=N

j

j

iX

N

j

j

i X

YX

ij

ij

ijij dcwithd

mdd

cmE

1 11 1

2))((1)(

Here, dXij and dYij denote the interpoint distance in feature space X andprojection space Y, respectively:

∑

∑

=

=

−=

−=

M

qjqiqX

d

qjqiqY

xxd

mymymd

ij

ij

1

2

1

2

)(

))()(()(


Minimization of Sammon´s stress using gradient descent is achieved byiterative computation of new coordinates for the pivot vectors in Y space:

Selected Dimensionality Reduction Methods Distance Preserving Nonlinear Mapping: Sammon´s NLM

The partial derivatives for gradient descent are given by

)(*)()1( myMFmymy iqiqiq ∆−=+

2

2

)()(/

)()()(

mymE

mymEmywith

iqiqiq ∂

∂∂∂

=∆

∑

∑

≠=

≠=

−−−

−=

∂∂

−

−

−=

∂∂

N

ijj Y

jqiq

XYiq

jqiq

N

ijj XYiq

ijijij

ijij

dyy

ddcymE

yyddcy

mE

13

2

2

2

1

)(112)(

)(112)(


NLM maps each point and preserves distances & structure in dataSalient visualization method for direct data projection & visual analysisMapping control:

Error E(m)>0.1 indicates unacceptable mapping result [Sammon 69]Computation of NLM has O(N2) complexityMethod becomes infeasible for large databasesGradient descent optimization is not capable to exactly preserve all distancesInevitable mapping error is introduced

Selected Dimensionality Reduction Methods Assessment of Sammon´s Nonlinear Mapping

Acceleration for approximated mapping by heuristics !


Lee, Slaggle & Blum´s Triangulation Mapping:Compute Minimal-Spanning-Tree (MST) of the data (O(N2) complexity !)Select initial data points from the MST and map to 2D spaceStep through MST & map each of the remaining points by triangulation

Selected Dimensionality Reduction Methods Accelerated Distance Preserving Mappings

Only 2N-3 distances are exactly preservedThe algorithm tends to unfold circular structures, misleading in analysis !Option: Choose a fixed, global reference point for focusing the mapping (ROI)


Visor Mapping [König et al. 94]:Fast mapping with only O(N) complexity as data previewerBases on the assumption to find three suited global pivot points for ensuingtriangulation mapping of the remaining N-3 data pointsMapping steps (no parameters):

Compute centroid of the data setDetermine data point most distant from centroidDetermine two more data points with maximum from centroid &each otherPlace these pivot points in the 2D-plane, exactly preserving theirmutual distancesPlace the remaining N-3 points by triangulation based on pivot points

Selected Dimensionality Reduction Methods Accelerated Distance Preserving Mappings

The concept is amenable to enhancement by reference or multiple pivot points


NLM and Visor mappingresults are displayed forIristrain dataMappings can be subject torotation & mirroringBasic global data structureis quite similarThe relevant information,e.g., for classificationsystem design can beextracted by fast previewQuantitative analysis ofmapping error required !

Selected Dimensionality Reduction Methods Comparing NLM and Visor Mappings


After initial dimensionality reducing mapping computation for a given dataset, single or groups of data points shall be subject to the same mapping, tooIn classification, test & application data shall be mapped in recallIn visualization, new data points shall be mapped & displayed

In both cases, the mapping of additional points shall take place withoutrecomputation of the entire mappingPCA & ANN´s satisfy this requirement, NLM originally does notRemedy 1: Train an ANN with NLM mapping data & use it for recallRemedy 2: Introduce NLM recall mapping (NLMR)

Selected Dimensionality Reduction Methods Regarding the Issue of Recall or Postmapping

12

M-1

M

1

2Initial Mapping

PostmappingX Y

Newdata


NLMR bases on the previously computed NLMNew data vectors subject to recall are placed according to their distances to the training data pointsMutual distances between test vectors are neglected in NLMRThus the cost function is modified to:

Selected Dimensionality Reduction Methods Introducing an NLM recall method (NLMR)

rivr

ijXd ˆtjvr

∑∑

∑

==

=

=−=

−=

K

jX

M

q

tjq

riqX

K

j X

YXi

ijij

ij

ijij

dcandvvdwith

d

mdd

cmE

1ˆ

1

2ˆ

1 ˆ

2ˆˆ

ˆ)(

))((

ˆ1)(ˆ

rr

K corresponds to the number of initially mapped training samples


Each point is randomly initialized in Y spaceIterative adjustments by gradient descent take place according to:

Selected Dimensionality Reduction Methods Introducing an NLM recall method (NLMR)

10)(ˆ)(ˆ

/)(ˆ)(ˆ

)(ˆ

)(ˆ*)(ˆ)1(ˆ

2

2

≤<∂∂

∂∂

=∆

∆−=+

MFandmymE

mymEmywith

myMFmymy

iq

i

iq

iiq

iqiqiq

Mapping of test data for recall orpostmapping is now feasibleThe number of iterations required per datavector mapping vary considerablyComputational savings can be achievedfor given Minimum Error threshold


NLM(R) can now serve as nonlinear alternative to PCA. Quantitativediscriminance comparison showed NLM(R) to be superior for given dim.

Selected Dimensionality Reduction Methods NLM recall method (NLMR) Application

NLMR Application example for Iristrain and Iristest data:


Compression from M- to 2D data for linear/nonlinear unsupervised methodsEvaluation by overlap qo and separability qs measures for training & test sets

Selected Dimensionality Reduction Methods Quantitative Assessment of Unsupervised Mapping Techniques

Method Dim. Train qo qs Test qo qsOriginal 4 Iristrain 0.95503 0.90666 Iristest 0.91683 0.88000PCA 2 Iristrain 0.91329 0.86667 Iristest 0.91970 0.89333NLM 2 Iristrain 0.94250 0.89333 Iristest 0.93128 0.89333BP 2 Iristrain 0.93008 0.90666 Iristest 0.91630 0.90667Original 24 Mech1 1.000 0.98933 Mech2 0.99799 0.96308PCA 2 Mech1 0.98959 0.97067 Mech2 0.94889 0.91384NLM 2 Mech1 0.97898 0.94400 Mech2 0.95730 0.91384BP 2 Mech1 0.97861 0.94400 Mech2 0.93619 0.89538

BP has 4-9-2-9-4 network topology, 1000 epochs of quickprop learningPCA is simple and excellent when applicable to the given dataNLM(R) outperforms PCA for nonlinear data & is easy to useBP results are very hard to obtain & approximately as those of NLM(R)This comparison assessed discriminance not structure preservation !


Supervised linear technique for dimensionality reduction [Fukunaga 90]Objective: Find a linear transformation that minimizes the within or intraclassdistances between data points and maximizes the between or interclassdistances, i.e., class regions shall become compact & well separated Scatter matrices are computed for this aim. In the parametric approachinterclass Sb and intraclass Sw scatter matrices are computed by:

Selected Dimensionality Reduction MethodsSupervised Mapping Techniques: Linear Discriminance Analysis

∑∑= =

−−=L

i

N

j

Tjjjjw

iiiii vv

NS

1 1))((1 ωωωω µµ rrrr

with ∑=

=i

ii

N

jj

i

vN 1

1 ωωµ rr

∑=

−−=L

i

Tjj

ib

ii

NN

NS

1

))((1 µµµµ ωω rrrrwith ∑∑

==

==N

j

iN

jj

i

NNv

N 11

1 ωµµ rrr


One possible measure of the underlying separability is given by Js with:

Selected Dimensionality Reduction MethodsSupervised Mapping Techniques: Linear Discriminance Analysis

Computation of Eigenvalues λi and Eigenvectors ψi of matrixLargest Eigenvalues correspond to components with largest discriminanceEigenvalues are sorted and the m Eigenvectors corresponding to the largestEigenvalues are selected for dimensionality reduction:

imi vym

rr..1.1

Ψ=

)( 1bws SSTrJ −=

)( 1bw SS −

Dimensionality is reduced and discriminance improved in one stepFor classification, m is chosen according to Scree plot of λi or one of themeasures given in the taxonomy of section 2Visualization by 2 (3) ψi, for m>2 combination with unsupervised methodsThe parametric approach is limited in scope and ability !


Here, denotes the mean vector of the k-nearest-neighbors

of with different class affiliations

The weighting factor for off-class-borders vectors gj is given by:

Extension of scatter matrices for nonparametric or even multimodal data[Fukunaga 90]Interclass scatter is computed based on k-nearest-neighbor technique fornonparametric scatter matrix Sb by:

Selected Dimensionality Reduction MethodsSupervised Mapping Techniques: Nonparametric Scatter Matrices

∑∑= =

≠≠ −−=L

i

N

j

TjNNjjNNjib

iiiii vvg

N 1 1))((1 ωωωω µµ rrrr

S

∑=

≠≠ =k

lljNN

i

NN

i vk 1

1 ωωµ rr

ijvωr

),(),()},(),,(min{

iiii

iiii

jNNjjNNj

jNNjjNNjj vdvd

vdvdg ωωςωως

ωωςωως

µµµµ

+=

with weight decay factor

[ [∞= ,0ς


In space Y, now Tr(Sw-1Sb) = Tr(Sb) holds (whitening according to Sw )

Computation of Eigenvalues λi and Eigenvectors ψi of matrix Sb

Eigenvalues are sorted and the m Eigenvectors corresponding to the largestEigenvalues are selected for dimensionality reduction:

The parametric intraclass scatter matrix Sw is retainedDecorrelation with regard to Sw , i.e., Sw in space Y will be Sw= IPattern data is transformed by


jTww

j vyM

rr ))(( 21

..1

−ΛΨ=

** )(..1.1 j

Tbj vy

mm

rrΨ=

Finally, the dimensionality reducing projection is computed by:

jTwwTb

mjTb

mj vyyMm

rrr ))(()()( 21

..1..1*

..1..1

−ΛΨΨ=Ψ=


Visualization example of nonparametric scatter matrix (NPSCM) applicationfor Mech1 data:


Numerical assessment of the method with regard to discriminance will followafter presentation of nonlinear supervised mapping methods


Koontz & Fukunaga extended Sammon´s stress in the NLM by an additionalterm dedicated to assessment of intraclass distancesThe KFM mapping pursues intraclass reduction & structure preservation andthus nonlinear discriminance analysis by the following cost function:

Selected Dimensionality Reduction MethodsSupervised Mapping Techniques: Koontz & Fukunaga´s Mapping

∑∑= =

−+=

N

j

j

i X

YXji

ij

ijijijY

d

mddmd

cmE

1 1

22 ))((ˆ)(),(1)(λωωδ

Here, c, dXij and dYij are the same terms defined for Sammon´s stress in theNLM and

1ˆ:0:ˆ

),( =

≠=

= αωωωωα

ωωδ withji

jiji


Minimization of KFM cost function using gradient descent is achieved byiterative computation of new coordinates for the pivot vectors in Y space:


The partial derivatives for gradient descent are given by

)(*)()1( myMFmymy iqiqiq ∆−=+

2

2

)()(/

)()()(

mymE

mymEmywith

iqiqiq ∂

∂∂∂

=∆

∑

∑

≠=

≠=

−−−−=

∂∂

−

−−=

∂∂

N

ijj Y

jqiq

XYX

ji

iq

jqiq

N

ijj XYX

ji

iq

ijijijij

ijijij

dyy

dddcymE

yydddcy

mE

13

2

2

2

1

)(11ˆ),(2)(

)(11ˆ),(2)(

λωωδ

λωωδ


Modification of the original approach by setting reduces parametersand gives convenient control from pure structure preservation ( ) topure separability achievement ( )Visualization of Iristrain and Mech1 with after 100 iterations


3.0ˆ =λ

λα ˆ1ˆ −=0.1ˆ =λ

0.0ˆ =λ


As for NLM, a recall procedure is not straight forward available for KFMKoontz & Fukunaga presented a (restricted) method using polynomial distanceapproximation and a pivot point approach for mappingEmployment of neural network, e.g., BP, as universal function approximatoris an interesting alternative


Network is supplied with original featuredata at the input & KFM data at the outputTraining phase aims on mapping errorminimization with generalizationTest data is mapped later for classificationor visualization employing the trained netProblem: Net configuration & convergence


Visualization of KFM applied to Iristrain and Iristest data with a 4-9-5-2 BPnetwork (training & test data mapped by BP network !)


Achieving network convergence & good approximation properties is notstraightforward for combined KFM/BP nonlinear mapping


Employment of BP neural network with a bottleneck topology for nonlinearmapping or nonlinear discriminance analysis (NDA) is an interestingalternative:

Selected Dimensionality Reduction MethodsSupervised Mapping Techniques: BP with Bottleneck Topology

Network is supplied with original featuredata at the input & class data at the outputTraining phase aims on classification errorminimization with given bottleneckThe number of neurons in the bottlenecklayer determine projection dimension dThe trained net is cut after bottleneckTest data is mapped later for classificationor visualization employing the cut netProblem: Net configuration & convergence


Result of BP_NDA for Mech1 and Mech2 data employing a 24-9-2-4 BPneural network with a bottleneck topology (d=2):

Selected Dimensionality Reduction MethodsSupervised Mapping Techniques: BP with Bottleneck Topology

Fast and rather easy training, only 130-150 epochs were requiredMuch easier & faster than KFM/BP, but no structure preservation


Compression from M- to 2D data for linear/nonlinear supervised methodsEvaluation by overlap qo and separability qs measures for training & test sets

Selected Dimensionality Reduction Methods Quantitative Assessment of Supervised Mapping Techniques

Method Dim. Train qo qs Test qo qsOriginal 4 Iristrain 0.95503 0.90666 Iristest 0.91683 0.88000NPSCM 2 Iristrain 0.98224 0.94666 Iristest 0.95788 0.90666BP 2 Iristrain 0.97536 1.000 Iristest 0.95295 0.94667KFM/BP 2 Iristrain 1.000 1.000 Iristest 0.94023 0.92000Original 24 Mech1 1.000 0.98933 Mech2 0.99799 0.96308NPSCM 2 Mech1 0.99908 0.98200 Mech2 0.98716 0.97536BP 2 Mech1 0.99988 0.99733 Mech2 0.99373 0.99385

KFM with trained BP network (only for Iris data, hard to train for Mech data)BP with 24-9-2-4 network topology for Mech1 fast to train & good resultsLinear NPSCM is simple and excellent when applicable to the given dataNonlinear BP, KFM/BP outperform NPSCM. Problem: overfitting/overlapBP & KFM/BP similar but KFM/BP results are very hard to obtain


AFS is a linear mapping, based onthe selection matrix AS with Feature selection: ,switch variables, 2M combinations Feature weighting:or arbitrary real numbers

Selected Dimensionality Reduction Methods Supervised Mapping Techniques: Automatic Feature Selection (AFS)

)(max)( χχ JXJ =

Quest: Find minimum feature subset with optimum discriminanceAFS chooses for a given sample set the subset of variables or configuration Xthat maximizes a given cost function J:

M

M

cc

cc

...0000...00...............0...000...00

1

2

1

−

AS= { }1,0∈ic

[ ]1,0∈ic

Cost function J can be one of the measures given in the taxonomy, e.g., qsiCombinatorial optimization problem, NP-completeExhaustive search for global optimum only feasible for small MWay out: Apply heuristics & optimization strategies to find at least a localoptimum with bounded time and effort

jsj xAy rr=


Selected Dimensionality Reduction Methods Supervised Mapping Techniques: First-Order Feature Selection

Simplification: J is computed for each individual feature and a selectedcombination of classes, e.g., pairwise class discriminanceConsiderable computational savings by neglecting possible higher ordercorrelations between feature pairs or tupels could serve as simple parametric measureFor each class separation feature are ranked according to their J valueRank table example for Iristrain data:

Selection takes place by choosing the features on top rank positions, e.g.,features 3 & 4 for first rank position (C=[0, 0, 1, 1]T)

ijlq

1

2

4

3

Rank

1.660

1.218

0.255

0.442

C 2-3

0.89041.0653v2

5.18024.3871v4

5.45114.1392v3

1.48231.0204v1

C 1-3RankC 1-2RankFeature



Assessment of Parametric First-Order Feature Selection (FOFS):The method is very fast and its complexity is O(M)However, for pairwise class separation, the complexity grows with O(L)The rank tables grow exponentially, visual inspection becomes infeasible !Only a local optimum solution can be expectedThe method can be extended to different class separation (one vs. all, all)Inclusion of lower ranking features can affect the solutionSummary: If parametric assumption is met, method can be fast & effectiveProblem: Nonparametric and multimodal one-dimensional distributions:

µ1 µ2

σ1 σ2

• Class 1• Class 2



Remedy: A nonparametric J is computed, e.g., qoi restricted to one dimensionExample for application data with two classes from visual inspection:

The global solution C=[0, 0, 0, 1, 1]T was determined by exhaustive searchThe nonparametric FOFS found the global optimum, parametric FOFS failedHowever, the number of rank positions salient for the given problem is notimmediately obviousFOFS can serve as first step to confine search space in selection hierarchy

0.927011.00472v5

0.821130.64894v2

0.880820.85473v4

0.705841.15581v3

0.697750.29175v1

C 1-2Rank (Nonpar.)C 1-2Rank Par.)Feature


Nonparametric cost function is computed for each configuration, i.e., thecurrently selected subset of featuresHeuristic search strategy is applied to confine the search spaceSequential-Backward/Forward-Selection (SBS/SFS):

M*(M-1)/2+M combinations have to be assessed for SBS/SFS O(M2) complexity

For M=16, a local optimum solution will be found in approx. 136 s (1 s perassessment assumed) in contrast to 18h for exhaustive search

Selected Dimensionality Reduction Methods Supervised Mapping Techniques: Higher Order Feature Selection

⇒

M features active

M-1 features active

M-2 features active1 features active

Tentative removal ofeach active feature

Permanent removal of thefeature maximizing J


Application example of SBS for Iris train:

Application example of SBS for visual inspection data:

Further applicable heuristics: Branch & bound, floating search, etc.


1 2 3 4 5 0.903846- 2 3 4 5 0.942308- - 3 4 5 0.961538- - - 4 5 0.961538- - - - 5 0.913462Optimum quality: 0.961538Significant Features: 4 5

1 2 3 4 0.90667- 2 3 4 0.94667- - 3 4 0.96000- - - 4 0.00000 Optimum quality: 0.96Significant Features:3 4


Stochastic methods, e.g., Simulated Annealing (SA), applicable as heuristicPerturbation Method (PM) is a simple variant, based on random state changesPM application for Iristrain:


Proposed Accepted1 - - - : 0.00000 0.0000001 - 3 - : 0.88000 0.8800001 - 3 - : 0.00000 0.8800001 - 3 4 : 0.94667 0.9466671 - 3 4 : 0.92000 0.9466671 - 3 4 : 0.90667 0.9466671 - 3 4 : 0.90667 0.9466671 - 3 4 : 0.88000 0.9466671 - 3 4 : 0.88000 0.9466671 - 3 4 : 0.88000 0.946667- - 3 4 : 0.96000 0.960000- - 3 4 : 0.00000 0.960000- - 3 4 : 0.00000 0.960000- - 3 4 : 0.00000 0.960000- - 3 4 : 0.94667 0.960000Best Quality: 0.96000Best Features: 3 4

Random initial stateRandom selection of variableto propose state transitionSelected variable is toggledand transition is accepted if

oldsi

newsisi qqq −=∆

Random tossing surprisinglyeffective at moderate effortDifferent results for each run !Several enhancements possible(multiple variable change or SA)


Vectors x that cannot be improved in any objective dimension withoutdegradation in another are known as Pareto optimalFor two vectors a,b from parameter space X, a dominates biff

Alternative to weighted sum: Pareto optimalityGeneral multiobjective optimization problem with objective vector y

))(,),(())(,),(),(()( 21 xdrxqxfxfxfxfy sin KK ===

)()(:},,2,1{)()(:},,2,1{

bfafnjbfafni

jj

ii

>∈∃∧≥∈∀

K

K

Selected Dimensionality Reduction Methods Supervised Mapping Techniques: Evolutionary Feature Selection

Multiobjective optimization for AFS, e.g., qsi, qoi and )0,(11

i

M

ic

Mdr ∑

=

∂=

)( ba f

)()( bfaforba =fAdditionally, a covers b iff)( ba f

All nondominated vectors in X are denoted as Pareto optimal front


Strength Pareto Approach (Zitzler &Thiele) employed for FS: FSSPEA


Individual´s fitness in external set:

)1,0[;1

∈+

== iii sN

nsf

Individual´s fitness in population:

´),1[;1,

Nfsf jjii

ij ∈+= ∑f

External set pruning (clustering)FSSPEA parameters (ICANNGA´01):☯ Tournament selection☯ N=20, N´=10☯ Mutation rate=0.07☯ Random initialization☯ Uniform crossover☯ 50 generations

N

N´



Method ORG SFS SBS PM SGA Train Test

0,880

0,900

0,920

0,940

0,960

0,980

1,000

Mechatronic

Method Mech1 Mech2 FeaturesORG 0,992500 0,981667 24SFS 0,997500 0,928333 3SBS 0,989333 0,981667 12PM 0,997500 0,959167 6SGA 0,997333 0,969167 12FSSPEA 0,997500 0,928333 3

Original feature space Feature space for FSSPEA

Results for Mech1 & Mech2:

SFS and FSSPEA find sameoptimum solution !Inclusion of further constraints


Interactive Data Visualization and Explorative Analysis

Data-base

Data-base

Pre-processing

Pre-processing

DimensionReduction

DimensionReduction

Attributes Visualization

Uncompresseddata

Features 2D-Coordinates

Visualization standard: Static (2D-) scatter plot Advanced Visualization by enhanced user interaction & CAD-functionality


Interactive Data Visualization and Explorative AnalysisGeneralization of Component Planes

Generalization fromSOM component planes

The values of onefeature are displayed atthe projection points

Like Hinton diagrams

Several planes can bedisplayed and analysed

Correlating and salientfeatures can be detected

Multicomponent plotsexploit human textureperception for analysis

Coding by shade Coding by size

Complementing class information Weight icons


Interactive Data Visualization and Explorative AnalysisMulticomponent Plots

Weight IconsIconified radar plot at eachprojection pointDenoted as Snow-Flakes


Interactive Data Visualization and Explorative AnalysisGrid Modes

Grid modes comple-menting the SOM mesh

Sequence order of datavectors can be displayed

Voronoi tesselation giveexact quantization and1-NN classificationboundaries for 2D-data

An adequate idea isgiven for projection ofmultidimensional data

Sequential threading

Class-specific VTVoronoi tesselation

Delaunay grid


Interactive Data Visualization and Explorative AnalysisOutlier and Identical Vector Visualization

Variable-sized fields and colored class-specific VT can alleviate this problemOutliers and isolated vectors in different class regions can easily be identifiedIdentical vectors are marked by an hexagonal shapeStacking by mouse click rotates identical or strongly occluding vectorsFast troubleshooting: Pathological vectors can be tracked-back in the database

Outliers and identical vectors are pathological cases for system designAt high point densities mutual occlusion of points becomes a problem


Interactive Data Visualization and Explorative AnalysisAttribute Display and Hierachical Database Access


Interactive Data Visualization and Explorative AnalysisInteractive Navigation and Hierarchical Analysis

Focus of interest can conveniently be shifted across scales by zooming in orout and in the same scale by shifting the navigation windowDescending down to local detail, mapping quality and errors become an issue

The methods described so far alleviate first insight in the data structureFor very large data sets with considerable local density variations, analysis ona single scale will not be feasibleGlobal to local analysis across scales (zoom/pan) offers superior performance:

Navigation Window

Current ROI


Interactive Data Visualization and Explorative AnalysisInteractive Navigation and Hierarchical Analysis

Scale is preset from menu entry or interactively by mouse operation

Different implementation in QuickCog with pop-up navigation window anddrag&drop navigation


With increasing intrinsic dimension, all mapping methods will introducemapping errorsVisualization results may be subject to twists and distortions that can lead toinvalid conclusion drawing & corrupt knowledge aquisitionHow can mapping quality be measured and mapping faults be disclosed ?First proposed measure: Distance preservation E with

Interactive Data Visualization and Explorative AnalysisAssessment of Mapping Quality and Mapping Errors

∑∑= =

−=

N

j

j

i X

YX

ij

ijij

d

dd

cE

1 1

2)(1

12

M-1

M

1

2

i

j j

i dYidX


Local quality qmi is computed by determining the n-nearest-neighbors NNji ofthe ith pattern

Second proposed measure: Topology preservation qm with

Interactive Data Visualization and Explorative AnalysisAssessment of Mapping Quality and Mapping Errors

∑=

=N

imm i

qNn

q1*3

1

]),1[],,1[( Njni ∈∈

Rank order of the nearest-neighbors in X and Y are compared and qmi valuesare assigned according to the following simple assignment scheme:

qmi = 3 if NNji in X = NNji in Yqmi = 2 if NNji in X = NNjl in Y with qmi = 1 if NNji in X = NNjt in Y withqmi = 0 else

ijnl ≠∈ ];,1[mnmnt <∈ ];,[

The topology preservation measure qm returns 1.0 for perfect preservation


Interactive Data Visualization and Explorative AnalysisApplication of Mapping Quality Measures

For typical parameter settings n=4 and m=10 an application example forIristrain and Mech1 data is given:

Data setqm E qm E

Iristrain 0.6667 0.0098 0.6711 0.0093Mech1 0.4527 0.0933 0.4207 0.1628

NLM Visor

In addition to the global measures E and qm, local information for each pointin the projection can be computed by local distance preservation Ei and localtopology preservation qmiThese can be superimposed on the mapping:

Local mapping faults can be disclosedO(N2) complexity of the quality measure !Consumes more time than the mapping itselfRemedy: Fast interactive local alternative

qmi plot for n=4 and m=10


Interactive Data Visualization and Explorative AnalysisActual Local Neighborhood Display

The actual nearest neighborsin X are computed for aninteractively chosen patternThe projection pointscorresponding to these NNin Y are threaded by a redline in the visualizationThis discloses mappingerrors at low computationalcost for valid conclusiondrawing and knowledgeacquisitionSimilar to qmiCompatible with navigation& hierarchical analysis


Interactive Data Visualization and Explorative Analysis Actual Local Neighborhood Display

Diminishing linewidthwith growing distancefrom reference pointimproves the visualization

Instead of point threadingby a polyline, a pairwiseconnection can bebeneficial & more lucid


Interactive Data Visualization and Explorative Analysis Advanced Projection and Visualization Techniques

None of the presented methods will return in general a satisfying global andlocal mapping qualityThree main strategies for improving mapping quality can be observed:

Hierachical clustering andmapping of the dataHierarchical mapping &navigation approach:Global to local shift of themapping stress implementedby a limited or shrinkingneighborhood during themapping computation


Interactive Data Visualization and Explorative Analysis Advanced Projection and Visualization Techniques: Example of Case 2

Artificial 3D sphere dataPoints distributed only on thesphere shellIntrinsically 3D data !Mapping error to 2D unavoidableSphere is flattended: ROIcontains vectors from front & rearRecomputation separates data !2D-Projection of 3D sphere

Selected ROIRemapping of data comprised in ROI


Computational effort & mapping quality can be traded-off in a hierachicaldata clustering and mapping approachData is first quantized by SOM, c-means, or other clustering methodThen, the K centroids of the quantized data are mapped by NLMThe complete data set is then mapped by NLM(R)Considerable speed-up can be achieved:


hier

complete

SASA

SMF =

KNiterKKiterSAand

NNiterationsSAwith

rthier

complete

**2

)1(**

2)1(**

+−

=

−=

Obviously, K controls computational effort & mapping qualityThe required computational effort for clustering must be included


Interactive Data Visualization and Explorative AnalysisHierarchical Mapping Based on NLM and NLM(R)

Quantization of Iris data by SOMOptional component plane visualization

Mapping of all 150 samples by NLM(R)

NLM Mappingof 16 centroids


Interactive Data Visualization and Explorative AnalysisHierarchical Mapping Based on NLM and BP network

Quantization of Iris data by SOM

Mapping of all 150 samples by BP recall

NLM Mappingof 16 centroids

Recall of BPwith centroids


Interactive Data Visualization and Explorative AnalysisHierarchical Mapping Based on NLM and NLM(R)

Mechatronic (1775 samples, 24-D) data K=64 centroids, SMF=26.73 (K=3, SMF=591 !)


Interactive Data Visualization and Explorative AnalysisVisual Assessment of Hierarchical Mapping

NLM(Flat) HierarchicalNLM/NLM(R)

HierarchicalNLM/ANN


Interactive Data Visualization and Explorative AnalysisQuantitative Assessment of Hierarchical Mapping

The introduced distance & topology preservation measures are employedfor quantitative comparison of three regarded data sets:

Data set Mapping E qmIris data NLM 0.01439 0.6366Iris data NLM(R) 0.01791 0.5000Iris data BP 0.66365 0.4794

Mechatronic NLM 0.02875 0.1671Mechatronic NLM(R) 0.05713 0.1130Mechatronic BP 0.75846 0.1059

RIAD NLM 0.05116 0.2421RIAD NLM(R) 0.10400 0.1754RIAD BP 0.14935 0.1340

The hierarchical mapping approach based on NLM & NLM(R)outperforms the NLM & ANN approach in terms of complexity, speed,user convenience, and results


Interactive Data Visualization and Explorative AnalysisMapping Quality vs. Computational Expense in Hierarchical Mapping

The introduced distance & topology preservation measures are employedfor quantitative comparison of three regarded data sets:

Gradual increase of topoloy preservation & decrease of distancepreservation error can be observed for increasing K. Nonmonotonicity isdue to chosen clustering procedures (single runs)

Mapping Quality vs. Codebook Size for Iris Data

00,10,20,30,40,50,60,7

3 5 7 9 15 25 35 45 55 65 100

No. of Codebook Vectors K

Map

ping

Qua

lity

Mea

sure

Distance preservation E

Topology preservation qm



Higher mapping quality by global to local shift of the mapping stress duringthe mapping processThis can be implemented by a limited fixed or shrinking neighborhood, asmet in the SOM algorithmAn experimental mapping method will be introduced to demonstrate theidea, a more practical modification of Sammon´s NLM will follow

TOPology preserving mApping of Sample sets (TOPAS):

TOPAS uses qm (n=N, only first row in assignment scheme) as cost functionCorrections occur if yi is not at correct rank or NN-position with regard to yj :

denotes a time & position dependent learning rate

[ ]ijY

jqiqi

Yjiqiq d

yyymmymy

−Ξ+=+ ))(,()()1( rα

)(,( iY

j ym rΞα



The temporal factor of the learnrate decays with timeThe neighborhood width also decays with time and the mapping error ismore and more governed by close neighborsThis global to local shift of the mapping stress supports data unfolding !NLM and TOPAS mapping of Cube data:

Details next slide



TOPAS shows better structure preservation propertiesA scaling problem during data unfolding can be observedReference point method (Lee et al.) salient and applicable for TOPASComputational complexity is O(N3), only practical for small data (sub)sets !



An enhancement of Sammon´s NLM can incorporate salient TOPAS featuresin a practical implementationThe Magic Factor MF is extended to time and position dependent learn rate:

For the neighborhood function , a Gaussian function withdecaying is chosen

)))(,(,()()()1( mdmymmymyijXniqtiqiq σαα ∆−=+

)(mσ))(,( md

ijXn σα

After achieving a globalarrangement, unfoldingand fine tuning of localdata structures takes placeReference point methodincluded in ENLMOnly O(N2) as NLM



NLM vs. ENLM (three runs) for Cube data:



ENLM is a viable approach: data unfolding without scaling problem withoutincrease in computational complexityThe reference point method allows to focus the mapping to a ROIHowever, parametrization of the method becomes increasingly difficultEven O(N2) complexity can be too much for practical application and largedatabasesMerging of Case 1 (Clustering & hierarchical mapping) with Case 3 (Globalto local shift of mapping stress) can be consideredFurther combination with Case 2 (Hierarchical mapping & navigationapproach) can be salient to optimize mapping quality, speed, & convenience

Numerous research activities can be observedMore results & methods on the way !


12

31

23 123 1

2

31

23

1

2

31

2

3

1

23

1

23

1

2

3 123 1

2

312

312

31

23 1

2

3

1

23 123

123

123

1

2

3123

1

2

31

23

1

2

3

Interactive Data Visualization and Explorative Analysis Tools for Interactive Multivariate Data Visualization: WeightWatcher

Research objective: Achievement of a transparent & intuitive MMI toaccess and analyse multivariate data for information processingEmployment of efficient projection techniques, enhanced user interaction,and CAD-inspired functionalityOptimum exploitation of the remarkable human perceptive and associativecapabilities

Thus, the WeightWatcher (WW)was conceived for visual analysisof neural networksThe concept was generalized toarbitrary multivariate data setsWW is part of QuickCog, aplatform for fast & transparentdesign of vision & cognition systems

Input (Datei),1

i

Input (Datei),2

i

RNN (L),1

i

Speichern (Datei),1

i

V isor,1

ii

Weight Watcher,1

ii


Interactive Data Visualization and Explorative Analysis Tools for Interactive Multivariate Data Visualization: Weight Watcher

WW in QuickCog serves for feature space visualization & assessmentAll described visualization techniques are implemented by WWThe QuickMine-Toolbox comprises projection & clustering methods with WW

An MS-Excel-Macro allowsimport of arbitraryapplication data

QuickMine can beemployed forgeneral datavisualization &analysis tasks


Interactive Data Visualization and Explorative Analysis Tools for Interactive Multivariate Data Visualization: Acoustic Navigator

AN is dedicated topsychoacoustics &sound engineeringIntegrated in theprocessing hierarchy ofPATS systemCombines visualization& auralization in theanalysis processGlobal & local sounddata visualizationSearch functions &directed navigationTentative edit functionsfor complex patternsynthesis [Sammon 69]


Interactive Data Visualization and Explorative Analysis Application Examples: Vision & Cognition System Design

Benefits of Dimensionality Reduction and Visualization for the Rapid Designof Vision & Cognition Systems:

Clustering & structure in the data can be observedDiscriminance of the computed feature set can be assessedAssessment of different feature computation methods is feasibleOpen-loop parameter optimization for feature computation is feasibleInteractive selection of significant features by WW component planesOutliers & identicals can be easily identified and eliminated from the data setHuman errors during data acquisition, labeling & filing can be detectedLarge unlabeled or partially labeled data sets can be manually or semi-automatically according to human to the human observation of data structureand implied similarity in the projectionClassification errors can be spatially located in the feature space projectionUnderstanding of design problems & trouble shooting is alleviated


Interactive Data Visualization and Explorative Analysis Application Examples: BGA X-Ray Inspection in Electronics Manufacturing

Training data

Test data

Misclassifications & outlierscan be tracked-back in thedatabase by mouseclick inthe WW displaySystem design, validation,optimization, and troubleshooting is alleviated


Interactive Data Visualization and Explorative AnalysisApplication Example: Eye-Tracking System

Feature space for FSSPEA

FS for Eye-Shape Classification:FSSPEA finds best solutionGeneralization is affected !SFS solution for HW-design

OR

G

SFS

SBS

PM

FSSP

EA (5

0)

FSSP

EA (1

0) Train

Test

0,7800,8000,8200,8400,8600,8800,9000,9200,9400,9600,9801,000

Eye-shapes

Method Train Test FeaturesORG 0,992500 0,981667 24SFS 0,997500 0,928333 3SBS 0,989333 0,981667 12PM 0,997500 0,959167 6SGA 0,997333 0,969167 12FSSPEA 0,997500 0,928333 3


Interactive Data Visualization and Explorative AnalysisApplication Example: Eye-Tracking System Hardware

RNN Low-Power Analog Eye-Shape Classifier:

12-D Gabor jet reduced to 6 features by AFSRNN constrained to Manhattan distance, TRNN=66 bit RAMDACs for weightsAMS 0.6µm CUQ CMOS-technologyCell size 1595 um x 184 um (conservative)Power dissipation 348,9 nW, 3,5x10-10 JClassification speed: 1ms/pattern, R=100%


Interactive Data Visualization and Explorative Analysis Application Examples: Medical Data Analysis

Application of interactive multivariate data visualisation to the analysis ofpatients findings in metabolic research (diabetes mellitus type 2)[TU Dresden, institute and outpatient clinic for clinical metabolic research]

Analysis of patientfindings (diabetes)RIAD (Risk factors inIGT for Athero-sclerosis & Diabetes)study: 453 subjects, 24features eachVisualization ofcomplete databaseshowed complexpictureLow qm value confirmshigh intrinsic dim.


Interactive Data Visualization and Explorative Analysis Application Examples: Medical Data Analysis

The database obviously was not directly amenable to analysis by projection andinteractive visualization due to high intrinsic dimensionality

Selection techniques werecombined with projectionFeature selection accordingto sex or diabetes stateThe database could bereduced to 7 featuresTwo clusters correlatingstrongly with patients sexcould be identifiedSeveral known correlationsbetween lab parameters &actual diabetes state wereconfirmed and new localones detected


Interactive Data Visualization and Explorative Analysis Application Examples: Wafer Data Analysis

Simple example of DR methodology potential for fab data analysisThe database was gathered from an MPC-Run with 10 Wafers and 4Parameter Extraction Sites per Wafer, 59 parameters each:

All chips with unsatisfactory behavior were on Wafer 5 (blue; 1-4, 6-10 yellow)Assume Wafer 5 processing was abnormal, what makes it different ?

No obvious structure in theprojection of the multivariate data



The lack of obvious structure can be due to absence of abnormity or toocclusion by a majority of normal parameters and high intrinsic dimensionThus, AFS is employed to find parameters supporting abnormity hypothesis

From 59 parameters AFS chose 3 for qsi/SBS and 7 for qoi/SBSThere is (weak) hypothesis support: Can these features be responsible ?

Selected features:1, 23, 31, 36, 37, 38, 57



The subcluster was found forthe assumption of Wafer 5abnormityThe parameters must bechecked for physicalplausibility with regard toobserved chip behaviorThe visualization & analysisresult is imposed on the originaldatabase in ExcelIn the given case, the detectedabnormity in the selectedparameters cannot be heldresponsibleProbably a design error, not amanufacturing problem !



This simple example only scratched the surface of the application potentialFeasibility demonstration of DR methodology for microelectronic data

In this case found parametersdistinguish Wafer 5 but are notrelevant to observed problemBut the answer was found very fast !The outlined approach offers fast,efficient, and transparent access tomultivariate, complex data met intoday´s manufacturing processesOther nonobvious information canbe found & employed for processoptimization & centeringCircuit design & design centering isanother potential application field


Summary

This tutorial gave an introduction to & a focused survey of theprinciples & techniques for dimensionality reduction andinteractive visualization of multivariate data

Relevant techniques were qualitatively & quantitativelyassessed and compared in a unifying approach

Basic problems of DR & some advanced concepts to copewith those were presented

Two dedicated tools, the WeightWatcher & the AcousticNavigator were presentedApplication examples were given, e.g., for database analysis


Future Trend

The topic of DR, visualization & explorative analysis enjoysincreasing interest in the context of data mining

More application fields & customized tools can be envisioned,e.g., file management or web-browsing

Improvements by hierarchical projection & mappingtechniques in terms of speed, efficiency, mapping reliability,& user convenience mandatory and on the way !

The survey must be extended due to numerous recentlypublished approaches, e.g., TNN SI on Data Mining May 2000

dimensionality reduction and interactive visualization of ... · systematic feature space reduction...

Documents