examples of high-dimensional data - rice universitygallen/examples_high_dim_data.pdf · examples of...
TRANSCRIPT
Examples of High-Dimensional Data
Genevera I. Allen
Statistical Learning: High-Dimensional Data
January 10, 2011
(Stat 699) High-Dimensional Data January 10, 2011 1 / 16
1 Large p, Small n: Biological Data
2 Random Fields Data
3 Collaborative Filtering Data
(Stat 699) High-Dimensional Data January 10, 2011 2 / 16
Example: Microarrays
Measure gene expression.
Often tens of thousands of genes(features).
Only tens of hundreds of samples.
arrays
genes
(Stat 699) High-Dimensional Data January 10, 2011 3 / 16
Review: Microarrays
(Stears, R. L. et. al, 2003)
(Stat 699) High-Dimensional Data January 10, 2011 4 / 16
Statistical Questions: Microarrays
Data pre-processing:I Normalization.I Missing data imputation.
Inference:I Which genes are significant?
Clustering:I Groups of genes, groups of
samples.
Model Building:I Small n, large p.
arrays
genes
(Stat 699) High-Dimensional Data January 10, 2011 5 / 16
Other Types of Biological Data
Genetics:
Deep Sequencing - Counts.
Micro RNA Expression - Continuous.
CGH (Copy Number Variation) - Continuous / Categorical.
SNPs (Single Nucleotide Polymorphisms) - Binary / Categorical.
Methalaytion - Continuous.
(Stat 699) High-Dimensional Data January 10, 2011 6 / 16
Other Types of Biological Data
Proteomics / Metabolomics (Chemometrics):
(H-NMR) Measures the chemical shift associated with variousmetabolites.
(Stat 699) High-Dimensional Data January 10, 2011 6 / 16
1 Large p, Small n: Biological Data
2 Random Fields Data
3 Collaborative Filtering Data
(Stat 699) High-Dimensional Data January 10, 2011 7 / 16
Example: Functional MRIs (fMRI)
Rows: Voxels.
Columns: Subjects (And/or replicates and times).
Measurement: Hemodynamic response (change in blood flow).
Slice 15 Slice 16 Slice 17
(Stat 699) High-Dimensional Data January 10, 2011 8 / 16
Review: fMRIs
(Heeger & Ress, 2002)
(Stat 699) High-Dimensional Data January 10, 2011 9 / 16
Statistical Questions: fMRIs
Inference:I Which voxels are significant?I Which groups of voxels ( regions
of interest) are significant?
Clustering:I Groups of voxels that behave
similarly - finding regions ofinterest.
Networks (FunctionalConnectivity):
I How are voxels or groups ofvoxels related to each other?
I How are voxels or groups ofvoxels related through time?
(Stat 699) High-Dimensional Data January 10, 2011 10 / 16
Others
Finance.I Time Series Data.
Climate Data.I Spatial Data.I Spatio-temporal Data.
Neuroimaging.I DTI - Diffusion Tensor Imaging.I Calcium-Florescence Imaging.I EEG & MEG.
(Stat 699) High-Dimensional Data January 10, 2011 11 / 16
1 Large p, Small n: Biological Data
2 Random Fields Data
3 Collaborative Filtering Data
(Stat 699) High-Dimensional Data January 10, 2011 12 / 16
Example: Netflix Movie Rating Data
Rows: Movies.
Columns: Customers.
Measurement: Movieratings (scale of 1 - 5).
Anne Ben Charlie Doug Eve . . .
Star Wars 2 5 4 4 3 . . .Harry Potter 3 4 5 3 ? . . .
Pretty Woman 4 ? 2 ? 5 . . .Titanic 5 ? 2 1 3 . . .
Lord of the Rings ? 5 5 4 4 . . ....
......
......
.... . .
(Stat 699) High-Dimensional Data January 10, 2011 13 / 16
Netflix Prize
Challenge: Predict un-rated movies with 10% improvement overCinematch.
Training Set: 480,000 customer ratings on 18,000 movies.
Around 98.7% missing ratings!
$1,000,000 prize!
Contest: October 2006 - August 2009.
Winners: Team led by Robert Bell and Yehuda Koren.
Methods: Variations on the SVD and k-nearest neighbors (Bell &Koren, 2008).
Fields: Recommender systems & Collaborative filtering.
(Stat 699) High-Dimensional Data January 10, 2011 14 / 16
Visualizing Netflix Data
Zipf
(Justin S. Dyer & Art B. Owen, 2010)
(Stat 699) High-Dimensional Data January 10, 2011 15 / 16
Visualizing Netflix Data
Copulas
(Justin S. Dyer & Art B. Owen, 2010)
(Stat 699) High-Dimensional Data January 10, 2011 15 / 16
Other Examples
Amazon
Yahoo!
(Stat 699) High-Dimensional Data January 10, 2011 16 / 16