compressed sensing techniques for sensor data using unsupervised learning

Compressed Sensing Techniques for Sensor Data Compressed Sensing Techniques for Sensor Data Compressed Sensing Techniques for Sensor Data Compressed Sensing Techniques for Sensor Data using Unsupervised Learning using Unsupervised Learning using Unsupervised Learning using Unsupervised Learning

SONG CUISONG CUISONG CUISONG CUI

COPYRIGHT©2013 SONG CUI. ALL RIGHTS RESERVED

Outline

• Concept of compressed sensing

• Basic theories with interpretations

• Case studies:

Medical imaging (Considered as a sensor network problem)

Wearable electroencephalography (EEG)

Recommendation system


Concept of compressed sensing


Sensor (limited computational capability, transmission power et al.)

Learning algorithms

Data compression

CPU or GPU (excellent computational capability)

Data analytics

• Feature extraction

• Prediction

• Decision making

Data acquisition

Compression techniques

• Traditional lossy compression techniques: JPEG, wavelet et al.

• Traditional compression techniques are not efficient for sparse data.


Sparse data Non-sparse data

Why JPEG is ineffective for sparse data

• Discrete cosine transform converts the image (2-D matrix) from spatial domain to frequency domain in JPEG.

• Insignificant coefficients in frequency domain are discarded in JPEG.• Sparse data have comparable coefficients in all frequencies.


Keep the first 5 coefficients in

frequency domain and

recover the signal

Data compression


Data compression

• Data compression is through underdetermined linear system.• It is a dimension reduction process in machine learning.

• Compression matrix C is pre-determined where the compression ratio is: m/n.


y Cx= , Where n<m

Comparison with PCA

• The compression matrix C in compressed sensing is pre-determined with some restrictions.

• PCA is more computational demanding:

Normalizing means and variances in training samples .

Finding the eigenvalues and eigenvectors for .

Picking up the first n principle eigenvectors to form C.

• Data compression is efficient and requires limited or no computational power which favors applications (e. g. mobile sensors) that have limited power, data storage, transmission, and computational capabilities.


1 2 3 4, , , ..., mx x x x x

T

1

( ( )) /m

i i

i

x x m=

∑

Data reconstruction


Data reconstruction

Can we fully recover the original signal x from its compressed version y?


y Cx= , Where n<m

1 2y x x= +The answer is no in general:

Data reconstruction

• However, the signal can be fully recovered if:

The sparsity of the original signal x is s (the maximum number of non-zero entry is s).

C must satisfy restricted isometry property (RIP) which means any s columns in matrix C are independent.


Sparsity is the key!

Data reconstruction method

y Cx=

• Ill imposed inverse Problem: find the x from y:

• L1 norm minimization is used in practice:


0min || || . t .y Cxx s =ɶ ɶ

If RIP holds and x is sparse

2 1|| || || ||y Cx xλ− +ɶ ɶMinimizing

E. Candès and T. Tao, IEEE Trans. Inform. Theory, vol. 51, no. 12, pp. 4203-4215 (2005).

1 2min || || . t . || y Cx ||x s δ− ≤ɶ ɶ

Why L1 regularization?


L1 norm minimization

L2 norm minimization

Original sparse signal

Numerical examples are from Prof. W.

K. Ma’s lecture notes

(http://www.ee.cuhk.edu.hk/~wkma).

Analogy to help

understand L1 and L2

regularizations:

Personal income tax

rate

2|| || || || (p 1or 2)py Cx xλ− + =ɶ ɶ

Parallel computing for large-scale datasets


1 2 1 1

2 2 1 2

3 2 1 3

|| || || ||

|| || || ||

|| || || ||

......

y Cx x x

y Cx x x

y Cx x x

λ

λ

λ

− + ⇒

− + ⇒

− + ⇒

ɶ ɶ ɶ

ɶ ɶ ɶ

ɶ ɶ ɶ

y Cx= 1 1 2 2 3 3( , ), ( , ), ( , ),...x y x y x yData compression:

Data recovery:

1 2 1 1|| || || ||y Cx x xλ− + ⇒ɶ ɶ ɶ

1 CPU Multi-thread computing in GPU or CPU clusters

2 2 1 2|| || || ||y Cx x xλ− + ⇒ɶ ɶ ɶ

3 2 1 3|| || || ||y Cx x xλ− + ⇒ɶ ɶ ɶ

…. ….

Distributed methods are available for single L1 regularization problem.

Case study 1: Medical imaging

• Positron emission tomography (PET) is capable of measuring positron-emitting radionuclides.

• It is a medical diagnostic instrument for oncology, neuroimaging, and cardiology applications.


Background


Compressed sensing multiplexing circuits

• Data is sparse in spatial domain.

• Data compression is implemented on PCB circuit boards with pre-amplifiers, resistors and capacitors.


P. D. Olcott et al., IEEE NSS-MIC Conference Record

p. 3224 (2011).

General applications to wireless sensor network


The figure is from J. Haupt et al., IEEE Signal Processing Mag.

pp. 92 Mar. 2008.

Case study 2: Wearable electroencephalography (EEG)


Figures are from: AJ. Casson et al., IEEE Eng. Med. Biol. Mag 29:44–56 (2010)

Trend of wearable EEG

• Long-term monitoring capability is needed and large-scale data can be generated.

• Data are processed with machine learning algorithms in the remote end with strong computational capabilities and large database.

• Wearable data transmission enables device miniaturization and body area network applications.

• Applications include sleep disorders and augmented cognition.


Challenges in wearable EEG

• Electrode Technology

• Battery power consumption

Data acquisition

Data transmission


One possible solution: Compressed sensing!

Compressed sensing for scalp EEG

The data is not sparse in time domain.

The data has sparse representations in terms of other basis functions.


A. M. Abdulghani et al., Med. Biol. Eng. Comput. 50:1137–1145 (2012).

Case study 3: Recommendation system

Item 1 Item 2 Item 3 Item 4

User 1 5 ? 3

User 2 2

User 3 4 3 2

User 4 5 4 ?

• Matrix factorization methods

• Baseline methods

…………


User – item rating matrix: MThink about how you build up

a recommendation system!

T

ui i ur u q p= +

ui i ur u b b= + +

Interpretation of matrix completion

• We assume that there are correlations among some users rating on the same items (dependency in rows).

• We assume there are correlations among ratings on some items from the same user.



User 1 5 ? 3

User 2 2

User 3 4 3 2

User 4 5 4 ?

User – item rating matrix: M

We hypothesis that rank (M) is small!

Problem formulation


iu iu iurank(X),s. t .X M (M )= ∈Ω


User 1 5 ? 3

User 2 2

User 3 4 3 2

User 4 5 4 ?

User – item rating matrix: MFind a matrix X which fills

up the unknown user-

item ratings and satisfy:

Minimize

E. Candès, et al., Foundations of Computational Mathematics,

vol. 9, pp. 717 (2009).

NP hard

problem

0rank(X) || || ,X U *V= Σ = Σwhere

* iu iu iu|| X || ,s. t .X M (M )= ∈ΩMinimize

*

1

|| X || ,r

i i

i

σ σ=

=∑where is the singular value

Connection with compressed sensing

• A more robust and computational efficient approach:

• The method has been tested on Netflix dataset.


2

iu iu

1

(X M )r

i

i u i

λ σ=

− +∑∑ ∑ 2 1|| || || ||y Cx xλ− +

Minimize

Minimize

Previous results:

N. Srebro and R. Salakhutdinov., Advances in Neural Information Processing Systems, vol. 23, pp. 2056

(2010).

*|| X M || || ||F Xλ− +

Interpretation: M is a compressed version

of X!

Challenges

• Does the hypothesis min rank (X) always match with the truth?

• How to deal with non-uniform sampling (e. g. Some user have much more ratings than others)?

Weighted regularization

• How to deal with cold start problems in the model?

• How to incorporate additional information such as sex, time drifting, and geography location in the model?


Personal opinions!

Summary

• Compressed sensing is a new data compression and recovery method.

• It is effective for sparse data.Sparse in time domain.

Sparse in frequency domain.

Sparse in other representations.

•It is useful for mobile sensors which has large-scale data transmission, limited battery power, computational capabilities and requires device miniaturization.

• It has seen applications in machine learning such as recommendation systems.


Extra materials and links

• Stanford course website: http://statweb.stanford.edu/~candes/stats330/index.shtml

• A very resourceful website: Nult Blanche’s blog: http://nuit-blanche.blogspot.com/p/teaching-compressed-sensing.html covers updates from theories and applications such as MRI and machine learning.

• You can also add me on Linkedin: http://www.linkedin.com/profile/view?id=71703589 or contact [email protected] if you want to have further discussions.
