adaptive sinusoidal modelingspcc.csd.uoc.gr/_docs/lectures2016/spcc16_stylianoui.pdf · adaptive...

Post on 05-Jan-2020

16 Views

Category:

Documents

0 Downloads

Preview:

Click to see full reader

TRANSCRIPT

Adaptive Sinusoidal Modeling

Yannis Stylianou

University of Crete, Computer Science Dept., Greece,

yannis@csd.uoc.gr

SPCC 2016

Outline Sinusoidal Modeling Advanced Modeling QHM aQHM eaQHM Applications Key papers References

1 Sinusoidal Modeling

2 Advanced Modeling

3 QHMValidation

4 aQHMDescriptionValidationAM-FM decomposition

5 eaQHM

6 ApplicationsSpeech technologies

7 Key papers

8 References

yannis@csd.uoc.gr Adaptive Sinusoidal Modeling 2/48

Outline Sinusoidal Modeling Advanced Modeling QHM aQHM eaQHM Applications Key papers References

Sinusoidal Models

Sinusoidal Model:

x(t) =K∑

k=−Kγke

j2πfk t

Special case, Harmonic Model:

x(t) =K∑

k=−Kγke

j2πkf0t

Estimation of parameters (Linear approach):

γ = Ffk s

Model Evaluation, Mean-Squared Error (MSE):

ε =

∫w|s(t)− x(t)|2 dt

yannis@csd.uoc.gr Adaptive Sinusoidal Modeling 3/48

Outline Sinusoidal Modeling Advanced Modeling QHM aQHM eaQHM Applications Key papers References

Sinusoidal Models

Sinusoidal Model:

x(t) =K∑

k=−Kγke

j2πfk t

Special case, Harmonic Model:

x(t) =K∑

k=−Kγke

j2πkf0t

Estimation of parameters (Linear approach):

γ = Ffk s

Model Evaluation, Mean-Squared Error (MSE):

ε =

∫w|s(t)− x(t)|2 dt

yannis@csd.uoc.gr Adaptive Sinusoidal Modeling 3/48

Outline Sinusoidal Modeling Advanced Modeling QHM aQHM eaQHM Applications Key papers References

Sinusoidal Models

Sinusoidal Model:

x(t) =K∑

k=−Kγke

j2πfk t

Special case, Harmonic Model:

x(t) =K∑

k=−Kγke

j2πkf0t

Estimation of parameters (Linear approach):

γ = Ffk s

Model Evaluation, Mean-Squared Error (MSE):

ε =

∫w|s(t)− x(t)|2 dt

yannis@csd.uoc.gr Adaptive Sinusoidal Modeling 3/48

Outline Sinusoidal Modeling Advanced Modeling QHM aQHM eaQHM Applications Key papers References

Sinusoidal Models

Sinusoidal Model:

x(t) =K∑

k=−Kγke

j2πfk t

Special case, Harmonic Model:

x(t) =K∑

k=−Kγke

j2πkf0t

Estimation of parameters (Linear approach):

γ = Ffk s

Model Evaluation, Mean-Squared Error (MSE):

ε =

∫w|s(t)− x(t)|2 dt

yannis@csd.uoc.gr Adaptive Sinusoidal Modeling 3/48

Outline Sinusoidal Modeling Advanced Modeling QHM aQHM eaQHM Applications Key papers References

iQHM, aQHM, eaQHM, aHM

Towards Adaptive Sinusoidal Models

yannis@csd.uoc.gr Adaptive Sinusoidal Modeling 4/48

Outline Sinusoidal Modeling Advanced Modeling QHM aQHM eaQHM Applications Key papers References

Estimating Sinusoidal parameters

Sinusoidal representation for a speech/signal frame:

x(t) =

(K∑

k=−Kake

j2πfk t

)w(t)

Methods:

FFT-based methods (i.e., QIFFT [Abe et al., 2004-05, [1] [2]])Subspace methodsLeast Squares (LS) method

Frequency mismatch:

fk = fk + ηk

yannis@csd.uoc.gr Adaptive Sinusoidal Modeling 5/48

Outline Sinusoidal Modeling Advanced Modeling QHM aQHM eaQHM Applications Key papers References

Estimating Sinusoidal parameters

Sinusoidal representation for a speech/signal frame:

x(t) =

(K∑

k=−Kake

j2πfk t

)w(t)

Methods:

FFT-based methods (i.e., QIFFT [Abe et al., 2004-05, [1] [2]])Subspace methodsLeast Squares (LS) method

Frequency mismatch:

fk = fk + ηk

yannis@csd.uoc.gr Adaptive Sinusoidal Modeling 5/48

Outline Sinusoidal Modeling Advanced Modeling QHM aQHM eaQHM Applications Key papers References

Estimating Sinusoidal parameters

Sinusoidal representation for a speech/signal frame:

x(t) =

(K∑

k=−Kake

j2πfk t

)w(t)

Methods:

FFT-based methods (i.e., QIFFT [Abe et al., 2004-05, [1] [2]])Subspace methodsLeast Squares (LS) method

Frequency mismatch:

fk = fk + ηk

yannis@csd.uoc.gr Adaptive Sinusoidal Modeling 5/48

Outline Sinusoidal Modeling Advanced Modeling QHM aQHM eaQHM Applications Key papers References

Estimating Sinusoidal parameters

Sinusoidal representation for a speech/signal frame:

x(t) =

(K∑

k=−Kake

j2πfk t

)w(t)

Methods:

FFT-based methods (i.e., QIFFT [Abe et al., 2004-05, [1] [2]])Subspace methodsLeast Squares (LS) method

Frequency mismatch:

fk = fk + ηk

yannis@csd.uoc.gr Adaptive Sinusoidal Modeling 5/48

Outline Sinusoidal Modeling Advanced Modeling QHM aQHM eaQHM Applications Key papers References

Estimating Sinusoidal parameters

Sinusoidal representation for a speech/signal frame:

x(t) =

(K∑

k=−Kake

j2πfk t

)w(t)

Methods:

FFT-based methods (i.e., QIFFT [Abe et al., 2004-05, [1] [2]])Subspace methodsLeast Squares (LS) method

Frequency mismatch:

fk = fk + ηk

yannis@csd.uoc.gr Adaptive Sinusoidal Modeling 5/48

Outline Sinusoidal Modeling Advanced Modeling QHM aQHM eaQHM Applications Key papers References

Estimating Sinusoidal parameters

Sinusoidal representation for a speech/signal frame:

x(t) =

(K∑

k=−Kake

j2πfk t

)w(t)

Methods:

FFT-based methods (i.e., QIFFT [Abe et al., 2004-05, [1] [2]])Subspace methodsLeast Squares (LS) method

Frequency mismatch:

fk = fk + ηk

yannis@csd.uoc.gr Adaptive Sinusoidal Modeling 5/48

Outline Sinusoidal Modeling Advanced Modeling QHM aQHM eaQHM Applications Key papers References

Quasi-Harmonic Model, QHM [5]

Frequency mismatch:

x(t) =

(K∑

k=−Kake

j2πfk t

)w(t)

QHM (de Prony 1795, Laroche [3] (1989), Stylianou 1993,Pantazis [4] (2008, 2011) ):

x(t) =

(K∑

k=−K(ak + tbk)e j2πfk t

)w(t)

yannis@csd.uoc.gr Adaptive Sinusoidal Modeling 6/48

Outline Sinusoidal Modeling Advanced Modeling QHM aQHM eaQHM Applications Key papers References

Quasi-Harmonic Model, QHM [5]

Frequency mismatch:

x(t) =

(K∑

k=−Kake

j2πfk t

)w(t)

QHM (de Prony 1795, Laroche [3] (1989), Stylianou 1993,Pantazis [4] (2008, 2011) ):

x(t) =

(K∑

k=−K(ak + tbk)e j2πfk t

)w(t)

yannis@csd.uoc.gr Adaptive Sinusoidal Modeling 6/48

Outline Sinusoidal Modeling Advanced Modeling QHM aQHM eaQHM Applications Key papers References

A few details of QHM

QHM in the frequency domain:

Xk(f ) = akW (f − fk) + jbk2π

W ′(f − fk)

Decomposition of bk : bk = ρ1,kak + ρ2,k jakThen

Xk(f ) = ak

[W (f − fk)−

ρ2,k

2πW ′(f − fk) + j

ρ1,k

2πW ′(f − fk)

]and taking into account:

W (f − fk −ρ2,k

2π) = W (f − fk)−

ρ2,k

2πW ′(f − fk)+

O(ρ22,kW

′′(f − fk))

Approximation of the kth component of QHM

Xk(f ) ≈ ak

[W (f − fk −

ρ2,k

2π) + j

ρ1,k

2πW ′(f − fk)

]yannis@csd.uoc.gr Adaptive Sinusoidal Modeling 7/48

Outline Sinusoidal Modeling Advanced Modeling QHM aQHM eaQHM Applications Key papers References

A few details of QHM

QHM in the frequency domain:

Xk(f ) = akW (f − fk) + jbk2π

W ′(f − fk)

Decomposition of bk : bk = ρ1,kak + ρ2,k jakThen

Xk(f ) = ak

[W (f − fk)−

ρ2,k

2πW ′(f − fk) + j

ρ1,k

2πW ′(f − fk)

]and taking into account:

W (f − fk −ρ2,k

2π) = W (f − fk)−

ρ2,k

2πW ′(f − fk)+

O(ρ22,kW

′′(f − fk))

Approximation of the kth component of QHM

Xk(f ) ≈ ak

[W (f − fk −

ρ2,k

2π) + j

ρ1,k

2πW ′(f − fk)

]yannis@csd.uoc.gr Adaptive Sinusoidal Modeling 7/48

Outline Sinusoidal Modeling Advanced Modeling QHM aQHM eaQHM Applications Key papers References

A few details of QHM

QHM in the frequency domain:

Xk(f ) = akW (f − fk) + jbk2π

W ′(f − fk)

Decomposition of bk : bk = ρ1,kak + ρ2,k jakThen

Xk(f ) = ak

[W (f − fk)−

ρ2,k

2πW ′(f − fk) + j

ρ1,k

2πW ′(f − fk)

]and taking into account:

W (f − fk −ρ2,k

2π) = W (f − fk)−

ρ2,k

2πW ′(f − fk)+

O(ρ22,kW

′′(f − fk))

Approximation of the kth component of QHM

Xk(f ) ≈ ak

[W (f − fk −

ρ2,k

2π) + j

ρ1,k

2πW ′(f − fk)

]yannis@csd.uoc.gr Adaptive Sinusoidal Modeling 7/48

Outline Sinusoidal Modeling Advanced Modeling QHM aQHM eaQHM Applications Key papers References

A few details of QHM

QHM in the frequency domain:

Xk(f ) = akW (f − fk) + jbk2π

W ′(f − fk)

Decomposition of bk : bk = ρ1,kak + ρ2,k jakThen

Xk(f ) = ak

[W (f − fk)−

ρ2,k

2πW ′(f − fk) + j

ρ1,k

2πW ′(f − fk)

]and taking into account:

W (f − fk −ρ2,k

2π) = W (f − fk)−

ρ2,k

2πW ′(f − fk)+

O(ρ22,kW

′′(f − fk))

Approximation of the kth component of QHM

Xk(f ) ≈ ak

[W (f − fk −

ρ2,k

2π) + j

ρ1,k

2πW ′(f − fk)

]yannis@csd.uoc.gr Adaptive Sinusoidal Modeling 7/48

Outline Sinusoidal Modeling Advanced Modeling QHM aQHM eaQHM Applications Key papers References

A few details of QHM

QHM in the frequency domain:

Xk(f ) = akW (f − fk) + jbk2π

W ′(f − fk)

Decomposition of bk : bk = ρ1,kak + ρ2,k jakThen

Xk(f ) = ak

[W (f − fk)−

ρ2,k

2πW ′(f − fk) + j

ρ1,k

2πW ′(f − fk)

]and taking into account:

W (f − fk −ρ2,k

2π) = W (f − fk)−

ρ2,k

2πW ′(f − fk)+

O(ρ22,kW

′′(f − fk))

Approximation of the kth component of QHM

Xk(f ) ≈ ak

[W (f − fk −

ρ2,k

2π) + j

ρ1,k

2πW ′(f − fk)

]yannis@csd.uoc.gr Adaptive Sinusoidal Modeling 7/48

Outline Sinusoidal Modeling Advanced Modeling QHM aQHM eaQHM Applications Key papers References

Approximations in QHM

we found previously:

Xk(f ) ≈ ak

[W (f − fk −

ρ2,k

2π) + j

ρ1,k

2πW ′(f − fk)

]Back to the time-domain:

xk(t) ≈ ak

[e j(2πfk+ρ2,k )t + ρ1,kte

j2πfk t]w(t)

Initially, we assumed:

xk(t) = ak

[e j(2π(fk+ηk ))t

]w(t)

in other words, it is suggested:

ηk = ρ2,k/2π

yannis@csd.uoc.gr Adaptive Sinusoidal Modeling 8/48

Outline Sinusoidal Modeling Advanced Modeling QHM aQHM eaQHM Applications Key papers References

Approximations in QHM

we found previously:

Xk(f ) ≈ ak

[W (f − fk −

ρ2,k

2π) + j

ρ1,k

2πW ′(f − fk)

]Back to the time-domain:

xk(t) ≈ ak

[e j(2πfk+ρ2,k )t + ρ1,kte

j2πfk t]w(t)

Initially, we assumed:

xk(t) = ak

[e j(2π(fk+ηk ))t

]w(t)

in other words, it is suggested:

ηk = ρ2,k/2π

yannis@csd.uoc.gr Adaptive Sinusoidal Modeling 8/48

Outline Sinusoidal Modeling Advanced Modeling QHM aQHM eaQHM Applications Key papers References

Approximations in QHM

we found previously:

Xk(f ) ≈ ak

[W (f − fk −

ρ2,k

2π) + j

ρ1,k

2πW ′(f − fk)

]Back to the time-domain:

xk(t) ≈ ak

[e j(2πfk+ρ2,k )t + ρ1,kte

j2πfk t]w(t)

Initially, we assumed:

xk(t) = ak

[e j(2π(fk+ηk ))t

]w(t)

in other words, it is suggested:

ηk = ρ2,k/2π

yannis@csd.uoc.gr Adaptive Sinusoidal Modeling 8/48

Outline Sinusoidal Modeling Advanced Modeling QHM aQHM eaQHM Applications Key papers References

Approximations in QHM

we found previously:

Xk(f ) ≈ ak

[W (f − fk −

ρ2,k

2π) + j

ρ1,k

2πW ′(f − fk)

]Back to the time-domain:

xk(t) ≈ ak

[e j(2πfk+ρ2,k )t + ρ1,kte

j2πfk t]w(t)

Initially, we assumed:

xk(t) = ak

[e j(2π(fk+ηk ))t

]w(t)

in other words, it is suggested:

ηk = ρ2,k/2π

yannis@csd.uoc.gr Adaptive Sinusoidal Modeling 8/48

Outline Sinusoidal Modeling Advanced Modeling QHM aQHM eaQHM Applications Key papers References

Constraints

W (f − fk −ρ2,k

2π) = W (f − fk)−

ρ2,k

2πW ′(f − fk)+

O(ρ22,kW

′′(f − fk))

yannis@csd.uoc.gr Adaptive Sinusoidal Modeling 9/48

Outline Sinusoidal Modeling Advanced Modeling QHM aQHM eaQHM Applications Key papers References

Influence of the Window

We would like small value for W ′′(f ) at fk

W ′′(f ) is influenced by the length, T , of the analysis window,i.e., for rectangular window: W ′′(f ) ∝ T 3

So ... we would like a short analysis window

But ... long analysis window provides robustness

Length of the window � bandwidth

yannis@csd.uoc.gr Adaptive Sinusoidal Modeling 10/48

Outline Sinusoidal Modeling Advanced Modeling QHM aQHM eaQHM Applications Key papers References

Influence of the Window

We would like small value for W ′′(f ) at fk

W ′′(f ) is influenced by the length, T , of the analysis window,i.e., for rectangular window: W ′′(f ) ∝ T 3

So ... we would like a short analysis window

But ... long analysis window provides robustness

Length of the window � bandwidth

yannis@csd.uoc.gr Adaptive Sinusoidal Modeling 10/48

Outline Sinusoidal Modeling Advanced Modeling QHM aQHM eaQHM Applications Key papers References

Influence of the Window

We would like small value for W ′′(f ) at fk

W ′′(f ) is influenced by the length, T , of the analysis window,i.e., for rectangular window: W ′′(f ) ∝ T 3

So ... we would like a short analysis window

But ... long analysis window provides robustness

Length of the window � bandwidth

yannis@csd.uoc.gr Adaptive Sinusoidal Modeling 10/48

Outline Sinusoidal Modeling Advanced Modeling QHM aQHM eaQHM Applications Key papers References

Influence of the Window

We would like small value for W ′′(f ) at fk

W ′′(f ) is influenced by the length, T , of the analysis window,i.e., for rectangular window: W ′′(f ) ∝ T 3

So ... we would like a short analysis window

But ... long analysis window provides robustness

Length of the window � bandwidth

yannis@csd.uoc.gr Adaptive Sinusoidal Modeling 10/48

Outline Sinusoidal Modeling Advanced Modeling QHM aQHM eaQHM Applications Key papers References

Influence of the Window

We would like small value for W ′′(f ) at fk

W ′′(f ) is influenced by the length, T , of the analysis window,i.e., for rectangular window: W ′′(f ) ∝ T 3

So ... we would like a short analysis window

But ... long analysis window provides robustness

Length of the window � bandwidth

yannis@csd.uoc.gr Adaptive Sinusoidal Modeling 10/48

Outline Sinusoidal Modeling Advanced Modeling QHM aQHM eaQHM Applications Key papers References

Influence of frequency mismatch

Assumey(t) = αe j(2πf t+ηt)

modeled by QHM as

x(t) = (a + tb)e j(2πf t) − T ≤ t ≤ T

LS solution provides:

a = αsin(ηT )

ηT

b = α 3j(sin(ηT )

η2T 3− cos(ηT )

ηT 2)

Frequency mismatch estimate:

η = 3

(1

ηT 2− cot(ηT )

T

)yannis@csd.uoc.gr Adaptive Sinusoidal Modeling 11/48

Outline Sinusoidal Modeling Advanced Modeling QHM aQHM eaQHM Applications Key papers References

Influence of frequency mismatch

Assumey(t) = αe j(2πf t+ηt)

modeled by QHM as

x(t) = (a + tb)e j(2πf t) − T ≤ t ≤ T

LS solution provides:

a = αsin(ηT )

ηT

b = α 3j(sin(ηT )

η2T 3− cos(ηT )

ηT 2)

Frequency mismatch estimate:

η = 3

(1

ηT 2− cot(ηT )

T

)yannis@csd.uoc.gr Adaptive Sinusoidal Modeling 11/48

Outline Sinusoidal Modeling Advanced Modeling QHM aQHM eaQHM Applications Key papers References

Influence of frequency mismatch

Assumey(t) = αe j(2πf t+ηt)

modeled by QHM as

x(t) = (a + tb)e j(2πf t) − T ≤ t ≤ T

LS solution provides:

a = αsin(ηT )

ηT

b = α 3j(sin(ηT )

η2T 3− cos(ηT )

ηT 2)

Frequency mismatch estimate:

η = 3

(1

ηT 2− cot(ηT )

T

)yannis@csd.uoc.gr Adaptive Sinusoidal Modeling 11/48

Outline Sinusoidal Modeling Advanced Modeling QHM aQHM eaQHM Applications Key papers References

Influence of frequency mismatch

Assumey(t) = αe j(2πf t+ηt)

modeled by QHM as

x(t) = (a + tb)e j(2πf t) − T ≤ t ≤ T

LS solution provides:

a = αsin(ηT )

ηT

b = α 3j(sin(ηT )

η2T 3− cos(ηT )

ηT 2)

Frequency mismatch estimate:

η = 3

(1

ηT 2− cot(ηT )

T

)yannis@csd.uoc.gr Adaptive Sinusoidal Modeling 11/48

Outline Sinusoidal Modeling Advanced Modeling QHM aQHM eaQHM Applications Key papers References

Combining Influences

−400 −300 −200 −100 0 100 200 300 400−400

−200

0

200

400

η1 (Hz)

(b)

er(η

1) (

Hz)

−400 −300 −200 −100 0 100 200 300 400−400

−200

0

200

400

η1 (Hz)

(a)

er(η

1) (

Hz)

RectWinHamming

no iter2 iter

• Iteratively, the bias can be removed when |η| < B/3, where B isthe bandwidth of the squared analysis window.yannis@csd.uoc.gr Adaptive Sinusoidal Modeling 12/48

Outline Sinusoidal Modeling Advanced Modeling QHM aQHM eaQHM Applications Key papers References

Robustness against additive noise

Signal contaminated by noise:

y(t) =4∑

k=1

akej2πfk + v(t)

Mean Squared Error (MSE):

MSE{fk} =1

M

M∑i=1

|fk(i)− fk |2

MSE{ak} =1

M

M∑i=1

|ak(i)− ak |2

Comparison with Cramer-Rao Bounds (CRB) and QIFFT(Abe et al. 2004)

10000 Monte Carlo simulations

yannis@csd.uoc.gr Adaptive Sinusoidal Modeling 13/48

Outline Sinusoidal Modeling Advanced Modeling QHM aQHM eaQHM Applications Key papers References

MSE of frequencies as a function of SNR.

0 10 20 30 40 50 60 70 8010

−10

10−5

100

SNR (dB)

MS

E(f

1)

CRBno iter3 iterFFT

0 10 20 30 40 50 60 70 8010

−10

10−5

100

SNR (dB)

MS

E(f

2)

CRBno iter3 iterFFT

0 10 20 30 40 50 60 70 8010

−10

10−5

100

SNR (dB)

MS

E(f

3)

CRBno iter3 iterFFT

0 10 20 30 40 50 60 70 8010

−10

10−5

100

SNR (dB)

MS

E(f

4)

CRBno iter3 iterFFT

yannis@csd.uoc.gr Adaptive Sinusoidal Modeling 14/48

Outline Sinusoidal Modeling Advanced Modeling QHM aQHM eaQHM Applications Key papers References

MSE of amplitudes as a function of SNR.

0 10 20 30 40 50 60 70 8010

−10

10−5

100

SNR (dB)

MS

E(a

1)

CRBno iter3 iterFFT

0 10 20 30 40 50 60 70 8010

−10

10−5

100

SNR (dB)

MS

E(a

2)

CRBno iter3 iterFFT

0 10 20 30 40 50 60 70 8010

−10

10−5

100

SNR (dB)

MS

E(a

3)

CRBno iter3 iterFFT

0 10 20 30 40 50 60 70 8010

−10

10−5

100

SNR (dB)

MS

E(a

4)

CRBno iter3 iterFFT

yannis@csd.uoc.gr Adaptive Sinusoidal Modeling 15/48

Outline Sinusoidal Modeling Advanced Modeling QHM aQHM eaQHM Applications Key papers References

Notes on QHM

The approximation made in QHM is valid provided that thefrequency mismatch lies in a specific interval.

This interval is a function of the bandwidth of the analysiswindow.

Robustness of QHM against noise was tested and verified.

It can be shown that iterative QHM is equivalent to (anapproximate) Gauss-Newton method.

There are also relations between QHM and ReassignedSpectrum

yannis@csd.uoc.gr Adaptive Sinusoidal Modeling 16/48

Outline Sinusoidal Modeling Advanced Modeling QHM aQHM eaQHM Applications Key papers References

Notes on QHM

The approximation made in QHM is valid provided that thefrequency mismatch lies in a specific interval.

This interval is a function of the bandwidth of the analysiswindow.

Robustness of QHM against noise was tested and verified.

It can be shown that iterative QHM is equivalent to (anapproximate) Gauss-Newton method.

There are also relations between QHM and ReassignedSpectrum

yannis@csd.uoc.gr Adaptive Sinusoidal Modeling 16/48

Outline Sinusoidal Modeling Advanced Modeling QHM aQHM eaQHM Applications Key papers References

Notes on QHM

The approximation made in QHM is valid provided that thefrequency mismatch lies in a specific interval.

This interval is a function of the bandwidth of the analysiswindow.

Robustness of QHM against noise was tested and verified.

It can be shown that iterative QHM is equivalent to (anapproximate) Gauss-Newton method.

There are also relations between QHM and ReassignedSpectrum

yannis@csd.uoc.gr Adaptive Sinusoidal Modeling 16/48

Outline Sinusoidal Modeling Advanced Modeling QHM aQHM eaQHM Applications Key papers References

Notes on QHM

The approximation made in QHM is valid provided that thefrequency mismatch lies in a specific interval.

This interval is a function of the bandwidth of the analysiswindow.

Robustness of QHM against noise was tested and verified.

It can be shown that iterative QHM is equivalent to (anapproximate) Gauss-Newton method.

There are also relations between QHM and ReassignedSpectrum

yannis@csd.uoc.gr Adaptive Sinusoidal Modeling 16/48

Outline Sinusoidal Modeling Advanced Modeling QHM aQHM eaQHM Applications Key papers References

Notes on QHM

The approximation made in QHM is valid provided that thefrequency mismatch lies in a specific interval.

This interval is a function of the bandwidth of the analysiswindow.

Robustness of QHM against noise was tested and verified.

It can be shown that iterative QHM is equivalent to (anapproximate) Gauss-Newton method.

There are also relations between QHM and ReassignedSpectrum

yannis@csd.uoc.gr Adaptive Sinusoidal Modeling 16/48

Outline Sinusoidal Modeling Advanced Modeling QHM aQHM eaQHM Applications Key papers References

QHM and Gauss-Newton method

Assuming we have N samples of x(t) = ce jωtw(t)

Approximate Gauss-Newton solution for frequency

ω(1) = ω(0) −R

{∑Nt=−N tw2(t)x(t)e−jω

(0)t

c(0)∑N

t=−N t2w2(t)

}QHM

x(t) = (c + tb)e jωtw(t)

withω(1) = ω(0) + ρ2

= ω(0) −R

{jb(0)

c(0)

}where

b(0) =∑N

t=−N tw2(t)x(t)e−jω(0)t∑Nt=−N t2w2(t)

yannis@csd.uoc.gr Adaptive Sinusoidal Modeling 17/48

Outline Sinusoidal Modeling Advanced Modeling QHM aQHM eaQHM Applications Key papers References

QHM and Gauss-Newton method

Assuming we have N samples of x(t) = ce jωtw(t)

Approximate Gauss-Newton solution for frequency

ω(1) = ω(0) −R

{∑Nt=−N tw2(t)x(t)e−jω

(0)t

c(0)∑N

t=−N t2w2(t)

}QHM

x(t) = (c + tb)e jωtw(t)

withω(1) = ω(0) + ρ2

= ω(0) −R

{jb(0)

c(0)

}where

b(0) =∑N

t=−N tw2(t)x(t)e−jω(0)t∑Nt=−N t2w2(t)

yannis@csd.uoc.gr Adaptive Sinusoidal Modeling 17/48

Outline Sinusoidal Modeling Advanced Modeling QHM aQHM eaQHM Applications Key papers References

QHM and Gauss-Newton method

Assuming we have N samples of x(t) = ce jωtw(t)

Approximate Gauss-Newton solution for frequency

ω(1) = ω(0) −R

{∑Nt=−N tw2(t)x(t)e−jω

(0)t

c(0)∑N

t=−N t2w2(t)

}QHM

x(t) = (c + tb)e jωtw(t)

withω(1) = ω(0) + ρ2

= ω(0) −R

{jb(0)

c(0)

}where

b(0) =∑N

t=−N tw2(t)x(t)e−jω(0)t∑Nt=−N t2w2(t)

yannis@csd.uoc.gr Adaptive Sinusoidal Modeling 17/48

Outline Sinusoidal Modeling Advanced Modeling QHM aQHM eaQHM Applications Key papers References

Evaluation: RMSE for frequency

B SNR = 0dB, N = 100 (left), N = 500 (right).CRB: Cramer-Rao lower bound for frequency estimation

2 4 6 8 10

10−4

10−3

10−2

Iterations(a)

RM

SE

GNiQHMCRB

2 4 6 8 10

10−4

10−3

10−2

RM

SE

Iterations(b)

GNiQHMCRB

yannis@csd.uoc.gr Adaptive Sinusoidal Modeling 18/48

Outline Sinusoidal Modeling Advanced Modeling QHM aQHM eaQHM Applications Key papers References

Evaluation: RMSE for amplitude

B SNR = 0dB, N = 100 (left), N = 500 (right).CRB: Cramer-Rao lower bound for amplitude estimation

2 4 6 8 10

10−1

100

Iterations(a)

RM

SE

2 4 6 8 10

10−1

100

Iterations(b)

RM

SE

yannis@csd.uoc.gr Adaptive Sinusoidal Modeling 19/48

Outline Sinusoidal Modeling Advanced Modeling QHM aQHM eaQHM Applications Key papers References

Reassigned Spectrum

Time relocation:

τ = −∂φ(τ, ω)

∂ω= τ −R

(XTw (τ, ω)

X (τ, ω)

)where

XTw (τ, ω) = −N∑

t=−Ntw(t)x(t + τ)e−jω(t+τ)

Frequency relocation:

ω = ω +∂φ(τ, ω)

∂τ= ω + I

(XDw (τ, ω)

X (τ, ω)

)where

XDw (τ, ω) = −N∑

t=−N

dw(t)

dtx(t + τ)e−jω(t+τ)

yannis@csd.uoc.gr Adaptive Sinusoidal Modeling 20/48

Outline Sinusoidal Modeling Advanced Modeling QHM aQHM eaQHM Applications Key papers References

Reassigned Spectrum and QHM

QHM: x(t) = (a + tb)e jωtw(t) with b = ρ1a + ρ2ja1

Then, it can be shown:

τ = τ +W2

W0ρ1

and (in case of using Gaussian windows):

ω = ω +W2

W0ρ2

where

Wk =N∑

t=−Ntkw(t)

yannis@csd.uoc.gr Adaptive Sinusoidal Modeling 21/48

Outline Sinusoidal Modeling Advanced Modeling QHM aQHM eaQHM Applications Key papers References

Reassigned Spectrum and QHM

QHM: x(t) = (a + tb)e jωtw(t) with b = ρ1a + ρ2ja1

Then, it can be shown:

τ = τ +W2

W0ρ1

and (in case of using Gaussian windows):

ω = ω +W2

W0ρ2

where

Wk =N∑

t=−Ntkw(t)

yannis@csd.uoc.gr Adaptive Sinusoidal Modeling 21/48

Outline Sinusoidal Modeling Advanced Modeling QHM aQHM eaQHM Applications Key papers References

Frequency mismatch estimation error

−600 −400 −200 0 200 400 600−2000

−1000

0

1000

2000

η1 (Hz)

er(

η1)

(H

z)

No Iterations

iQHM

−600 −400 −200 0 200 400 600−2000

−1000

0

1000

2000

η1 (Hz)

er(

η1)

(H

z)

RS−QHM

yannis@csd.uoc.gr Adaptive Sinusoidal Modeling 22/48

Outline Sinusoidal Modeling Advanced Modeling QHM aQHM eaQHM Applications Key papers References

From QHM to Adaptive QHM, aQHM [6]

QHM (stationarity assumption):

x(t) =

(K∑

k=−K(ak + tbk)e2πjfk t

)w(t)

Adaptive QHM (aQHM):

x(t) =

(K∑

k=−K(ak + tbk)e jφk (t)

)w(t)

where

φk(t) = 2π

∫ t

0fk(s)ds + ϕ, t ∈ [0,T ]

is the estimated instantaneous phase.

yannis@csd.uoc.gr Adaptive Sinusoidal Modeling 23/48

Outline Sinusoidal Modeling Advanced Modeling QHM aQHM eaQHM Applications Key papers References

From QHM to Adaptive QHM, aQHM [6]

QHM (stationarity assumption):

x(t) =

(K∑

k=−K(ak + tbk)e2πjfk t

)w(t)

Adaptive QHM (aQHM):

x(t) =

(K∑

k=−K(ak + tbk)e jφk (t)

)w(t)

where

φk(t) = 2π

∫ t

0fk(s)ds + ϕ, t ∈ [0,T ]

is the estimated instantaneous phase.

yannis@csd.uoc.gr Adaptive Sinusoidal Modeling 23/48

Outline Sinusoidal Modeling Advanced Modeling QHM aQHM eaQHM Applications Key papers References

From QHM to aQHM; Graphically

fk(t)

fk(t)

t

tl

tl+T

tl+Tt

l−T

tl−T

}ρ2,k

tl

t(b)

(a)

yannis@csd.uoc.gr Adaptive Sinusoidal Modeling 24/48

Outline Sinusoidal Modeling Advanced Modeling QHM aQHM eaQHM Applications Key papers References

Frame Rate in aQHM

One sample: no interpolation between estimations

Higher rates (i.e., 5ms, 10ms): Interpolation betweenestimates is required:

Amplitudes are linearly interpolatedFrequencies are interpolated with splinesPhases are interpolated by integration of instantaneousfrequency

yannis@csd.uoc.gr Adaptive Sinusoidal Modeling 25/48

Outline Sinusoidal Modeling Advanced Modeling QHM aQHM eaQHM Applications Key papers References

Example of estimation in aQHM: no iteration(QHM)

200 300 400 500 600 700 800 900 1000 1100900

920

940

960

980

1000

1020

1040

1060

1080

1100

Fre

quen

cy (

Hz)

Time (sample)

instantaneous frequency

estimated

yannis@csd.uoc.gr Adaptive Sinusoidal Modeling 26/48

Outline Sinusoidal Modeling Advanced Modeling QHM aQHM eaQHM Applications Key papers References

Example of estimation in aQHM: one iteration

200 300 400 500 600 700 800 900 1000 1100900

920

940

960

980

1000

1020

1040

1060

1080

1100F

requ

ency

(H

z)

Time (sample)

instantaneous frequency

estimated

yannis@csd.uoc.gr Adaptive Sinusoidal Modeling 27/48

Outline Sinusoidal Modeling Advanced Modeling QHM aQHM eaQHM Applications Key papers References

Example of estimation in aQHM: twoiterations

200 300 400 500 600 700 800 900 1000 1100900

920

940

960

980

1000

1020

1040

1060

1080

1100

Fre

quen

cy (

Hz)

Time (sample)

instantaneous frequency

estimated

yannis@csd.uoc.gr Adaptive Sinusoidal Modeling 28/48

Outline Sinusoidal Modeling Advanced Modeling QHM aQHM eaQHM Applications Key papers References

Validation

Let us consider as example:

x(t) = (11− 340t + 4000t2)e j2π(280t+19500t2)

and frequency mismatch: |fk − ζk | = 35Hz , using Hammingwindow

Consider simplified approaches: Sinusoidal-based analysis(SM)

Mean Absolute Error (MAE) [frame rate: one sample]

AM FM (Hz)

QHM, 10ms 0.46 2.15

aQHM (1) 0.008 0.006

SM, 10ms 0.44 3.08

yannis@csd.uoc.gr Adaptive Sinusoidal Modeling 29/48

Outline Sinusoidal Modeling Advanced Modeling QHM aQHM eaQHM Applications Key papers References

Validation

Let us consider as example:

x(t) = (11− 340t + 4000t2)e j2π(280t+19500t2)

and frequency mismatch: |fk − ζk | = 35Hz , using Hammingwindow

Consider simplified approaches: Sinusoidal-based analysis(SM)

Mean Absolute Error (MAE) [frame rate: one sample]

AM FM (Hz)

QHM, 10ms 0.46 2.15

aQHM (1) 0.008 0.006

SM, 10ms 0.44 3.08

yannis@csd.uoc.gr Adaptive Sinusoidal Modeling 29/48

Outline Sinusoidal Modeling Advanced Modeling QHM aQHM eaQHM Applications Key papers References

Validation

Let us consider as example:

x(t) = (11− 340t + 4000t2)e j2π(280t+19500t2)

and frequency mismatch: |fk − ζk | = 35Hz , using Hammingwindow

Consider simplified approaches: Sinusoidal-based analysis(SM)

Mean Absolute Error (MAE) [frame rate: one sample]

AM FM (Hz)

QHM, 10ms 0.46 2.15

aQHM (1) 0.008 0.006

SM, 10ms 0.44 3.08

yannis@csd.uoc.gr Adaptive Sinusoidal Modeling 29/48

Outline Sinusoidal Modeling Advanced Modeling QHM aQHM eaQHM Applications Key papers References

Validation

Let us consider as example:

x(t) = (11− 340t + 4000t2)e j2π(280t+19500t2)

and frequency mismatch: |fk − ζk | = 35Hz , using Hammingwindow

Consider simplified approaches: Sinusoidal-based analysis(SM)

Mean Absolute Error (MAE) [frame rate: one sample]

AM FM (Hz)

QHM, 10ms 0.46 2.15

aQHM (1) 0.008 0.006

SM, 10ms 0.44 3.08

yannis@csd.uoc.gr Adaptive Sinusoidal Modeling 29/48

Outline Sinusoidal Modeling Advanced Modeling QHM aQHM eaQHM Applications Key papers References

QHM reminder

QHM:

x(t) =

(K∑

k=−K(ak + tbk)e2πjfk t

)w(t)

Instantaneous parameters:

Ak(t) =√

(aRk + tbRk )2 + (aIk + tbIk)2

Φk(t) = 2πfkt + atanaIk+tbIkaRk +tbRk

Fk(t) = fk +1

aRk bIk − aIkb

Rk

M2k (t)

= fk +1

2πρ2,k

yannis@csd.uoc.gr Adaptive Sinusoidal Modeling 30/48

Outline Sinusoidal Modeling Advanced Modeling QHM aQHM eaQHM Applications Key papers References

QHM reminder

QHM:

x(t) =

(K∑

k=−K(ak + tbk)e2πjfk t

)w(t)

Instantaneous parameters:

Ak(t) =√

(aRk + tbRk )2 + (aIk + tbIk)2

Φk(t) = 2πfkt + atanaIk+tbIkaRk +tbRk

Fk(t) = fk +1

aRk bIk − aIkb

Rk

M2k (t)

= fk +1

2πρ2,k

yannis@csd.uoc.gr Adaptive Sinusoidal Modeling 30/48

Outline Sinusoidal Modeling Advanced Modeling QHM aQHM eaQHM Applications Key papers References

QHM and AM-FM decomposition

AM-FM signal

y(t) =

K(t)∑k=1

ak(t)cos(φk(t)),

Taylor series expansion of the instantaneous phase of kthcomponent:

φk(t) = 2πζkt +∞∑i=0

φk,it i

i !

Instantaneous frequency of the kth component at t = 0:

νk(0) = ζk +φk,12π

... and previously we had:

Fk(0) = fk +ρ2,k

yannis@csd.uoc.gr Adaptive Sinusoidal Modeling 31/48

Outline Sinusoidal Modeling Advanced Modeling QHM aQHM eaQHM Applications Key papers References

QHM and AM-FM decomposition

AM-FM signal

y(t) =

K(t)∑k=1

ak(t)cos(φk(t)),

Taylor series expansion of the instantaneous phase of kthcomponent:

φk(t) = 2πζkt +∞∑i=0

φk,it i

i !

Instantaneous frequency of the kth component at t = 0:

νk(0) = ζk +φk,12π

... and previously we had:

Fk(0) = fk +ρ2,k

yannis@csd.uoc.gr Adaptive Sinusoidal Modeling 31/48

Outline Sinusoidal Modeling Advanced Modeling QHM aQHM eaQHM Applications Key papers References

QHM and AM-FM decomposition

AM-FM signal

y(t) =

K(t)∑k=1

ak(t)cos(φk(t)),

Taylor series expansion of the instantaneous phase of kthcomponent:

φk(t) = 2πζkt +∞∑i=0

φk,it i

i !

Instantaneous frequency of the kth component at t = 0:

νk(0) = ζk +φk,12π

... and previously we had:

Fk(0) = fk +ρ2,k

yannis@csd.uoc.gr Adaptive Sinusoidal Modeling 31/48

Outline Sinusoidal Modeling Advanced Modeling QHM aQHM eaQHM Applications Key papers References

QHM and AM-FM decomposition

AM-FM signal

y(t) =

K(t)∑k=1

ak(t)cos(φk(t)),

Taylor series expansion of the instantaneous phase of kthcomponent:

φk(t) = 2πζkt +∞∑i=0

φk,it i

i !

Instantaneous frequency of the kth component at t = 0:

νk(0) = ζk +φk,12π

... and previously we had:

Fk(0) = fk +ρ2,k

yannis@csd.uoc.gr Adaptive Sinusoidal Modeling 31/48

Outline Sinusoidal Modeling Advanced Modeling QHM aQHM eaQHM Applications Key papers References

Reconstruction errors with QHM, aQHM, SM

0.05 0.1 0.15 0.2 0.25 0.3 0.35−0.5

0

0.5

Time (s) (a)

0.05 0.1 0.15 0.2 0.25 0.3 0.35−0.1

−0.05

0

0.05

0.1

Time (s) (b)

0.05 0.1 0.15 0.2 0.25 0.3 0.35−0.1

−0.05

0

0.05

0.1

Time (s) (c)

0.05 0.1 0.15 0.2 0.25 0.3 0.35−0.1

−0.05

0

0.05

0.1

Time (s) (d)

Original

QHM Recon. Error

aQHM Recon. Error

SM Recon. Error

yannis@csd.uoc.gr Adaptive Sinusoidal Modeling 32/48

Outline Sinusoidal Modeling Advanced Modeling QHM aQHM eaQHM Applications Key papers References

aQHM: AM-FM decomposition of VoicedSpeech

Signal Reconstruction

x(t) =K∑

k=1

Ak(t) cos (Φk(t))

Test on 5 minutes of 20 female and male voiced speech(TIMIT)

Average Signal-to-Error Reconstruction Ratio (in dB)

Male Female

QHM 23.9 29.1

aQHM (3) 29.1 34.1

SM 17.2 21.1

yannis@csd.uoc.gr Adaptive Sinusoidal Modeling 33/48

Outline Sinusoidal Modeling Advanced Modeling QHM aQHM eaQHM Applications Key papers References

aQHM: AM-FM decomposition of VoicedSpeech

Signal Reconstruction

x(t) =K∑

k=1

Ak(t) cos (Φk(t))

Test on 5 minutes of 20 female and male voiced speech(TIMIT)

Average Signal-to-Error Reconstruction Ratio (in dB)

Male Female

QHM 23.9 29.1

aQHM (3) 29.1 34.1

SM 17.2 21.1

yannis@csd.uoc.gr Adaptive Sinusoidal Modeling 33/48

Outline Sinusoidal Modeling Advanced Modeling QHM aQHM eaQHM Applications Key papers References

aQHM: AM-FM decomposition of VoicedSpeech

Signal Reconstruction

x(t) =K∑

k=1

Ak(t) cos (Φk(t))

Test on 5 minutes of 20 female and male voiced speech(TIMIT)

Average Signal-to-Error Reconstruction Ratio (in dB)

Male Female

QHM 23.9 29.1

aQHM (3) 29.1 34.1

SM 17.2 21.1

yannis@csd.uoc.gr Adaptive Sinusoidal Modeling 33/48

Outline Sinusoidal Modeling Advanced Modeling QHM aQHM eaQHM Applications Key papers References

Extended aQHM

Recall aQHM:

x(t) =K∑

k=−K(ak + tbk)e jφk (t)

Extended aQHM:

x(t) =K∑

k=−K(ak + tbk)α(t)e jφk (t)

yannis@csd.uoc.gr Adaptive Sinusoidal Modeling 34/48

Outline Sinusoidal Modeling Advanced Modeling QHM aQHM eaQHM Applications Key papers References

AM-FM modeling: aQHM

0 0.05 0.11

1.5

2

2.5

3

Time (s) (a)

AM

1

0 0.05 0.1600

650

700

750

800

Time (s) (b)

FM

1 (

Hz)

0 0.05 0.11

1.5

2

2.5

3

Time (s) (c)

AM

2

0 0.05 0.1900

950

1000

1050

1100

Time (s) (d)

FM

2 (

Hz)

TrueEstimated

TrueEstimated

TrueEstimated

TrueEstimated

yannis@csd.uoc.gr Adaptive Sinusoidal Modeling 35/48

Outline Sinusoidal Modeling Advanced Modeling QHM aQHM eaQHM Applications Key papers References

AM-FM modeling: extended aQHM

0 0.05 0.11

1.5

2

2.5

3

Time (s) (a)

AM

1

0 0.05 0.1600

650

700

750

800

Time (s) (b)

FM

1 (

Hz)

0 0.05 0.11

1.5

2

2.5

3

Time (s) (c)

AM

2

0 0.05 0.1900

950

1000

1050

1100

Time (s) (d)

FM

2 (

Hz)

TrueEstimated

TrueEstimated

TrueEstimated

TrueEstimated

yannis@csd.uoc.gr Adaptive Sinusoidal Modeling 36/48

Outline Sinusoidal Modeling Advanced Modeling QHM aQHM eaQHM Applications Key papers References

Comparing Adaptive Models

0 0.1 0.2 0.3 0.4−0.5

0

0.5

Time (s)

Orig

inal

Sig

nal

0 0.1 0.2 0.3 0.4−0.5

0

0.5

Time (s)

aQH

M R

ec. S

igna

l

0 0.1 0.2 0.3 0.4−0.5

0

0.5

Time (s)

eaQ

HM

Rec

. Sig

nal

0 0.1 0.2 0.3 0.4−0.01

0

0.01

Time (s)

aQH

M R

ec. E

rror

0 0.1 0.2 0.3 0.4−0.01

0

0.01

Time (s)

eaQ

HM

Rec

. Err

or

yannis@csd.uoc.gr Adaptive Sinusoidal Modeling 37/48

Outline Sinusoidal Modeling Advanced Modeling QHM aQHM eaQHM Applications Key papers References

Stop modeling

0.05 0.1 0.15−0.05

0

0.05

0.1

Time (s)

Original signal /t/

0.05 0.1 0.15−0.05

0

0.05

0.1

Time (s)

adaptive QHM reconstruction

0.05 0.1 0.15−0.05

0

0.05

0.1

Time (s)

extended adaptive QHM reconstruction

0.05 0.1 0.15−0.05

0

0.05

0.1

Time (s)

Sinusoidal reconstruction

yannis@csd.uoc.gr Adaptive Sinusoidal Modeling 38/48

Outline Sinusoidal Modeling Advanced Modeling QHM aQHM eaQHM Applications Key papers References

Stop modeling

yannis@csd.uoc.gr Adaptive Sinusoidal Modeling 39/48

Outline Sinusoidal Modeling Advanced Modeling QHM aQHM eaQHM Applications Key papers References

Large scale evaluation

Global Signal to Reconstruction Error Ratio (dB)

Model /p/ /t/ /k/ /b/ /d/ /g/

SM 12.7 12.8 12.4 16.6 14.9 15.3

aQHM 19.9 20.6 21.7 28.3 26.9 27.5

eaQHM 25.4 25.7 27.2 32.9 32.2 32.9

yannis@csd.uoc.gr Adaptive Sinusoidal Modeling 40/48

Outline Sinusoidal Modeling Advanced Modeling QHM aQHM eaQHM Applications Key papers References

Speech Modifications

High quality pitch scale modification

Within glottal cycle formant tracking and modificationstowards voice conversion

Free pre-echo effect in time scaling

Emotion detection and control

yannis@csd.uoc.gr Adaptive Sinusoidal Modeling 41/48

Outline Sinusoidal Modeling Advanced Modeling QHM aQHM eaQHM Applications Key papers References

Speech Modifications

High quality pitch scale modification

Within glottal cycle formant tracking and modificationstowards voice conversion

Free pre-echo effect in time scaling

Emotion detection and control

yannis@csd.uoc.gr Adaptive Sinusoidal Modeling 41/48

Outline Sinusoidal Modeling Advanced Modeling QHM aQHM eaQHM Applications Key papers References

Speech Modifications

High quality pitch scale modification

Within glottal cycle formant tracking and modificationstowards voice conversion

Free pre-echo effect in time scaling

Emotion detection and control

yannis@csd.uoc.gr Adaptive Sinusoidal Modeling 41/48

Outline Sinusoidal Modeling Advanced Modeling QHM aQHM eaQHM Applications Key papers References

Speech Modifications

High quality pitch scale modification

Within glottal cycle formant tracking and modificationstowards voice conversion

Free pre-echo effect in time scaling

Emotion detection and control

yannis@csd.uoc.gr Adaptive Sinusoidal Modeling 41/48

Outline Sinusoidal Modeling Advanced Modeling QHM aQHM eaQHM Applications Key papers References

Speech Synthesis

Unit selection based: fast synthesis, smooth trajectories,high quality prosodic modifications, voice conversion

Statistical modeling: naturally smooth trajectories,articulators tracking, adaptation and controllability,modulations

Sinusoidal models work nicely with DNNs

yannis@csd.uoc.gr Adaptive Sinusoidal Modeling 42/48

Outline Sinusoidal Modeling Advanced Modeling QHM aQHM eaQHM Applications Key papers References

Speech Synthesis

Unit selection based: fast synthesis, smooth trajectories,high quality prosodic modifications, voice conversion

Statistical modeling: naturally smooth trajectories,articulators tracking, adaptation and controllability,modulations

Sinusoidal models work nicely with DNNs

yannis@csd.uoc.gr Adaptive Sinusoidal Modeling 42/48

Outline Sinusoidal Modeling Advanced Modeling QHM aQHM eaQHM Applications Key papers References

Speech Synthesis

Unit selection based: fast synthesis, smooth trajectories,high quality prosodic modifications, voice conversion

Statistical modeling: naturally smooth trajectories,articulators tracking, adaptation and controllability,modulations

Sinusoidal models work nicely with DNNs

yannis@csd.uoc.gr Adaptive Sinusoidal Modeling 42/48

Outline Sinusoidal Modeling Advanced Modeling QHM aQHM eaQHM Applications Key papers References

Speech Recognition

Robust to noise filter bank estimation

Data driven dynamic information

High-resolution modulations

Fits well to the DNN framework.

yannis@csd.uoc.gr Adaptive Sinusoidal Modeling 43/48

Outline Sinusoidal Modeling Advanced Modeling QHM aQHM eaQHM Applications Key papers References

Speech Recognition

Robust to noise filter bank estimation

Data driven dynamic information

High-resolution modulations

Fits well to the DNN framework.

yannis@csd.uoc.gr Adaptive Sinusoidal Modeling 43/48

Outline Sinusoidal Modeling Advanced Modeling QHM aQHM eaQHM Applications Key papers References

Speech Recognition

Robust to noise filter bank estimation

Data driven dynamic information

High-resolution modulations

Fits well to the DNN framework.

yannis@csd.uoc.gr Adaptive Sinusoidal Modeling 43/48

Outline Sinusoidal Modeling Advanced Modeling QHM aQHM eaQHM Applications Key papers References

Speech Recognition

Robust to noise filter bank estimation

Data driven dynamic information

High-resolution modulations

Fits well to the DNN framework.

yannis@csd.uoc.gr Adaptive Sinusoidal Modeling 43/48

Outline Sinusoidal Modeling Advanced Modeling QHM aQHM eaQHM Applications Key papers References

Speech Pathology

Jitter and shimmer estimators

Vocal fatigue

Robust estimation in spasmodic dysphonia

Vocal tremor.

yannis@csd.uoc.gr Adaptive Sinusoidal Modeling 44/48

Outline Sinusoidal Modeling Advanced Modeling QHM aQHM eaQHM Applications Key papers References

Speech Pathology

Jitter and shimmer estimators

Vocal fatigue

Robust estimation in spasmodic dysphonia

Vocal tremor.

yannis@csd.uoc.gr Adaptive Sinusoidal Modeling 44/48

Outline Sinusoidal Modeling Advanced Modeling QHM aQHM eaQHM Applications Key papers References

Speech Pathology

Jitter and shimmer estimators

Vocal fatigue

Robust estimation in spasmodic dysphonia

Vocal tremor.

yannis@csd.uoc.gr Adaptive Sinusoidal Modeling 44/48

Outline Sinusoidal Modeling Advanced Modeling QHM aQHM eaQHM Applications Key papers References

Speech Pathology

Jitter and shimmer estimators

Vocal fatigue

Robust estimation in spasmodic dysphonia

Vocal tremor.

yannis@csd.uoc.gr Adaptive Sinusoidal Modeling 44/48

Outline Sinusoidal Modeling Advanced Modeling QHM aQHM eaQHM Applications Key papers References

Key papers to read

R. McAulay and T. Quatieri: Speech analysis/Synthesis based on a sinusoidal representation IEEE TASSP,Vol.34, No.4, Aug 1986, pp 744-754.

Y. Pantazis, O. Rosec, and Y. Stylianou: Adaptive AM-FM Signal Decomposition with Application toSpeech Analysis IEEE TASLP, Vol.19, No.2, Feb 2011, pp 290-300.

Y. Pantazis, O. Rosec, and Y. Stylianou: Iterative Estimation of Sinusoidal Signal Parameters. IEEE SPL,Vol.17, No.5, May 2010, pp. 461-464

P. Babu and P. Stoica: Comments on Iterative Estimation of Sinusoidal Signal Parameters. Comments onIEEE SPL, pp.1022-1023, Vol.17, No.12, Dec 2010

Y. Pantazis, O. Rosec, and Y. Stylianou: Reply to ”Comments on ’Iterative Estimation of Sinusoidal SignalParameters’ ” IEEE SPL, pp. 1024-1026, Vol.17, No.12, Dec 2010

G. Degottex and Y. Stylianou: Analysis and Synthesis of Speech using an Adaptive Full-band HarmonicModel IEEE TASLP, Vol.21, Issue 10, pp 2085-2095, 2013

yannis@csd.uoc.gr Adaptive Sinusoidal Modeling 45/48

Outline Sinusoidal Modeling Advanced Modeling QHM aQHM eaQHM Applications Key papers References

M. Abe and J. S. III, “CQIFFT: Correcting Bias in a Sinusoidal Parameter Estimator based on Quadratic

Interpolation of FFT Magnitude Peaks,” Tech. Rep. STAN-M-117, Stanford University, California, Oct 2004.

M. Abe and J. S. III, “AM/FM Estimation for Time-varying Sinusoidal Modeling,” in Proc. IEEE Int. Conf.

Acoust., Speech, Signal Processing, (Philadelphia), pp. III 201–204, 2005.

J. Laroche, “A new analysis/synthesis system of musical signals using Prony’s method. Application to

heavily damped percussive sounds.,” in Proc. IEEE Int. Conf. Acoust., Speech, Signal Processing, (Glasgow,UK), pp. 2053–2056, May 1989.

Y. Pantazis, O. Rosec, and Y. Stylianou, “On the Properties of a Time-Varying Quasi-Harmonic Model of

Speech,” (Brisbane), Sep 2008.

Y. Pantazis, O. Rosec, and Y. Stylianou, “Iterative Estimation of Sinusoidal Signal Parameters,” vol. 17,

no. 5, pp. 461–464, 2010.

Y. Pantazis, O. Rosec, and Y. Stylianou, “Adaptive AMFM signal decomposition with application to speech

analysis,” IEEE Trans. on Audio, Speech, and Language Processing, vol. 19, pp. 290–300, 2011.

T. F. Quatieri, Discrete-Time Speech Signal Processing.

Engewood Cliffs, NJ: Prentice Hall, 2002.

yannis@csd.uoc.gr Adaptive Sinusoidal Modeling 46/48

Outline Sinusoidal Modeling Advanced Modeling QHM aQHM eaQHM Applications Key papers References

Acknowledgments

My colleagues: Yannis Pantazis, Olivier Rosec (Voxygen),Vassilis Tsiaras, Gilles Degottex

My ex-PhD Student: Giorgos Kafentzis

Tom Quatieri and Prentice Hall for gave me the permission touse figures from Tom’s book[7]

yannis@csd.uoc.gr Adaptive Sinusoidal Modeling 46/48

Outline Sinusoidal Modeling Advanced Modeling QHM aQHM eaQHM Applications Key papers References

THANK YOU

for your attention on this part as well!

yannis@csd.uoc.gr Adaptive Sinusoidal Modeling 47/48

top related