SPEECH SEGMENTATION USING EXTREMA-BASED SIGNAL TRACK LENGTHMEASURE

Prasanta Kumar Ghosh

Speech Analysis and Interpretation LaboratoryDepartment of Electrical Engineering, University of Southern California

Los Angeles, CA 90089prasantg@usc. edu

ABSTRACT

We introduce a novel temporal feature of a signal, namelyextrema-based signal track length (ESTL) for the problem ofspeech segmentation. We show that ESTL measure is sen-sitive to both amplitude and frequency of the signal. Theshort-time ESTL (STLESTL) shows a promising way to cap-ture the significant segments of speech signal, where the seg-ments correspond to acoustic units of speech having distincttemporal waveforms. We compare ESTL based segmentationwith ML and STM methods and find that it is as good as spec-tral feature based segmentation, but with lesser computationalcomplexity.

Index Terms- Speech analysis, Speech processing

1. INTRODUCTION

Automatic segmentation of speech is an important problemthat is useful in speech recognition, synthesis and coding.There has been many approaches to the problem of automaticsegmentation. In 1966 D.R. Reddy [1] has developed a speechsegmentation scheme using the variation of intensity levelsand zero-crossing counts, and other program parameters wereobtained by visual inspection of the waveform. More recentlyHMM based automatic phonetic segmentation has been re-ported [2]; it requires extensive training data but they havereported very high degree of segmentation accuracy. The pop-ularly used feature vector based methods for speech segmen-tation are Spectral Transition Measure (STM) and MaximumLikelihood (ML) segmentation [3]. Of the spectral domainmethods, ML is widely used for phone based segmentation.

It was shown in [4] that MFCC serves as a robust pa-rameter for speech segmentation. Due to co-articulation ef-fect, the spectral transition across some phoneme boundariesis not clearly defined and this results in segment boundariesto appear in wrong locations. Hence, temporal domain fea-tures have been explored for the problem of segmentation [5],which compare favorably with spectral domain methods.

We introduce a new concept of signal track length mea-sure, which is found to be sensitive to both signal amplitude

and frequency. Since acoustic segments have almost distinctwaveform structure, which is reflected in amplitude and fre-quency of the the signal, we utilize signal track length to trackthe change in either amplitude or frequency of the signal.

2. EXTREMA-BASED SIGNAL TRACK LENGTHMEASURE

Let x(t) be a continuous-time signal. The signal track length(STL) of x(t) over the interval [tl, t2] can be obtained byintegrating the infinitesimal track length dr (as shown in Fig.1) over time t =t to t = t2. Let us consider the infinitesimal

x(t) + dcx(t)

dr

Tdxcdt

ti t t+dt t2

Fig. 1. Signal track length measure.

time duration from t to t + dt. Also let x (t + dt) = x (t) + dx.Then dr = /(dt)2 + (dx)2. Therefore, STL of x(t) fromt = tl to t = t2 is

STL(tI, t2)Jt=t2 ot=t2It dr(=/ t t)2 + (d)2t tl ttl

dt=l1/ ( t) dt (d

ot=t2 dx

1 + (X (t))2dt (1)t=tj

where x' (t) is the first derivative of x(t). Since the functionalform of x (t) or x' (t) is not known, one has to resort to numer-

1-4244-0728-1/07/$20.00 C2007 IEEE IV - 1065 ICASSP 2007

ical integration using samples of x(t). However, we estimateSTL using the extrema information of the signal. Let t = t,,and tei+ be the locations of two consecutive extrema of x (t);i.e., x (te, )=O and X (te, 1_)=O. Then the STL from t = te, tot = te1i+ can be approximated as follows

STL(te, tei+ ) (te1te)2 + ((tei+1 X(tei))2 (2)

This we call extrema-based signal track length (ESTL) forthe interval [tei, tei+ ] denoted by ESTL (tei, te,+_ ). ESTLof x(t) from t = t1 to t = t2 can be calculated by finding theextrema points in between t1 and t2 and summing all inter-extrema signal track length. If there are M extrema in be-tween t1 and t2, ESTL(tl, t2) can be calculated as follows,

ESTL(ti, t2) =/(tel -tl)2 + (X(tel_ X(ti ))2M-1

+ (te+1te)2 + ((tei+1) X(tei))2

+ (t2- tem)2 + (X(t2) X(tem ))2 (3)

where the first and last term calculate the ESTL from t1 tofirst extrema and Mth extrema to t2 respectively.

2.1. Essential property of ESTL

The special property of ESTL is that it is sensitive to bothsignal amplitude and signal frequency; i.e., ESTL(tI, t2) ofx(t) is determined by the amplitude and frequency content ofthe signal over the time interval [tl, t2] . This can be easilyshown by considering x(t) = Asin(wot + q), where A is theamplitude of the sinusoid; wo = 2o = 27fo, where To is theperiod of the sinusoid; X is the initial phase of the signal.

Since the sinusoid has one extrema for every TO second,the number of extrema of x(t) between t1 and t2 is given by

M 21 t2 =t 2(t2 t) [assuming t2 -tl > To] (4)To/2 TO

By neglecting the first and last term in (3), (since t2 -tl >To) we can write

The condition under which (5) reduces to (6) can be writtenas Afo > .25, i.e., Afo > 2.5. For A = 1 this means that thefrequency of the sinusoid has to be greater than 2.5 Hz, whichis a feasible assumption for most practical signals.

From (6) it is clear that ESTL is proportional to Afo.Thus, short-time ESTL of a signal will exhibit significant changeif either amplitude or frequency of the signal changes signifi-cantly. However, if the amplitude and frequency change in asignal is such that their product remains constant, the short-time ESTL (STLESTL) does not change. This is shown fora synthetic signal in Fig. 2. The Teager Energy Operator

K N hvv E0 0.2 0.4

0.2t (sec)

Fig. 2. ESTL measure for a synthetic signal: (a) Signalconsisting offour sinusoidal segments each of 0.1 sec hav-ing amplitude andfrequency combinations as (0.5, 200Hz),(1, 1OOHz), (0.5, 1OOHz) and (1, 200Hz) respectively. (b)STJESTL computed using short-time window of 30 ms andshift of5 ms.

(TEO) of the signal x(t) is proportional to A2f 2 [6]. Thus,the square of ESTL(ti, t2) is also a measure of the local sig-nal energy in the sense of TEO.

2.2. ESTL for discrete-time signals

For a discrete-time signal x [n], extrema locations (EL) andextrema amplitude (EA) are estimated as follows

M-1 T 2A)2E ( )2[)M»

M ( 2 )+ (2A)2 [M >> 1]

ith EL Tj = n

ifx[n- 1] < x[n] > x[n + 1],or x[n- 1] > x[n] < x[n + 1],

and ith EA Sj = x[rji](t2-t1) 4A TO2

t2-t)TO 4A)

For O <« 1

ESTL(t1, t2) (t2 -tl) = (t2To-tl)4Afo

(5) To minimize error in estimating extrema information in case

of discrete-time signals we upsample x [n] to a high samplingfrequency (F8 = , T, Sampling period), say 48 kHz forspeech. Using these extrema information, we calculate ESTLfrom n = nj to n = n2 by replacing te, and ti by T8rhi and

(6) T2,ni respectively in (3).

IV- 1066

ESTL (t, t2) (7)

(8)

0.4

3. SPEECH SEGMENTATION USING STLESTL

Considering speech signal to be a multicomponent signal [7],different speech events have distinguished amplitude and fre-quency information. Since ST-ESTL changes with the changein both amplitude and frequency, we use STAESTL as a meansof identifying different speech events. Let us look at a typicalSTAESTL plot of a speech file 'sal.wav' taken from TIMITdatabase, shown in Fig. 3. Form Fig. 3(b) it is clear that

a)

1 (a)0 02 04 06 08 1 1.2 1 4 1.6

30

C) 20

CO 10

, 30

() 20

CO 10

, 30

w 20

Cl) 10

02 04 06 08

02 04 06 08

02 04 06 08

t (in sec)

Fig. 3. ESTL measure on speech dcSTIESTLfor 20 msframes, with 1020 dB SNR condition (d) STISTL]f

STAESTL has very small value durat the very beginning of the sentenSTAESTL has significant values wh(tude is high (0.42-0.5 sec, 0.8-0.9 shigh (0.15-0.25 sec, 1.04-1.18 sec)cation at which STAESTL has signithem as segment boundaries. Thesegment boundaries (given the requia speech signal) is described below

* Step 1: Given a speech signaplitude to +1 to -1.

* Step 2: Use analysis windov10 msec to calculate STAEST]index.

* Step 3: Let ESTLmin and E'mum and maximum value of tSTAESTLm will yield small 1

gion of the signal, choose a thy+ 0.1{ESTLmax-ESTLmin}segment boundaries, where SIThis gives a gross segmentatiispeech from silence. Supposesegment boundaries from this

(b)12 14 16

* Step 4: Consider all valleys in STAESTL plot with am-plitude A, such that A, > ESTLth. Let us considerith valley with amplitude A, and the respective leftand right peak with amplitude Al, and A,,. We ar-range all valleys in ascending order according to the

valley depth factor defined as Aij: Out of these(Al, A,j) 2

arranged valleys we select the first (P -1-1) valleysas segment boundaries.

We also explore the noise robustness property of STAESTLcurve for segmentation at high and medium SNR condition.From Fig. 3(b) and (c) it can be noted that the peak and valleylocations of STAESTL remain unaltered for signal with addi-tive noise upto 10 dB SNR. Hence the same algorithm is usedto find segment boundaries in noisy cases.

4. EXPERIMENTS AND RESULTS

It was shown in [4] that MFCC parameters provide the mostrobust feature for segmentation. To computeMFCC (16 MFCC

1 12 14 16(c) coefficients per frame), we have used a analysis window lengthof 20 ms, and window shift of 10 ms. To compare the per-formance of ESTL based segmentation, we have chosen two

(d) spectral domain methods:1 1.2 1.4 1.6 (1)ML segmentation, using MFCC with a symmetric lifter

(1+Asin2 (v)), proposed in [4].ita: (a) Speech signal (b) (2)Spectral Transition Measure (STM) using the feature vec-ms shift (c) STIESTL for tor and lifter combination in (1).

'or 10 dB SNR condition. The experiments have been conducted on 50 male and 50female speakers' sentence (Sampling frequency = 16 kHz)

ing silence region (either taken from TIMIT database. The experiment was performedce or in between words). on these clean speech sentence as well as on noisy speech,n either the signal ampli- with SNR of 30, 20 and 10 dB. The no of manual segmentsec) or signal frequency is given by TIMIT database is used as input to the ESTL based. Hence, we pick the lo- segmentation algorithm. For ML segmentation (without anyificant changes and mark duration constraint), we have assumed the same number ofalgorithm for finding the segments. STM requires a global thresholding; to circumventired no of segments P for the problem, we have used STM with same number of seg-

ments, and only those number of largest peaks are detected forsegmentation. If the obtained boundary is within ±20 ms of a

x1t[nj normalize its am- TIMIT boundary, we call it a 'match' (M). If two consecutiveboundaries match, we count it as a 'segment match'(S). Also,

v of 20 msec and shift of insertion(I) and deletions(D) are noted, keeping the ±20 ms

Lm, where m is the frame constraint.From table 1 we see that the segmentation performance

of ESTL method is better than STM but worse than that ofSTLmax denote the mini- ML. The performance is steady at medium SNR due to noisethe STAESTLm. Since the robustness property of the ESTL. As the SNR of the sig-values for non-speech re- nal decreases, %M for ESTL-based method remains almostreshold ESTLth = ESTLmin same but the insertion rate increases. This is because evenand mark the frames as though the shape of STAESTL curve remain same, with ad-FESTL plot crosses ESTLth. ditive noise spurious vallyes appear in the STAESTL curveon, which mainly isolates which results in increasing number of wrong segment bound-, we obtain P1 number of aries. The ESTL based segmentation is computationally sim-step. pler thanML as we need to pick only the valleys of the one di-

IV- 1067

Table 1. Segmentation performance ofESTL and other spec-tral domain methods

Method f[ SNR(dB) || M% 1|% D% S%ESTL clean 80.1 11.9 19.9 53.4ML clean 85.1 11.7 14.9 56.1STM clean 69.0 25.4 31.0 48.9ESTL 30 80.0 12.7 20.0 52.0ML 30 86.0 12.5 14.0 55.3STM 30 69.3 25.1 30.7 42.1ESTL 20 79.4 15.7 20.6 48.3ML 20 84.3 16.9 15.7 53.7STM 20 71.7 26.8 28.3 42.9 iESTL 10 ] 78.9 20.8 21.1 43.7ML 10 ]I 76.9 } 22.1 23.1 49.82STM fj 10 1168.8 27.7 31.2 40.5 i

mensional feature contour, unlike the full-search optimizationusing multi-dimensional spectral features in ML. Simulationsshow that the cpu time for running segmentation algorithmusing the ESTL is almost 25% of that of the ML segmenta-tion.

We present (in Fig. 4 and Fig. 5), the spectrogram andtime domain plot of two signal segments along with ST±ESTLof the signal. In both figures part (a) shows the spectrogramwith overlaid segment boundaries as in TIMIT database, part(b) shows the time domain plot of the speech signal and part(c) shows the STAESTL with segment boundaries obtainedfrom STAESTL based algorithm.

ey p p apc p rh sh th dx h#

~4000-

2N6000-lfltlI';16l 11 hPill0"iL

0 l -faw 1,1I' _IMN] w ll#kl "Wi M 91i ]OKIII#I!F114011,f Iil'!0 l(a0 0.2 0.4 0.6 0.8 1 1.2 1.4 1.6

1

0 0.2 0.4 0.6 0.8 1 1.2 1.4 1.6

I'l A

20

10(-

Fig. 4. Comparison of manual and ESTL boundaries for'si1271.wav' over last 1. 7 sec.

As can be seen from both the figures, most of the acousticunits in speech can be identified by either amplitude or fre-quency changes. Vowel and diphthongs have high energy,while fricatives have higher frequency content. Therefore,they are well segmented using the algorithm applied on STESTLplot. Nasal and stops are also segmented easily. Amplitudevariation within a phonemic segment has given rise to inser-

tion error (Ishl at 0.6 sec, Fig. 4). Also in 'programs' (0.82-1.46sec Fig. 5), Iplrlowlglrlaelm!, Inr phone (part of consonantclusters Igiri) and /Il (0.45 sec in Fig. 5) have been missed.

kclk ix kcl k el pcl p r ow gDb r ae m

j 1 Iai 11Nl

Ii 1 k1i 1 1 ,1 1 1it {| 1||tIiI (a)

10k

0 .- N I- -,I - 1, -(C)

Fig. 5. Comparison of manual and ESTL boundaries for'sx146.wav' over 0-1. 6 sec.

5. CONCLUSION

We have presented a temporal domain method for the prob-lem of speech segmentation. ESTL is found to capture bothamplitude and frequency changes in the signal and hence issuitable for finding segments of speech which have distinctamplitude or frequency content.

6. ACKNOWLEDGEMENT

The author wishes to thank Prof. T.V. Sreenivas. This workwas done under his supervision at speech lab, Indian Institureof Science, Bangalore, India.

7. REFERENCES

[1] D.R. Reddy, "Segmentation of Speech Sounds",J.Acoustic.Soc.Am., 1966, Vol. 40, No. 2, pp 307-312.

[2] D.T. Toledano, L.A. Hemandez Gomez and L.V.Grande, "Automatic Phonetic Segmentation", IEEETrans. Speech and Audio Proc., Vol. 11, No. 6, Nov.2003, pp 617-625.

[3] T. Svendsen and F.K. Soong, "On the Automatic Seg-mentation of Speech Signals", Proc. ICASSP, Dallas,1987, pp 77-80.

[4] A.K.V. SaiJayram, V. Ramasubramanian and T.V.Sreenivas, "Robust parameters for automatic segmenta-tion of speech", Proc. ICASSP, May 2002, pp 1-513-516.

[5] Anindya Sarkar and T.V. Sreenivas, "Automatic SpeechSegmentation using Average Level Crossing Informa-tion", Proc. ICASSP, March 2005, pp 1-397 - 1-400.

[6] J.F. Kaiser, "On Teager's energy algorithm and its gener-alization to continuous signals", Proc. 4th IEEE DigitalSignal Proc. Workshop, Sep 1990.

[7] R.J. McAulay and T.F. Quatieri, "Speech analy-sis/Synthesis based on a sinusoidal representation",IEEE Trans. ASSP, Vol. 34, Issue 4, pp. 744 - 754, Aug1986.

IV- 1068

T ,, ,, 1 1 1., ,, Tgyliaxiu rzlGlJl T T T rr rh 1 i,[ 1 1 ,51

0 0.2 0.4 0.6 0.8 1 1.2 1.4 1.6

1

0.5

_jk. AL,0 .r'r I '111W

-0.5

0 0.2 0.4 0.6 0.8 1 1.2 i.4 1.6

30 r- Ti

-o

c:

0.2 0.4 0.6 0.8

t (sec)1 1 1.4 1.6

0.2 0.4 0.6 0.8

t (sec)1 1.2 1.4 1.6

h# S8000

. r 6000

. M:0-P .- 4000

2000

0

Oo

-O