389MONETA ET AL. Biol Res 41, 2008, 389-395Biol Res 41: 389-395, 2008 BRMeasuring emotion in the voice during psychotherapyinterventions: A pilot study

MARÍA E. MONETA1, 3, *, MARIO PENNA1, HUGO LOYOLA1, ANNA BUCHHEIM2,and HORST KÄCHELE2

1 Faculty of Medicine, University of Chile2 Department of Psychosomatic Medicine and Psychotherapy, University of Ulm, Germany3 University Diego Portales* To whom all correspondence should be addressed.

ABSTRACT

The voice as a representation of the psychic world of patients in psychotherapeutic interventions has not beenstudied thoroughly. To explore speech prosody in relation to the emotional content of words, voices recordedduring a semi-structured interview were analyzed. The subjects had been classified according to theirchildhood emotional experiences with caregivers and their different attachment representations. In this pilotstudy, voice quality as spectral parameters extracted from vowels of the key word “mother” (German:“Mutter”) were analyzed. The amplitude of the second harmonic was large relative to the amplitude of thethird harmonic for the vowel “u” in the secure group as compared to the preoccupied group. Such differencesmight be related to the subjects’ emotional involvement during an interview eliciting reconstructed childhoodmemories.

Key terms: mother, attachment, vocal cues, emotion.

emotional involvement of speakers. Vocalpatterns denote involuntary physiologicalchanges in the speaker’s speech productionsystem (Scherer, 1986), as well asculturally accepted speaking styles (Schereret al., 2001; Campbell and Erickson, 2004;Erickson 2005). Physiological arousal andthe appraisal of the emotion experiencedexert a particularly strong influence on theconfiguration of vocal cues duringdiscourse (Siegmann and Boyle, 1993; Pell,2001).

A number of studies have shown thatdifferent emotions are cued by variouscombinations of acoustic parameters;speech rate and the fundamental frequencypresumably exerting the strongest effect(Banse and Scherer, 1996, Murray andArnolt, 1993; Pell , 2001).The meanfundamental frequency and speech rate aregenerally higher for emotions associatedwith high sympathetic arousal, like anger

Correspondence concerning this article should be addressed to Dr. Maria Eugenia Moneta, Faculty of Medicine, Universityof Chile, Santiago de Chile. E-mail adress: mmoneta@med.uchile.cl

Received: August 4, 2008. In Revised form: January 13, 2009. Accepted: January 19, 2009

INTRODUCTION

Spoken language is considered to be themain psychotherapeutic tool (Russel, 1993);however, the relevance of voice quality forthe communicative process, i.e. the effectsof emotion on speech intonation and tempo,has received little attention. Dittman andWynne (1961) proposed that para-linguisticvariables like pitch and speech rate maycapture emotional signals from discourse,but they could be independent of thesubject’s emotional expression in aninterview. Studies on speech structure andemotion imply various components,spanning from the emotions underlyingcognitive activity to the accompanyingphysiological responses during differentemotions (Scherer, 1986; Gobl andChasiade, 2003; Campbell, 2004).

Modern techniques for voice analysis areaccurate enough to unveil shifts in the

MONETA ET AL. Biol Res 41, 2008, 389-395390

and fear, as well as happiness andanxiousness (Banse and Scherer,1996;Ellgring and Scherer, 1996; Johnston andScherer, 1999; Scherer 2003; Hachizaga etal., 2004; Erickson 2005). Fundamentalfrequency and speech rate are theparameters that give a better assessment ofthe emotional state of speakers as evaluatedby listeners (Banse & Scherer, 1996;Breitenstein et al., 2001).

An important, but difficult-to-analyzeaspect of expressive speech, is voice quality,defined as the quality of “a sound by which alistener can tell that two sounds of the sameloudness and pitch are dissimilar” (ANSI,Psycho-acoustical terminology, 1973;Campbell and Mokhtari, 2003). Changes invoice quality can signal both paralinguisticinformation in terms of changes in thespeaker’s emotional state, mood or attitudeto the message and listener, and non-linguistic information in terms of thespeaker’s social or geographical background,as well as personal characteristics related tothe speaker’s physical constitution or health(Mokthari, 2003; Erickson, 2005). Changesin voice quality are the result of changes inthe configuration of the vocal tract, laryngealand glottal source (Mokthari, 2003;Skakibara and Imagawa, 2004). Thesechanges can be quantified by comparing theamplitude levels of different spectralcomponents, among which the first andsecond harmonic and the first formant havebeen used (Gordon and Ladfoged, 2001). Inaddition, spectral slope has been measuredby splitting the spectrum in third-octavebands and fitting a line to their respectiveenergies as an indication of harshness orsoftness (Schroeder, 2004).

Rice & Kerr (1986) developed aqualitative approach for studying vocalexpression during client therapistinteraction and outcome in psychotherapy.However, no quantitative analysis of thestructure of vocal patterns of either patientor therapist has yet been attempted. Incontrast to the lack of studies on vocalcorrelates of emotion, facial expressionduring psychotherapy has receivedconsiderable attention in recent years(Bänninger-Huber, 1992; Krause et al.1992; Krause, 1998; Dreher et al., 2001).

In this study, presented here as apreliminary report, we explored voicequality through vocal correlates ofemotional styles evoked by the word

“mother” (German: Mutter) in subjectsselected for psychotherapy. A standardizedinterview, the Adult Attachment Interview(AAI; George et al., 1994), was used toclassify the subjects in three categoriesaccording to three attachmentrepresentations (Secure, Dismissing andPreoccupied). Secure individuals give anopen, coherent and consistent account oftheir childhood memories, regardless if theywere positive or negative. These personscan easily address the topics asked aboutand convey an emotional balance about therelationships with their parents. Adults withthe Dismissing classification giveincoherent, incomplete accounts of theexperience with their parents and oftenshow gaps in memory. As a defense againstpainful memories, they minimize theimportance of attachment experiences.These people insist on the normality oftheir affections and on their innerindependence from their parents.

Preoccupied adults recall childhoodexperiences in an angry, excessive and non-objective way. A characteristic of thisgroup is the oscillation between positiveand negative evaluations without beingconscious of this contradiction. Thelanguage employed seems confused,unclear and vague. We predicted that thespectral structure of the word “Mutter”during recall of childhood memories differsamong subjects of these three categories,affecting their voice quality.

METHODS

Subjects

The study is based on the Adult AttachmentInterview (AAI) conducted during the firsttherapeutic session with ten female subjectsat the Psychosomatic Clinic of theUniversity of Ulm. All subjects have hadtheir first baby in the last two months andwere between 27 and 32 years old. Theywere instructed about the AAI, in which

391MONETA ET AL. Biol Res 41, 2008, 389-395

they are asked to remember personalemotional issues in a psychotherapeuticsetting. Patients were asked about theirwillingness to collaborate in this study. Allsubjects were interviewed by the sametherapist (A.B.) and were native-Germanspeakers. The AAI measures the currentrepresentations of past and presentattachment experiences based on narrativeaccounts. The inter-individual differencesin the assessed attachment representationsform three main categories: “Secure”,“Dismissing”, “Preoccupied” (Main &Goldwyn, 1994). Three Secure, fourDismissing and four Preoccupied subjectswere analyzed according to their vocalspectrum.

Procedure

For the present study, the word “mother”(German: Mutter) was selected from theaudiotapes of AAIs of subjects of the threeattachment groups to compare the affectivequality of speech. Recordings of theinterviews containing the word “Mutter” inresponse to specific questions wereacquired at a sampling rate of 22050 Hzwith a Macintosh computer (Power PC7100), using the Sound Edit 16 software.The acquired sounds were further analyzedwith the Signalyze 3.12 software. Twelverepetitions of the word “mother” inresponse to the first half of the interview,containing emotionally loaded questionsfrom the AAI in relation to the motherswere analyzed for each subject. Powerspectra (0-5500 Hz, 20 Hz resolution) ofthe two vowels of the word “mother” wereobtained at the midpoint of the vowels uand e (“Mutter”). The amplitudes andfrequencies of the fundamental frequencyof the following spectral peaks weremeasured, fundamental frequency (Fo) andthe second and third (H2 and H3,respectively). The differences in amplitudesand frequencies of these three spectralpeaks were computed, and averages ofthese measurements calculated for eachindividual and compared among individualsof the three groups with a One-wayANOVA and the Duncan’s statistical test(P< 0.05). In addition to power spectra,

oscillograms and sonograms of thecomplete word were obtained for graphicrepresentation of this sound.

RESULTS

Fig. 1 shows the oscillogram, sonogram andpower spectra of the word “Mutter” fromone individual of the secure group. Peakscorresponding to Fo, H2 and H3 from thepower spectra were taken from the middlepart of vowels u and e.

Power spectra showed that Fo was about206.20 ±12.27 Hz for vowel u and 205.92±18.15 Hz for vowel e when all the subjectswere pooled for analysis, and no significantdifferences occurred between groups(ANOVA: F2= 0.659, P = 0.719). However,the amplitude differences between Fo, H2and H3 varied considerably amongindividuals and groups. Amplitudedifferences in dB between Fo and H3 (Fo-H3) and between H2 and H3 (H2-H3) werealways positive for vowel u, i.e.: Fo and H2always had larger amplitudes relative to H3.A tendency to higher values occurred forsome of these amplitude differences in theSecure group relative to the Preoccupiedgroup, yielding significant differences inthe amplitude of H2 and H3 for vowel ubetween both groups (H2-H3: ANOVA: F2= 6.727, P = 0.0346; Duncan’s test, P =0.045, Fig. 2). The Dismissing groupshowed intermediate values for thesemeasurements and did not differsignificantly from the values of thePreoccupied and Secure groups (Duncan’stest, P = 0.640).

DISCUSSION

In this preliminary study on voice qualitymeasurements related to a psychotherapeuticinterview, we have found differences in therelative amplitudes of harmonics H2 and H3between subjects of the Preoccupied andSecure groups for the vowel “u”, but not forthe vowel “e”. The spectral variation invowel “u” is probably related to its emphasisand longer duration in the phonetics of theGerman word “Mutter” (mother). Our data

MONETA ET AL. Biol Res 41, 2008, 389-395392

suggest subtle variations in the voice for thePreoccupied group relative to the Secure andDismissing groups, while pronouncing theword “Mutter”. As mentioned in theIntroduction, this could imply a difference invoice quality due to emotional arousal.Changes in voice quality are related tomodifications of the vocal tract; involuntarychanges in tonicity and thus, we believe,speech production is a vehicle for emotionand mood.

Although preliminary, our data point tothe usefulness of measuring the amplitudeof harmonics in detecting affectiveattributes of speech. Measurements of thefirst harmonics could indicate more subtlechanges in the voice than those detected byvariations of the fundamental frequency andformants.

The lack of differences in Fo among

subjects in our study could be related to asimilar level of arousal for all individualsinterviewed by the same therapist. Suchinvariance could account for a convergencein fundamental frequency between speakersduring conversation, as reported by Gregoryet al (2000).

Although a number of studies havefocused on pitch variables, in particularfundamental frequency (Scherer, 1986;Tolkmitt and Scherer, 1986; Murray andArnolt 1993), little is known about the roleof voice quality in communicating affection.As pointed out by Scherer (1986) and lateron by Gobl and Chasaide (2003), thetendency has been to concentrate on thoseparameters that are relatively easy tomeasure, such as fundamental frequency andtiming variables. Johnston and Scherer(2001) have found that fundamental

Figure 1: Oscillogram, sonagram and power spectra of the word “Mother” (German: Mutter) fromone individual of the secure (S) group. Peaks corresponding to the fundamental frequency (Fo),second (H2) and third (H3) harmonics are shown in the power spectra of the vowels u and e.

SECURE INDIVIDUAL

393MONETA ET AL. Biol Res 41, 2008, 389-395

Figure 2: Differences between the amplitudes of the spectral peaks of the fundamental frequency(Fo) and second (H2) and third (H3) harmonics for the vowels u and e for the three attachmentgroups. Each circle indicates the average value for an individual. Significant differences (Duncantest, P < 0.05) occurred only for vowel u (amplitude H2-amplitude H3) between the preoccupied (P)and secure (S) groups.

MONETA ET AL. Biol Res 41, 2008, 389-395394

frequency varies with emotional state, beinglower for states related to boredom anddepression and higher for happy and anxiousstates. However, the study of voice qualityimplying more detailed analysis of spectralcontents has been restricted because ofmethodological difficulties involved.

Voice quality signals informationconcerning the speaker’s attitude towardsthe interlocutor, the subject matter and thesituation. Furthermore, the listener’sevaluations of emotions appeared to beprimarily determined by voice quality cuesother than fundamental frequency (Gobl,1989; Lee and Childres, 1991; Gobl andChasaide, 2003). Gobl and Chasaide (2003)in particular, have emphasized that voicemodifications are more related to attitudes,states and mood rather than to specificemotions. A human’s ability to “listenbetween-the-lines” is heavily dependent onvoice quality (Campbell, 2004). In terms ofsubjective perception of the word “mother”(Mutter) by non-instructed listeners, theycould not appreciate any audible differencesbetween the subjects in our study.

Changes in the spectral structure ofspeech that may denote emotionaldifferences have been reported bymeasuring formants at different stages oftherapeutic interventions (Tolkmit &Scherer, 1986). Fujimura and Erickson(2004) have proposed a model thatincorporates speech rhythm aspects, as wellas the linguistic and sociolinguistics aspectsof expressive speech during conversation.Nevertheless, a comprehensive model ofexpressive speech is still lacking.

It has been established that the infantbrain, as early as seven months after birth,detects emotionally loaded words andshows differential attentional responses,depending on their emotional valence(Grossmann et al., 2005). Infants are able tointerpret prosody and recognize and imitatevocal patterns and rhythms, discriminatingintonational aspects of human voice(Fernald, 1993; Spence and Freeman, 1996;Floccia et al., 2000). They can also respondto variations in frequency, intensity andtemporal patterning of sounds (implicitknowledge) signaling affective states(Bebbe et al., 1997; Kuhl et al., 1997;

Papousek and Papousek 1981). Such anearly disposition to react to the emotionalcontents of speech in an implicit waysuggests the need for further research onprosody influences on interpersonalencounters in the therapeutic context. AsBeebe points out, words are not enough;music is needed (Bebbe, 2007).

REFERENCES

ANSI (American National Standard report , 1973)Psychoacoustical terminology. Technical Report, S,3:30

BÄNNINGER-HUBER E (1992) Prototypical affectivemicrosequences in psychotherapeutic interaction.Psychotherapy Res, 4: 291-306

BANSE R, SCHERER K (1996) Acoustic profiles in vocalemotion expressions. J Person Soc Psychol, 70: 614-636

BEEBE B, JAFFE J, LACHMANN F (1997) Mother-Infantinteraction structures and pre-symbolic self and objectrepresentations. J Relation Perspectives, 7: 133-182

BEBBE B, (2007) On Intersubjective Theories: Theimplicit and the explicit in interpersonal relationships.Carli L. Rodini C (Eds)

BREITENSTEIN C, VAN LACKER D, DAUM I (2001)The contribution of speech rate and pitch variation tothe perception of vocal emotions in a German and anAmerican sample. Cognition and Emotion, 15, 1: 57-79

DITTMANN A, WYNNE L (1961). Linguistic techniquesand the analysis of emotionality in interviews. JAbnormal Psychol, 63: 201-204

DREHER M, MENGELE U, KRAUSE R, KAEMMERERA (2001) Affective Indicators of the PsychotherapeuticProcess. An empirical case study. Psychotherapy Res,11: 99-117

CAMPBELL N (2004) Listening between the lines: Astudy of paralinguistic information carried by tone ofvoice. Proc. Int. Symp.Tonal Aspects of Language withEmphasis on Tone Languages, Beijing (TAL) 13-16

CAMPBELL N, ERICKSON D (2004) What do peoplehear? A study on the perception of nonverbal affectiveinformation in conversational speech. J Phonet Soc Jpn8:9-28

CAMPBELL N, MOKHTARI P (2003) Voice quality: The4th prosodic parameter. Proc15th Int Congr PhoneticSciences: 2417-2420

CHILDERS D G, LEE K (1991) Vocal quality factors:Analysis, Synthesis and perception. J Acoust Soc Am,90: 2394-2410

ELLRING H, SCHERER K (1996) Vocal indicators ofmood change in depression. J Non-verbal Behav, 20, 2:83-110

ERICKSON D (2005) Expressive speech: production,perception and application to speech synthesis. AcoustSci and Tech 26, 4:317-325

ERICKSON D, YOSHIDA K, MOCHIDA T, SHIBUYA Y(2004) Acoustic and articulatory analysis of sadJapanese speech. Phonet Soc Jpn, Fall Meet.:113-118

ERICKSON D, MOKHTARI P, MENEZES C, FUJINO A(2003) Voice quality and other acoustic changes in sadJapanese speech. Tech. Rep: Inst Electron Inf CommunEng: 43-48

FERNALD A. (1993). Approval and disapproval: Infant

395MONETA ET AL. Biol Res 41, 2008, 389-395

responsiveness to vocal affect in familiar andunfamiliar languages. Child Dev, 64: 657-674

FLOCCIA C, NAZZI T, BERTONCINI J (2000).Unfamiliar voice discrimination for short stimuli innewborns. Dev Science, 3, 3: 333-343

FUJIMURA H, ERICKSON D (2004) The C/D Model forprosodic representation of expressive speech in EnglishProc Autumn Meet Acoust Jpn: 271-272

GEORGE C, KAPLAN N, MAIN M (1984) The AdultAttachment Interview. Unpublished manuscript,University of California, Berkeley.

GOBL C (1989) A preliminary study on acoustic voicequality correlates. STL-QPSR1, Speech, Music andHearing, Royal Institute of technology, Stockholm 9-21

GOBL C, CHASIADE NI (2003) The role of voice qualityin communicating emotion, mood and attitude. SpeechCommun, 40:189-212

GREGORY S W, GREEN B E, CARROTHERS R, DAGANK, WEBSTER S (2000) Verifying the primacy of voicefundamental frequency in social status accommodation.Language and Commun, 21:37-60

GROSSMANN T, STRIANO T, FEDERICCI A (2005)Infant’s electric brain activity responses to emotionalprosody. NeuroReport 16: 1825-1828

GORDON M, LADFOGED P (2001) Phonation types: Across-linguistic overview. J Phonet, 29:383-406

HASHIZAWA Y, TAKEDA M, HAMZAH M, OHBYAMAG (2004) On the difference of prosodic features ofemotional expressions in Japanese speech according tothe degree of emotion. Proc Speech prosody Nara, 655-658

JOHNSTONE T, SCHERER K (2001) VocalCommunication of emotion. In M.Lewis and J.Haviland (Eds.) The handbook of emotions (2nded.)New York: Guilford Press, pp: 226-235

KRAUSE R, STEIMER-KRAUSE, E ULRICH B (1992) Useof affect research in dynamic psychotherapy. In M.Leuzinger-Bohleber, H. Schneider, & R. Pfeifer (Eds.),Two Butterflies on my Head. Psychoanalysis in theInterdisciplinary Dialogue Berlin, Springer, pp: 277-291

KUHL P, ANDRUSKI J E, CHISTOVICH I A,KOZHEVNIKOVA EV, RYSKINA V L,STOLYAROVA E I, SUNDBERG U, LACERDA F(1997) Cross-language analysis of phonetics units inlanguage addressed to infants. Sciences 227:684-686

MAIN M, GOLDWYN R (1994) Adult Attachment Scoringand Classification Systems. Unpublished manuscript,University of California, Berkeley (1st edition 1985)

MOKTHARI P (2003) Parameterization and control oflaryngeal voice quality by principal components ofglottal waveforms (2003) J Phonet Soc Jpn, 7:40-54

MURRAY I, ARNOLT J I (1993) Towards a simulation of

emotion in synthetic speech: A review of the literatureon human vocal emotion. J Acoust Soc Am, 93: 1097-1108

PAPOUSEK M, PAPOUSEK H (1981). Die Bedeutungmusikalischer Elemente in der frühen Kommunikationzwischen Eltern und Kind. Sozialpädiatrie in Praxisund Klinik, 3: 412-415

PELL M (2001). Influence of emotion and focus locationon prosody in matched statements and questions. JAcoust Soc Am, 109: 1668-1680

RICE L, KERR G (1986) Measures of client and therapistvocal quality In L.Greenberg & W. Pinsof (Eds.) Thepsychotherapeutic process. A research Handbook NewYork: Guilford Press. pp: 73-105

RUSSELL R L (Ed.) (1993) Language in Psychotherapy:Strategies of Discovery. New York: Plenum Press.

SAKAKIBARA K, IMAGAWA H (2004) Acousticalinterpretation of certain laryngeal settings using aphysical model. Proc Speech Prosody, Nara: 637-640

SCHERER K, JOHNSTON T, BANZINGER T (1998)Automatic verif ication of emotionally stressedspeakers: The problem of individual differences. ProcIntern Workshop on Speech and computer, St .Petersburg.

SCHERER K (1986) Voice, stress and emotion. In M. H.Appley & Trumbull (Eds.) Dynamic of stress. NewYork: Plenum Press pp: 159-181.

SCHERER K, BANSE R, WALLBOTT HG (2001).Emotion inferences from vocal expression correlateacross languages and cultures. J Cross-culturalPsychol, 32, 1:76-92

SCHERER K (2003) Vocal communication of emotion: “Areview of research Paradigms.” Speech Commun, 40:227-256

SCHROEDER M (2004) Speech and Emotion research: anoverview of research framework and a dimensionalapproach to emotional speech synthesis. DoctoralThesis, Phonus 7, Res Inst Phonetics, SaarlandUniversity

SIEGMAN A W, BOYLE S (1993) Voices of fear andanxiety and sadness and depression: The effects ofspeech rate and loudness on fear and anxiety andsadness and depression. J Abnorm Psychol, 10:430-437

SPENCE M J, FREEMAN MS (1996) Newborn infantsprefer the maternal low-pass filtered voice, but not thematernal whispered voice. Infant Behav Dev, 18, 15:727-735

TOLKMITT P, SCHERER K (1986) Effects ofexperimentally induced stress on vocal patterns. JExperim Psychol: Human Perception and Performance,12: 302-313