Perception of Humanoid Social Mediator in Two-PersonDialogs

Yasir TahirInstitute for Media Innovation

Nanyang TechnologicalUniversity (NTU),Singaporeyasir001@e.ntu.edu.sg

Umer RasheedSchool of EEE

Nanyang TechnologicalUniversity (NTU),Singaporeumer1@e.ntu.edu.sg

Shoko DauwelsCIRCQL

Nanyang TechnologicalUniversity (NTU),Singaporesdauwels@ntu.edu.sg

Justin DauwelsSchool of EEE

Nanyang TechnologicalUniversity (NTU),Singaporejdauwels@ntu.edu.sg

ABSTRACTIn this work we present a humanoid robot (Nao) that pro-vides real-time sociofeedback to participants taking part intwo-person dialogs. The sociofeedback system quantifiesspeech mannerism and social behavior of participants in anongoing conversation, determines whether feedback is re-quired, and delivers feedback through Nao. For example,Nao alarms the speaker(s) when the voice is too high or toolow, or when the conversation is not proceeding well dueto disagreements or numerous interruptions. In this study,participants are asked to engage in two-person conversationswhile the Nao robot acts as mediator. They then assess thereceived sociofeedback with respect to various aspects, in-cluding its content, appropriateness, and timing. Partici-pants also evaluate their overall perception of Nao as socialmediator via the Godspeed questionnaire.

Categories and Subject DescriptorsH.5.2 [Information Interfaces and Presentation]: Mis-cellaneous; D.2.8 [Robotics]: Commercial Robots and Ap-plications; J.4 [Computer Applications]: Social and Be-havioural Sciences

General TermsHuman Factors, Design, Experimentation

KeywordsReal-Time Sociofeedback, Humanoids, Human Robot Inter-action, Social Mediator.

Permission to make digital or hard copies of part or all of this work for personal orclassroom use is granted without fee provided that copies are not made or distributedfor profit or commercial advantage, and that copies bear this notice and the full ci-tation on the first page. Copyrights for third-party components of this work must behonored. For all other uses, contact the owner/author(s). Copyright is held by theauthor/owner(s).HRI’14, March 3–6, 2014, Bielefeld, Germany.ACM 978-1-4503-2658-2/14/03.http://dx.doi.org/10.1145/2559636.2559831.

1. INTRODUCTIONOne of the key objectives of research and development

in robotics is to come up with various robots than can as-sist humans in everyday domestic environments. Nowadays,robots are increasingly being viewed as social entities to beintegrated in our daily lives. Socially interactive robots areused to communicate, express and perceive emotions, main-tain social relationships, interpret natural cues, and developsocial competencies [1, 2].With increasing demand of robotsfor domestic environments, research on human-robot inter-action (HRI) has gained more and more importance. Inorder to enhance human-robot interaction, the need for in-tegration of social intelligence in such robots has become anecessity [3, 4]. In this work we integrate real-time sociofeed-back with a humanoid (Nao) robot. We limit ourselves to sixsocial states that are most relevant to a meeting scenario.These social states are Normal conversation, uninterestedspeakers, aggressive conversation, one speaker talking toomuch in a conversation, very low volume and very high vol-ume by the speakers. Using the audio cues the system deter-mines the social state of each speaker and provides feedbackvia Nao robot if required.

2. SOCIOFEEDBACK SYSTEMWe adopt easy-to-use portable equipment for recording

conversations, it consists of lapel microphones for each ofthe two speakers. Each speaker’s audio is captured on sepa-rate channels so that interesting features based on interplaybetween both channels can be computed. We compute thefollowing conversational cues natural turns, speaking per-centage, mutual silence percentage, turn duration, naturalinterjections, speaking interjections, interruptions, failed in-terruptions, speaking rate and response time. Once thespeech cues are calculated, they are fed to machine learn-ing algorithms such as Support Vector Machines (SVMs) todeduce social state of participants [5]. Speaking mannerismare quantitatively assessed by low-level speech metrics suchas volume, rate, and pitch of speech. The social behavior isquantified by sociometrics including level of interest, agree-ment, and dominance. Together, they provide a comprehen-sive picture of the social state of participants in dialogs.

300

Figure 1: System Overview. The system records audio data, computes several conversational and prosodicfeatures, and from those features, determines levels of interest, agreement, and dominance via support vectormachines (SVM).

3. NAO AS SOCIAL MEDIATORThe aim of the conducted experiment was to investigate

whether Nao can interact as a social mediator. Social me-diaotor is someone who observes an on-going conversationand acts as a mediator or judge and provides feedback tospeakers if required. In our case Nao interfaced with so-ciofeedback system monitored several conversations and pro-vided feedback based on social state estimated by sociofeed-back system (see Fig.1).

The experiment consisted of 20 participants, who wereinvited to participate in scenario based conversations eachbeing one minute in duration. We observed in earlier record-ings that it is very difficult for participants to act out cer-tain social state while thinking about what to say at thesame time, therefore we had scripts for each social scenario.The scripts were everyday conversations where one or bothspeakers acted out in a certain way. Each participant par-ticipated in six conversations as we consider six social statesfor this experiment. At the end of the conversation the Naorobot provided feedback using audio message and a corre-sponding gesture which the participant then rated. Therewere two questionnaires first one after each conversation toget participant input about feedback provided, second onewas Godspeed questionnaire [6] which participants filled atthe end to rate their perception of Nao as a social medi-ator. Godspeed questionnaire asks the user rating for an-thropomorphism, animacy, likability, perceived intelligenceand perceived safety. In our case all these criteria wererated high by the participants except perceived safety be-cause high score of perceived safety means participant wereagitated therefore low scores are desirable for this criterion.The questions asked to rate feedback are shown in Table1 along with the average rating score out of 5 where 5 ismaximum.

4. CONCLUSIONWe presented a user study to assess how Nao is perceived

by people in the role of social mediator in two-person di-alogs. Overall, this study suggests that sociofeedback bythe Nao robot can be accurately identified and is appreci-ated by participants. In future, we aim to further improvethe social state estimation. We will collect multi-modal (au-dio and video) datasets for training the system. Secondly,we will attempt to scale the proposed system to multi-partydialogs. We also intend to further improve the feedback de-livery of Nao so that in future it can be a part of real worldgroup discussions.

Feedback AssessmentQ1 Did you notice when the socio-feedback

system was addressing you?4.5

Q2 Did you notice when the socio-feedbacksystem was addressing others?

4.5

Q3 Was the timing of socio-feedback appropri-ate?

3.3

Q4 Did the socio-feedback system interruptthe conversation?

2.3

Q5 Was the interaction natural? 4.2Q6 Did you understand the message given by

the socio-feedback?4.8

Q7 Do you agree with the given feedback? 4.7Q8 Did you enjoy using the socio-feedback sys-

tem?4.8

Table 1: Questions of the assessment form. Thecolumn on the right shows the average rating score.

5. ACKNOWLEDGEMENTSInstitute for Media Innovation (Seed Grant M4080824)

Nanyang Technological University (NTU).

6. REFERENCES[1] T. Fong, I. Nourbakhsh, and K. Dautenhahn, “A survey

of socially interactive robots,” Robotics and autonomoussystems, vol. 42, no. 3, pp. 143–166, 2003.

[2] H. Li, J.-J. Cabibihan, and Y. K. Tan, “Towards aneffective design of social robots,” International Journalof Social Robotics, vol. 3, no. 4, pp. 333–335, 2011.

[3] D. Feil-Seifer and M. Mataric, “Robot-assisted therapyfor children with autism spectrum disorders,” inProceedings of the 7th international conference onInteraction design and children, pp. 49–52, ACM, 2008.

[4] F. Papadopoulos, K. Dautenhahn, and W. C. Ho,“Exploring the use of robots as social mediators in aremote human-human collaborative communicationexperiment,” Paladyn, vol. 3, no. 1, pp. 1–10, 2012.

[5] U. Rasheed, Y. Tahir, S. Dauwels, and J. Dauwels,“Real-time comprehensive sociometrics for two-persondialogs,” in Human Behavior Understanding,pp. 196–208, Springer, 2013.

[6] C. Bartneck, E. Croft, and D. Kulic, “Measuring theanthropomorphism, animacy, likeability, perceivedintelligence and perceived safety of robots,” in Metricsfor HRI Workshop, Technical Report, vol. 471,pp. 37–44, Citeseer, 2008.

301