Touching Sound: Vulnerability and Synchronicity

Abstract In this paper, vulnerability is considered in terms of our ability to synchronise in time with others, and how the consequences of being unable to do so leads to isolation and a range of social, emotional, and psychological issues. We describe a project to develop a shared musical instrument to facilitate the process of joint interaction and the emergence of moments of synchronising in time with others. The design involves a multidisciplinary and multi-practice team that includes music therapists, artists, performance artists, musicians, music psychologists, music technologists, software and hardware sound engineers.

Author Keywords Synchrony; Cooperative Action; Music Therapy; Entrainment; Aspergers; Autism; Design.

ACM Classification Keywords H.5.m. Information interfaces and presentation (e.g., HCI), H.5.2.User interfaces, H.5.3 Computer supported cooperative work, evaluation/methodology, synchronous interaction, theories and models.

General Terms Human Factors; Design. Copyright is held by the author/owner(s).

CHI’13, April 27 – May 2, 2013, Paris, France.

ACM 978-1-XXXX-XXXX-X/XX/XX.

Sam Aaron Computer Lab, University of Cambridge, UK. Samaaron@gmail.com

Phil Barnard MRC Cognition and Brain Sciences Unit, Cambridge, UK. Philip.Barnard@mrc-cbu.cam.ac.uk

Ian Cross, Satinder Gill Faculty of Music, University of Cambridge, UK. ic108@cam.ac.uk spg12@cam.ac.uk Tommi Himberg Brain Research Unit, O.V. Lounasmaa Laboratory, Aalto University, Finland. tommi.himberg@aalto.fi

Richard Hoadley, Helen Odell-Miller, Department of Music and Performing Arts, Anglia Ruskin University, Cambridge UK. richard.hoadley@anglia.ac.uk Helen.Odell-Miller@anglia.ac.uk Rob Toulson Cultures of the Digital Economy Research Institute (CoDE), Anglia Ruskin University, Cambridge UK. Rob.Toulson@anglia.ac.uk

Introduction Being unable to connect in time with another human being can lead to isolation, loneliness, and frustration. For most of us, we take our ability to move in synchrony with each other for granted and only become aware of this capacity when the timing is not quite right, leaving an awkward feeling. For some, however, this capacity is inhibited, and notably for people with autism. At the mild end of the autism spectrum, those with Aspergers suffer depression and degrees of intolerance from others that comes with this discrepancy in timed interaction. They are fully aware that they have difficulties in establishing bonds with others in the same way as those around them, and although at the surface they appear to be like anyone else, their behavioural reactions are slightly different which results in a lack of understanding towards them. Those with Aspergers are vulnerable in their emotions and their isolation.

In his work on autism, amongst other disorders, Condon [1] identified that the temporal coordination between someone with autism and someone without, created problems in being able to understand the other due to the temporal delay of perceiving and responding to sound.

Musicians have developed therapeutic methods [2] to facilitate moments of connection through making music together, seeking to help people with a range of emotional and psychological issues to synchronise in time with another person.

Over the past two years, a team of us including musicians, music psychologists, music therapists, performance artists, software and hardware designers,

have been developing a design process and prototype of a shared musical instrument to address how to support synchronisation in time and facilitate the capacity to cooperate with another person. Such an instrument would be tangible and pleasurable, requiring at least two people to play by touching it with their hands to create musical sounds. It would be a complement to the current instruments and methods used by music therapists, and we see it as something that, in the future, could be used by families and friends to engage with those they love and seek to support. At the heart of music therapy is experiencing each other through sound and movement.

In the past, music therapists have sometimes been disappointed and wary in working with technologists as the design process and end result has tended to be technology driven rather than therapy and human engagement driven, and thereby fallen short of therapeutic needs [3]. From the beginning, we realised that we needed to work together in order for us to develop any meaningful and useful interface for vulnerable children and adults. This paper presents our work in progress, and in the next section we discuss our design method and framework, followed by examples of some findings from the pilot work, and concluding with some reflections on what needs to be done.

Design and Research Our shared musical instrument investigates how a gestural and tangible interface can facilitate synchrony and cooperative action in musical interaction. The design involves an iterative process of developing and testing what we have termed Generic Interfaces for Socio-Musical Orientation (GISMOs). We are integrating

gesture-based interfaces with interactive digital music systems to produce an instrument that responds to the degrees of cooperation exhibited in the joint interaction with it. The patterns of communicative interaction, both verbal and gestural are analysed to gain an understanding of the emergence of cooperative behaviour, and the findings are fed back into the design to optimise the interface’s facilitation of joint interaction.

The research draws on work in Interactive Musical Digital systems, Computer-Supported Cooperative  Work (CSCW), and tangible interfaces. In the field of interactive digital music, many systems have been developed in which performers use gesture to shape or generate sound. Murray-Rust & Smaill [4] provide a preliminary system for describing and evaluating the dynamics of digital musical interactivity in generic terms. Cross [5] has integrated aspects of this system in a framework to guide and to test the development of digital music technologies for cooperative interaction.

This research builds on recent musicological research [6] which views music as a medium for cooperative interaction that engenders social solidarity. Music achieves this by aligning the affective states of interacting participants, by aligning their actions in time through processes of entrainment [7]. Musical interaction supports behaviours such as empathy, mimicry or synchrony of gesture [8, 9], providing a means for exploring the conditions for the emergence and maintenance of cooperation.

While collaborative or cooperative interactions lie at the heart of much research in CSCW, their study is primarily considered with regard to the achievement of

the goals of the collaboration [10]. CSCW has traditionally focused on transactional features involved in the coordination of joint, goal-directed action. However, research (e.g. [11]) has shown that interactive features other than those directly concerned with the extrinsic goals of the interaction, are highly influential for the success of interaction. These features, characterised as phatic or relational, serve to set up and sustain cooperative social relationships. The CSCW and Tangible Embedded and Embodied Interfaces (TEI) literature is beginning to consider this in the last decade (see, e.g. [12] and [13]).

The relational dimension is central to the maintenance and success of interaction, and mediating interfaces need to be sensitive to cues of both transactional and relational dimensions. The design of the interface of the shared musical instrument is being developed with sensitivity to these cues.

The “physicality” of touch and movement are also part of cooperative interaction. The area of TEI addresses this aspect in the design of interactive and cooperative touch and movement-based interfaces that explore human emotion, sharing of experience, and sensory interaction (e.g. [14] on the experience of intimacy through the cooperative manipulation of sound). The sensory immediacy of tangibility, combined with real-time connectivity, allows for the sharing of experience that is essential for ‘being with others’ [15]. In HCI there is a growing interest in temporality and bodily interaction (e.g. Mobile Life lab at Stockholm University).

Designing The Prototype The team decided that due to sensitivity, early

prototyping of the GISMO should be tested out on people who are not vulnerable. This is for ethical reasons so as not to cause any disturbance and harm. The music therapists will be able to gauge when the prototype has reached a level where vulnerable people can use it.

The design of the first prototype was undertaken following discussions in the team. One of the music therapists working with children asked us to think of the interface as being simple to engage with and ‘dribble proof’. Another therapist suggested that one of the simplest interfaces might just involve the laying down of hands. In design terms, this was easily accomplished using electrically conductive paint which is sensitive to the skin’s natural capacitance, and a perspex surface was used that emphasised and protected the interface’s functionality. The two hand prints were configured so as to emphasise the requirement that the device should be operated by two different people, although it is accepted that this could be made more clear in future versions. Subsequent to testing and the pilot study it was also decided that any future version would be improved by the inclusion of raised, moulded hand shapes on the surface of the Perspex. These would help make the type of contact required, clearer.

One of the interactive goals of the prototype is simultaneous synchrony. Music therapists term this the ‘pivotal moment’. It is brief but significant, and is a felt experience by both the client and the therapist. The music technologist, software engineer, and sound engineer selected a high pitch musical sound to cue simultaneous synchrony. This sound would only occur when two people touched the GISMO at the same time.

The prototype has been tested in the pilot study briefly presented below (see www.touchingsound.rt60.co.uk).

The Pilot Study The Pilot study was conducted in the Centre for Music and Science, with five pairs of subjects, all with experience in music. For the Pilot study, as we were testing what could be understood about the interface, we deliberately did not inform the participants about the relationship between their actions and the sound outputs. We asked them discover what is possible, and once they felt that had achieved this they were asked to play with the GISMO together. Each session lasted 20 minutes and was videotaped.

PRELIMINARY OBSERVATIONS The participants seemed to find the organ sound to be the most pleasurable. Opinions differed about the high pitched sound of tinkling bells, yet when it occurred the participants were curious as to what actions caused it.

One pair liked the organ sound so much that they focused on discovering how to achieve it, yet their coordination with each other at the surface of the GISMO appeared to be correlated to the higher pitched musical sound. In another pair, where one of the participants found it somewhat irritating, the pair still found themselves oriented to achieving it. The higher pitched sound seemed to facilitate getting people to come together in moments of simultaneous synchrony.

They all came to realise that the high pitched sound occurred when they touched the surface at the same time, but they were not so clear as to the correlation of other sounds to other forms of synchronised coordination.

GISMO Prototype

The participants used a wide range of gestures from finger movements, to sliding the hand across the surface, to waving the hand in choreographed movements around the surface, and in the case of one pair (both pianists), used both their hands and all ten fingers. Most of them realised that the palm or whole hand needs to be placed on the surface or just above the surface.

Participants entrained each other in movement at the levels of individual and joint entrainment to sound, both on and above the surface of the GISMO. Consider this interaction starting at 2.43 mins into the session:

2.43 mins - both have hands touching and flat on the Gismo;

2.44 mins – A lifts her hand and B follows;

2.45 mins - B places her hand down and A follows;

2.46 mins - both simultaneously lift their hands;

2.47 mins - A places hand down and B follows, then A then lifts her hand before B touches down as she hears the organ sound, causing B's hand to hover above Gismo's surface; both then lift their hands higher at the organ sound.

There was a common held view that one person was the 'leader', in 'charge', because one of the hands seemed to activate the sound more than the other.

The observations raise a complex range of design issues, including: what is a pleasurable sound? What kinds of gestures does the interface need to afford? How do we correlate sounds with different timings in coordinated synchronisation? How do we ensure the system does not bias leadership possibilities? How would patterns of self and self-with other entrainment to sounds evolve with the design?

For the GISMO to be applied in the context of music therapy, and be used by families and with friends, it will need to be able to handle the particular aesthetics of sounds for the vulnerable person, and adapt to their interaction capabilities. It would also need to be sensitive to the touch sensitivity of participants to particular textures, materials, and physical feedback.

Conclusion This project is informed from its outset by music therapy, understanding its practices and experiential knowledge of the therapist and the experiential knowledge of the vulnerable person. We consider vulnerability here as lying in the difficulties of moving in time with another human being that ordinarily allows us to be connected, understand, and be understood by others at the phatic and relational level. The result can be isolation, loneliness, and in cases, depression.

In the development of this shared musical instrument, we aim to help elucidate principles that underlie the emergence of cooperation in real-time interaction, using music as a medium through which interactions can be conducted, and apply these to the development of GISMO as an interface that affords moments of sharing an experience with another person.

Acknowledgements We thank Amelia Oldfield, Bonnie Kempske, Jane Turner, and Cecily Morrison for their advice on the Touching Sound project and the design of GISMO.

References [1] Condon, WS, Ogston WD. Sound film analysis of normal and pathological behaviour patterns. Journal of Nervous and Mental Diseases, 143, 4 (1996), 338-347.

[2] Wigram, T. Improvisation. Methods and techniques for Music Therapy Clinicians, Educators and Students. Jessica Kingsley Publishers, 2004.

[3] Magee, W. L., Bertolami, M., Kubicek, L., LaJoie, M., Martino, L., Sankowski, A., et al. Using Music Technology in Music Therapy With Populations Across the Life Span in Medical and Educational Programs. Music and Medicine, 3, 3 (2011): 146-153.  

[4] Murray-Rust, D., & Smaill, A. Towards a model of musical interaction and communication. Artificial Intelligence, 175, 9-10 (2011), 1697-1721.

[5] Cross, I. (in press). "Does not compute"? Music as real-time communicative interaction. AI & Society.

[6] Turino, T. Music as social life. London: University of Chicago Press, 2008.

[7] Clayton, M., Sager, R., & Will, U. In time with the music: The concept of entrainment and its significance for ethnomusicology. ESEM counterpoint, 1 (2005), 1-45.

[8] Miles,  L.  K.,  Nind,  L.  K.,  &  Macrae,  C.  N.  The  rhythm  of  rapport:  Interpersonal synchrony and social perception. Journal of Experimental Social Psychology, 45, 3 (2009): 585-589.

[9] Rabinowitch, T.-C., Cross, I., & Burnard, P. Long-term musical group interaction has a positive influence on empathy in children. Psychology of Music (2012) 1-15.

[10] Schmidt, K. Cooperative Work and Coordinative Practices: Contributions to the Conceptual Foundations of Computer-Supported Cooperative Work (CSCW). London: Springer, 2011.

[11] Coupland, J., Coupland, N., & Robinson, J. D. "How Are You?": Negotiating Phatic Communion. Language in Society, 21, 2 (1992), 207-230.

[12] Oleksik, G., Jones, R., & Milic-Frayling, N. Use of gestures in a multi-device environment (Technical Report MSR-TR-2010). Redmond, WA 98052: Microsoft Research, 2010.

[13] Olson, I. C., Leong, Z. A., Wilensky, U. & Horn, M. S. (2011). “It’s just a toolbar!” Using tangibles to help children manage conflict around a multi-touch tabletop. In Proceedings of TEI’11, (2011), 29-36.

[14] Balaam, M., Fitzpatrick, G., Good, J., & Harris, E. Enhancing interactional synchrony with an ambient display, Proceedings of the 2011 annual conference on Human factors in computing systems, ACM Press (2011), 867-876.

[15] Nagargoje, A., Maybach, K., Sokoler, T. Social Yoga Mats: Designing for Exercising/Socialising Synergy. In Proceedings of TEI’12, (2012), 87-90.