material interactions with tangible tabletops: a pragmatist perspective

Draft version – please don’t spread.

Material Interactions with Tangible Tabletops: a Pragmatist Perspective

Nicolai Brodersen Hansen CAVI & PIT

Aarhus University [email protected]

Kim Halskov CAVI & PIT

Aarhus University [email protected]

ABSTRACT

We investigate how the interaction with tangible interactive tabletops can be seen as a material exploration of form and sound. As the theoretical foundation for our analysis we build on John Dewey’s pragmatism as well as recent efforts to appropriate pragmatism for interaction design research. As the research platform for this investigation we developed an interactive tabletop, the Radar Table, which allows users to create soundscapes by manipulating tangible objects. The Radar Table was deployed ‘in the wild’ at a major Danish music festival, and based on video recordings we examine people’s dynamic exploration of sound through the interactive tabletop. The main contribution of the paper is the development of the theoretical foundation for understanding tangible tabletops as material interfaces that can be shaped and experimented with. We build on three of the basic concepts of pragmatism: situation, inquiry, and technology, which we develop further for the study of the dynamics of material interactions with tangible tabletops as part of a research strategy of appropriating pragmatism for use in interaction design and HCI research.

Author Keywords Interactive tables, tangible tabletops, materials, pragmatism, sound

ACM Classification Keywords H.5.m. Information interfaces and presentation (e.g., HCI): Miscellaneous.

INTRODUCTION There is a growing interest in design as a material practice within HCI, as evident by a number of recent publications [2, 8, 17, 25, 33, 34]. These research contributions consider software and hardware as materials that can be molded and

shaped by designers and users in order to reach a stable form as a product or interactive installation.

Vallgårda and Redström [34] describe information technology as a computational composite, a material that may be combined with other materials as part of design practice. This way of considering software and hardware as intertwined has by Fernaus and Sundström been labeled as the material move [8], which explores how designers work with new and challenging materials.

One such computational composite is tangible tabletops, in which physical objects provides an interface unlike for example the materials of touchscreens, mouse or keyboards. In this paper we discuss tangible tabletops that allows users to experiment with interactive sounds. In the form of music, sound is something almost everyone has a relation to, can enjoy and have fun with. With a material interface we wanted to make the experimentation with sound available to everyday users. To do so and in order to explore the potentials tangible tabletops as material interaction we have developed an interactive installation, the Radar Table, which makes sounds available to people for playful engagement.

Figure 1 The Radar Table at the SPOT Festival

The Radar Table turns the conventional understanding of what to expect from a concert or musical experience on its head. Radar Table is an interactive, audiovisual installation, a kind of music instrument, and a platform for creative and social interaction, which transcends the boundary between performer and audience by offering people the opportunity

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]. NordiCHI '14, October 26 - 30 2014, Helsinki, Finland Copyright is held by the owner/author(s). Publication rights licensed to ACM. ACM 978-1-4503-2542-4/14/10�$15.00. http://dx.doi.org/10.1145/2639189.2639200


to collaboratively express themselves musically as part of a performance and as well as a personal experience – without any prerequisites for playing music.

The Danish musician, composer and producer, Henrik Munch, has created the musical material, which is available for people to shape and transform into a musical experience. By interacting with tangible objects on the table surface, the audience defines how the music sounds and how it evolves. Cube-shaped objects represent music loops, which can be combined, layered, and offers control of the volume levels of each element. Cylindrical tangibles representing musical effects (reverb, delay, etc.) can easily be applied to the musical loops to create complex expressions. The flexibility enabled by the tangible objects provides people the opportunity to mold the music in unanticipated ways - and in essence creating a unique music experience.

Radar Table is the second iteration of the table. It has recently been re-designed and developed by our research laboratory [11] and tested over several days at the SPOT Festival, which is both a conventional music festival and a platform for Nordic musicians to develop their artistic and commercial potential, (see Figure 1). A previous version, the DJStation, was developed as part of a research project and exhibition experiment, which aimed to explore how young people, - the so called Digital Natives - use, consume, and interact with digital and social media.

Radar Table belongs to the category of new interfaces for musical experiences and among the most prominent ones is the Reactable [14, 15] and as a more recent example we find mixiTUI [27]. The research teams behind both cases report that the two installations has been very well received by users and provided a platform for engaging experiences and fun. The Radar Table has spent a very active life outside of the laboratory and seeing people explore the interactive table with seemingly great excitement has also intrigued us. We have been motivated by the apparent success of the Radar Table and begun to ponder the nature of this particular kind of interactive experience. For that purpose we have video recorded the use of the Radar Table at the SPOT festival. The video material has enabled us to start digging into the dynamics of interacting with tangible tabletops and present a detailed account interacting with the material interfaces of tangible tabletops and sound experiences by looking into the moment-to-moment dynamics of interactive experiences in order to complement other studies, for instance Pedersen and Hornbæk’s [27] evaluation of mixiTUI based on a questionnaire.

In order to provide a theoretical foundation for our analysis we build on Dewey’s pragmatism as well as recent efforts to appropriate pragmatism for interaction design research [9] and [4]. We further develop three of the basic concepts of pragmatism: situation, inquiry, and technology, which we appropriate for the study of the material interactions with tangible tabletops as part of a broader research interest

in appropriating pragmatism for use in interaction design research. In particular we contribute to an understanding of tangible tabletops as material and interactive sound as composite materials by drawing on the concept of pragmatic technology [12]. By doing so we contribute to Fernaus and Sundströms call for a “language of materials” [8].

We first present related work regarding interactive tables as well as give an overview of the Radar Table with regards to technical setup and collection of data in our research project. We then outline our initial understanding of design materials as pragmatist technology drawing on the philosophy of Dewey, before moving on to analyze the table in use. Through this case we explore tangible tabletops as material, and condense an initial understanding of the kind of experimentation that goes on in use. Finally we wrap up with a discussion and future work for going forward.

TANGIBLE TABLETOPS Tabletop interfaces represent an area of interface technology, which recently have seen a dramatic increase in the application of multi-touch interfaces, where several users can simultaneously interact directly with a flat surface, using their fingertips [5]. The idea of tangible interaction was coined by Ishii and Ullmer [13] to describe interactive physical objects that can serve as input and output devices. Jordà and colleagues [15] trace the roots of tangible interaction even further back, to Shneiderman’s [32] conception of direct manipulation. Shaer and Hornecker [31] have compiled an excellent overview of the state-of-the art in tangible user interfaces. In this article we focus on one particular kind of tangible user interface, namely, tangible tabletop interfaces, which uses physical objects as interface component on top of a tabletop. Such interactive tabletops have been explored in a number of settings, including gaming [21], museums [22], and marketing [24].

The most direct source of inspiration for the Radar Table is the Reactable [14, 15]. The Reactable is an electronic music instrument built upon a tangible tabletop interface. By placing acrylic pucks on the tabletop surface one or more performers can create music by manipulating the tangible objects. According to [15] the objects can be categorized into six functional groups: audio generators, audio filters, controllers, control filters, mixers and global objects and each of these six elements is represented by a unique physical shape with a graphic symbol representing the functionality. The position and movements of the tangible objects is registered by using computer vision tracking software reacTIVision [18].

When placed on the table surface the objects are illuminated and they interact with neighboring pucks according to their position and proximity. Auras around the physical objects display information about their behavior on


the table and the interaction between the pucks is visualized on the tabletop surface by lines with various shape, e.g. jagged lines. By rotating the objects one of their internal parameters may be controlled, for instance related to the frequency of sounds. The Radar Table share many similarities with the Reactable but, as we will return to in one of the subsequent sections, also exhibits a number of differences.

The Reactable has been exposed at more than 200 presentations and concerts in more than 30 countries and according to [14] has gained extensive popularity due to the design’s balance between expressivity, richness and control. Jordà and co-authors [15] have estimated that people after 5 to 10 minutes grasps the basic principles and that users who spend more that 10 minutes often become addicted, but apart from such general statements no detailed evaluation is available. We have experienced that The Radar Table is being received with a similar kind of engagement and we have therefore been motivated to looking into more detailed descriptions of how people use the Radar Table.

The Reactable has been an important inspiration for many other tangible interfaces, including the mixiTUI, which is a tangible framework for performing pre-composed music rather than a tangible instrument [27]. The mixiTUI has been evaluated in terms of a concert attended by 117 persons where the feedback from the audience was collected in terms of questionnaires. In our research into tangible interfaces we have a similar interest in understanding how the Radar Table is used but in contrast to the research mentioned above we have based our evaluation on video recordings of people using the interface.

Our study of the Radar Table build on previous research into tangible interfaces with a focus the dynamics of interaction as discussed by Nielsen and co-authors [24] in a study of a tabletop with two sets of boxes with Lego Bionicle figures. Their study is based on four hours of video recordings and the analysis of the data identifies a number of interaction forms concerning the initiation and social interaction relevant to an understanding of the interactive table.

As will be presented in more detail in the methodology section our study is based on using video in order to gain insight into what Fernaeus [7] has labeled an action centric perspective on interaction together with a concern for supporting sharable use, rather than primarily individual use.

METHODOLOGY In general we employ a research through design approach, recognizing the dynamic nature of this approach [1]. As such, we are interested in capturing interaction and use, and utilize a detailed analysis of this as a springboard for further research. This explains the different iterations of the Radar Table discussed below, and also explains our motivation for

the detailed study employed here – we want to understand in depth how and why it works. The Radar Table has been our research instrument for an exploration the dynamics of tangible tabletops as material interaction.

In the case examined, the Radar Table was deployed ‘in the wild’ at a major Danish music festival. All interactions were video recorded by a GoPro camera mounted high above the table and at the same time we recorded both the sound generated and people’s conversation. This gave us access to roughly six hours of video material that we watched, identifying instances of single and group interactions. This enabled us to select a few examples for detailed analysis here, since we do not have the space here to present a detailed analysis of all six hours of use. We have selected three sequences on the premise that they represent different ways of interacting with the table and instance of a certain complexity of interaction.

THE RADAR TABLE The Radar Table developed by our research laboratory, [11], consists of a translucent table surface (80 cm × 107 cm) under which a projector and a camera is mounted. The tangible objects are equipped with visual markers and tracked by the camera connected to a computer, which together with our custom designed software identifies the position and rotation of each tangible object fitted with visual markers beneath their bases. The projector beneath the table displays visuals on the table.

The Radar Table was designed starting out from four design principles:

1. Support walk up and use

2. Encourage exploration

3. Support both individual and social play

4. Enable emergent use

Based on these four principles, Radar Table was designed with very basic interaction in mind. We operate with two kinds of tangibles: cubes and cylinders. Associated with each side of the cubes we have a single music loop and when introduced on the table the loop associated with the side of the cube facing the table is played from the start. By rotating the cube the volume may be controlled. In total we have seven cubes and with each of the six sides associated with a single loop. The only visual identifier on the cube surfaces is the visual marker used for tracking of the object. To encourage exploration it has been a conscious design decision not to support users in selecting a specific loop. Each loop is of a different length ranging from four to 31 bars with the implication that relation between loops changes over time.

The loops are produced with two parameters in mind, key and rhythm. The keynotes of each loop all belong to a contemporary western music tradition, though in some instances slightly dissonant, and some of the loops are in


minor and other in major. Moreover the rhythm or time signature of the loops fits together, for instance 3/4 time or 4/4 time, which offers the opportunity to play poly-rhythmic music. Those loops that do not match well are associated with the same cube hereby excluding that they are played at the same time.

Some of individual loops are based on unfinished materials and ‘darlings’ from the composer’s archive, which he has earlier produced and often put a lot of work into but not included in any of his works. Each of the samples has been modified to fit the total collection of samples and supplemented with additional newly produced material. The process of producing the loops resembles in many ways a conventional composition process of an exploratory nature with elements of trial and error. The goal was to provide a consistent multifarious universe with several music styles.

Figure 2 - Radar Table deployed at the SPOT Festival

The cylinders represent sound effects and when introduced to the table, the sound effect affects the sound played by cubes nearby the cylinder. The sound effects are implemented using the music software Ableton Live 8 (www.ableton.com). A total of eight sound effects are available:

1. Low pass filter: cuts frequencies below a specified frequency,

2. Ring modulator: frequency shifter,

3. Resonator: intensification and prolongation of a sound,

4. Distortion: distortion of a sound,

5. Delay: records an input signal and plays it back after a period of time,

6. Reverb: simulates the acoustics of real or imaginary rooms,

7. Beat repeat: creative beat mangling with controlled or randomized repetitions of an incoming signal,

8. Bit crusher: sample-rate reduction.

By rotating a cylinder a parameter of the effect is modified as for instance in the case of distortion where rotating the cylinder controls the frequency interval affected by the effect. Moreover the power of an effect is depended on the distance to cubes.

The Radar table has undergone a very active life outside of the laboratory: Screen Media Expo London 2012, Infocom Las Vegas 2012, Multitouch Helsinki 2012, LEGO World Copenhagen 2012 and 2013, CHI 2013 Paris, and SPOT Interactive Aarhus 2013.

Comparison with Mixi Tui and Reactable Reactable, Mixi Tui, and Radar Table are all tangible tabletops for music generation. Reactable is a kind of synthesizer where each of the tangibles represents a synthesizer component for either generating or modifying sounds. Both Mixi Tui and Radars Table are rather sequencers storing sequences of musical nodes (loopes/sound samples) where some of the tangibles represent sound samples and other tangible represents sound effects. Based on Mixi Tui presentations available on the Internet [mixitui.com] we would claim Mixi Tui offers a more narrow range of sound samples than the collection of sound samples available at the Radar Table. In this way Radar Table provides the opportunity to create more diverse music than Mixi Tui, and Reactable being the most generic platform, a synthesizer.

The Radar Table and Reactable both use cubes for generating sound, but whereas the Radar Table has only one shape of effect-tangible, cylinders, each of the Reactable effects has its unique two-dimensional shape. All Mixi Tui tangibles are two-dimensional objects belonging to three categories: Loops, Controls and effect.

The visual material on the tabletop surface provides in all three cases feedback concerning the state of the tangible object (e.g. volume) as well as the relation between the tangibles. In general the tabletop surface of the Radar Table is much less complex that the ones of the two others, Mixi Tui being the most complex one, for instance with visually marked areas on the tabletop surface dedicated ways of controlling the sound production.

MATERIALS AND MATERIALITY IN HCI Our understanding of the Radar Table as a material interface to sound may be positioned in relation to ongoing discussions of materials and materiality in HCI.

Historically, the material of interaction design has been software in sense that software were programmed and then presented on a screen. But as the computer gradually reached out [10] a plethora of theoretical foundations dealing with the numerous new configurations of software and hardware emerged [28]. At the same time the distinction between software and hardware, perhaps always contentious, became blurred.


Löwgren and Stolterman [20] has conceptualized information technology as a material without qualities and likewise Ozenc et al. [25] casts software as having immaterial material qualities, which can be difficult for designers to work with. According to Ozenc et al. [25], the variety of configurations and possibilities of software and hardware combinations pose new challenges for designers.

Vallgårda and Redström [34] discuss information technology as computational composites and considers software, hardware and physical things as materials. Computational composites are blends of different materials that, like other composite materials, may be combined to give rise to new unique qualities, which are more than just the sum of the parts. As an example, the smart material [19] muscle-wire is a is an extremely thin wire made from nitinol (a nickel-titanium alloy) that is known for its ability to contract when an electric current is applied. By combining this wire with computational devices such the Lilypad Arduino [3], one has a way of controlling the state of a material [34].

In essence, this range of materials and ways of combining them has almost limitless applications. Jung and Stolterman [16] highlights how the pervasiveness of computational technology means that we now consider computers as materials for interaction design. They discuss how interaction design and HCI have previously given preference to functionality as compared to form and material qualities, leading to minimal physical expressions as embodied by the touchscreen. This runs counter to product design, architecture and industrial design, where materiality and selections of material have been vital. Thus according to Jung and Stolterman [16] there is an opportunity for furthering interaction design and HCI by exploring the material qualities and possibilities of computational technology as a material. This leads them to suggest a study using a material probe, a kind of cultural probe, containing different physical materials, which recipients can then compare to digital materials in their local area.

Building on these developments, our work here can be seen as a way of exploring one specific material, tangible tabletops producing interactive sound, while at the same time developing theory to discuss its use and properties.

PRAGMATISM In order to provide a theoretical foundation for our analysis of tangible tabletops and interactive sound in HCI and interaction design we turn to Dewey’s pragmatism as well as recent efforts to appropriate pragmatism for interaction design research ([4],[9]). Our ambition is not just to build on their work, but develop theory that can serve in a descriptive role [28], for designers and researchers offering utility beyond the specific case of the Radar Table. We do this by developing a set of theoretical concepts for understanding how the material nature of the Radar Table

allows users to experience and play with sound thus building on the work already done by others in appropriating pragmatism for use in interaction design.

One of the most prominent pragmatism scholars is Donald Schön, whose conceptualization of designing as a reflective conversation with the materials of a design situation [29, 30] has been very influential, and many of the discussions of materiality and materials from the previous section are indebted to this definition, either implicitly or explicitly. More generally speaking, a strong body of research has highlighted how pragmatism can serve as contribution to the theoretical foundations of HCI and interaction design (e.g. [4, 9, 23, 26] ).

All of these approaches build on one of the most prominent pragmatist philosophers, John Dewey (1859-1952), who has contributed to a wide range of discussions within philosophy of education, psychology, journalism and ethics. In the following we build on primarily on the elaborations of Deweys work found in [9, 4, 12] to outline how the theory of inquiry [6] can contribute to conceptualizing the Radar Table as a material installation that makes interactive sound available for users. We do this by offering three distinct concepts: kinds of problematic situations encountered; inquiry strategies utilized by users to resolve these problematic situations; and how inquiry is always supported by technology.

Problematic situations Dewey sees life as a constant stream of experiences. Most of these experiences do not stand out and give rise to inquiry. They are non-reflective [12] – we are able to pass by them by without focusing specifically on any aspect of them, such as for instance riding a bike. However sometimes problematic situations occur, that requires us to enter into a mode of inquiry. It is important to emphasize that Dewey use the term “problematic” in a very broad sense; it is simply a situation that has a wide range of possible outcomes, and therefore requires an investigation or decision to be made in order to progress. Thus, listening to a poem and considering the interpretation of the lyrics is also considered “problematic” in Deweyan terms, in that inquiry is the act of interpreting the lyrics. Based on this, we want to examine the categories and traits of the different problematic situations that the users of the Radar Table encounter through use. By doing so, we enable ourselves and others to start asking questions about which problematic situations are desirable and undesirable, and how these are supported by the technology available.

Inquiry strategies When we encounter a problematic situation, we begin to formulate hypotheses and try them out in order to resolve the problematic situation. In a pragmatist view, we progress experimentally through inquiry by transforming the constituents of the problematic situation. For instance, a problematic situation such as the one with interpreting


poem might be resolved in several steps of transformations. First the reader might look up the multiple meanings of a word thus transforming himself through learning, before proceeding by writing down different metaphors, before finally arriving at an interpretation he is happy with. In inquiry each step builds upon the former– it is not the case that transformations are discrete steps that are preplanned. Rather they are what Schön [30] calls experimental moves, all designed to test out the usefulness of for instance a given interpretation, or theory about the meaning of the poem in question.

Technology supporting inquiry Inquiry unfolds through transformations and formations of hypotheses. According to Dewey, all kind of inquiry is supported by technology and in pragmatism it is a vital point that technology is anything that scaffolds inquiry [4].

Technology in a pragmatist understanding of the word can be said to take on dual meaning and role, representing both an aspect of tangible tabletops as material, and as building blocks and tools of moving the inquiry forward. In pragmatism, technology is in the understanding of [12] shaping as well as being shaped by use in action. [9] highlights how this can be seen as making the world part of the cognition, drawing our attention to how thinking and doing are closely intertwined, which lends itself well to an analysis of a complex social phenomena like the Radar Table.

ANALYSIS In the previous section we have introduced three basic concepts from pragmatism: problematic situation, inquiry strategies, and technology, which we now apply in the analysis of three examples of using the Radar Table at the SPOT festival. The three examples have been selected in order to investigate how a pragmatist perspective can elucidate the ways in which the material qualities of the Radar Table made interactive sounds available for experimentation.

For each example we first describe the situation as it unfold and then analyze each of them through the employment of the three concepts outlined above.

Example one – exploring the Radar Table The first example one of initial exploration of the table, answering the dual set of questions of what the table is, and what the interplay of sounds and tangible objects are.

A younger female who has been lurking around the installation while another user was trying it out. Together with her are two males and one female friend, standing in a group in the periphery of the installation. She walks up to the table and turns over one of the sample-cubes as her friend approaches. This produce a subtle change of the Radar Table as her action changes the loop being played. However, since there are currently a lot of other sample-

cubes left over from the previous user, she seems not to notice this. She then tries rotating one of the effect-cylinders, to see if that does anything. Since it is an effect-cylinders that is not currently close enough to a sample-cube her action has no influence, but her initial experimentation has drawn in some of her friends so now five people experimenting with the table at the same time. They do this by playfully, turning over cubes, rotating them, and moving them around, while chatting to each other.

The young female takes the lead, at one point sharing some of her discoveries with her friends - as an answer to the question of whether the "music is made by the table?" she says "yeah yeah, look, when I lift this, the music stops!" as she lifts up one of the sample cubes. Having unraveled this relationship between the cubes and the music playing, the group removes everything from the table. As the music stops, a few whoops are uttered, serving as confirmation of the pleasure of having figured out the relationship between the tangibles and the music playing.

The problematic situation encountered by the users at this initial encounter with the Radar Table is somewhat open and confusing. Since the young female who initiates use has little understanding of how the table works, she seems to be posing the question of “what is this?” as well as “what can I do here?” The traits of this problematic situation highlights the fact that in pragmatism, the term situation denotes the whole context of use – in this case both the table, the tangibles and the other people the young female collaborates with.

The users can then be said to utilize an inquiring strategy that is aimed at answering these two questions posed in the problematic situation. By turning over cubes and ultimately removing them all, they are employing an inquiry strategy that consists of more than just thinking – it is a physical strategy as well, in which thought and manipulation of the tangibles are closely intertwined. The group proceeds experimentally, showing how their inquiry strategy is not a definite and ordered testing, but rather different attempts at seeing how the table functions. These early experiments are reminiscent of Schöns exploratory moves [30], in that they aim to answer the question of “what happens when I do this?”

The material configuration of the Radar Table shows how, in a pragmatist view, technology supports and mediates inquiry. Both the constitution of the problematic situation as well as the support for the inquiring strategy is partly constituted by the material configuration of the Radar Table. Considering the material of the Radar Table as pragmatic technology, it becomes clear that the transformations offered and supported by the layout of the table are both what enables and constrains the actions of the users. As they move through resolving their problematic situations, the different options and possibilities of the Radar Table opens up.


In the next example, we exemplify how the same users build on this accumulated knowledge to begin experimenting more systematically with the materials at hand at the Radar Table.

Example two – collaborative composing This second example concerns the exploration of the possibilities of producing music collaboratively using the Radar Table through the shared interface. It shows how this shared surface can be seen to both inhibit and enable certain types of inquiry. Following the same group as in the previous example we show how their understanding of the table develops, and how they begin to experiment with the different sounds available at the Radar Table.

Having developed a basic understanding of the relationship between sound and the cubes on the table, the participants proceed more systematically than before. One participant starts out by placing down an effect-cylinder on the now empty table, which does nothing in itself, as it requires a sound-cube. Another participant pushes the effect-cylinder away, and places a sound-cube on the table instead, which results in a drum-kick emanating from the speakers of the Radar Table. The participants continue to take turns adding or removing tangibles – it is not the case that they systematically take turn, but rather that each action by one participant offers opportunities for another.

For example, when one participant added a sound-cube, another participant would connect to the sound-cube by adding and rotating an effect-cylinder. Rotating the effect-cylinder turns up or down the power of the effect, demonstrating the effect of her move. This means that doing something to the Radar Table through a combination of physical manipulation changes in the overall soundscape.

The problematic situation differs from the previous example in that the participants are aware of the fact that the different cubes allow them to change the music coming from the table. Inquiry then moves forward through experimentation, as seen by the collaborative adding and removal of sample-cube effect-cylinders Again, we are reminded that in pragmatism, situation encompasses both the physical and digital materials, as well as the other people present at the given point in time. As such, the group seeks to answer questions on what the interplay between the different kinds of tangibles is, as well as how they can be manipulated.

The problematic situation is then resolved through different inquiring strategies, attempts at experimentation with and transforming of the constituents of the situation. One such strategy consists of simply discussing theories about the workings of the table, suggesting improvements to the current sounds playing. In other instances, the movement of a cube seems to inspire a participant to an action, like adding for instance a reverb to a drum-sound just placed by someone else, thus building on their actions.

Figure 3 - Group interaction around the Radar Table

Our second example also demonstrates how Radar Table’s technology supports inquiry - it offers up opportunities for specific transformations of the problematic situation, but at the same time inhibit the potential moves. Not everyone can physically manipulate the same sample-cube at a given time, something that constricts the kind of music that can be created. This actually worked to the advantage of the participants in our example here, in that it nudged them towards collaboration, creating a social and dynamic experience, rather than four single and isolated experiences.

Example 3 – a solo performance In this example we focus on a lone user of the Radar Table. Through his careful and playful exploration of the possibilities of different sounds in combination, he can be said to enjoy the full technological capabilities of the Radar Table. By doing so, he shows how inquiry can be a focused as well as joyful exploration of sound through the Radar Table.

A young man approaches the table just as two others are leaving it and immediately begins to place different sample-cubes on the table one by one, removing them when they do not produce the sound he seems to appreciate. He takes his time to turn over the sample-cubes, selects a sample from each cube and modifies the volume turning each of the cubes. In contrast to the two previous examples this situation is more fluid and dynamic, presumably as there is only one person interacting with the table. Each change is made, as the young man listens to the music coming out of the speakers, beginning to add effect-cubes, twisting and turning them. He seems not to be heading towards a clear goal, but rather experimenting and playing with the soundscape. After a few minutes he starts to turn effect-cubes up and down, “riffing” the different samples, by for


instance turning the reverb on a baseline up and down, much like a DJ would be mixing at live-set.

Figure 4 - Solo use of the Radar Table

He keeps on manipulating the tangibles, adding or removing sample-cubes, making quite subtle changes, as well as playing with effect-cubes by adjusting the effect up and down. After about five minutes he leaves the table as others arrive.

The young man using the Radar Table in this way can be said to enjoy the full capabilities of the technology, due to his competence in using the interface as well as desire to create a coherent musical performance. The problematic situation for him is something much different from the one faced by the group in the two previous examples. He is engaged in creating a pleasing coherent soundscape, as well as exploring the capabilities of the Radar Table.

Perhaps due the fact that he is alone at the table his inquiry strategies are different. He initiates inquiry by first exploring the different potential sounds, gradually letting each sound transform the situation into an emerging soundscape. Each added or modified cube or cylinder is a move that transforms the situation through an on-going inquiry. These inquiring strategies are defined and supported by the technology of the Radar Table. While it is obvious that the experience had by this young man differs from the two previous examples due to his point of departure, it is still the case that the Radar Table constrains and enables the way his inquiry can unfold. So, the young man’s experience is one where the focus of inquiry is on the music itself through the Radar Table, as opposed to the previous focus on the Radar Table directly.

THE DYNAMICS OF INTERACTIVE SOUND We have been investigating three specific examples of use, representing a diversity of situations. All three examples show how users of the Radar Table have been able to, in different ways, experiment with interactive sound. We examined the three examples have been analyzed through our pragmatist framework, focusing specifically on the

concepts of problematic situations, inquiry strategies, and technology supporting inquiry. Here we gather and reflect on the findings for each of these categories, before going on to discuss our future.

Problematic situations By taking as the point of departure the idea that certain situations stand out in the flow of experience, it makes sense to ask about the character of the situations encountered when using the Radar Table.

A key point here is, that it is, in the view of pragmatism, not a feature of the Radar Table in itself that creates the problematic situations. Rather, a situation in a pragmatist view is the totality at that exact moment in time, meaning that as inquiry unfolds through learning, experimentation and conversation with others, the problematic situation changes.

• What can I do here?

• What is the interplay between the table and sound?

• What is the relationship between the different tangibles?

• How can I manipulate individual sounds?

• What would make the current soundscape sound better?

• What would I do if this were a "live performance"?

Figure 5 - Problematic situations

In example 2, when the group has understood that the tangibles affect the sounds from Radar Table, they find themselves in a changed problematic situation. This highlights a key advantage in using pragmatism as a framework in HCI and design, one that Dalsgaard also discusses [4]. Since pragmatism conceptualizes situations as always in flux and changing through inquiry, it seems well suited to analyze situations of emergent situations. Such situations are rife when interacting with the Radar Table – indeed, it is designed to be an emergent experience, in that it does not have a set start and end, as our examples above also show.

Inquiry strategies

Having encountered a problematic situation, the users will, depending on their own point of departure, attempt to resolve them in different ways, through working their way through inquiry experimentally. It then makes sense to ask which different kinds of inquiry for resolving the situations were employed throughout the examined case.

Broadly speaking, it seems that there are two different approaches, one involving experimenting with the tangibles, and one consisting of collaborating with others.


• Turn over tangibles

• Add/remove tangibles

• Remove all tangibles

• Slowly and methodically tweak tangibles

• Observe others

• Converse with others

Figure 6 - Inquiry strategies

Returning to our pragmatist outset, this can also be seen as a form of experimentation. By conversing with others, one is inquiring into the situation, establishing a Schönian conversation with the material [30]. This highlights how an installation like the Radar Table can utilize both its tangible and visual assets as well as the opportunity and gentle nudge towards socializing that is built into the design. Emergence and experimentation is at the heart of our installation.

Technology supporting inquiry The Radar Table offers a simple material interface providing opportunities for complex sound experiences. While it is easy to move the cubes and cylinders around creating different soundscapes it is also an open-ended experience. The Radar Table enables and constrains the potential transformations of the situation in ongoing inquiry. Going back to our two previous concepts, problematic situations and inquiry strategies, it is clear that both are constituted by technology – the Radar Table is an important constituent element of the problematic and at the same time resource for inquiry.

• People

• Sound

• Tangibles

Figure 7 - Technology supporting inquiry

However, the users are an integral part of problematic situation and inquiry strategy as well, as their expectations and ideas is part of the dynamic flow of the situation. Furthermore their collaboration, while not explicitly technology in a traditional sense, can actually be considered technological in a pragmatist understanding of the word. Technology in pragmatism is, according to [12] what supports inquiry. In that sense, other people around the table can be considered a kind of technology, as well as previous knowledge of making music, experience with tangibles, and even language. However we have chosen to focus squarely on the interactions between users, other people and tangibles, and sound.

Conclusion In this paper we have started out from the growing interest in material practice within HCI and applied as well as

developed three concepts from the philosophy of pragmatism in order to gain insight into the unfolding dynamics of using the materials of a tangible tabletop.

Our research has been limited by looking into a single case and by analyzing in detail only three specific interaction sequences of a few minutes length. We have examined the three sequences in certain detail, and we are confident in the potential of developing a more systematic model based on pragmatism for more detailed analysis.

Such a model could enable comparison of use of installations, for the Radar Table, Reactable, and mixiTui, in order to gain further insight into sound as material. In a more general perspective such a model could be applied and developed in other areas as well, for instance interaction processes in general as well as design processes.

REFERENCES [1] Basballe, D.A. and Halskov, K. 2012. Dynamics of Research Through Design. Dynamics of Research Through Design. Proceedings of the Designing Interactive Systems Conference. ACM. 58-67.

[2] Belenguer, J.S., Lundén, M., Laaksolhati, J. and Sundström, P. 2012. Immaterial materials: designing with radio. Proceedings of the Sixth International Conference on Tangible, Embedded and Embodied Interaction (2012), 205-212.

[3] Buechley, L., Eisenberg, M., Catchen, J. and Crockett, A. 2008. The LilyPad Arduino: using computational textiles to investigate engagement, aesthetics, and diversity in computer science education. Proceedings of the twenty-sixth annual SIGCHI conference on Human factors in computing systems. 423-432.

[4] Dalsgaard, P. 2009. Designing engaging interactive environments: A pragmatist perspective. Aarhus University, Department of Aesthetics and Communication-Information and Media Studies.

[5] Dalsgaard, P. and Halskov, K. 2012. Tangible 3D tabletops: combining tangible tabletop interaction and 3D projection. Proceedings of the 7th Nordic Conference on Human-Computer Interaction: Making Sense Through Design. ACM. 109-118.

[6] Dewey, J. 1925. Logic: The Theory of Inquiry (1938). The later works. 1953, (1925), 1-549.

[7] Fernaeus, Y., Tholander, J. and Jonsson, M. 2008. Towards a new set of ideals: consequences of the practice turn in tangible interaction. Proceedings of the 2nd international conference on Tangible and embedded interaction. ACM. 223-230.

[8] Fernaeus, Y. and Sundström, P. 2012. The material move how materials matter in interaction design research. Proceedings of the Designing Interactive Systems Conference. ACM. 486-495.


[9] Gedenryd, H. 1998. How designers work: making sense of autentic cognitive activities. Lund University.

[10] Grudin, J. 1990. The computer reaches out: the historical continuity of interface design. Proceedings of the SIGCHI conference on Human factors in computing systems: Empowering people. 261-268.

[11] Halskov, K. 2011. CAVI - An interaction design research lab. interactions, 18(4), 92-95.

[12] Hickman, L. 1992. John Dewey's pragmatic technology. Indiana University Press.

[13] Ishii, H. and Ullmer, B. 1997. Tangible bits: towards seamless interfaces between people, bits and atoms. Proceedings of the ACM SIGCHI Conference on Human factors in computing systems. 234-241.

[14] Jordà, S. 2010. The reactable: tangible and tabletop music performance. CHI'10 Extended Abstracts on Human Factors in Computing Systems. 2989-2994.

[15] Jordà, S., Geiger, G., Alonso, M. and Kaltenbrunner, M. 2007. The reacTable: exploring the synergy between live music performance and tabletop tangible interfaces. Proceedings of the 1st international conference on Tangible and embedded interaction. ACM. 139-146.

[16] Jung, H. and Stolterman, E. 2011. Form and materiality in interaction design: a new approach to HCI. CHI '11 Extended Abstracts on Human Factors in Computing Systems. ACM. 399-408.

[17] Jung, H. and Stolterman, E. 2011. Material probe: exploring materiality of digital artifacts. Proceedings of the fifth international conference on Tangible, embedded, and embodied interaction. ACM. 153-156.

[18]Kaltenbrunner, M. and Bencina, R. 2007. reacTIVision: a computer-vision framework for table-based tangible interaction. Proceedings of the 1st international conference on Tangible and embedded interaction. ACM. 69-74.

[19] Kamila, S. 2013. Introduction, Classification and Applications of Smart Materials: An Overview. American Journal of Applied Sciences. 10, 8 (Jul. 2013).

[20] Löwgren, J. and Stolterman, E. 2007. Thoughtful Interaction Design: A Design Perspective on Information Technology. The MIT Press.

[21] Marco, J., Cerezo, E. and Baldassarri, S. 2012. ToyVision: a toolkit for prototyping tabletop tangible games. Proceedings of the 4th ACM SIGCHI symposium on Engineering interactive computing systems. 71-80.

[22] McCrindle, C., Hornecker, E., Lingnau, A. and Rick, J. 2011. The design of t-vote: a tangible tabletop application

supporting children's decision making. Proceedings of the 10th International Conference on Interaction Design and Children. 181-184.

[23] Melles, G. 2008. An Enlarged Pragmatist Inquiry Paradigm for Methodological Pluralism in Academic Design Research. ARTIFACT. 2, 1 (Nov. 2008), 3–11.

[24] Nielsen, R., Fritsch, J., Halskov, K. and Brynskov, M. 2009. Out of the box: exploring the richness of children's use of an interactive table. Proceedings of the 8th International Conference on Interaction Design and Children. 61-69.

[25] Ozenc, F.K., Kim, M., Zimmerman, J., Oney, S. and Myers, B. 2010. How to support designers in getting hold of the immaterial material of software. Proceedings of the 28th international conference on Human factors in computing systems. ACM. 2513-2522.

[26] Östman, L.E. 2005. A pragmatist theory of design: The impact of the pragmatist philosophy of John Dewey on architecture and design. Karlstad University.

[27] Pedersen, E.W. and Hornbæk, K. 2009. mixiTUI: a tangible sequencer for electronic live performances. Proceedings of the 3rd International Conference on Tangible and Embedded Interaction. ACM. 223-230.

[28] Rogers, Y. 2012. HCI theory: classical, modern, and contemporary. Synthesis Lectures on Human-Centered Informatics. 5, 2 (2012), 1-129.

[29] Schön, D.A. 1992. Designing as reflective conversation with the materials of a design situation. Knowledge-Based Systems. 5, 1 (1992), 3-14.

[30] Schön, D.A. 1983. The reflective practitioner: how professionals think in action. Basic Books.

[31] Shaer, O. and Hornecker, E. 2010. Tangible user interfaces: past, present, and future directions. Foundations and Trends in Human-Computer Interaction. 3, 1--2 (2010), 1-137.

[32] Shneiderman, B. 1993. 1.1 direct manipulation: a step beyond programming languages. Sparks of Innovation in Human-Computer Interaction. 17, (1993).

[33] Sundström, P., Taylor, A.S. and O'Hara, K. 2011. Sketching in software and hardware Bluetooth as a design material. Proceedings of the 13th International Conference on Human Computer Interaction with Mobile Devices and Services. ACM. 405-414.

[34] Vallgårda, A. and Redström, J. 2007. Computational composites. Proceedings of the SIGCHI Conference on Human Factors in Computing Systems. ACM. 513-522.

material interactions with tangible tabletops: a pragmatist perspective

Documents