dynamic physical modeling for designing music interactions
TRANSCRIPT
SIGCHI Extended Abstracts Sample File: Note Initial CapsEdgar
Berdahl Louisiana State University Baton Rouge, LA, USA
Eric Sheffield Louisiana State University Baton Rouge, LA, USA
The author(s) retain copyright, but ACM receives an exclusive publication license.
Abstract Physical models can be employed to design music interac- tions as well as other human-computer interactions. Synth- A-Modeler is proposed for compiling physical models into a wide range of targets. Synth-A-Modeler is typically em- ployed to program force-feedback haptic devices such as the FireFader, Falcon, and Plank devices; however, it is easily extensible to other devices, including touchscreens augmented with vibrotactile tokens. It is proposed to collab- orate on any of the many research contexts in which Synth- A-Modeler is useful, including: designing new widgets for touchscreen GUIs, designing new musical instruments, per- ception/cognition experiments, composing music, etc.
Author Keywords Virtual models, physical models, haptic interfaces, electro- magnetic actuator design, Synth-A-Modeler, Faust audio
ACM Classification Keywords H.5.5 [Information interfaces and presentation]: Sound and Music Computing; H.5.2 [Information interfaces and presen- tation]: Haptic I/O
Introduction Physical models can provide rich metaphors for designing music interactions as well as other human-computer inter- actions. If a computer simulates some aspect of a virtual
physical system, then human users will tend to have some intuitions about how to successfully interact with it. The “desktop metaphor” at Xerox PARC provided an early test bed for this, with users identifying how moving icons around in a virtual environment could correspond to relocating files on a hard disk drive [6]. However, back then, physics simu- lations were limited by the relatively small amounts of com- putation available.
Name Symbol
Mass-like objects
Table 1: Currently implemented objects in Synth-A-Modeler.
Flashing forward to today, laptop, desktop, embedded and mobile computers are orders of magnitude more power- ful. This enables modern computers to simulate complex physical models in real time, even at audio sampling rates. There are now many opportunities for incorporating these into user interactions, but first they need to be categorized, studied, expanded, and evaluated in greater detail.
Consider the following list of some gestures commonly available on commercial multitouch devices of today: click, tap, drag, stretch, slide/swipe, twist, pinch, magnify, etc. In addition, on commercial devices, a small number of ges- tures are emerging that incorporate virtual inertias, such as bouncing or shaking icons. However, given that so many different virtual objects could be incorporated to enhance the dynamics (see Table 1), it can be argued that the future opportunities for richer dynamics are vast. This argument is strengthened by the trend of new mobile computers inte- grating a wide variety of low noise sensors. These sensors can aid in precisely realizing interactions in multiple de- grees of freedom. Consider for example the iPhone 6S+, which can not only detect the positions of multiple fingers simultaneously, but it can also estimate the individual down- ward pressure applied by each these fingers.
Virtual Physics Modeling Using Synth-A-Modeler Synth-A-Modeler takes descriptions of physical models and compiles them into signal flow diagrams [3]. For exam- ple, the objects in Table 1 can be interconnected, such as shown in Figure 1. Completed models can be compiled into a wide variety of target formats, as indicated in Figure 2.
Figure 1: Via the port object, the user plucks a mass (in red) that bounces in between a resonator “modal synthesis” object (in purple) and a digital waveguide string (in blue).
Synth-A-Modeler integrates the three previously particularly popular physical modeling paradigms in computer music: digital waveguide modeling [7], mass-link modeling [4], and modal synthesis [1]. This makes it straightforward to imple- ment models that incorporate all three kinds of objects to discover new interactions such as exemplified in Figure 1.
Overview of Haptic Control Using Synth-A-Modeler Besides providing audio output, models from Synth-A- Modeler can provide graphical output (see the display object in Table 1) or haptic control signals (see the port ob- ject in Table 1). These haptic control signals can be used for controlling force-feedback devices such as the Fire- Fader, the NovInt Falcon, the Plank, or other impedance- controlled haptic device. However, they can alternatively be used for driving less expensive vibrotactile actuators. The next section describes a method we use to interface Synth- A-Modeler models with high-resolution sensors on mobile devices, which are further augmented using custom vibro- tactile haptic actuators. The goal is to provide a wireless
Figure 2: Once a physical model is encoded as an MDL “model” file, Synth-A-Modeler and Faust can compile it into many different targets in C, C++, Java, or JavaScript. The toolchain is completely open source, and Synth-A-Modeler is accompanied by a series of models introducing its capabilities and syntax.
technology for specific, vibrotactile-enhanced interactions that operate with high fidelity over a wide frequency range.
Vibrotactile Tokens for Touchscreen Interaction with Virtually Vibrating Constraints To create a vibrotactile actuator that is wireless from the point of view of the user, and to interface the user with the actuator in a high-fidelity fashion, it has been decided for the user to effectively hold on to a magnet (e.g. rotor). To enhance the interaction, the magnet is housed within a 3D- printed, conductive “token” (see Figure 3) that can supplant the role of a finger pressing on the touchscreen. These to- kens are unconstrained [8], with the idea being that haptic signals can help indicate virtual constraints. Each token is designed for a specific (music) interaction (e.g. a plectrum, bow, ring, dial, etc.). The electromagnetic design is novel compared to prior HCI systems. For example, only a single coil is employed for simple vibrotactile stimulation [9], and the actuator is responsive over a wide frequency range [10] even if the finger is not moving [2].
The electromagnetic design works as follows. A touch- screen is laid flat on a table, a coil is placed around the perimeter of the touchscreen, and a current is made to run counterclockwise around the screen perimeter (into the page on the right and out of the page on the left in Fig- ure 3). If a permanent magnet in is placed on top of the touchscreen in the field (see the “N S” (purple) rectangle in Figure 3), a torque and force can be exerted on it [5]. For example, a dipole magnetic moment vector u can be drawn progressing from the south pole S of the magnet through the north pole N of the magnet. The exerted torque (see the dash-dotted red arrow in Figure 3) will primarily serve to rotate the magnet so that its vector u aligns locally with the magnetic field induced by the solenoid (see the simulated field lines with many black arrows in Figure 3).
Figure 3: Magnetic design of a wireless, vibrotactile token. The permanent magnet wants to rotate to align its magnetic moment u with the field lines (in black with arrows) induced by the coil.
Figure 4: Synth-A-Modeler integrates the digital waveguide, mass-interaction, and modal synthesis physical modeling paradigms.
Results So Far Some of the tokens created so far are shown in Figures 5 and 6. Music interactions that have been implemented in- clude 1) using the pressure sensitivity of an iPhone 6S+ to control the pluck stiffness for plucking a virtual harp of strings (x-position adjusts the position of the plectrum along the length, causing a timbreal change), 2) using a dial token to rotationally pluck a virtual string (x-position adjusts pitch bend and y-position adjusts the brightness), and 3) using a joystick token (not shown) to crossfade among four syn- chronized rhythms. A user study is currently in progress, and a demonstration video can be viewed at https://www.cct.lsu.edu/~eberdahl/VibrotactileTokens.m4v Finally, the authors look forward to composing new music for the wireless vibrotactile tokens.
Figure 5: A wireless, vibrotactile plectrum token for touchscreen interactions.
Figure 6: A wireless, vibrotactile dial token for touchscreen interactions.
References [1] Jean-Marie Adrien. 1991. The Missing Link: Modal
Synthesis. MIT Press, Cambridge, MA, USA, Chapter Representations of Musical Signals, 269–297.
[2] Olivier Bau, Ivan Poupyrev, Mathieu Le Goc, Laureline Galliot, and Matthew Glisson. 2012. REVEL: Tactile Feedback Technology for Augmented Reality. In ACM SIGGRAPH. Los Angeles, CA, USA, Article 89, 11
pages. [3] Edgar Berdahl and Julius O. Smith III. 2012. An Intro-
duction to the Synth-A-Modeler Compiler: Modular and Open-Source Sound Synthesis using Physical Models. In Proceedings of the Linux Audio Conference. Stan- ford, CA, USA.
[4] Claude Cadoz, Annie Luciani, and Jean-Loup Florens. 1993. CORDIS-ANIMA: A Modeling and Simulation System for Sound and Image Synthesis—The General Formalism. Computer Music Journal 17, 1 (Spring 1993), 19–29.
[5] Umran Inan and Aziz Inan. 1999. Engineering Elec- tromagnetics. Addison Wesley Longman, Menlo Park, CA.
[6] Bill Moggridge. 2007. Designing Interactions. MIT Press, Cambridge, MA.
[7] Julius O. Smith. 2010. Physical Audio Signal Process- ing: For Virtual Musical Instruments and Audio Effects. W3K Publishing, http://ccrma.stanford.edu/~jos/pasp/.
[8] Brygg Ullmer, Hiroshi Ishii, and Robert J. K. Jacob. 2005. Token+Constraint Systems for Tangible Inter- action with Digital Information. ACM Trans. Comput.- Hum. Interact. 12, 1 (March 2005), 81–118.
[9] Malte Weiss, Chat Wacharamanotham, Simon Voelker, and Jan Borchers. 2011. FingerFlux: Near-surface Haptic Feedback on Tabletops. In Proceedings of the 24th Annual ACM Symposium on User Interface Soft- ware and Technology (UIST ’11). Santa Barbara, Cali- fornia, USA, 615–620.
[10] L. Winfield, J. Glassmire, J.E. Colgate, and M. Peshkin. 2007. T-PaD: Tactile Pattern Display through Variable Friction Reduction. In Joint EuroHaptics Con- ference and Symposium on Haptic Interfaces for Vir- tual Environment and Teleoperator Systems. 421– 426.
Overview of Haptic Control Using Synth-A-Modeler
Vibrotactile Tokens for Touchscreen Interaction with Virtually Vibrating Constraints
Results So Far
Eric Sheffield Louisiana State University Baton Rouge, LA, USA
The author(s) retain copyright, but ACM receives an exclusive publication license.
Abstract Physical models can be employed to design music interac- tions as well as other human-computer interactions. Synth- A-Modeler is proposed for compiling physical models into a wide range of targets. Synth-A-Modeler is typically em- ployed to program force-feedback haptic devices such as the FireFader, Falcon, and Plank devices; however, it is easily extensible to other devices, including touchscreens augmented with vibrotactile tokens. It is proposed to collab- orate on any of the many research contexts in which Synth- A-Modeler is useful, including: designing new widgets for touchscreen GUIs, designing new musical instruments, per- ception/cognition experiments, composing music, etc.
Author Keywords Virtual models, physical models, haptic interfaces, electro- magnetic actuator design, Synth-A-Modeler, Faust audio
ACM Classification Keywords H.5.5 [Information interfaces and presentation]: Sound and Music Computing; H.5.2 [Information interfaces and presen- tation]: Haptic I/O
Introduction Physical models can provide rich metaphors for designing music interactions as well as other human-computer inter- actions. If a computer simulates some aspect of a virtual
physical system, then human users will tend to have some intuitions about how to successfully interact with it. The “desktop metaphor” at Xerox PARC provided an early test bed for this, with users identifying how moving icons around in a virtual environment could correspond to relocating files on a hard disk drive [6]. However, back then, physics simu- lations were limited by the relatively small amounts of com- putation available.
Name Symbol
Mass-like objects
Table 1: Currently implemented objects in Synth-A-Modeler.
Flashing forward to today, laptop, desktop, embedded and mobile computers are orders of magnitude more power- ful. This enables modern computers to simulate complex physical models in real time, even at audio sampling rates. There are now many opportunities for incorporating these into user interactions, but first they need to be categorized, studied, expanded, and evaluated in greater detail.
Consider the following list of some gestures commonly available on commercial multitouch devices of today: click, tap, drag, stretch, slide/swipe, twist, pinch, magnify, etc. In addition, on commercial devices, a small number of ges- tures are emerging that incorporate virtual inertias, such as bouncing or shaking icons. However, given that so many different virtual objects could be incorporated to enhance the dynamics (see Table 1), it can be argued that the future opportunities for richer dynamics are vast. This argument is strengthened by the trend of new mobile computers inte- grating a wide variety of low noise sensors. These sensors can aid in precisely realizing interactions in multiple de- grees of freedom. Consider for example the iPhone 6S+, which can not only detect the positions of multiple fingers simultaneously, but it can also estimate the individual down- ward pressure applied by each these fingers.
Virtual Physics Modeling Using Synth-A-Modeler Synth-A-Modeler takes descriptions of physical models and compiles them into signal flow diagrams [3]. For exam- ple, the objects in Table 1 can be interconnected, such as shown in Figure 1. Completed models can be compiled into a wide variety of target formats, as indicated in Figure 2.
Figure 1: Via the port object, the user plucks a mass (in red) that bounces in between a resonator “modal synthesis” object (in purple) and a digital waveguide string (in blue).
Synth-A-Modeler integrates the three previously particularly popular physical modeling paradigms in computer music: digital waveguide modeling [7], mass-link modeling [4], and modal synthesis [1]. This makes it straightforward to imple- ment models that incorporate all three kinds of objects to discover new interactions such as exemplified in Figure 1.
Overview of Haptic Control Using Synth-A-Modeler Besides providing audio output, models from Synth-A- Modeler can provide graphical output (see the display object in Table 1) or haptic control signals (see the port ob- ject in Table 1). These haptic control signals can be used for controlling force-feedback devices such as the Fire- Fader, the NovInt Falcon, the Plank, or other impedance- controlled haptic device. However, they can alternatively be used for driving less expensive vibrotactile actuators. The next section describes a method we use to interface Synth- A-Modeler models with high-resolution sensors on mobile devices, which are further augmented using custom vibro- tactile haptic actuators. The goal is to provide a wireless
Figure 2: Once a physical model is encoded as an MDL “model” file, Synth-A-Modeler and Faust can compile it into many different targets in C, C++, Java, or JavaScript. The toolchain is completely open source, and Synth-A-Modeler is accompanied by a series of models introducing its capabilities and syntax.
technology for specific, vibrotactile-enhanced interactions that operate with high fidelity over a wide frequency range.
Vibrotactile Tokens for Touchscreen Interaction with Virtually Vibrating Constraints To create a vibrotactile actuator that is wireless from the point of view of the user, and to interface the user with the actuator in a high-fidelity fashion, it has been decided for the user to effectively hold on to a magnet (e.g. rotor). To enhance the interaction, the magnet is housed within a 3D- printed, conductive “token” (see Figure 3) that can supplant the role of a finger pressing on the touchscreen. These to- kens are unconstrained [8], with the idea being that haptic signals can help indicate virtual constraints. Each token is designed for a specific (music) interaction (e.g. a plectrum, bow, ring, dial, etc.). The electromagnetic design is novel compared to prior HCI systems. For example, only a single coil is employed for simple vibrotactile stimulation [9], and the actuator is responsive over a wide frequency range [10] even if the finger is not moving [2].
The electromagnetic design works as follows. A touch- screen is laid flat on a table, a coil is placed around the perimeter of the touchscreen, and a current is made to run counterclockwise around the screen perimeter (into the page on the right and out of the page on the left in Fig- ure 3). If a permanent magnet in is placed on top of the touchscreen in the field (see the “N S” (purple) rectangle in Figure 3), a torque and force can be exerted on it [5]. For example, a dipole magnetic moment vector u can be drawn progressing from the south pole S of the magnet through the north pole N of the magnet. The exerted torque (see the dash-dotted red arrow in Figure 3) will primarily serve to rotate the magnet so that its vector u aligns locally with the magnetic field induced by the solenoid (see the simulated field lines with many black arrows in Figure 3).
Figure 3: Magnetic design of a wireless, vibrotactile token. The permanent magnet wants to rotate to align its magnetic moment u with the field lines (in black with arrows) induced by the coil.
Figure 4: Synth-A-Modeler integrates the digital waveguide, mass-interaction, and modal synthesis physical modeling paradigms.
Results So Far Some of the tokens created so far are shown in Figures 5 and 6. Music interactions that have been implemented in- clude 1) using the pressure sensitivity of an iPhone 6S+ to control the pluck stiffness for plucking a virtual harp of strings (x-position adjusts the position of the plectrum along the length, causing a timbreal change), 2) using a dial token to rotationally pluck a virtual string (x-position adjusts pitch bend and y-position adjusts the brightness), and 3) using a joystick token (not shown) to crossfade among four syn- chronized rhythms. A user study is currently in progress, and a demonstration video can be viewed at https://www.cct.lsu.edu/~eberdahl/VibrotactileTokens.m4v Finally, the authors look forward to composing new music for the wireless vibrotactile tokens.
Figure 5: A wireless, vibrotactile plectrum token for touchscreen interactions.
Figure 6: A wireless, vibrotactile dial token for touchscreen interactions.
References [1] Jean-Marie Adrien. 1991. The Missing Link: Modal
Synthesis. MIT Press, Cambridge, MA, USA, Chapter Representations of Musical Signals, 269–297.
[2] Olivier Bau, Ivan Poupyrev, Mathieu Le Goc, Laureline Galliot, and Matthew Glisson. 2012. REVEL: Tactile Feedback Technology for Augmented Reality. In ACM SIGGRAPH. Los Angeles, CA, USA, Article 89, 11
pages. [3] Edgar Berdahl and Julius O. Smith III. 2012. An Intro-
duction to the Synth-A-Modeler Compiler: Modular and Open-Source Sound Synthesis using Physical Models. In Proceedings of the Linux Audio Conference. Stan- ford, CA, USA.
[4] Claude Cadoz, Annie Luciani, and Jean-Loup Florens. 1993. CORDIS-ANIMA: A Modeling and Simulation System for Sound and Image Synthesis—The General Formalism. Computer Music Journal 17, 1 (Spring 1993), 19–29.
[5] Umran Inan and Aziz Inan. 1999. Engineering Elec- tromagnetics. Addison Wesley Longman, Menlo Park, CA.
[6] Bill Moggridge. 2007. Designing Interactions. MIT Press, Cambridge, MA.
[7] Julius O. Smith. 2010. Physical Audio Signal Process- ing: For Virtual Musical Instruments and Audio Effects. W3K Publishing, http://ccrma.stanford.edu/~jos/pasp/.
[8] Brygg Ullmer, Hiroshi Ishii, and Robert J. K. Jacob. 2005. Token+Constraint Systems for Tangible Inter- action with Digital Information. ACM Trans. Comput.- Hum. Interact. 12, 1 (March 2005), 81–118.
[9] Malte Weiss, Chat Wacharamanotham, Simon Voelker, and Jan Borchers. 2011. FingerFlux: Near-surface Haptic Feedback on Tabletops. In Proceedings of the 24th Annual ACM Symposium on User Interface Soft- ware and Technology (UIST ’11). Santa Barbara, Cali- fornia, USA, 615–620.
[10] L. Winfield, J. Glassmire, J.E. Colgate, and M. Peshkin. 2007. T-PaD: Tactile Pattern Display through Variable Friction Reduction. In Joint EuroHaptics Con- ference and Symposium on Haptic Interfaces for Vir- tual Environment and Teleoperator Systems. 421– 426.
Overview of Haptic Control Using Synth-A-Modeler
Vibrotactile Tokens for Touchscreen Interaction with Virtually Vibrating Constraints
Results So Far