proceedings of the international computer music conference 2012
TRANSCRIPT
PROCEEDINGS OF THE INTERNATIONAL COMPUTER MUSIC CONFERENCE 2012IRZU _INSTITUTE FOR SONIC ARTS RESEARCH
ISBN: 978-0-9845274-1-0
_450 _451
Note C D E F G A B CPythagorean Tuning 204 204 90 204 204 204 90Ptolemaic Tuning 204 182 112 204 182 204 112Mean-tone Temperament 193 193 117 193 193 193 117Werckmeister I 192 198 108 198 192 204 108Equal Temperament 200 200 100 200 200 200 100
Table 1. Diatonic overview of several historical tuning systems. All interval values are expressed in cents.
99 231 363 495 633 756 897 10351167
100200300400500600
Pitch (cent)
Num
ber
of
anno
tati
ons
Figure 4. This composition contains an equal tempera-ment of 9 tones per octave
can be seen in Figure 4. Each interval counts 133 cents,which entails the occurrence of 9 major thirds and 9 aug-mented fifths in the scale as well. It provides the piecea scale that is built on a mixture of unknown and morefamiliar intervals. However Tarsos did retrieve all ninepitch classes, also three small deviations of pitch classeswere noticed. Each of these three pitch classes measureconsequently 38 cents lower then the three notes from theintended scale (namely 231 633 and 897 cents). They oc-curred in a specific octave, and not over the entire ambitus.More research could tell the intention of these tones.
3.3. Historical scales
One can upload any of the historical scales that is listed inScala, or made manually, and convert any classical sym-bolic score available in MIDI into audio. As for exampleBach’s Das Wohltemperierte Klavier, BWV 846-893, canbe listened to in equal-temperament or in well-temperament.Interesting opposition since there is still a discussion onwhich tuning Bach intended these compositions for[5].Another use case is rendering some baroque compositions,that are known for their sensitivity towards affective the-ory, in several tuning systems. As a teaser, table1 givesan overview of some historical tunings. Notice the smallvariations in the different diatonic scales.
4. TARSOS LIVE
Tarsos can be used real-time: when this option is selected,any tone or set of tones that is presented is directly analy-sized. The scale that is played arises on the graphicalaxes. By selecting the peaks of the annotations, the pro-gram allows you to play together with the live musicianin that specific scale. Many possibilities come forward,an interesting one is that Western classical musicians can
now play together with any scale that is presented by mu-sicians, ranging from alternative scales to ethnic instru-ments. Any alteration in the scale is noticed directly, andcan the scale can be adjusted.
5. FUTURE WORK
The interface of Tarsos will be provided with a scale visu-alization that does not refer to the Western keyboard andthat comprises the size of the intervals as an ecologicaluser interface. Another feature will be the display of non-octave bound organisation of scales, as for example the88CET or Bohlen-Pierce. Where the user can (re)set theinterval of the octave towards any personal choice. Tarsoswill be applied on the entire RMCA archive intending abetter insight in African tone scales.
6. REFERENCES
[1] C. Cannam, “The vamp audio analysis plugin api: Aprogrammer’s guide,” http://vamp-plugins.org/guide.pdf.
[2] A. de Cheveigne and K. Hideki, “Yin, a fundamentalfrequency estimator for speech and music,” The Jour-nal of the Acoustical Society of America, vol. 111,no. 4, pp. 1917–1930, 2002.
[3] T. De Mulder, “Recent improvements of an auditorymodel based front-end for the transcription of vocalqueries,” in Proceedings of the IEEE InternationalConference on Acoustics, Speech and Signal Process-ing, 2004.
[4] P. McLeod and G. Wyvill, “A smarter way to findpitch,” in Proceedings of International Computer Mu-sic Conference, ICMC, 2005.
[5] S. M. Ruiz, “Temperament in Bach’s Well-TemperedClavier. A historical survey and a new evaluationaccording to dissonance theory,” Ph.D. dissertation,Universitat Autnoma de Barcelona, 2011.
[6] J. Six and O. Cornelis, “Tarsos - a Platform to ExplorePitch Scales in Non-Western and Western Music,” inProceedings of the 12th International Symposium onMusic Information Retrieval (ISMIR 2011), 2011.
[7] G. Tzanetakis, A. Kapur, W. A. Schloss, andM. Wright, “Computational ethnomusicology,” Jour-nal of Interdisciplinary Music Studies, vol. 1, no. 2,2007.
THE XYOLIN, A 10-OCTAVE CONTINUOUS-PITCH XYLOPHONE,AND OTHER EXISTEMOLOGICAL INSTRUMENTS
Steve Mann and Ryan JanzenUniversity of Toronto, Faculties of Engineering, Arts&Sci., and Forestry
ABSTRACTA class of truly acoustic, yet computational musical in-struments is presented. The instruments are based on physi-phones (instruments where the initial sound-production isphysical rather than virtual), which have been outfittedwith computation and tactuation, such that the final sounddelivery is also physical.
In one example, a single plank of wood is turned intoa continuous-pitch xylophone in which the initial soundproduction originates xylophonically (i.e. as vibrationsin wood), as input to a computational user-interface. Butrather than using a loudspeaker to reproduce the computer-processed sound, the final sound delivery is also xylo-phonic (i.e. the same wood itself is set into mechanical vi-bration, driven by the computer output). This xylophone,which we call the “Xyolin”, produces continuously vari-able pitch like a violin. It also covers more than 10 oc-taves, and includes the entire range of human hearing,over its 122 centimeter length, logarithmically (1 semi-tone per centimeter).
Other examples include pagophones in which initialsound generation occurs in ice, and final sound output alsooccurs in the ice.
More generally, we propose an existemological (ex-istential epistemology, i.e. “learn-by-being”) frameworkwhere any found material or object can be turned intoa highly expressive musical instrument in which soundboth originates and is output idiophonically in the samematerial or object, which may include some or all of theplayer’s own body as part of the instrument.
1. NON-COCHLEAR SOUND
The theme of this year’s ICMC conference is Non-Cochlearsound. The notion of non-cochlear sound is suggestive oftwo things:
1. sound that is perceived by other than the cochlea,e.g. tactile sound (sound that can be felt throughthe whole body rather than only heard); and
2. a metaphor likened to Marcel Duchamp’s “non-retinal”visual art, broadening our perception of what is meantby art, through “Readymades” (ordinary found ob-jects as art, for example). Likewise Non-CochlearSonic Art can be thought of as broadening our un-derstanding of sonic art in the Seth Kim-Cohen senseof “Non-Cochlear”[Kim-Cohen, 2009].
This paper presents a methodology and philosophy ofinstrument-building that embraces non-cochlear in both
these senses, i.e. instruments that are tactile (and can thus,for example, be played and enjoyed without the ear—theycan even be enjoyed by the deaf), and instruments that are“Readymades” in the Duchamp/Seth Kim-Cohen sense (withthe existential self-deterimination of the DIY “maker” cul-ture).
2. BACKGROUND AND PRIOR WORKThe work presented in this paper can be thought of asan extension of the concept of physiphones [Mann, 2007](using the natural acoustic sound production in physicalmaterial and objects for computer input devices), which it-self may be regarded as an extension of hyperinstruments[Machover, 1991].
2.1. Computer music and user-interfacesTraditional computer-music is generated by using vari-ous kinds of Human-Computer Interfaces (input devices),connected to a computer system, which synthesizes thesound we hear through a loudspeaker system. See Fig. 1,in relation to Fig. 2 3 4, to be described in what follows.
Some of the input devices used for computer musicare very creative. For example, Hiroshi Ishii of the MIT(Massachusetts Institute of Technology) Media Lab hasworked extensively to develop TUIs (Tangible User Inter-faces) [Ishii and Ullmer, 1997].
TUIs have been extensively used as user-interfaces[Vertegaal and Ungvary, 2001] [Alonso and Keyson, 2005].Many of these user-interfaces are extensive and creative,and use real-world objects as input devices. For example,Luc Geurts and Vero Vanden Abeele have used a bowl ofwater with electrical contacts in the water as a computerinput device so that splashing the water triggers the play-back of a pre-recorded sound sample[Geurts and Abeele, 2012]. Others have created systemsthat allow anyone to easily turn any objects such as fruit,plants, human skin, water, paintbrushes, or other objectsinto musical instruments [Silver et al., 2012].
Thus the piano keyboard symbol of Fig. 1 is meant tostand for any of the wide variety of Human-Computer in-put device in common usage, which can include real worldphysical objects, such as a bowl of water, as input devices.
2.2. Machover’s Hyperinstruments
In 1986, Tod Machover, from the MIT Media Lab de-veloped the concept of hyperinstruments, in which realphysical objects such as a violin, cello, or piano, are fit-ted with sensors as input devices to a computer which
_452 _453
USER
INPUT DEVICE
COMPUTER
SPEAKER
Figure 1: A common computer music methodology: A user interacts with a user-interface that is connected to a computer, which generates sound through amplifi-cation and a speaker system.
USER
OUTPUT
(ACOUSTIC)
INPUT DEVICE
COMPUTER
OUTPUT
(SUNTHETIC)
Figure 2: Machover’s hyperinstrument: A user interacts with a real physical objectsuch as a violin or cello. The real physical object includes various sensors that alsofunction as input devices to a computer, which synthesizes computer music thatis reproduced by a speaker system, and can be heard simultaneously with (i.e. inaddition to) the real physical object’s own acoustic sound production.
synthesizes sound to accompany the real physical instru-ment [Machover, 1991]. One example is the hyperpiano,in which “MIDI data generated by performer on a YamahaDisklavier is manipulated by various Max/MSP processesas accompaniment and augmentation of keyboard perfor-mance” (http://en.wikipedia.org/wiki/Tod Machover)[Machover, 1991].
2.3. Mann’s Hyperacoustic insruments
Throughout the 1980s and 1990s Steve Mann created avariety of input devices that use the real world itself asthe user-interface, for which he coined the terms “Real-ity User Interface” and “Natural User Interface” (NUI)[Mann, 2001] (before Microsoft Corporation began usingthis term in a narrower sense to denote tabletop interfaces.).In this paper, we use the term “Natural User Interface” inits original sense to denote interfaces that both (1) use nat-ural human capabilities (i.e. capabilities which come nat-urally to us), and (2) use nature itself as a user-interface(i.e. real-world physical objects, and natural philosophy,i.e. physics).
Some embodiments of these interfaces used the acous-tic disturbances in real physical objects as computer input[Mann, 2001] [Mann, 2007]. See Fig 3.
Some of these “Natural User Interfaces” included turn-ing public fountains such as Dundas Square (Canada’scultural and civic centre, akin to Times Square in the UnitedStates), various municipal ice rinks, and Lake Simcoe (On-tario, Canada) itself, into giant musical instruments. Thesewere not merely input devices to control sound synthsiz-ers, but, rather, instruments in which mechanical vibra-tions in the water, ice, earth, concrete, or the like, werecaptured with listening devices, and modified by computerin such a way as to make an expressive musical instru-ment. Such instruments are called hyperacoustic instru-ments [Mann et al., 2007].
It is easy to make an instrument that makes new andunfamiliar sounds. But just as a painter like Picasso hadto first prove himself with realism, before creating some-thing new, the hyperacoustic instruments were first used
USER SPEAKER
COMPUTER
OPTIONAL ADDITIONAL SENSORS
MIC. OR PICKUP
Figure 3: Mann’s physiphones (hyperacoustic instruments): A user interacts witha real physical object such as block of wood, ice, earth, water fountain, or the like[Mann, 2007][Mann et al., 2007]. The sensor is a microphone or other listeningdevice (sound pickup). Rather than synthesize sound, the computer modifies thesounds actually generated by the real physical object, such as by pitch-correction(pitch transposing) to notes on a musical scale. The natural physiphonically gen-erated sounds [Mann, 2007] are heard by an amplifier and speaker system, afterbeing modified by the computer.
USER
COMPUTER
OPTIONAL ADDITIONAL SENSORS
MIC. OR PICKUP
Figure 4: Proposed system: “acoustic physiphones”. A hyperacoustic system isboth the original source of the computer-modified sound, as well as the deliverymechanism of that modified sound. There is no speaker. Instead the physical ob-ject itself vibrates, both to generate the original sound, as well as to deliver theprocessed sound to the audience and player(s).
to play various classical music and jazz standards, in or-der to prove to the world that they were real instruments,and then were subsequenty used to play new music com-posed for them.
One example performance was Mann’s “Adaggio forFingernails and Chalkboard” (performed 2010 May 8th)in which actual acoustic sound, captured by contact mi-crophones on each of his fingernails was pitch-transposedand pitch-corrected to musical notes, first to play some fa-miliar classical and jazz repertoire, and then to play thenew Adaggio.
Another example of a hyperacoustic instrument is theuse of one or more wooden blocks or any other foundscraps of wood as a xylophone in which the natural soundof the wood is pitch-transposed to a musical pitch. In thisway, any found object can be turned into a non-electrophicmusical instrument, i.e. an idiophone, in which the soundis generated acoustically, then modified by computer.
It should be noted that such instruments are not merelyinput devices to computerized sound generators, as shownin Fig 1, but, rather, use the original sound itself, and arethus much more expressive and natural. For example, anordinary desk can be turned into a xylophone in whichtapping on the desk can make sounds like a bell, whereasrubbing on it can make more sustained notes like that ofa violin or cello. This sonic expressivity is due to the factthat the original sound, not a synthsized sound, is used.
Various found objects, such as a bath tub that werefound in a dumpster, were turned into expressive musi-cal instruments that could play any classical or jazz reper-toire, intricate Bach fugues, etc., as well as being able toplay newly composed music written specifically for thenew instruments.
A hyperacoustic instrument built into a SpaBerry hottub was used as the main instrument for the main act inNorth America’s largest winter festival, to perform forCanada’s Prime Minister and Governor General, in front
of an audience of more than 10,000 people. The resultinginstrument was a variation of the hydraulophone known asthe balnaphone.
Additionally, hyperacoustic instruments facilitate trulynatural user interfaces such as, for example, turning a liv-ing tree into a xylophone in which the sound originates xy-lophincally. Players strike the tree branches with mallets,and the actual sounds from the tree are picked up by listen-ing devices attached to the tree. The natural sounds pro-duced by tapping, scratching, or rubbing the tree are pitch-transposed to musical notes. The target pitch of the pitchtransposition is dependent on where the tree is struck. Thisis determined by using an array of listening devices withsound localization (time-of-flight), and/or a vision system(camera(s) and computer input frame grabber) that also“watches” to see where the tree is struck. With regards toFig. 3, the camera(s), if present, is/are the “optional addi-tional sensor(s)”.
Individual parts of the tree can then be labeled withchalk, e.g. A, B-flat, B, C., C-sharp, etc..
2.4. Orchestrions, player-pianos, and other actuatedinstrumentsOur work differs from computer-controlled musical in-struments like player-pianos, solenoid-activated xylophones,and other computer actuated musical instruments[Overholt et al., 2011] in the sense that we are not tryingto get the computer to play the instrument. In fact, quitethe opposite: we’re trying to get the computer to help usget “closer to nature”!
3. PROPOSED INSTRUMENTS
In this paper, we propose “acoustic physiphones” whichare natural user-interfaces in which:
• the initial sound production (sound generation) isnatural, i.e. acoustic, as with physiphones;
• the final sound delivery (sound reproduction) is byway of the natural material. Thus if the sound origi-nated xylophonically (from vibrations in wood), theprocessed sound is also reproduced xylophonically(i.e. by way of vibrations in wood). Preferablythe same wood that is used to generate the originalsound is used to deliver the processed (e.g. pitch-transposed) sound.
See Fig 4.The software used for the work done in this paper was
written in the “C” programming language, on specializedembedded computers that we designed and built to be com-pletely waterproof and environmentally sealed, so as tooperate in a natural environment. We used GNU Linuxand wrote our own device drivers to extend the operatingsystem to adapt to the new hardware we built.
3.1. The “Xyolin”We now present an example of an acoustic physiphone,which we call the “Xyolin”, named and invented by au-thor S. Mann. It is a xylophone, but it has infinitely con-
COMPUTER
HIGH
VOLTAGE
AMPLIFIER
XYLOPHONE PLANK, WHICH IS ALSO
ITS OWN SOUNDBOARD
TRANSMIT/RECV
DUPLEXER
MALLET WITH
TRANSDUCER
INSIDE IT
Figure 5: System architecture of the Xyolin (single-plank xylophone). Four smalltransducers, one in each corner of the plank, capture acoustic vibrations in the plankand convey these to the computer. A high voltage amplifier was adapted from anold vacuum tube amplifier found in a dumpster. The computer thus drives one largetransducer located in the middle of the plank. All of the transducers are capableof being transmitters or recievers, but the sound heard by the audience is primarilydue to vibrations induced in the board by the large central transducer.
tinous pitch like a violin. It can be played by striking, orby rubbing or bowing (thus giving it the capability to beplayed either percussively or with infinite sustain for notesof whatever duration are desired).
Various single-plank xylophones were built from highquality Sitka Spruce soundboards. But one of these instru-ments was made from a piece of rough plywood found in agarbage dumpster. It was fitted with four transducers, onein each corner, which could sense and effect vibrationsin the wood. Originally these were used as both listen-ing devices and excitatory devices, but later a much largertransducer was put in the center of the board. See Fig 5. Inaddition to position tracking by listening (time-of-arrivaldifferences in the various receive transducers, etc.), vari-ous other position sensing technologies were used in thiswork. These included a 24.360 GHz home-made radar setadapted for close range, an ultrasonic range sensor, andan overhead camera to improve the position-sensing (es-pecially while rubbing, where the onset of sound was lessdiscernible), and to recognize various mallets, sticks, ges-tures, etc..
Additionally, fine granules of brightly colored sand wereoften placed on the board, so as to form cymatics, visibleto the overhead camera. In this way the camera can “see”the nodal patterns in the vibrating wood, and this informa-tion can be used as part of the feedback loop in driving thetransmit transducer(s) to affect the vibrations in the wood.Other variations used ripple tanks as, or on, the vibratingmedium of the instrument.
The board becomes both the input device as well asthe soundboard for the instrument, delivering a variety ofpublic performances without the need to use a PA (publicaddress) system.
See Fig 6. When hitting the board with one or moremallets or sticks, the surface texture had little effect onthe sound production or sound delivery. But when rubbingthe surface with a mallet or stick, the surface texture of theboard was found to be very important.
It was found that rough plywood, covered in violinrosin, worked best for generating long sustained violin-like notes, through rubbing with a stick also coated in vi-olin rosin.
_452 _453
USER
INPUT DEVICE
COMPUTER
SPEAKER
Figure 1: A common computer music methodology: A user interacts with a user-interface that is connected to a computer, which generates sound through amplifi-cation and a speaker system.
USER
OUTPUT
(ACOUSTIC)
INPUT DEVICE
COMPUTER
OUTPUT
(SUNTHETIC)
Figure 2: Machover’s hyperinstrument: A user interacts with a real physical objectsuch as a violin or cello. The real physical object includes various sensors that alsofunction as input devices to a computer, which synthesizes computer music thatis reproduced by a speaker system, and can be heard simultaneously with (i.e. inaddition to) the real physical object’s own acoustic sound production.
synthesizes sound to accompany the real physical instru-ment [Machover, 1991]. One example is the hyperpiano,in which “MIDI data generated by performer on a YamahaDisklavier is manipulated by various Max/MSP processesas accompaniment and augmentation of keyboard perfor-mance” (http://en.wikipedia.org/wiki/Tod Machover)[Machover, 1991].
2.3. Mann’s Hyperacoustic insruments
Throughout the 1980s and 1990s Steve Mann created avariety of input devices that use the real world itself asthe user-interface, for which he coined the terms “Real-ity User Interface” and “Natural User Interface” (NUI)[Mann, 2001] (before Microsoft Corporation began usingthis term in a narrower sense to denote tabletop interfaces.).In this paper, we use the term “Natural User Interface” inits original sense to denote interfaces that both (1) use nat-ural human capabilities (i.e. capabilities which come nat-urally to us), and (2) use nature itself as a user-interface(i.e. real-world physical objects, and natural philosophy,i.e. physics).
Some embodiments of these interfaces used the acous-tic disturbances in real physical objects as computer input[Mann, 2001] [Mann, 2007]. See Fig 3.
Some of these “Natural User Interfaces” included turn-ing public fountains such as Dundas Square (Canada’scultural and civic centre, akin to Times Square in the UnitedStates), various municipal ice rinks, and Lake Simcoe (On-tario, Canada) itself, into giant musical instruments. Thesewere not merely input devices to control sound synthsiz-ers, but, rather, instruments in which mechanical vibra-tions in the water, ice, earth, concrete, or the like, werecaptured with listening devices, and modified by computerin such a way as to make an expressive musical instru-ment. Such instruments are called hyperacoustic instru-ments [Mann et al., 2007].
It is easy to make an instrument that makes new andunfamiliar sounds. But just as a painter like Picasso hadto first prove himself with realism, before creating some-thing new, the hyperacoustic instruments were first used
USER SPEAKER
COMPUTER
OPTIONAL ADDITIONAL SENSORS
MIC. OR PICKUP
Figure 3: Mann’s physiphones (hyperacoustic instruments): A user interacts witha real physical object such as block of wood, ice, earth, water fountain, or the like[Mann, 2007][Mann et al., 2007]. The sensor is a microphone or other listeningdevice (sound pickup). Rather than synthesize sound, the computer modifies thesounds actually generated by the real physical object, such as by pitch-correction(pitch transposing) to notes on a musical scale. The natural physiphonically gen-erated sounds [Mann, 2007] are heard by an amplifier and speaker system, afterbeing modified by the computer.
USER
COMPUTER
OPTIONAL ADDITIONAL SENSORS
MIC. OR PICKUP
Figure 4: Proposed system: “acoustic physiphones”. A hyperacoustic system isboth the original source of the computer-modified sound, as well as the deliverymechanism of that modified sound. There is no speaker. Instead the physical ob-ject itself vibrates, both to generate the original sound, as well as to deliver theprocessed sound to the audience and player(s).
to play various classical music and jazz standards, in or-der to prove to the world that they were real instruments,and then were subsequenty used to play new music com-posed for them.
One example performance was Mann’s “Adaggio forFingernails and Chalkboard” (performed 2010 May 8th)in which actual acoustic sound, captured by contact mi-crophones on each of his fingernails was pitch-transposedand pitch-corrected to musical notes, first to play some fa-miliar classical and jazz repertoire, and then to play thenew Adaggio.
Another example of a hyperacoustic instrument is theuse of one or more wooden blocks or any other foundscraps of wood as a xylophone in which the natural soundof the wood is pitch-transposed to a musical pitch. In thisway, any found object can be turned into a non-electrophicmusical instrument, i.e. an idiophone, in which the soundis generated acoustically, then modified by computer.
It should be noted that such instruments are not merelyinput devices to computerized sound generators, as shownin Fig 1, but, rather, use the original sound itself, and arethus much more expressive and natural. For example, anordinary desk can be turned into a xylophone in whichtapping on the desk can make sounds like a bell, whereasrubbing on it can make more sustained notes like that ofa violin or cello. This sonic expressivity is due to the factthat the original sound, not a synthsized sound, is used.
Various found objects, such as a bath tub that werefound in a dumpster, were turned into expressive musi-cal instruments that could play any classical or jazz reper-toire, intricate Bach fugues, etc., as well as being able toplay newly composed music written specifically for thenew instruments.
A hyperacoustic instrument built into a SpaBerry hottub was used as the main instrument for the main act inNorth America’s largest winter festival, to perform forCanada’s Prime Minister and Governor General, in front
of an audience of more than 10,000 people. The resultinginstrument was a variation of the hydraulophone known asthe balnaphone.
Additionally, hyperacoustic instruments facilitate trulynatural user interfaces such as, for example, turning a liv-ing tree into a xylophone in which the sound originates xy-lophincally. Players strike the tree branches with mallets,and the actual sounds from the tree are picked up by listen-ing devices attached to the tree. The natural sounds pro-duced by tapping, scratching, or rubbing the tree are pitch-transposed to musical notes. The target pitch of the pitchtransposition is dependent on where the tree is struck. Thisis determined by using an array of listening devices withsound localization (time-of-flight), and/or a vision system(camera(s) and computer input frame grabber) that also“watches” to see where the tree is struck. With regards toFig. 3, the camera(s), if present, is/are the “optional addi-tional sensor(s)”.
Individual parts of the tree can then be labeled withchalk, e.g. A, B-flat, B, C., C-sharp, etc..
2.4. Orchestrions, player-pianos, and other actuatedinstrumentsOur work differs from computer-controlled musical in-struments like player-pianos, solenoid-activated xylophones,and other computer actuated musical instruments[Overholt et al., 2011] in the sense that we are not tryingto get the computer to play the instrument. In fact, quitethe opposite: we’re trying to get the computer to help usget “closer to nature”!
3. PROPOSED INSTRUMENTS
In this paper, we propose “acoustic physiphones” whichare natural user-interfaces in which:
• the initial sound production (sound generation) isnatural, i.e. acoustic, as with physiphones;
• the final sound delivery (sound reproduction) is byway of the natural material. Thus if the sound origi-nated xylophonically (from vibrations in wood), theprocessed sound is also reproduced xylophonically(i.e. by way of vibrations in wood). Preferablythe same wood that is used to generate the originalsound is used to deliver the processed (e.g. pitch-transposed) sound.
See Fig 4.The software used for the work done in this paper was
written in the “C” programming language, on specializedembedded computers that we designed and built to be com-pletely waterproof and environmentally sealed, so as tooperate in a natural environment. We used GNU Linuxand wrote our own device drivers to extend the operatingsystem to adapt to the new hardware we built.
3.1. The “Xyolin”We now present an example of an acoustic physiphone,which we call the “Xyolin”, named and invented by au-thor S. Mann. It is a xylophone, but it has infinitely con-
COMPUTER
HIGH
VOLTAGE
AMPLIFIER
XYLOPHONE PLANK, WHICH IS ALSO
ITS OWN SOUNDBOARD
TRANSMIT/RECV
DUPLEXER
MALLET WITH
TRANSDUCER
INSIDE IT
Figure 5: System architecture of the Xyolin (single-plank xylophone). Four smalltransducers, one in each corner of the plank, capture acoustic vibrations in the plankand convey these to the computer. A high voltage amplifier was adapted from anold vacuum tube amplifier found in a dumpster. The computer thus drives one largetransducer located in the middle of the plank. All of the transducers are capableof being transmitters or recievers, but the sound heard by the audience is primarilydue to vibrations induced in the board by the large central transducer.
tinous pitch like a violin. It can be played by striking, orby rubbing or bowing (thus giving it the capability to beplayed either percussively or with infinite sustain for notesof whatever duration are desired).
Various single-plank xylophones were built from highquality Sitka Spruce soundboards. But one of these instru-ments was made from a piece of rough plywood found in agarbage dumpster. It was fitted with four transducers, onein each corner, which could sense and effect vibrationsin the wood. Originally these were used as both listen-ing devices and excitatory devices, but later a much largertransducer was put in the center of the board. See Fig 5. Inaddition to position tracking by listening (time-of-arrivaldifferences in the various receive transducers, etc.), vari-ous other position sensing technologies were used in thiswork. These included a 24.360 GHz home-made radar setadapted for close range, an ultrasonic range sensor, andan overhead camera to improve the position-sensing (es-pecially while rubbing, where the onset of sound was lessdiscernible), and to recognize various mallets, sticks, ges-tures, etc..
Additionally, fine granules of brightly colored sand wereoften placed on the board, so as to form cymatics, visibleto the overhead camera. In this way the camera can “see”the nodal patterns in the vibrating wood, and this informa-tion can be used as part of the feedback loop in driving thetransmit transducer(s) to affect the vibrations in the wood.Other variations used ripple tanks as, or on, the vibratingmedium of the instrument.
The board becomes both the input device as well asthe soundboard for the instrument, delivering a variety ofpublic performances without the need to use a PA (publicaddress) system.
See Fig 6. When hitting the board with one or moremallets or sticks, the surface texture had little effect onthe sound production or sound delivery. But when rubbingthe surface with a mallet or stick, the surface texture of theboard was found to be very important.
It was found that rough plywood, covered in violinrosin, worked best for generating long sustained violin-like notes, through rubbing with a stick also coated in vi-olin rosin.
_454 _455
Figure 6: Xyolin during an evening performance. A single wooden plank is fittedwith position sensors that sense the position of one or more mallets or sticks. Theresult is a simple uncluttered artistic performance instrument. The plank and someof the mallets or sticks are fitted with listening devices that capture the actual soundof the wood being struck or rubbed with the mallets or sticks. The acoustic soundfrom hitting or rubbing the wood is passed through one or more position-dependentbandpass filters, implemented on a computer system. The final output from thecomputer is amplified and fed back to the very plank that first generated the sound.
The xylophone pictured in Fig 6 covers just over 10octaves, with a resolution of exactly one centimeter persemitone (i.e. 12 centimeters per octave). The centimetersare marked with lines, as is every octave (in bolder lines)but the user can hit the plank between markings to getquarter tones or any other microtonal intervals.
The frequency range of the instrument is from E-flat 0(19.45 Hz) to E10 (21,096.16 Hz). Thus it spans the entirerange of human hearing from less than 20Hz to greaterthan 20kHz, over its 122 cm (122 semitone) length.
Position is determined by an array of listening deviceson the underside of the plank (using initial time-of-flightestimation in the wood, corrected for the differences in thespeed of sound going along the grain versus going cross-grain, etc.). Additionally, a side-looking K-band complex(in-phase and quadrature) radar set and an overhead cam-era run a machine vision algorithm with background sub-traction [Yao and Odobez, 2007]. This provides improvedtracking accuracy and distinguishes between various mal-lets and sticks which each have a uniquely colored bandattached near the tip, or a Luneberg radar lens (or both).
The stick in the player’s right hand (the stick picturedto the audience’s left) in Fig 6 is equipped with its ownpickup. This pickup feeds back at a high enough gain toprovide infinite sustain if it is kept touching the wood. Inthis way it will cause the wood to vibrate at any frequencyfrom 20 Hz to 20kHz depending on its position. The otherstick (the one without the pickup) simply excites the pick-ups in the wooden plank.
3.2. Acoustic feedback, with dynamic range compres-sion, and position-dependent bandpass filterA dynamic range compressor is a device that makes quietsounds louder and loud sounds quieter, thereby “compress-ing” an audio signal’s dynamic range. Compressors areoften used, for example, to process the output of vocal mi-crophones to reduce the dynamic range of a human voice.
Ordinarily in audio applications, acoustic feedback ishighly undesirable, and dynamic range compression canbe precarious in a live theatre because it can lead to feed-back. However, deliberate use of feedback is often used(e.g. when a guitarist stands next to a speaker to get longsustained violinesque tones).
P gg
(a)
T R
OutputLevel(dB)
Input Level (dB)
Threshold
2:1
4:1
∞:1
1:1
(b)Figure 7: (a) Uncontrolled feedback through an acoustic physical material (havingtransfer function P ), using amplified transmit and receive transducers with gainsgT and gR, respectively. (b) Input/output relationship of a simple dynamic rangecompressor, with various compresion ratios. [second image in the public domain,via Wikimedia Commons]
gRP P
gP
D
P
C*
ENV
LPF
C
g
COMPRESSOR
INITIATINGVIBRATIONS
PHYSICAL ACOUSTIC MATERIAL
GA
IN
TIM
E D
ELA
Y
RE
FLE
CTIO
NS
,
RE
SO
NA
NC
E
GAIN
TRANSMITTRANS−DUCER
RECEIVETRANS−DUCER
R
T
x
FPOSITION−DEPENDENT FILTER
Figure 8: Acoustic feedback assisted by dynamic range compression. We alsouse a position dependent bandpass filter F to tune the resonance according to theposition of the player’s hand or mallet, as detected by radar set and computer vision.
A simple feedback system is shown in Fig. 7(a), withgT representing an amplified transmit transducer (turns anelectrical signal into acoustic vibrations), P representingthe physical material through which the sound is fed back,and gR representing a receive transducer with amplifier(turns acoustic vibrations into an electrical signal). In con-trol theory P is often used to represent a “plant” (e.g. ajoint in a robot), and here P literally is a plant when weare using a tree branch. The system in Fig 7(a) is typicallyunstable and difficult to operate. That is, if we turn up thegains gT and gR high enough such that a vibration occurs,the vibration can suddenly grow out of control, in the pos-itive feedback loop, and the transducers have to be liftedoff the acoustic material before damage occurs!
Compressors typically act on a signal in the mannershown in Fig 7(b), acting on the amplitude of a signal (de-termined over several periods of the waveform) rather thanbeing applied at each point in time through the waveform(which would add harmonics due to a nonlinear effect onthe shape of the waveform itself). Therefore the naturalsound of the acoustic process is preserved, and feedbackis controlled and maintained.
In this paper we present controlled feedback in idio-phonic media, using adaptive computational processing(e.g. compression, filtering, etc.). to control and sustainfeedback with a pitch, timbre, and amplitude that can beaccurately and reliably controlled by the player. See Fig 8.
Even though the compressor is nonlinear, we can takea small segment of time over which the compressor’s gainC∗ is static, to first order (it gradually varies over thecourse of many waveform cycles), thus creating a linearfeedback system. Over the course of a single waveform,then, the input-output transfer function simply becomes:
PgT ⋅gR⋅FPC∗+1
. This mathematically describes the acous-tic response to a mallet strike or any vibration created bythe player, represented by input x in Fig 8. The compres-sor adjusts C∗ to ensure the feedback is sustained.
Figure 9: Derivation of a new shape for the 10-octave “Xyolin”....
3.3. Reshaping the “Xyolin”The embodiment of the one-plank xylophone pictured inFig 6 works quite well, but we wished to improve both itssound, and its aesthetic form.
There is something nice about the aesthetics of a stan-dard xylophone, as the higher notes have shorter bars. Wewish to mimick this exponential shape, both for appear-ance and for improved sound.
Conceptually, imagine we make a xylophone that has12 wooden bars per octave. A two-octave xylophone willhave 25 bars (12*2 + 1 to complete the octave), as shownin Fig 9(leftmost). Notice that the rightmost bar is halfthe length of the leftmost bar, since the fundamental fre-quency of vibration varies inversely with the square of thelength, i.e. half the length results in four times the fre-quency [Lapp, 2010]. Thus length ∝ √f (length is in-versely proportional to the square root of the frequency).
No suppose we make a microtonal xylophone, withquartertones, thus having 51 bars for the same two oc-taves (the rightmost bar still being half the length of theleftmost bar.
In the limit, as the pitch increment approaches zero,and the number of bars approaches infinity, we obtain thearrangement shown in Fig 9(center).
In Fig 9(center) we have just one piece of solid wood.The rightmost side is half the height of the leftmost side.
Now if we actually had a xylophone that ran 10 octavesfrom 20Hz to 20480Hz (20∗210 Hz), the lowest (longest)bar would be 32 times longer than the shortest bar.
This frequency range is really amazing when we thinkabout it, and it is due to the fact that length∝√f , i.e. theratio of longest to shortest bar is much smaller than theratio of highest to lowest frequency.
Therefore, we generate a continuous exponential shapethat runs over the entire 10 octave range, as shown inFig 9(rightmost). The left side of this shape is 32 timestaller than the right side.
3.4. A single-plank exponentially shaped xylophoneCutting out the plank in this shape, gives our instrumenta nice new shape, although the number of receive trans-ducers was reduced from 4 down to 3 (and the transmittransducer was moved to a new location closer to the fat-ter end of the plank). The new artistic aesthetic serves apractical purpose. For example, it is now obvious whichend is the end for low notes and which end is the end forhigh notes. The extreme differences between the two endsalso helps to make apparent the extreme range of pitchesthat the instrument is capable of producing. See Fig. 10.
However, the shape goes beyond mere aesthetics. Nowthe lower modes of vibration in the wood tend to occur
Figure 10: The “Xyolin”, a single plank xylophone with an exponential taper. Thisshape has both aesthetic value (e.g. it is obvious which end of the plank is for lownotes, which end is for high notes, and the extremes in size clearly indicate itsbroad compass), as well as functional value. Rightmost: we see the view from theoverhead camera used for computer-vision (tracking positions of the mallets andsticks, etc.).
Figure 11: Acoustic physiphone made from a fallen tree branch found in a forest.The transmit transducer is shown toward the left, hanging downwards. The re-ceive transducers were acoustically coupled to various smaller branches with pipeclamps.
more strongly at the larger end, and the higher modes ofvibration tend to occur more strongly at the smaller end.Thus we hear low notes emanate mainly from the largeend, high notes mainly from the small end, while midtonesemanate mainly from the middle of the plank.
Moreover, when using a stick or mallet with a pickupin it, the infinite sustain actually works better with thisnew tapered shape. For example, the very narrow end canvibrate easily at very high frequencies, up to and beyondthe range of human hearing. The large end works betterat low frequencies, especially as it can move more of thesurrounding air in the room, in order to better reproducelow pitches. We also preferred the timbral changes to thesound arising from the tapered shape, especially the im-proved clarity of long sustained high notes.
3.5. Natural User InterfacesA walk in the forest with a rubber mallet will often revealfallen tree branches that are very sonorous. Accordingly,a fallen branch of Sitka Spruce was found, which soundedquite nicely on its own.
This piece of fallen tree was made into an acousticphysiphone, by fitting it with a transmit transducer anda number of receive transducers. See Fig 11. The resultis a highly expressive and sonorous instrument that can beused to play highly intricate recognizable songs and clas-sical or jazz reperetoire (including intricate Bach fugues,etc.) as well as new experimental music, owing to the mi-crotonal character and high degree of timbral variability.
_454 _455
Figure 6: Xyolin during an evening performance. A single wooden plank is fittedwith position sensors that sense the position of one or more mallets or sticks. Theresult is a simple uncluttered artistic performance instrument. The plank and someof the mallets or sticks are fitted with listening devices that capture the actual soundof the wood being struck or rubbed with the mallets or sticks. The acoustic soundfrom hitting or rubbing the wood is passed through one or more position-dependentbandpass filters, implemented on a computer system. The final output from thecomputer is amplified and fed back to the very plank that first generated the sound.
The xylophone pictured in Fig 6 covers just over 10octaves, with a resolution of exactly one centimeter persemitone (i.e. 12 centimeters per octave). The centimetersare marked with lines, as is every octave (in bolder lines)but the user can hit the plank between markings to getquarter tones or any other microtonal intervals.
The frequency range of the instrument is from E-flat 0(19.45 Hz) to E10 (21,096.16 Hz). Thus it spans the entirerange of human hearing from less than 20Hz to greaterthan 20kHz, over its 122 cm (122 semitone) length.
Position is determined by an array of listening deviceson the underside of the plank (using initial time-of-flightestimation in the wood, corrected for the differences in thespeed of sound going along the grain versus going cross-grain, etc.). Additionally, a side-looking K-band complex(in-phase and quadrature) radar set and an overhead cam-era run a machine vision algorithm with background sub-traction [Yao and Odobez, 2007]. This provides improvedtracking accuracy and distinguishes between various mal-lets and sticks which each have a uniquely colored bandattached near the tip, or a Luneberg radar lens (or both).
The stick in the player’s right hand (the stick picturedto the audience’s left) in Fig 6 is equipped with its ownpickup. This pickup feeds back at a high enough gain toprovide infinite sustain if it is kept touching the wood. Inthis way it will cause the wood to vibrate at any frequencyfrom 20 Hz to 20kHz depending on its position. The otherstick (the one without the pickup) simply excites the pick-ups in the wooden plank.
3.2. Acoustic feedback, with dynamic range compres-sion, and position-dependent bandpass filterA dynamic range compressor is a device that makes quietsounds louder and loud sounds quieter, thereby “compress-ing” an audio signal’s dynamic range. Compressors areoften used, for example, to process the output of vocal mi-crophones to reduce the dynamic range of a human voice.
Ordinarily in audio applications, acoustic feedback ishighly undesirable, and dynamic range compression canbe precarious in a live theatre because it can lead to feed-back. However, deliberate use of feedback is often used(e.g. when a guitarist stands next to a speaker to get longsustained violinesque tones).
P gg
(a)
T R
OutputLevel(dB)
Input Level (dB)
Threshold
2:1
4:1
∞:1
1:1
(b)Figure 7: (a) Uncontrolled feedback through an acoustic physical material (havingtransfer function P ), using amplified transmit and receive transducers with gainsgT and gR, respectively. (b) Input/output relationship of a simple dynamic rangecompressor, with various compresion ratios. [second image in the public domain,via Wikimedia Commons]
gRP P
gP
D
P
C*
ENV
LPF
C
g
COMPRESSOR
INITIATINGVIBRATIONS
PHYSICAL ACOUSTIC MATERIAL
GA
IN
TIM
E D
ELA
Y
RE
FLE
CTIO
NS
,
RE
SO
NA
NC
E
GAIN
TRANSMITTRANS−DUCER
RECEIVETRANS−DUCER
R
T
x
FPOSITION−DEPENDENT FILTER
Figure 8: Acoustic feedback assisted by dynamic range compression. We alsouse a position dependent bandpass filter F to tune the resonance according to theposition of the player’s hand or mallet, as detected by radar set and computer vision.
A simple feedback system is shown in Fig. 7(a), withgT representing an amplified transmit transducer (turns anelectrical signal into acoustic vibrations), P representingthe physical material through which the sound is fed back,and gR representing a receive transducer with amplifier(turns acoustic vibrations into an electrical signal). In con-trol theory P is often used to represent a “plant” (e.g. ajoint in a robot), and here P literally is a plant when weare using a tree branch. The system in Fig 7(a) is typicallyunstable and difficult to operate. That is, if we turn up thegains gT and gR high enough such that a vibration occurs,the vibration can suddenly grow out of control, in the pos-itive feedback loop, and the transducers have to be liftedoff the acoustic material before damage occurs!
Compressors typically act on a signal in the mannershown in Fig 7(b), acting on the amplitude of a signal (de-termined over several periods of the waveform) rather thanbeing applied at each point in time through the waveform(which would add harmonics due to a nonlinear effect onthe shape of the waveform itself). Therefore the naturalsound of the acoustic process is preserved, and feedbackis controlled and maintained.
In this paper we present controlled feedback in idio-phonic media, using adaptive computational processing(e.g. compression, filtering, etc.). to control and sustainfeedback with a pitch, timbre, and amplitude that can beaccurately and reliably controlled by the player. See Fig 8.
Even though the compressor is nonlinear, we can takea small segment of time over which the compressor’s gainC∗ is static, to first order (it gradually varies over thecourse of many waveform cycles), thus creating a linearfeedback system. Over the course of a single waveform,then, the input-output transfer function simply becomes:
PgT ⋅gR⋅FPC∗+1
. This mathematically describes the acous-tic response to a mallet strike or any vibration created bythe player, represented by input x in Fig 8. The compres-sor adjusts C∗ to ensure the feedback is sustained.
Figure 9: Derivation of a new shape for the 10-octave “Xyolin”....
3.3. Reshaping the “Xyolin”The embodiment of the one-plank xylophone pictured inFig 6 works quite well, but we wished to improve both itssound, and its aesthetic form.
There is something nice about the aesthetics of a stan-dard xylophone, as the higher notes have shorter bars. Wewish to mimick this exponential shape, both for appear-ance and for improved sound.
Conceptually, imagine we make a xylophone that has12 wooden bars per octave. A two-octave xylophone willhave 25 bars (12*2 + 1 to complete the octave), as shownin Fig 9(leftmost). Notice that the rightmost bar is halfthe length of the leftmost bar, since the fundamental fre-quency of vibration varies inversely with the square of thelength, i.e. half the length results in four times the fre-quency [Lapp, 2010]. Thus length ∝ √f (length is in-versely proportional to the square root of the frequency).
No suppose we make a microtonal xylophone, withquartertones, thus having 51 bars for the same two oc-taves (the rightmost bar still being half the length of theleftmost bar.
In the limit, as the pitch increment approaches zero,and the number of bars approaches infinity, we obtain thearrangement shown in Fig 9(center).
In Fig 9(center) we have just one piece of solid wood.The rightmost side is half the height of the leftmost side.
Now if we actually had a xylophone that ran 10 octavesfrom 20Hz to 20480Hz (20∗210 Hz), the lowest (longest)bar would be 32 times longer than the shortest bar.
This frequency range is really amazing when we thinkabout it, and it is due to the fact that length∝√f , i.e. theratio of longest to shortest bar is much smaller than theratio of highest to lowest frequency.
Therefore, we generate a continuous exponential shapethat runs over the entire 10 octave range, as shown inFig 9(rightmost). The left side of this shape is 32 timestaller than the right side.
3.4. A single-plank exponentially shaped xylophoneCutting out the plank in this shape, gives our instrumenta nice new shape, although the number of receive trans-ducers was reduced from 4 down to 3 (and the transmittransducer was moved to a new location closer to the fat-ter end of the plank). The new artistic aesthetic serves apractical purpose. For example, it is now obvious whichend is the end for low notes and which end is the end forhigh notes. The extreme differences between the two endsalso helps to make apparent the extreme range of pitchesthat the instrument is capable of producing. See Fig. 10.
However, the shape goes beyond mere aesthetics. Nowthe lower modes of vibration in the wood tend to occur
Figure 10: The “Xyolin”, a single plank xylophone with an exponential taper. Thisshape has both aesthetic value (e.g. it is obvious which end of the plank is for lownotes, which end is for high notes, and the extremes in size clearly indicate itsbroad compass), as well as functional value. Rightmost: we see the view from theoverhead camera used for computer-vision (tracking positions of the mallets andsticks, etc.).
Figure 11: Acoustic physiphone made from a fallen tree branch found in a forest.The transmit transducer is shown toward the left, hanging downwards. The re-ceive transducers were acoustically coupled to various smaller branches with pipeclamps.
more strongly at the larger end, and the higher modes ofvibration tend to occur more strongly at the smaller end.Thus we hear low notes emanate mainly from the largeend, high notes mainly from the small end, while midtonesemanate mainly from the middle of the plank.
Moreover, when using a stick or mallet with a pickupin it, the infinite sustain actually works better with thisnew tapered shape. For example, the very narrow end canvibrate easily at very high frequencies, up to and beyondthe range of human hearing. The large end works betterat low frequencies, especially as it can move more of thesurrounding air in the room, in order to better reproducelow pitches. We also preferred the timbral changes to thesound arising from the tapered shape, especially the im-proved clarity of long sustained high notes.
3.5. Natural User InterfacesA walk in the forest with a rubber mallet will often revealfallen tree branches that are very sonorous. Accordingly,a fallen branch of Sitka Spruce was found, which soundedquite nicely on its own.
This piece of fallen tree was made into an acousticphysiphone, by fitting it with a transmit transducer anda number of receive transducers. See Fig 11. The resultis a highly expressive and sonorous instrument that can beused to play highly intricate recognizable songs and clas-sical or jazz reperetoire (including intricate Bach fugues,etc.) as well as new experimental music, owing to the mi-crotonal character and high degree of timbral variability.
_456 _457
Figure 12: An ensemble of xylophones was constructed from real living trees in aforest. Here we see a transmit transducer hanging from a branch at the left, a re-ceive transducer acoustically coupled to the branch near the right, and an overheadcamera assisting with the identification and position tracking of a variety of sticksand mallets. Additionally a data projector is incorporated into the camera for usein late-night concerts, as well as turning the branch into an interactive touch screenof sorts.
Figure 13: Public pagophone performance made using ice that was made moresonorous by computer processing. Transducers embedded in the ice cause it tovibrate at musical pitches. The two large slabs of ice each produce 12 perfectlytuned musical notes that remain in perfect tune even as the ice melts. The smallerslabs each produce a single note.
Finally, a forest concert was prepared, in which numer-ous trees were turned into xylophonic ensemble of mu-sical instruments. Special mounting brackets were de-veloped to attach transmit and recieve transducers to treebranches, to softly “grasp” the gree branches without dam-aging them. See Fig 12.
4. OTHER ACOUSTIC PHYSIPHONES
The same principles that apply to our Xyolin, in all itsReadymade embodiments, from office desks, to woodenplanks, to branches, to forests, etc., can also be applied toother materials. This work was the opening keynote forACM (Association of Computing Machinery) TEI confer-ence, by way of a performance using ice as an interactivemusical medium.
In this performance, we used four transmit transducers,and 12 receive transducers, arranged on and in blocks ofice. Some of the transducers were frozen right into the iceblocks, and others were coupled acoustically to the ice.See Fig. 13.
We also invited audience members to bring forwardany object that they wished to turn into a musical instru-ment. We took requests, e.g. “Can you play Pachelbel’sCanon on this rubber boot?” or “Can you play Gershwin’sSummertime on this soft-cover book?”, which we did.
We then performed some original music on the ice, andon the objects selected or supplied by the audience mem-bers. See Fig. 14.
Figure 14: Public performance with Readymade instruments: acoustic physi-phones made from items supplied by audience members. A transmit transducerexcites the object to regenerate its own acoustic vibrations as picked up by a re-ceive transducer. An overhead camera tracks an active or passive stick or mallet.In this figure, the stick is a “magic wand” containing an active illumination sourcetracked by the camera, as well as an audio pickup to sense vibrations in the sup-plied objects. An overhead camera and projection system mounted to a microphoneboom can be placed over any supplied objects to turn them into interactive touchsurfaces that augment these acoustic physiphones. Leftmost: a rubber boot; Right-most: a smartphone (we also wrote a smartphone app that turns anything into amusical instrument).
Figure 15: Readymade bath instrument showing innards.
5. READYMADE FOUNTAINS
The proposed method of creating acoustic physiphonesfrom nearly any found objects is not limited to idiophonicsound creation.
As an example of another form of sound creation, amusical instrument was made from a bath tub found in adumpster. After cleaning out the tub it was fitted with var-ious hydrophones (12 receive hydrophones and two trans-mit hydrophones), and some waterproof computer equip-ment. Four wheels were installed, under the tub, one ineach corner, to create a kind of “bathmobile”. A propaneheater was fitted to the tub, so that it could be rolled aroundwhile being played. A circulatory system was createdfrom electric pumps running from a car battery and powerinverter installed in the underside of the tub, together withthe various computational and sensory equipment.
The resulting readymade bathmobile is an instrumentin which sound:
• originates as vibrations in water, by playing any ofthe 12 water jets installed on the tub;
• is delivered to the audience by vibrations in the samewater.
See Figs. 15 and 16. Sound production and sound deliveryare thus hydraulophonic, with computational capabilitiesand a wide range of acoustic timbres and capabilities.
Moreover, the sound is truly tactile, in the sense thatparticipants can feel the sound in their fingertips, and also
Figure 16: Readymade bath instrument during a rolling street performance.
Figure 17: The Readymade bath instrument is inherently tactile and visual. Aswell as hearing, we can also feel and see the vibrations in the water which producethe sound. As a result, hearing impaired musicians can also enjoy the instrument.
see the sound vibrations in the water. See Fig. 17.As a result, hearing impaired musicians can also en-
joy the instrument. For example, hearing impaired per-cussionist Evelyn Glennie played on the instrument, andwas able to play and feel melodies and harmonies on it.Thus, like the idiophones presented in this paper, the bathinstrument is non-cochlear in both senses of the word: itcan be experienced without the cochlea, and it also trulyreferences the work of Marcel Duchamp, in many ways!
6. SCIENCE OUTREACH
STEM is an acronym for Science, Technology, Engineer-ing, and Mathematics, and an agenda of public educationis integrating these disciplines.
Other interdisciplinary efforts like MIT’s Media Lab-oratory focus on Art + Science + Technology. Design isalso an important discipline, so we might consider DAST= Design + Art + Science + Technology.
DAST could put a “heart and soul” into STEM, e.g.going beyond “multidisciplinary” to something we call“multipassionary” or “interpassionary” or “transpassion-ary”, i.e. passion is a better master than discipline (AlbertEinstein said that “love is a better master than duty”).
Consider, for example, DASTEM = Design + Art +Science + Technology + Engineering + Mathematics(“dastemology”), or perhaps DASI = Design + Art + Sci-ence + In(ter)vention or Innovation.
Perhaps what we want to nurture is the “inventopher”(inventor+philosopher), through existemology (existentialepistemology), i.e. “learn-by-being”.
This goes beyond the “learn by doing” (the construc-tionist education of Minsky and Pappert at MIT).
A simple example of putting existemology into prac-tice is when we teach our children how to measure some-thing, using anthropomorphic units (measurements basedon the human body) (wikipedia.org/wiki/Anthropic units)like inches (width of the thumb) or feet. The human body
itself becomes the ruler. We learn about rulers and mea-surement by becoming the measurement instrument.
Consider a four-year-old learning about water pressure:Daddy: This gauge is in kilopascals.Christina (age 4): Why “kill a pascal”?Daddy: Kilo means 1000, so its 1000 pascals.Christina: What’s pascal?Daddy: A French physicist, also one newtwon persquare meter.Christina: What’s newton?Daddy: Another physicist....
The same child had no problem understanding water pres-sure in pounds per square inch or Christinas (her own bodyweight) per square Stephanie (her sister’s area). The veryinaccuracy of anthropomorphic units, especially when usedacross various age groups, is why the concept is so pow-erful as a teaching tool: it is OK to make mistakes, to takeguesses, and to get a rough imprecise understanding of theworld around us.
Another example of existemology is wearable comput-ing: we learn about computers by “becoming” the tech-nology in the “cyborg” sense, Learning by Being: ThirtyYears of Cyborg Existemology, INTERNATIONAL HAND-BOOK OF VIRTUAL LEARNING ENVIRONMENTS,2006, Part IV, 1571-1592.
Much like the Suzuki method for teaching music, the“Mann method” (author S. Mann) of teaching is based onexistemology. The human body itself becomes a musicalinstrument that teaches physics, states-of-matter, mathe-matics, and the like.
An example around this idea is Pipe Dreams, a seriesof performances and demonstrations in 2011, in which au-thor S. Mann played instruments while sleeping. A skullcap with 64 brainwave electrodes was connected to a com-puter that played four instruments, one in each state-of-matter: chimes made of pipes (solid matter); a hydraulo-phone (liquid matter); a pipe organ (gaseous matter); anda plasmaphone (sound from the fourth state-of-matter).
When the solid, liquid, and gas pipes are arrayed to-gether around the sleeping subject, they form an interest-ing sculptural form as well. The tubular glockenspiel haspipes that vary in length inversely as the square root ofthe frequency, whereas the pipe organ has pipes that varyinversely with linear frequency, and the hydraulophonepipes vary inversely with the square of the frequency:
Xylophone or glockenspiel Pipe organ Hydraulophonelength ∝√f length ∝ f length ∝ f2
Moreover, the chimes (glockenspiel) are velocity-sensing,whereas the pipe organ is displacement sensing, and thehydraulophone is absement sensing. Absement is the time-integral of displacement. More generally, hydraulophonesgive rise to a new kinematics (“scitamenik”) that includesnegative derivates-of-displacement, in the sequence: {...,absounce, abserk, abseleration, absity, absement, displace-ment, velocity, acceleration, jerk, jounce, ...}. See Fig 18.
These simple and fundamental aspects like state-of-matterand kinematics allow us to see the world in new ways,beyond music. For example, others have recognized thedidactic value of this new kinematics philosophy:
_456 _457
Figure 12: An ensemble of xylophones was constructed from real living trees in aforest. Here we see a transmit transducer hanging from a branch at the left, a re-ceive transducer acoustically coupled to the branch near the right, and an overheadcamera assisting with the identification and position tracking of a variety of sticksand mallets. Additionally a data projector is incorporated into the camera for usein late-night concerts, as well as turning the branch into an interactive touch screenof sorts.
Figure 13: Public pagophone performance made using ice that was made moresonorous by computer processing. Transducers embedded in the ice cause it tovibrate at musical pitches. The two large slabs of ice each produce 12 perfectlytuned musical notes that remain in perfect tune even as the ice melts. The smallerslabs each produce a single note.
Finally, a forest concert was prepared, in which numer-ous trees were turned into xylophonic ensemble of mu-sical instruments. Special mounting brackets were de-veloped to attach transmit and recieve transducers to treebranches, to softly “grasp” the gree branches without dam-aging them. See Fig 12.
4. OTHER ACOUSTIC PHYSIPHONES
The same principles that apply to our Xyolin, in all itsReadymade embodiments, from office desks, to woodenplanks, to branches, to forests, etc., can also be applied toother materials. This work was the opening keynote forACM (Association of Computing Machinery) TEI confer-ence, by way of a performance using ice as an interactivemusical medium.
In this performance, we used four transmit transducers,and 12 receive transducers, arranged on and in blocks ofice. Some of the transducers were frozen right into the iceblocks, and others were coupled acoustically to the ice.See Fig. 13.
We also invited audience members to bring forwardany object that they wished to turn into a musical instru-ment. We took requests, e.g. “Can you play Pachelbel’sCanon on this rubber boot?” or “Can you play Gershwin’sSummertime on this soft-cover book?”, which we did.
We then performed some original music on the ice, andon the objects selected or supplied by the audience mem-bers. See Fig. 14.
Figure 14: Public performance with Readymade instruments: acoustic physi-phones made from items supplied by audience members. A transmit transducerexcites the object to regenerate its own acoustic vibrations as picked up by a re-ceive transducer. An overhead camera tracks an active or passive stick or mallet.In this figure, the stick is a “magic wand” containing an active illumination sourcetracked by the camera, as well as an audio pickup to sense vibrations in the sup-plied objects. An overhead camera and projection system mounted to a microphoneboom can be placed over any supplied objects to turn them into interactive touchsurfaces that augment these acoustic physiphones. Leftmost: a rubber boot; Right-most: a smartphone (we also wrote a smartphone app that turns anything into amusical instrument).
Figure 15: Readymade bath instrument showing innards.
5. READYMADE FOUNTAINS
The proposed method of creating acoustic physiphonesfrom nearly any found objects is not limited to idiophonicsound creation.
As an example of another form of sound creation, amusical instrument was made from a bath tub found in adumpster. After cleaning out the tub it was fitted with var-ious hydrophones (12 receive hydrophones and two trans-mit hydrophones), and some waterproof computer equip-ment. Four wheels were installed, under the tub, one ineach corner, to create a kind of “bathmobile”. A propaneheater was fitted to the tub, so that it could be rolled aroundwhile being played. A circulatory system was createdfrom electric pumps running from a car battery and powerinverter installed in the underside of the tub, together withthe various computational and sensory equipment.
The resulting readymade bathmobile is an instrumentin which sound:
• originates as vibrations in water, by playing any ofthe 12 water jets installed on the tub;
• is delivered to the audience by vibrations in the samewater.
See Figs. 15 and 16. Sound production and sound deliveryare thus hydraulophonic, with computational capabilitiesand a wide range of acoustic timbres and capabilities.
Moreover, the sound is truly tactile, in the sense thatparticipants can feel the sound in their fingertips, and also
Figure 16: Readymade bath instrument during a rolling street performance.
Figure 17: The Readymade bath instrument is inherently tactile and visual. Aswell as hearing, we can also feel and see the vibrations in the water which producethe sound. As a result, hearing impaired musicians can also enjoy the instrument.
see the sound vibrations in the water. See Fig. 17.As a result, hearing impaired musicians can also en-
joy the instrument. For example, hearing impaired per-cussionist Evelyn Glennie played on the instrument, andwas able to play and feel melodies and harmonies on it.Thus, like the idiophones presented in this paper, the bathinstrument is non-cochlear in both senses of the word: itcan be experienced without the cochlea, and it also trulyreferences the work of Marcel Duchamp, in many ways!
6. SCIENCE OUTREACH
STEM is an acronym for Science, Technology, Engineer-ing, and Mathematics, and an agenda of public educationis integrating these disciplines.
Other interdisciplinary efforts like MIT’s Media Lab-oratory focus on Art + Science + Technology. Design isalso an important discipline, so we might consider DAST= Design + Art + Science + Technology.
DAST could put a “heart and soul” into STEM, e.g.going beyond “multidisciplinary” to something we call“multipassionary” or “interpassionary” or “transpassion-ary”, i.e. passion is a better master than discipline (AlbertEinstein said that “love is a better master than duty”).
Consider, for example, DASTEM = Design + Art +Science + Technology + Engineering + Mathematics(“dastemology”), or perhaps DASI = Design + Art + Sci-ence + In(ter)vention or Innovation.
Perhaps what we want to nurture is the “inventopher”(inventor+philosopher), through existemology (existentialepistemology), i.e. “learn-by-being”.
This goes beyond the “learn by doing” (the construc-tionist education of Minsky and Pappert at MIT).
A simple example of putting existemology into prac-tice is when we teach our children how to measure some-thing, using anthropomorphic units (measurements basedon the human body) (wikipedia.org/wiki/Anthropic units)like inches (width of the thumb) or feet. The human body
itself becomes the ruler. We learn about rulers and mea-surement by becoming the measurement instrument.
Consider a four-year-old learning about water pressure:Daddy: This gauge is in kilopascals.Christina (age 4): Why “kill a pascal”?Daddy: Kilo means 1000, so its 1000 pascals.Christina: What’s pascal?Daddy: A French physicist, also one newtwon persquare meter.Christina: What’s newton?Daddy: Another physicist....
The same child had no problem understanding water pres-sure in pounds per square inch or Christinas (her own bodyweight) per square Stephanie (her sister’s area). The veryinaccuracy of anthropomorphic units, especially when usedacross various age groups, is why the concept is so pow-erful as a teaching tool: it is OK to make mistakes, to takeguesses, and to get a rough imprecise understanding of theworld around us.
Another example of existemology is wearable comput-ing: we learn about computers by “becoming” the tech-nology in the “cyborg” sense, Learning by Being: ThirtyYears of Cyborg Existemology, INTERNATIONAL HAND-BOOK OF VIRTUAL LEARNING ENVIRONMENTS,2006, Part IV, 1571-1592.
Much like the Suzuki method for teaching music, the“Mann method” (author S. Mann) of teaching is based onexistemology. The human body itself becomes a musicalinstrument that teaches physics, states-of-matter, mathe-matics, and the like.
An example around this idea is Pipe Dreams, a seriesof performances and demonstrations in 2011, in which au-thor S. Mann played instruments while sleeping. A skullcap with 64 brainwave electrodes was connected to a com-puter that played four instruments, one in each state-of-matter: chimes made of pipes (solid matter); a hydraulo-phone (liquid matter); a pipe organ (gaseous matter); anda plasmaphone (sound from the fourth state-of-matter).
When the solid, liquid, and gas pipes are arrayed to-gether around the sleeping subject, they form an interest-ing sculptural form as well. The tubular glockenspiel haspipes that vary in length inversely as the square root ofthe frequency, whereas the pipe organ has pipes that varyinversely with linear frequency, and the hydraulophonepipes vary inversely with the square of the frequency:
Xylophone or glockenspiel Pipe organ Hydraulophonelength ∝√f length ∝ f length ∝ f2
Moreover, the chimes (glockenspiel) are velocity-sensing,whereas the pipe organ is displacement sensing, and thehydraulophone is absement sensing. Absement is the time-integral of displacement. More generally, hydraulophonesgive rise to a new kinematics (“scitamenik”) that includesnegative derivates-of-displacement, in the sequence: {...,absounce, abserk, abseleration, absity, absement, displace-ment, velocity, acceleration, jerk, jounce, ...}. See Fig 18.
These simple and fundamental aspects like state-of-matterand kinematics allow us to see the world in new ways,beyond music. For example, others have recognized thedidactic value of this new kinematics philosophy:
_458 _459
Absity Absement Displacement(Distance)
AccelerationAbseleration
ddt
dt
ddt
dt
ddt
dt
ddt
dt
ddt
dt
... ...
+
−
f(x)
a bx
y
ddtdt
Velocity(Speed)
Two-stageHydraulophone
Hydraulophone isAbsement-sensitive
Organ isDisplacement-sensitive
Piano isVelocity-sensitive
Kinematics and Musical Instruments
Integration DifferentiationFigure 18: Hydraulophones reveal and exhibit a completely new way of under-standing and thinking about kinematics: negative derivatives of displacement!
Although time-integrated charge is a some-what unusual quantity in circuit theory, it maybe considered as the electrical analogue of amechanical quantity called absement. Basedon this analogy, simple mechanical devicesare presented that can serve as didactic ex-amples to explain memristive, meminductive,and memcapacitive behavior.[Jeltsema, 2012]
6.1. Water, Forestry, and First Nations instrumentsThe water instruments allow a natural element —- water—- to itself become a musical instrument. We are work-ing to combine water and forestry in a series of musicalperformances in various forests. One such performancecontextualizes the forest canopy as a “cathedral” of sorts,where native flutes are played high in the forest canopy,along a canopy walkway. Additionally, various water in-struments are played on and in natural bodies of water inthe forest.
In one of the compositions there are three elements:• Earth: Native Drums, forest, and tree instruments,
including the Xyolin. These instruments are playedon the ground;
• Water: Hydraulophones, which are played on and innatural bodies of water in the forest. Some of theseinstruments are actually played underwater;
• Air: Native Flutes played high in a forest canopywalkway.
Thus we have Earth on the ground, Water on and in thewater, and Air up in the air.
The use of the five Elements (Earth, Water, Air, Fire,Idea) is part of our work at the nexus of art, science, tech-nology (engineering), and design to support ”DAST” (De-sign, Art, Science, and Technology) outreach.
Lateral thinking within this new “states-of-matter” mu-sical instrument ontology (physical organology) can leadto the invention and rapid prototyping of many new mu-sical instruments in a DIY readymade context well-suitedto existemological outreach.
7. CONCLUSION
We have created several instances of a new kind of computer-based musical instrument in which the sound (a) origi-nates acoustically, and (b) is conveyed to the audienceacoustically, i.e. by acoustic vibrations in the physicalbody of the instrument.
Examples include the “Xyolin”, a xylophone that hasinfinitely many notes and covers the entire audio rangeof human hearing, where sound originates as vibrationsin wood, and is conveyed to the audience by vibrations inwood, as well as the pagophone, in which sound originatesin vibrations in ice, and is conveyed to the audience byway of vibrations in ice.
The instruments can play any jazz or classical reper-toire, intricate Bach fugues, etc., but they can also playa wide range of original works not possible on any otherinstrument.
Moreover, these new instruments give rise to a newway of thinking about and learning about science, suchas states-of-matter, and a new perspective on kinematicsthat includes negative derivatives of displacement.
8. ACKNOWLEDGEMENTS
The authors wish to thank Andrew Kmiecik, Jason Huang,Valmiki Rampersad, Raymond Lo, Queen’s University,NSERC, and AMD.
References[Alonso and Keyson, 2005] Alonso, M. B. and Keyson, D. V. (2005). MusicCube:
making digital music tangible. ACM CHI.[Geurts and Abeele, 2012] Geurts, L. and Abeele, V. V. (2012). Splash con-
trollers: Game controllers involving the uncareful manipulation of water. InProceedings of the ACM Tangible Embedded and Embodied Interaction, pages183–186, Kingston, Ontario, Canada.
[Ishii and Ullmer, 1997] Ishii, H. and Ullmer, B. (1997). Tangible bits: Towardsseamless interfaces between people, bits and atoms. Proceedings of the ACMCHI 97 Human Factors in Computing Systems Conference, pages March 22–27,1997, Atlanta, Georgia, pp. 234–241.
[Jeltsema, 2012] Jeltsema, D. (February 15-17, 2012). Memory elements: Aparadigm shift in lagrangian modeling of electrical circuits. Vienna, Austria.In proc. 7th Vienna Conference on Mathematical Modelling (MathMod), Nr.448,.
[Kim-Cohen, 2009] Kim-Cohen, S. (2009). In the Blink of an Ear: Toward aNon-Cochlear Sonic Art. Continuum.
[Lapp, 2010] Lapp, D. R. (2010). In THE PHYSICS OF MUSIC AND MU-SICAL INSTRUMENTS, pages 99–101, http://staff.tamhigh.org/lapp/book.pdf,WRIGHT CENTER FOR INNOVATIVE SCIENCE EDUCATION TUFTSUNIVERSITY MEDFORD, MASSACHUSETTS.
[Machover, 1991] Machover, T. (1991). Hyperinstruments: A composer’s ap-proach to the evolution of intelligent musical instruments. In Freeman, W.,editor, Cyberarts. Spartan Books, San Francisco.
[Mann, 2001] Mann, S. (2001). Intelligent Image Processing. John Wiley andSons. ISBN: 0-471-40637-6.
[Mann, 2007] Mann, S. (2007). Physiphones... In Proc. New Interfaces for Musi-cal Expression.
[Mann et al., 2007] Mann, S., Janzen, R., and Meier, J. (2007). The electric hy-draulophone: A hyperacoustic instrument with acoustic feedback. In Proc.International Computer Music Conference, ICMC ’07, August 27-31, Copen-hagen, Denmark, volume 2, pages 260–7.
[Overholt et al., 2011] Overholt, D., Berdahl, E., and Hamilton, R. (2011). Ad-vancements in actuated musical instruments. Organized Sound, 16(2):154–165.
[Silver et al., 2012] Silver, J., Rosenbaum, E., and Shaw, D. (2012). Makeymakey: Improvising tangible and nature-based user interfaces. In Proceedingsof the ACM Tangible Embedded and Embodied Interaction, pages 367–370,Kingston, Ontario, Canada.
[Vertegaal and Ungvary, 2001] Vertegaal, R. and Ungvary, T. (2001). Tangiblebits and malleable atoms in the design of a computer music instrument. In CHI’01: CHI ’01 extended abstracts on Human factors in computing systems, pages311–312, New York, NY, USA. ACM Press.
[Yao and Odobez, 2007] Yao, J. and Odobez, J.-M. (2007). Multi-layer back-ground subtraction based on color and texture. CVPR, pages 1–8.
The Investment of Play: Expression and Affordances in
Digital Musical Instrument Design
Joanne Cannon Stuart Favilla
Interaction Design Lab
Computing & Information Systems
Melbourne School of Engineering
The University of Melbourne
Bent Leather Band
Christmas Hills
Victoria, Australia
ABSTRACT
This paper introduces the investment of play, its role and
significance in the design and development of digital
musical instruments (DMIs). Dimension map analyses
are used to create a qualitative numerical estimate of
DMI expression. Expression is then longitudinally
compared to data sets spanning a 16year study epoch of
the Bent Leather Band. This study identifies multiplicity
of control and other parameters, as significant
affordances for DMI musical expression and skill
development. The paper argues that Expression is
proportional to the sum of invested play and the
processional affordances latent within the DMI system.
1. INTRODUCTION
Many computer music practitioners strive to build
expressive digital musical instruments (DMIs) for
virtuosic performance. An often-used quote “low entry
fee with no ceiling on virtuosity” [17] typifies what
many consider to be the optimal qualities of a DMI, i.e.
an expressive instrument that can be played
immediately. Also encouraging the development of
virtuosic skill in the years to come. DMI virtuosity
however, is yet to be clearly understood. Question: How
is virtuosity attained with a DMI? Can it be attributed to
the expressive potential of the DMI or the artist(s)?
Playing music can be categorized into a number of
activities including but not limited to performing,
practicing, improvising, exploring and self-expressing.
It can be pursued for recreation, self-development, or as
a career, music can be played alone or in an ensemble.
The importance of playing music is well understood by
traditional musicians. A significant investment of play is
considered necessary to develop the requisite
psychomotor, timing and aural perception skills for
music. Many DMIs promote ease of use, requiring little
to no investment of play or the development of skill.
Additionally, the appropriation of game controllers and
mobile phones brings highly specialized usability based
design features with them. Are these features compatible
with longer-term artistic endeavour and the development
of virtuosity or expression? To paraphrase: “ease of use
may offer a low entry fee along with a low ceiling on
skill development”.
Recent studies [9], [13] have identified the evaluation
and understanding of computer music interaction to be a
highly subjective practice, lacking a coherent overview
and theoretical framework. DMIs are often dynamic
evolving systems, undergoing continual modification to
their physical or software components. This makes them
difficult to study. Attempts to define the expressive
potential of one DMI over another are hotly contested
due to DMI practitioners’ diversity encompassing
repertoire players, improvisers, DJ (beat) artists, and
installation sound artists. As a result, the field of DMIs
has remained nascent, adapting and appropriating the
latest forms of technology to novel and often short-term
musical ends.
A culture of disposable instruments now reigns,
where instruments are made and discarded before any
long-term play is invested. Although this is an
interesting development in the history of musical
instruments, it constitutes a profound disconnection with
the art of instrument playing, and its highly evolved and
formalized practice. Disposable instruments do not
promote the facilitation of skill nor do they encourage
skilled musicians to want to play them. Disposable
instruments confine DMI practitioners to a technological
ghetto, focused solely on technologic innovation.
A number of unique and specialized digital
instrument musicians have developed their practice on
one instrument system over extended periods. Andrew
Schloss and the Radio Drum, Michel Waiswicz and his
instrument the Hands, Serje de Laubier and his Meta
Instrument, Mark Applebaum and the Mousketeers, the
Hyperstring Instruments by Jon Rose, are each
examples of instrument systems performed over
extended timeframes, exceeding decades in some cases.
This list is not exhaustive yet it is generally accepted
within the field that these artists display a highly
developed skill and sense of expression, i.e. digital
musical instrument (DMI) virtuosity.
Dobrian and Koppelman [2] define virtuosity as
“complete mastery of an instrument”. They argue that
although an instrument affords expression (i.e. J.J.
Gibson’s affordances of Ecological Psychology) [5] it is
the musician’s virtuosity that facilitates expression.
Technically speaking, virtuosity is the attribute of the
musician and not the instrument. Expression they argue,
originates from the player and not the controller
interface, “control ! expression” [2].
Gibson’s Theory of Affordances [5] compares actions
between an animal (subject) and its environment (or
object) based on the animal’s capabilities and the
environment’s qualities. “The affordances of the
environment are what it offers the animal, what it
provides or furnishes, either for good or ill…. It implies
the complementarity of the animal and the
environment...”(ibid). Affordances are best thought of as
measurable properties. “For instance we perceive
stairways in terms of their climbability, a measurable