the deckle project : a sketch of three sensors · the deckle project : a sketch of three sensors...
TRANSCRIPT
The Deckle Project : A Sketch of Three Sensors
Hongchan ChoiCenter for Computer
Research in Music andAcoustics(CCRMA)Stanford UniversityStanford, CA, USA
John GranzowCenter for Computer
Research in Music andAcoustics(CCRMA)Stanford UniversityStanford, CA, USA
Joel SadlerMechanical Engineering
Stanford UniversityStanford, CA, USA
ABSTRACTThe Deckle Group1 is an ensemble that designs, builds andperforms on electroacoustic drawing boards. These draw-ing surfaces are augmented with Satellite CCRMA Beagle-Boards and Arduinos2.[1] Piezo microphones are used inconjunction with other sensors to produce sounds that arecoupled tightly to mark-making gestures. Position trackingis achieved with infra-red object tracking, conductive fabricand a magnetometer.
KeywordsDeckle, BeagleBoard, Drawing, Sonification, Performance,Audiovisual, Gestural Interface
1. INTRODUCTION
Figure 1: Drawing boards proportional to their re-spective sensors.
A variety of musical practices have emerged from an im-pulse to draw sound. In the early twentieth century, com-posers like Arseny Avraamov and Rudolph Pfefiinger paintedpatterns on the optical tracks of film. This practice initi-ated an early form of sound synthesis through direct graphic
1https://ccrma.stanford.edu/groups/deckle/2http://www.arduino.cc/en/Guide/ArduinoNano/
Permission to make digital or hard copies of all or part of this work forpersonal or classroom use is granted without fee provided that copies arenot made or distributed for profit or commercial advantage and that copiesbear this notice and the full citation on the first page. To copy otherwise, torepublish, to post on servers or to redistribute to lists, requires prior specificpermission and/or a fee.NIME ’12, May 21 – 23, 2012, University of Michigan, Ann Arbor.Copyright remains with the author(s).
manipulation of the medium, a process used to great effectby artists like Norman McLaren and Daphne Oram.[4] Inthese practices there is a delay between marking the ma-terial and the production of sound via the photosensitiveelement on the projector. Our drawing ensemble aims toclose that gap between mark and sound by amplifying andprocessing drawings as they are made.
To do this, the first step was simply to draw on an ampli-fied surface and attend to the sound of an otherwise visualprocess. Brought to the fore, this fricative of graphite onpaper is no longer merely the muted outcome of a visualprocess; now gestures arise as much for their sound as theresulting image. The drawings that emerge reflect in theirform these entangled modes of attention.
This platform for composing and performing with draw-ing gestures is in part inspired by a recent piece by Mark Ap-plebaum, Straitjacket where amplified easels are used per-cussively in the fourth movement. We also look forwardto the emerging work Drawn Together by The OpenEnd-edGroup3 where computer animations and sound augmentthe real time sketch.
2. INSTRUMENT DESIGNThree instruments emerge from these experiments. Theirhardware design is determined in part by the affordances ofeach sensing technique. To explore these techniques we useBeagleBoards running PureData for sound synthesis anduser interaction.
2.1 Tiny and Mighty: A New Form FactorOne of our objectives is to leverage the tools of computersynthesis without the usual dependence on a peripheral lap-top. For this it was necessary to embed a powerful computerin the instrument itself. The Beagleboard, a micro-sized yetfull-featured computer, was implemented in each drawingboard to make them self contained with minimal cables.
The Beagleboard is still developing and we encountereda few constraints. For example, in the absence of a devicedriver for OpenGL implementation, we had to take the visu-alization/interaction module out of the box to run it on thelaptop instead. Such experiments reveal present limitationsof the Beagleboard, but also suggest possible directions forits ongoing development.
2.2 Using Magnetometer for 2D SensingTo keep sound and gesture closely coupled, it was necessarynot only to process resonance from the board, but also totrack the position of the drawing utensil. To acquire this2D position data, we used a magnetometer, a sensor thatis conventionally used to detect the surrounding magneticfield resulting in a 3D vector oriented to the north pole.
3http://openendedgroup.com/
Figure 2: Mag : magnetometer drawing board
We used a small magnet fixed to the drawing utensil tovary the orientation of the sensor inside the board and getmeaningful data related to the drawing.
A magnetometer (Honeywell HMC-5883L)4 is connectedto an Arduino Nano board, and communicates with a Bea-gleBoard via the serial port. This setup requires specificcircuitry and firmware. We programmed the firmware toget a 3D vector using a custom Arduino library for theHMC-5883L magnetometer [5]. With this 3D vector ori-ented to the north end of a magnet, we projected it into a2D plane to get ”magnitude”, or distance between the sen-sor and the magnet, and the heading angle to the magnet.This implementation gives us 2D polar coordinates.
2.3 Pseudo-2D Sensing with Layered Conduc-tive Fabrics
The fabric board is a also 2D position sensing system using aEeontex conductive fabric (NW170PI-900)5 with 900 Ohmsper square inch resistance. The Eeontex fabric is a soft con-ductive textile that has a variable resistance depending onapplied pressure. Two sheets of Eeontex fabric (10 by 10inches) can be layered orthogonally on top of each other inorder to create the equivalent of an one dimensional slidingpotentiometer. One fabric has a voltage applied at oppo-site ends of the fabric (0V and +5V respectively), whilethe other fabric has an analog sensing line connected to theanalog input pin of an Arduino Nano. When pressure isapplied to a point on the surface, a variable resistance isproduced depending on the position of touch. In order toachieve 2D position sensing, the two layers are alternatedrapidly at 25-50ms, so that the horizontal and vertical po-sition are alternatively polled at any point on the circuit.The Arduino Nano controls the switching parameters in itsfirmware. (See Figure 4.)
2.4 Infra-Red Optical Tracking with WiimoteUsing the Wiimote to track specific objects is now a widespreadmethod [3]. We also implemented an optical tracking sys-tem with Wiimote, but took it one step further in termsof user interaction and musical application, discussed inthe next section. In order to achieve a solid platform forinfra-red tracking, we deployed an algorithm for 4-points
4http://www.sparkfun.com/products/105305http://www.marktek-inc.com/eeontexconductextiles.htm
Figure 3: Fab: conductive fabric drawing board
homographic calibration based on Processing with wrj4p56
library, which provides reliable and stable platform for theperformance. The 4-points calibration process compensatesfor distortions arising from the tilted viewing angle of thecamera and ensures that tracking points are projected cor-rectly in a virtual 2D plane. Using this procedure, we canmap a tracking point to our musical system without adjust-ing physical setup of the infra-red camera, for example, itsviewing angle and position. In addition to that, the wrj4p5
library guarantees highly stable Bluetooth pairing betweenthe Wiimote and a laptop, which can be otherwise unreli-able.
Optical tracking in the above process depends both on aBluetooth connection and an OpenGL visualization, bothof which are not working properly on the current versionof the BeagleBoard. Therefore, it is necessary in this caseto run Processing on a laptop whose applets handle all thecomplex operations such as Wiimote paring, OpenGL visu-alization, matrix transformation, and collision detection inthe Harmonic Table. The final parameters such as pitch,moving speed and angle are sent to a PureData patch run-ning on the BeagleBoard via OpenSoundControl protocol.
6http://sourceforge.jp/projects/wrj4p5/
Figure 4: two conductive fabrics for pseudo-2Dsensing
Figure 5: Optik : drawing board with infra-red op-tical tracking
3. SOUND SYNTHESIS AND INTERACTIONDESIGN
We implemented a package of encapsulated objects for Pure-Data. The package hcPDX7 features many useful featuressuch as onset detection, envelope following, and basic build-ing blocks for sound synthesis. Creating hcPDX package wasnecessary in establishing sonic consistency between threedrawing boards and facilitating rapid experiments and col-laboration. We found the monitoring and mapping featuresin hcPDX were useful for analyzing the incoming data fromvarious sensors including piezo microphones, helping us toaccelerate a process of ”error and trial” by plugging in pa-rameters to synthesis engine with minimum effort. Withthis robust analyses of the available data, we could thenmap the boards to different types of synthesis, as well as in-corporate the overall sound of the three boards playing to-gether. The following sections describe the software design,which is mostly about interpreting user input and mappingdata to sound synthesis.
3.1 "Mag": playing with feedback delaysSince the implementation of magnetometer required a cus-tom firmware for Arduino, we deployed a comport object inPureData and devised a custom serial data stream parser.This board utilizes the audio signal from a piezo microphoneattached underneath the surface as a source material, andits sound processing is mainly based on hcPDelay~. This ob-ject consists of two delay processors vd~ with variable delaytime and homographic interpolation which allows the per-former to change the delay time without noticeable glitches.This use of delay units as playable musical parameters hasbeen quite popular in electronic music. With extensive feed-back, these delay units sound like a pitched string instru-ment.
The magnitude data transformed by hcScale is mappedto the feedback parameters for two delay units, and theheading angle is mapped to their delay time. Since theresponsiveness of the magnetometer is fairly high, the as-sociation between the position of magnet and parametersof hcPDelay~ enables a performer to transform the soundof strokes on the drawing board into dynamic and ever-changing electronic sound while maintaining its organic soundcharacteristics. We found this drawing board was the mostsuccessful of the three in terms of expressivity and robust-
7https://ccrma.stanford.edu/groups/deckle/hcPDX
Figure 6: signal flow diagram of Mag drawing board
ness.
3.2 "Fab": cross-synthesis on strokesAlthough pseudo-2D tracking was achieved on the fabricboard, the noise floor in the data streams was considerablyhigh and we had to apply cascaded low-pass filters with hc-
Smooth. This filtering introduced latency to the responsetime of the sensors. A 3Hz cutoff frequency was used tomake this latency inaudible which also made the onset de-tection work properly. We used thehcXDetect2~ object foronset detection and envelope following, allowing the per-former to trigger sound and control the overall loudnesscontour. A pair of hcScale objects estimates horizontaland vertical pressure points on the board, and produces pa-rameters (pitch, modulation index) for the sound synthesismodule.
In this board, hcFMOp2~ and hcXSynth~ objects performsound synthesis; hcFMOp2~ object generates synthesized soundbased on FM synthesis technique and hcXSynth~ performscross-synthesis, superimposing the input audio signal fromthe drawing board on the sound from the FM synthesizer.This synthesis scheme aligns with our goal of drawing soundas we can introduce a sense of pitch while maintaining thenatural timbre of the sketch. At the end of the synthesisroutine, hcPDelay~ creates an echo-like reverberation help-ing the sound to be integrated into the overall timbre of theensemble.
Figure 7: signal flow diagram of Fab drawing board
3.3 "Optik": drawing on Harmonic TableWith many more degrees of freedom, the infra red trackingsystem motivated us to design a more sophisticated systemwhere a Harmonic Table was projected on the board. Theperformer could draw through this map of hexagonal re-gions, triggering discrete notes. Drawing geometric shapeson this board brings diverse musical results; linear marksresult in melodic contours whereas a triangle on adjacenthexagons plays minor or major chord depending on the di-rection of the shape.[6] The sound synthesis engine is basedon FM synthesis with phase modulation and onset detec-tion. To feature this instrument as a pitched system, wedid not include the sound from the microphone in the syn-thesis loop. The sound from the microphone is used solelyfor onset detection and envelope tracking, which is doneby hcXDetect2~. Also the speed and heading angle of thedrawing utensil are mapped to the timbral quality of theFM synthesizer and the cut off frequency of a resonant low-pass filter respectively. This combination results in a highlyexpressive and responsive musical instrument: tapping theboard triggers an individual note and scratching the boardwith variable pressure creates a contour of notes with a dy-namic change in timbre.
Figure 8: projected visualization of Harmonic Tableon ”Optik” drawing board
As shown in figure 8, we projected the visualization ofHarmonic Table on the drawing board in order to providea performer with visual feed back. Being able to draw cer-tain geometrical shapes on a board brings diverse musicalpossibilities; a linear trajectory of drawing gesture resultsin playing a certain musical scale and drawing a triangleon adjacent hexagons plays minor or major chord tones ac-cording to the direction of the shape.
4. HAPTICSThe benefits of vibrotactile feedback in electronic instru-ments is well established. O’Modhrain and Chafe discussthe increased control on the theremin when an elastic bandis secured between the hand and the instrument.[7] Danionet al. showed the same result with a virtual damped springthat improved motor control as well as facilitated learningof tasks.[2] We anticipate that similar results would arise ifyou compared control with drawings in the air with thoseon paper.
These instruments appropriate the very familiar tactilefeedback of pencil and paper. The use of an already prac-ticed series of gestures (drawing) with familiar sensory feed-
back makes the interface engaging and almost immediatelyaccessible for exploration. The already known precision ofshading and texture one can achieve with a pencil is hereco-opted for sound.
Our objective was to keep the tactile feedback associatedwith drawing unchanged, while introducing engaging sonicoutcomes beyond the simple amplification of the surface.The sensors are therefore tailored to these aims. For ex-ample, we use only two of the three axes provided by themagnetometer. The z-axis could sonify vertical gestureswhen the writing implement is lifted from the page. Thishowever would dissociate sound from the mark and the en-gagement with material. It would become like the theremin.The piezo sensor under the board is used to trigger onset,and as such limits sound to the plane where the artist alsofeels the resistance associated with sketching. The sensor isconstrained to correspond to a physical configuration, pro-ducing a kind of ventriloquy for the performer and listener,where the sound is perceived as emerging from the markitself.
5. CONCLUSIONWe have demonstrated 3 alternative instruments to sonifydrawings using position and acoustic onset sensing. Theinfluence of the sensors on the overall outcome is most ap-parent in the varying dimensions of the sound boards. Eachsensor had a set of affordances that influenced our choicesin design and code. We look forward to implementing othersensing technologies in the next set of instruments.
6. ACKNOWLEDGMENTSWe would like to thank Ed Berdahl and Wendy Ju for theirexpertise in human computer interaction and design. Weare also grateful for the indefatigable David Kerr who pro-vides many ideas for future direction. And finally we wouldlike to thank previous instructors who have helped developthis project, Chris Chafe and Ge Wang.
7. REFERENCES[1] E. Berdahl. Satellite ccrma: A musical interaction and
sound synthesis platform. Proceedings of theInternational Conference on New Interfaces forMusical Expression, 2011.
[2] F. Danion, J. S. Diamond, and J. R. Flanagan. Therole of haptic feedback when manipulating non-rigidobjects. Journal of Neurophysiology, pages 443–441,2011.
[3] J. Lee. Hacking the nintendo wii remote. PervasiveComputing, IEEE, 7(3):39–45, 2008.
[4] T. Levin. ”tones from out of nowhere”: Rudolphpfenninger and the archaeology of synthetic sound.Grey Room, pages 32–79, 2003.
[5] LoveElectronics. Hmc5883l tutorial and arduinolibrary.https://www.loveelectronics.co.uk/Tutorials/8/
hmc5883l-tutorial-and-arduino-library. RetrievedFebruary 2012.
[6] C. MUSIC. Harmonic table note map. http://www.c-thru-music.com/cgi/?page=layout_notemap.Retrieved February 2012.
[7] M. Sile, C. Chafe, et al. Incorporating haptic feedbackinto interfaces for music applications. In In Proceedingsof International Symposium on Robotics andApplications(ISORA). World Automation Conference,2000.