ecoacoustic and shamanic technologies for multimedia …mburtner/files/organized... ·...

Organised Sound 10(1): 3–19 © 2005 Cambridge University Press. Printed in the United Kingdom. doi:10.1017/S1355771805000622

Ecoacoustic and shamanic technologies formultimedia composition and performance

MATTHEW BURTNER

Virginia Center for Computer Music (VCCM), McIntire Department of Music, University of Virginia, Charlottesville, VA 22903, USAE-mail: [email protected]

The author’s close connection to traditional Alaskan culturehas inspired the creation and implementation of newmultimedia instruments based on the use of ritual objects inshamanic cultures of the far north. Simultaneously, themusical processes articulated by this music are structurallytied to environmental systems in a technique discussed here as‘ecoacoustics’. In this work, performer/composer interaction,musical composition theory, multimedia performance, andmusical instrument design have been transformed in responseto these influences.

1. OVERVIEW

In the recent multimedia performance artworks dis-cussed here, formal and informal Alaskan anthropo-logical and ecological research has been merged withmy ongoing work in computer music and composition.The large-scale multimedia artworks Ukiuq Tulugaq(Winter Raven) (2001) and Kuik (2004–2006, inprogress) involve the integration of specially designedcomputer interfaces based on traditional shamanicobjects into an ecoacoustic compositional process.The specially made techno-shamanic interfaces areused to evoke the mythic powers of traditional ritualsand their close connection to environmental forms. Anattempt is made to create a new kind of staged musicalwork informed by traditional Alaskan performanceart. Technology serves as a bridge between physicaland virtual movement and sound, exploring a contem-porary manifestation of a non-Western-European artform in which culture and nature are closely linked.

These new pieces were created in the context of acomposer working primarily for the concert stage in atradition of experimental, technology informed musicborrowing heavily from the structures, procedures andinstruments of the Western-European classical tradi-tion. In the Western-European musical tradition thereis a precedent for appropriating non-Western musicalmaterials to reinvigorate its own forms. For thisreason, it is worth noting that rather than borrowingfrom Alaskan music, in this work the compositionalprocess and performance modalities themselves havebeen transformed in terms of a shifting artisticapproach influenced by natural systems and sounds.As opposed to a Western-European music borrowing

from Alaskan culture, I view this music rather asAlaskan music using a Western-European framework.

Central to the discussion is the group of piecesmaking up Ukiuq Tulugaq or Winter Raven, anevening-length multimedia work for instrumentalensembles, surround computer sound, multiple videoprojections, dance and movement art, and dramaticstaging. Earlier compositions are considered as forma-tive to the techniques employed in this large-scalepiece, and more recent pieces and works in progressare discussed as building on techniques implementedin Winter Raven.

2. INTRODUCTION: THE SYMBIOSIS OFTECHNOLOGY AND ENVIRONMENTALISM

The complex relationship between the persistence ofnatural systems and human technological develop-ment deserves a special mention in a paper devotedspecifically to forming a close connection betweenthe two. Western social conditioning often positsopposing stereotypes of the ‘environmentalist’ and the‘technocrat’, the two inhabiting mutually exclusivecontexts. In Western culture, political conditioningalso gives us opposing stereotypes of the ‘developer’for whom nature stands in the way of progress, andthe ‘conservationist’ who views technological develop-ment as an oppressively encroaching force. Theseover-generalisations are complex and not invalid, theresult of generations of Western culture’s relationshipto its natural surroundings. While prevalent, thesestereotypes are also not by any means universal. Wefind, for example, groups of eco-friendly technocratsor techno-paranoid anti-environmentalists.

Non-Western-European cultures, by contrast, canhave strikingly different relationships to the interplayof human technological development and the naturalenvironment. In extreme environments such as thefar north, the necessity of seeking a symbiosis betweentechnology and environmentalism is more pro-nounced. After being born into and living many yearsin such environments, I observed another kind ofrelationship particularly evolving from the collision ofWestern technology and an indigenous approach tothe environment. Such a symbiosis can be illustrated

4 Matthew Burtner

through the development of any number of tools thatallow people to exist in harsh environments.

3. BACKGROUND/FOREGROUND:MULTIMEDIA IN ALASKAN SHAMANICCULTURE, AND SHAMANISM INMULTIMEDIA TECHNO-CULTURE

Over thousands of years, the indigenous cultures ofAlaska have cultivated a strong tradition of multime-dia performance including dance, music, drumming,masks, costumes, story telling, and more. In practice,long before any encounter with European and Russianexplorers, this unique art form was highly tied to anecological awareness shared in the culture, relying ona calendar of seasonal subsistence living. The designprinciples for the new instruments discussed here drawon a perceived mutability of physical objects in tradi-tional shamanic cultures. The mythology of Alaska,for example, reveals fluidity between the human,animal, natural and spiritual worlds. The shamannegotiates the relationships between these worlds bymeans of carefully orchestrated rituals. At the centreof these rituals, characteristic objects such as masksand other adornment actuate the ceremonial joiningbetween the metaphysical and physical worlds.

3.1. Description of the traditional Alaskan multimediaartwork

The ceremonial performance tradition of the Inuitpeople of Alaska was an elaborate theatre (Kaplan1982). As the primary interpreter of the spirit world,the shaman would prepare the people by guidingthem through a variety of ceremonies. Through theseactivities the cosmological views of the people wereexpressed. The creation of special masks was animportant part of this ceremonial preparation. TheInuit tradition of mask building and the presentationof these masks in a multimedia artwork involvingmusic, dance, story and art can only be understoodin the context of the subsistence culture that gives riseto the art. The celebration season during which thesemultimedia events were performed comes within anannual cycle, scripted by nature, and including a sub-sistence season lasting from approximately Februaryuntil September (see figure 1).

As the long harvest season ends the people preparefor the ceremonial season, a time of spiritual reflectionand renewal. The shaman was often inferior in tradi-tional hunting skills but his or her strength camerather from his/her coupling with a Tuunrat, a super-natural helper visible only to the shaman. Indeed,the word for shaman, angalkuq, means ‘the one whocan see’. The Tuunrat is the personification of variousnatural forces or objects, or the spirits of animals.In masks carved by shaman, the Tuunrat was oftendepicted as a contorted human face.

3.2. Techno-shamanism in art

The rich possibilities of integrating new technologywith principles derived from traditional shamanicpractices has been addressed by artists such asLawrence Paul Yuxweluptun in the Inherent Rights,Vision Rights installation (Yuxweluptun 1996), andDiana Domingues’ TRANS-E, My body, My bloodinstallation, among others (Domingues 1999). Thesepieces reveal a tradition of merging biological andvirtual reality as a means of exploring the relationshipbetween body and spirit. Yuxweluptun’s installationplaces the audience within a smokehouse ritual, andDomingues’ work points to a post-biological worldwhere electronic rituals engulf bodies into a space miti-gating the transcendental physical and the embodiedvirtual. This post-biological artistic vision is set in rela-tion to the shaman’s ‘supra-biological’ metaphysicalstate; the most apparent difference between the two isthe mediating force of computer technology.

Like visual art, music has experienced similarembodiment trends in recent years. Physical modellingsound synthesis and human computer interaction have

Figure 1. The Inuit harvest festival was an elaborate multi-media theatre supported by dance, drumming, music, storytelling, masks and costumes. In this image, taken by AlfredMilotte in 1946, a pair of masked dancers representingwalruses and another elaborately costumed dancer performwith drums on a stage. In the foreground, the audience sits,facing the stage (from Alaska State Museum, Juneau, neg.

no. 1103).

Ecoacoustic and shamanic technologies 5

developed to the point where highly sensitive inter-faces can control equally responsive synthesis algo-rithms in real time (Burtner and Serafin 2002, Cook2001). Using computer controllers and sound syn-thesis facilitates a fusion between the embodiment ofa performer on the physical stage and a disembodiedcounterpoint of forms in the virtual stage. Abettingthe exchange between virtual and physical systems,advances in sonification allow for the analysis andremapping of complex data systems from one mediumto another (Kramer et al. 2004). This has openedthe possibility for new structural forms derived fromnatural systems.

4. DEVELOPING A TECHNO-SHAMANMUSICAL CONTEXT

The theory of liminality, proposed by anthropologistVictor Turner (1974) refers to an ambiguity arisingfrom everyday tasks being reinterpreted as symbolicactivities. The transformation from common activityinto ritual activity is paramount to an understandingof shamanism. Ethnographically, rituals are describedas ‘the collectively patterned performance formsthrough which processes of cultural or sacred signifi-cation are integrated into consciousness and socialpractices’ (Tomaselli 1996: 50). Liminality describesthis passing from common reality, into a symbolicunderstanding that enacts a change in an individual’spersonal relationship with herself/himself and withsociety. The symbolisation of common reality throughorchestrated ritual creates the sense of transcendenceexperienced by the participants.

Shamanic traditions in Alaska employ the use ofspecial objects to manifest spiritual understanding.Face and hand masks, staffs or clothing can act as thesymbolic axis around which liminal transformationoccurs. In the case of human-computer interaction(HCI), any common object can act as a computer con-troller. The mouse and keyboard are two simple HCIinterfaces, but the type of data communicated betweenthe human and computer can be housed in a varietyof instruments. This aspect of HCI, called ‘modulatedobject theory’, has been discussed in Burtner (2004).

My research into traditional Alaskan culture hasinspired multimedia works that take advantage ofexpressive interfaces to construct virtual reality worldsevoking the transformative rituals of shamanic explo-ration. The controllers themselves create a real-timeinterface between the artist and the media, analogousto the shamanic ritual object’s interface with thespiritual world. Here technology is used as a meansof representing the magic observed in mythology.Taruyamaarutet is a formative example of thisapproach, defining certain characteristics. The masksand staff in Ukiuq Tulugaq and Kala Alak’s ShamanHands of Kuik are then examined as applications ofthe liminal technology.

4.1. Taruyamaarutet (Twisted Faces in Wood) (1995)

Taruyamaarutet for voice, bass clarinet, marimba,percussion, computer, dancer and projected imageswas my first attempt to incorporate shamanic Alaskanritual into a musical context. This piece focuses ona special kind of ‘twisted face’ mask sometimescalled ‘Taruyamaarutet’. I first came into contact withthe Yupik twisted-face masks in Unalakleet, Alaska.Further research on the masks drew on the seminalwork by Ann Fienup-Riordan (1996), and theextensive mask collection of the Anchorage Museumof Art.

In collaboration with actress and movement artist,Melanie Anastasia Brown, dance, lighting and pro-jected images of the masks were combined with musicon the concert stage to create a multimedia spectacle.In Taruyamaarutet, the instrumentalists and computershare the stage with the movement artist and areintegrated into the theatrical context. The vocal partin particular, sung by Haleh Abghari, pairs with thedramatic movement of the dancer. Brown’s choreo-graphy emphasises the vocal part allowing it to affecther movement.

The projected images change slowly, creating ashifting staged context for the more active dance.While the singer connects with the dance, the com-puter is closely related to the projected images. Thecomputer part, performed live and conducted like theother instruments, uses my polyrhythmicon computerhyper-instrument to articulate the formal structuralunity of the composition (Burtner 1997). Twistingpolymetric systems, underlying the highly rhythmictexture, are used as an analogy to the mask images.

Taruyamaaarutet exhibits an approach to mixed-media composition that has served as a model forthe works discussed below. In terms of technology, theintegration of the computer into the ensemble, andits close connection to the media, suggested a deeperinteractive relationship between the mixed media.Ukiuk Tulugaq (Winter Raven) introduced the notionof ecoacoustics and increased the level of interactivitybetween the computer, dance and video.

4.2. Ukiuq Tulugaq (Winter Raven) (2001)

Ukiuq Tulugaq (Winter Raven) is a large-scale multi-media work for instrumental ensemble, surroundsound electronics, interactive video, dance and thea-tre. The piece draws on an Inupiaq myth in which theworld is created by Raven from snow. As the story istold, in the beginning of the world only Raven existedflying through the darkness of space in the fallingsnow. As the snow gathered and fell from Raven’swings, some of it clumped together into a smallsnowball. Raven threw the snowball into space and itgathered more snow until it was large enough to standon.

6 Matthew Burtner

Winter Raven (Ukiuq Tulugaq) metaphoricallyconnects this story with the ecological seasonalapproach of winter. Snow was present originally alongwith Raven, and so winter is taken as a symbol forrenewal and genesis.

The dramatic form of Winter Raven is a changefrom fall into winter. This linear structure is filled withnonlinear narratives, as if stories or dreams are experi-enced along the way. The movements use widely diver-gent media. Each of the three acts explores a differentontological state based on the juxtaposition of time inrelation to the seasonal change. At the first perfor-mance of Winter Raven at the University of Virginiaon 18 March 2003, the programme described the plotof the piece in this way:

Act I takes place before winter. It is fall. The family ispreparing wood, leaves are falling, and there is abundantsunlight.

Act II presents the psychological transformation intowinter. It is the most dramatic section of the piece, inwhich the stark northern landscape represented byKunikluk becomes a backdrop for the juxtaposition of thespirit/flesh and the industry/voice in the Speaking Fleshand Industrial Garden/Lost Voices movements. The com-ing of ice and the freezing of everything break industrysuddenly.

In Act III it snows. The wind blows, leaving impressionson the snow. The light changes and shadows emerge. Theanimals seek shelter, their fading prints creating anothertype of pattern on the snow. While the act is predomi-nantly about moving forward into a still place of winter,it also revolves around the notion of memory and cyclicalprocesses in general. The focus on wind itself is a memoryof Act I, and Raven appears, invoking the memory of thefamily preparing wood for the winter. We are remindedof the continuation and cohabitation of humanity andnature. While the music is still and cold at the end it ispregnant with the possibility of rebirth and resurrection;ideas that give hope for the future.

Each of the three acts contain an ecoacousticchamber ensemble piece with video (discussed in thenext section and excerpted on the Organised Sound10(3) CD as AV4-Tingnivik, AV5-Kunikluk and AV6-Snowprints), and an unipkaaq ‘story’ involving music,dance/movement art, interactive video, and a speciallyconstructed mask (excerpted on the Organised SoundCD as AV1-Siknik, AV2-Siku and AV3-Anugi). Thereis another type of movement in each act, more looselydefined in terms of media, involving some aspect of thehuman voice or body such as a wood cutter humming,a human body played as a percussion instrument,layers of spoken texts and construction, and the finalUkiuq Tulugaq movement in which Raven appears. Atthe end of the piece when Raven appears, his arrivalis articulated by three representations; the ‘Voice ofRaven’ played by an electric violin triggering ravenaudio samples, the ‘Memory of Raven’ invoked by the

masked dancer using an FM radio transceiver, and the‘Spirit of Raven’ sung by the soprano voice.

In Ukiuq Tulugaq, a dancer portraying the Shamancharacter personifies the natural forces of sun, iceand wind. A desire to evoke the shamanic relationshipbetween human and nature on the stage inspired thecreation of the masks and a shaman staff. The shamanstaff allows the dancer to capture the detailed chang-ing movement of the masks she wears. A computerreceives and processes the video signal, projecting itonto a stage screen. Video tracking technology is usedas an analogy of the shaman’s ability to effect realityby entering a dream-like spirit world.

Figure 2 shows the performance setup for theshaman masks and staff; figure 3 shows the differentmasks used. Affixed to the top of the wooden staffformed from interwoven dried vines are a light anda wireless camera. The dancer holds the staff andperforms a ‘mask video choreography’, moving thecamera and mask to change the image. The signal isreceived and processed by a computer. The processedimage is then projected onto a screen behind theperformer. Signal processing of the wireless video feedis accomplished by a combination of automation andinteraction from the staff itself.

Aniseh Khan Burtner created the video choreo-graphy. In talking about how she created thechoreographic movements, she said:

Each of the masks signified an element or a spirit, andwhat I tried to do in the choreography is channel thatelement or spirit. The Sun movement was flowing, roundand warm in the dance; and Ice was the opposite – slow,cold, frigid and tight movements. The last one wasWind and it used grand and quick movements. Thechoreography was very organic.

I saw the masks before I heard the music, so at first themovement was inspired by the masks. Then it changedbecause it was the music I was actually dancing with, andputting on the masks intensified it. The video wasn’t alarge part of the choreography because it was behindme in rehearsals and performance. During the SpeakingFlesh movement I was watching the video during theperformance, and I was lost in it. It was very profoundto watch those faces and hear my body as the music.(Khan-Burtner 2004)

The movements were mapped into software by proces-sing the incoming signal and modulating it with othervideo. In Siknik Unipkaak (Sun), a video of sparklinglight gives a changing sheen to the colourised andblurred input signal. In Siku Unipkaak (Ice), an accu-mulating fractal algorithm of the processed inputsignal gradually spreads from the centre of the screenoutwards, as if crystals are being generated through-out the piece. Finally, in Anugi Unipkaak (Wind), thescreen is split, one side is a turbulent shifting greytexture and the other side gives space for the mask tobe seen moving in and out of the turbulence. Figure 4


Figure 2. Diagram of the shaman mask / shaman staff concert setup for interactive video.

Figure 3. Each shaman mask is supported by its own choreography, music and video processing (Aniseh Khan Burtner createdthe video choreography).

8 Matthew Burtner

shows each of the three video choreography move-ments. The top of each column is a still from the result-ing video and the bottom shows the correspondingstaged movement of the dancer. The first threeaudio-video excerpts on the Organised Sound CD,AV1-Siknik, AV2-Siku and AV3-Anugi, show exam-ples from each of the staged movements. The videoshows the staged action with interactive video pro-jected on the screen behind the dancers. Aniseh Khancreated the video choreography and Sage Blaskacreated the dance choreography for this production.The footage is from a live performance on 18 March2003 in Old Cabell Hall at the University of Virginia.

In Siku Unipkaaq, the dancer’s mask image iscolourised by removing all the reds and greens. Itis then slightly blurred with a Gaussian algorithm,and filtered to allow only the changed pixels to passthrough. The mask appears brighter if it is movingfaster, and dimmer if it is still. A movie of ice is layeredwith the live mask. This image is first colourised byenhancing the blues and removing reds, then playedback in a manner that continuously scrolls a singleline of the source video horizontally outwards fromthe centre to the edges. The result is a continuouslyexpanding blue and green lattice texture, changingorganically but forming a crystal-like structure(figure 4, centre; audio-video excerpt 2, AV2-Siku).

For Anugi Unipkaaq, the dancer’s input video isprocessed by taking the preceding two frames of videoand retaining randomly selected pixels. This signal isthen sent through a Gaussian blur to the output mixer.The source video is repositioned on the right sideof the screen partially obscuring the mask when itmoves into the same side of the screen (figure 4, right;

audio-video excerpt 3, AV3-Anugi). Siknik Unipkaaquses two independent streams of sunlight on water,combining the first with the live mask input and layer-ing this mix into the second. The videos are colourisedand blurred differently to create an effect of warmth(figure 4, left; audio-video excerpt 1, AV1-Siknik).

Although simple, the wireless video staff and maskchoreography of Winter Raven were successful inestablishing an interactive media context. Encouragedby the results, I began developing more sophisticatedinstruments that could give greater control to thedancer. Kuik is a multimedia opera based on theflow of water from mountain glaciers to the oceanin Western Alaska. A Dena’ina legend about the flowof this water forms the basis of a libretto made upof place names along the Kvichak river system. Thenatural process of transformation observed in the flowof water creates the form of the piece, and the diversecultures that have made their home along the rivercontribute stories and mythologies about the water.The work involves anthropological and ecologicalfield research in an attempt to communicate throughmusic the harmony between environment and cultureexpressed in native Alaskan cultures.

This idea creates a setting for the multimediacomposition using new technologies to communicatethe magic expressed in the mythology of the culture.In Kuik, new interfaces allow theatrical gesture of thecharacters to control sound synthesis and video pro-cessing. The first of these instruments to be completedwere the ‘shaman hands’, worn by the Kala Alak char-acter. Like the shaman staff and masks, these handsare theatrical costume objects designed to function astechnological instruments.

Figure 4. Each column above shows the processed video (above) and mask dancer (below). The rows from left to right show:Sun (Siknik Unipkaaq), Ice (Siku Unipkaaq) and Wind (Anugi Unipkaaq).


4.3. Shaman hands of the Kala Alak character fromKuik (2004)

Hand-based costume objects are commonly used intraditional Alaskan dance forms. From the women’s‘dance fans’ to the giant shaman hands, the accentua-tion of hand movements contributes to the uniquechoreographic forms of these performances.

In the field of human-computer-interaction (HCI),interfaces that extend the human hands are prevalent.Michel Waisvisz’s Hands, Laetitia Sonoma and BertBonger’s Lady Glove, Butch Rovan’s SoundGlove, theReality Quest N64 NGlove (figure 5, left), the P5 3DVirtual Glove (figure 5, right), and the DT Data Glove(figure 5, centre) are just a few of the many ‘glove’controllers in use.

The shaman hands controller is based on my contactwith a pair of wooden hands in southwest Alaska.These particular hands, in use approximately 100years ago, are heavy, dramatically oversized, andvisually striking (figure 6). The stories of hands withmagical properties, including healing and travellingpowers, inspired me to use them as a model for amultimedia controller interface.

The shaman hands differ from other hand-basedcomputer interfaces in that they make no attempt tobe ergonomic. On the contrary, they are unwieldy andheavy objects. Physically, these hands are designedfor broad rather than subtle movements. They implylarge-scale theatrics. The size of the hands influencesthe approach to audiovisual mapping in several ways.Micro-level control mappings are not idiomatic forthese hands in the way they are for many hand-basedcontrollers. These hands would not be idiomaticallyemployed as a video game controller, for example.Rather, the use of broadly articulated gesturessuggests macro-level control mapping strategies.As a result, the performer works with individualcontrapuntal layers of rich material, controllingbroad relational interdependencies. Subtle gesturesare also mapped to signal processors, but because

Figure 5. Three of the many glove controllers in use: Reality Quest N64 NGlove (left), P5 3D Virtual Glove (centre), and the5DT Data Glove (right).

Figure 6. Picture of a Togiak shaman (ca 1906) in BristolBay, Alaska, showing the hands sheltering a sick boy(photo by John E. Thwaites, Alaska State Library/Thwaites

Collection/PCA-18-497).

small movements are difficult to reproduce with theinstrument, such movements are relegated to reactivefluctuations within larger interactive contexts. Spatialaudio processing mappings are idiomatic since theshaman hands visibly traverse large areas.

Following traditional principles of ritual objectdesign, special wood was gathered to constructthe shaman hands. This involved carefully choosingmaterials based on symbolic meaning as well as func-tionality. The use of 100-year-old Bristol Bay cannerysalt-fish barrels was intended to draw focus to thecomplex socio-political history of the fishing industry

10 Matthew Burtner

on the Kvichak River, and its relationship to thesalmon that traverse the river each year. Three fittedslats of a barrel were used for each hand. The slatswere attached together with leather strips and woodglue. Following the pattern from the photographs, theboards were then cut into the shape of the hands,sanded and the wood treated. This process yieldeda thin curved hand shape (see figure 7). Leather armstraps and handles on the back of the hands allow thedancer to strap the hands to her arms. Despite theirweight and size, the hands feel surprisingly freeing dueto the dramatic exaggeration of arm movements theyprovide (see figure 8).

The controller is designed to capture touch,turn and bend information from the performer (seefigure 8). Two Memsic 2125 dual-axis accelerometerchips, mounted on small circuit boards, are embeddedin the hands. The accelerometers measure movementin two dimensions independently for each hand.Earlier work in tilt sensing used an Analog DevicesADXL 202-series accelerometer employing a massand spring technique to sense tilt (Burtner 2002). Bycontrast, the Memsic 2125 senses tilt by heating apocket of air that passes by thermopiles and is detectedas it moves inside the sensor (Williams 2002).

Bend sensors attached to the wrists of the handsmeasure the elbow joint movement of the dancer. Thebend sensors are connected through an RC circuitdesign to convert analogue voltages to digital signalsfrom each pin. Force sensing resistors on the fingersprovide the hands with a sense of touch. The shamanhands software runs on a Parallax BIISX micro-processor converting the signal to a MIDI message(figure 9).

5. ECOACOUSTIC AND IMMERSIVE AUDIOENVIRONMENTS

Ecoacoustics is the name used here for an approachderiving musical procedures from abstracted environ-mental processes, remapping data from the ecologicalinto the musical domain. In the most general sense,ecoacoustics is a type of environmentalism in sound,an attempt to develop a greater understanding of thenatural world through close perception. In the field ofcomposition, this takes the form of musical proceduresand materials that either directly or indirectly draw onenvironmental systems to structure music.

The data from nature may be audio informationfrom wind or ocean waves, for example, but it mayequally be some other parameter such as temperature,geological change, etc. Ecoacoustics draws heavilyfrom the related areas of soundscape composition(Truax 1978/99, 1994; Westerkamp 2002) andsonification (Kramer et. al. 2004). It embraces thesonification of ecological models, a technique wellarticulated by Damian Keller as acoustic ecology(Keller 1999, 2000).

The approach I took to ecoacoustics in these piecesbegan as procedurally causal such that data is mappedsimply and directly. The point was first to achieve aperceivable sonic representation of the environmentalsystem. The limitations of such sonification mappingstrategies in music are clear. Mapping temperaturechanges to frequency changes, for example, is an arbi-trary assignment that disregards the inherent percep-tual differences between the senses. As a result, it couldbe said that such mappings fail because the listenerdoes not perceive the original data construct througha new medium in the same way. This failure, however,Figure 8. Several distinct shaman hands movements.

Figure 7. Outside view of completed shaman hands. Thewhite dots are fingerprints as seen on the original hands.


Figure 9. Shaman hands block diagram showing sensor placement on hands.

12 Matthew Burtner

is precisely what makes this process compelling. Inthis sense, the sonification is seen as a compositionaldistortion, rather than a failure. Composing withsonification is thus a process of controlling the degreeof distortion, mapping it in a way that maintains char-acteristics of the source data while simultaneouslyfunctioning as compelling music.

5.1. Sonification strategies in chamber music

Sonification of environmental systems has provensuccessful in forming a close link between the naturalworld and musical structure. Giacinto Scelsi (1905–1988) devised a new kind of harmonic language,now commonly called spectralism, by building musicalstructures on the natural overtone series. His com-positions, written for acoustic instruments, openedup the rich world of timbre as a harmonic languagefor musical formal exploration. By employing signalanalysis technology, composers such as Gerard Grisey(1946–1998) and Jonathan Harvey (b. 1939) were ableto extend the specificity of the spectral structures bydrawing on a more finely varied array of harmonicdifferentiations and transformations (Bundler 1996;Harvey 1999). Other composers such as IannisXenakis (1922–2001) utilised complex sonificationstrategies to map physical behaviours of natural, oftenchaotic phenomena into instrumental music (Xenakis1992). Real-time tracking and computing methodshave allowed composers to design systems in which thecomputer directly interacts with the environment.Sound installations such as Garth Paine’s Reeds forlily pond, or Judith Shatin’s Tree Music are two fineexamples of a number of real-time sonification works(Paine 2004; Shatin and Topper 2004).

My own chamber music regularly makes use ofecoacoustic structures to organise large-scale form.Patterns from wind, tides, waves, seasonal transfor-mation, melting ice, snow, etc. are common musicalresources for my chamber music. The chamber musicand video movements of Winter Raven – Tingnivik,Kunikluk and Snowprints develop a number ofecoacoustic structural techniques. In these pieces, themusic is not meant to be heard as overtly environmen-tal. Rather, its structure is infused with environmentalmodalities of time and texture. It is intended tocommunicate to the listener through shared experi-ences of natural systems. Like the changes in environ-mental systems, such as temperature, light, airpressure, etc., this music changes slowly and doesnot necessarily fit into predictable musical forms. Thepieces attempt to capture the experiential rituals ofenvironmental natural processes, imbuing the musicwith a sense of engagement we experience whencoming into contact with natural systems. Threeaudio-visual excerpts on the Organised Sound CD,

AV4-Tingnivik, AV5-Kunikluk and AV6-Snowprints,correspond to these three movements.

Tingnivik for viola, alto saxophone, piano, noisegenerators and video, from the first act of WinterRaven, maps the shifting of daylight and temperatureas the season of fall approaches into the musical mate-rial. The title is an Inupiaq word meaning ‘the time ofleaves falling and birds flying’. In this way, the pieceexplores the ecological changes of ‘fall’ as an acousti-cal system. Each instrument is assigned a specificparameter of the environmental system. The evolutionof each of these components generates the develop-ment of the instrumental lines. Like experiencing anenvironment, the musical macro timbre is the result ofthe summation of these parameters. The viola repre-sents temperature, the saxophone, light, the piano iswind, and the noise generators move throughout thehall, creating a context in which the piece sounds. Thesampled environment tracks changes over mid-dayinto evening, extrapolating this into an acousticalrepresentation of the coming of fall.

A page from the score of Tingnivik shows the viola’srising glissando as a result of the environment cooling(figure 10). The saxophone’s changing multiphonicsrepresents the intermittent light through the trees, andthe piano is bowed, fluctuating like the wind. Threeor more additional performers carry wireless noisegenerators, and move about the audience creating anenvironmental context through filtered noise.

Figure 11 shows three still images from the video ofTingnivik. The video underscores the musical processby changing from warm browns and yellows to coolerblues, by transforming from images of leafy to leaflesstrees. The video becomes increasingly windy anddistant, evoking the onset of winter. The audio-videoexcerpt AV4-Tingnivik on the Organised Sound CDcorresponds with the above music.

In Kunikluk for ensemble, computer noise generatorand video, the accompanying image of a horizon lineslightly obscured by blowing snow or ice (‘kunikluk’ inInupiaq) creates a real-time still life within which thedramatic music sounds. At one point in the video araven flies across the field of vision but other than thatmoment, the video appears as if it is a still image. Thispoised stillness of the flat, permafrost-covered arcticlandscape contrasts with the extreme annual fluctua-tion of temperature and light. Kunikluk explores thisecological texture as a musical system. Audio-videoexcerpt five, AV5-Kunikluk, corresponds to the scoreexcerpt in figure 12.

The computer noise part, created in Common LispMusic, uses a type of granular noise generator thatallows the density of randomly generated componentsto be controlled along with their distribution in aband-limited field. The result is a quiet granular noisesound, analogous to the granular ice mists visible inthe arctic landscape.


Figure 10. Score excerpt from Tingnivik.

Figure 11. Video stills from Tingnivik.

Figure 12. Score and video still excerpt from Kunikluk.

14 Matthew Burtner

Snowprints for flute, cello, piano, electronics andvideo explores snow both metaphorically and soni-cally. In this piece, a recorded walk in the snow aboveAnchorage, Alaska, serves as a departure for a compo-sition utilising a wide array of snow sounds and snowforms. Photographs of snow formations correspondto audio recordings of snow sounds and the layering ofthese ‘print’ images and sounds created the musicalstructure. In snow, prints of bodies are captured andtransformed by wind and changing temperature. Thewind leaves impressions in the form of drifts; changinglight creates shadow prints on its surface; and animalsleave their own fading tracks (figure 13).

Many types of movement in different kinds of snowwere recorded. The sounds were then mixed directlyinto the electronic part, combined with three ‘digital

prints’ of the acoustic trio. The digital prints werecreated from a scanned synthesis string, a physicalmodel of a flute, and granular synthesis piano. Theorchestration of the composition is thus an acoustictrio of flute, cello and piano; and a digital trio ofvirtual flute, cello and piano. The noisy sounds ofthe snow bind the sonic world, creating a backgroundenvironment for the instrumental/digital prints.

In the video, three types of ‘prints’ are used in thepiece: wind formations, animal tracks, and shadows.All images in the video are naturally occurringand were not digitally enhanced in any way. A cor-responding audio-video excerpt, AV6-Snowprints, isprovided on the Organised Sound CD. The score fromSnowprints (figure 14) shows the instrumental parts, agraphical representation of the snow sounds, and the

Figure 13. Video stills from Snowprints.

Figure 14. Score excerpt from Snowprints.


electronic digital prints. As in the video, there are threetypes of digital prints corresponding to each of thethree instruments.

The musical form also falls into three sectionscorresponding to different approaches to the ‘print’metaphor. This form is articulated in the video byphotographs of the three kinds of prints appearing atkey points in the music, modulating from the surfaceof the snow to shadow effects, which reveal the pres-ence of external bodies and a directional light source.This transformation is supported in the music, whichmoves from an abstraction of the print metaphor intoa kind of soundscape composition at the end wherethe footsteps of the recording engineer are heard.Here, the music reveals its source, bringing the soundwalk into focus and drawing the listener out of envi-ronmentalism and abstraction into the presence ofhuman intention.

5.2. Ecoacoustic and multichannel composition usingSOS synthesis

Like ecoacoustics, multichannel composition has abasis in acoustic ecology through soundscape compo-sition. Multichannel soundscape compositions recon-struct sonic environments through the sampling andredistribution of distinct sounds to construct exter-nally referential environments. In the approach toimmersive audio environments taken here, naturalsystems are mapped into the multichannel domain asa way of creating structure, but not as an attempt toreconstruct the environment figuratively.

Spatio-operational spectral synthesis, or SOS, is asignal processing technique developed at the VCCMby myself and David Topper (Burtner, Topper andSerafin 2002). SOS is based on recent research inauditory perception on audio objecthood as a resultof streaming theory (Bregman 1999; Kubovy andValkenburg 2001). The technique breaks apart anexisting algorithm (i.e. additive synthesis, physicalmodelling synthesis, etc.) into salient spectral compo-nents treated independently in the spatial domain.Due to the inherent limitations of audition, the listenercannot readily decode the location of specific spectra,and at the same time perceive the assembled signal. Inthis sense, the nature of the auditory object is alteredby situating it on the threshold of streaming, betweenunity and multiplicity.

When spectral parameters are spatialised in a cer-tain manner, the components fuse and it is impossibleto localise the sound. Yet when they are spatialised dif-ferently, the localisation or movement is predominantover any type of spectral fusion. Creatively modulat-ing between fusion and separation is where SOS comesinto being.

SOS explores the nature of sonic unity and multi-plicity through spatialisation of the frequency domain.The technique exploits what can be described as a

‘persistence of audition’ insofar as the listener is awarethat auditory objects are moving, but not alwayscompletely aware of where or how. This level of spatialperception on the part of the listener can also becontrolled by the composer with specific parameters.Figure 15 illustrates the basic notion of SOS asdemonstrated in the example of a square wave.

5.3. SOS ecoacoustic immersive environments

In practice, SOS has proven to be a flexible andinteresting addition to available synthesis tools.Figure 15 describes a basic procedure, but this prin-ciple is variable when the individual componentsbecome spatialised independently. In the examplesdescribed here from Winter Raven, SOS techniqueswere implemented for a multimedia ecoacousticcontext. The electronic parts for the three Unipkaaqmovements discussed above are each supported bypercussion and eight-channel computer-generatedsound diffusion using SOS techniques. In the first ofthese three pieces, Siknik Unipkaaq (the story of sun),a group of interlocking concentric paths were created(figure 16).

A tempo ratio of 1 : 2 : 3 : 4 : 5 : 6 : 7 : 8 was appliedto spatial modulation for the eight independent pathsof audio. The base tempo of the structure is modulatedglobally, accelerating from a time base of 1= 120' toa time base of 1= 20'. This yields a time structureof 120' : 60' : 40' : 30' : 24' : 20' : 17' : 15', graduallycompressing into ratios of 20' : 10' : 6.7' : 5' : 4' : 3.3' :

Figure 15. SOS recombinant principle.

16 Matthew Burtner

Figure 18. Siku Unipkaaq SOS ‘shaking’ algorithm.

Figure 17. Instrumental score for Siknik Unipkaaq.

Figure 16. Siknik Unipkaaq SOS processing.

2.8' : 2.5'. A percussion ensemble helps articulate thisprocess (figure 17).

In Siku Unipkaaq (the story of ice), a ‘shaking’ algo-rithm was employed to model the freezing of motionin the spatial domain. Each component of the icesound pans between two randomly selected pointsvery rapidly and gradually reduces movement,increasing frequency. The panning occurs in the order

of 600 to 20 milliseconds, varying for each particle ofsound. The result is a feeling of gravity pulling thesound towards a single point between the two spatialanchors. Thus the sound is ‘frozen’ into multifacetedcrystals, continually spawning new paths that areagain frozen. At any given time there are four simulta-neous spatial paths. Figure 18 depicts this motiontype.


Figure 19. Instrumental score for Siku Unipkaaq.

culture have altered my approach to performer/composer interaction, musical composition theory,multimedia performance, and musical instrumentdesign. Through the design and implementation ofshamanic and ecoacoustic multimedia technologies,this influence is brought to the forefront of the musicalartwork.

A gradual reductive process is articulated by fourglockenspiel performers. Gradually the density andvariety of pitches are reduced, focusing the frequencyenergy into increasingly reduced bands of sound.Finally, the voices slow and stick onto individualrepeated notes (figure 19).

Anugi Unipkaaq (the story of wind) utilises thesound of wind recorded in Alaska and band-passfiltered into four discrete frequency regions. Eachexcerpted wind channel is panned rapidly betweengroups of randomly selected speakers, acceleratinglogarithmically as they approach the target point. Infigure 20, each straight line represents this acceleratingcurve. Amplitude is linked to spatial change such thatthe wind sounds crescendo into each new location. Thebands of wind rush simultaneously around the space,creating a kind of ‘SOS blizzard’.

The piece is scored for a solo percussionist playinga battery of tom toms and drums, and three playersplaying a single large bass drum. At the end of thepiece, as the rhythmic structure concentrates into asingle common rhythm, the solo percussionist joins theother players at the large bass drum and the groupends together (figure 21).

6. CONCLUSION

The work described here is one example of an artistrethinking the implementation of new technologiesthrough a non-Western-informed perspective. Myexperiences growing up close to traditional Alaskan

Figure 20. Anugi Unipkaaq SOS spatial motion ‘blizzard’algorithm.

18 Matthew Burtner

Figure 21. Instrumental score for Anugi Unipkaaq.

At the beginning of Formalized Music, IannisXenakis describes a successful work of art:

Art, and above all, music has a fundamental function,which is to catalyse the sublimation that it can bringabout through all means of expression. It must aimthrough fixations which are landmarks to draw towardsa total exaltation in which the individual mingles, losinghis consciousness in a truth immediate, rare, enormous,and perfect. If a work of art succeeds in this undertakingeven for a single moment, it attains its goal . ..

This transcendental truth reminds us of the shamanicapproach to liminal transformation discussed above.Xenakis continues by challenging artists to strive toattain the ‘meta-art’ while still being able to trace itsmechanisms:

But this transmutation of every-day artistic materialwhich transforms trivial products into meta-art is asecret. The ‘possessed’ reach it without knowing its‘mechanisms’. The others struggle in the ideological andtechnical mainstream of their epoch which constitutes theperishable ‘climate’ and the stylistic fashion. Keeping oureyes fixed on this supreme meta-artistic goal we shallattempt to define in a more modest manner the pathswhich can lead to it from our point of departure, which isthe magma of contradictions in present music.

This ambition for art to transcend mere expression, isat the core of the Inuit subsistence festival, led by theshaman, who bears a strong resemblance to Xenakis’description of the artist. Edward William Nelson,the explorer/anthropologist who travelled throughoutWestern Alaska, and preserved many indigenousartefacts, describes the qualities of a successful festi-val: ‘A good festival must not only please the spirits; itmust also be long remembered so as to guide the futureactions and thoughts of individuals’.

Beyond the naturalist meta-art lies the realm ofcultural memory and spirituality.

REFERENCES

Bregman, A. S. 1999. Auditory Scene Analysis: the Percep-tual Organization of Sound. Cambridge, MA: MIT Press.

Bundler, D. 1996. Interview with Gerard Grisey. In 20th

Century Music. March, San Anselmo, CA.Burtner, M. 1997. Rhythm/Noise/Space. Masters Thesis,

Peabody Institute of the Johns Hopkins University,Baltimore, MD.

Burtner, M. 1999. Portals of Distortion. Innova Records,American Composers Forum, CD-512. St. Paul, MN.

Burtner, M. 2001. Ukiuq Tulugaq. Doctorate Thesis,Stanford University, Stanford, CA.


Burtner, M. 2002. The Metasaxophone: concept, implemen-tation and mapping strategies of a new computer musicinstrument. Organised Sound 7(2).

Burtner, M. 2004. Shamanic controller design based on atheory of modulated objects. In Proc. New Interfaces forMusical Expression (NIME-04). Hamamatsu, Japan.

Burtner, M., and Serafin, S. 2002. The Exbow Metasax:compositional applications of bowed string physicalmodels using instrument controller substitution. Journalof New Music Research 22(5). Lisse, Netherlands.

Burtner, M., and Topper, D. 2004. Recombinant spatiali-zation for ecoacoustic immersive environments. In Proc.Linux Audio Developers Conference (LAD 2). Karlsruh,Germany.

Burtner, M., Topper, D., and Serafin, S. 2002. S.O.S.(spatio-operational spectral) synthesis. In Proc. DigitalAudio Effects (DAFX). Hamburg, Germany.

Cook, P. 2001. Principles for designing computer musiccontrollers. In Proc. of New Interfaces for MusicalExpression (NIME-01). Seattle, USA.

Domingues, D. 1999. Interactivity and Ritual: bodydialogues with artificial systems. ACM SIGGRAPH.http://www.siggraph.org/artdesign/gallery/S99/essays/dianafull.html

Fienup-Riordan, A. 1996. The Living Tradition of Yup’ikMasks: Agayuliyararput, Our Way of Making Prayer.Seattle, WA: University of Washington Press.

Harvey, J. 1999. In Quest of Spirit: The Musical Thought ofJonathan Harvey. Berkeley, CA: University of CaliforniaPress.

Kaplan, S. A. 1982. In W. W. Fitzhugh (ed.) Inua: SpiritWorld of the Bering Sea Eskimo. Washington, DC:Smithsonian Institution Press.

Keller, D. 1999. touch’n’go: Ecological Models in Com-position. Master of Fine Arts Thesis, Simon FraserUniversity, Burnaby, BC.

Keller, D. 2000. Compositional process from an ecologicalperspective. Leonardo Music Journal 10. San Francisco,CA.

Khan-Burtner, W. A. 2004. Personal Interview. Charlot-tesville, VA.

Kramer, G. (chair), Walker, B., Bonebright, T., Cook, P.,Flowers, J., Miner, N., Newhoff, J., Bargar, R., Barrass,S., Berger, J., Evreinov, G., Fitch, W. T., Gröhn, M.,Handel, S., Kaper, H., Levkowitz, H., Lodha, S.,Shinn-Cunningham, B., Simoni, M., and Tipei, S. 2004.

Sonification Report: Status of the Field and ResearchAgenda. International Community for AuditoryDisplay (ICAD-10). http://www.icad.org/websiteV2.0/References/nsf.html. Sydney, Australia.

Kubovy, M., and Valkenburg, D. V. 2001. Auditory andvisual objects. In Cognition.

McAdams, S., and Bregman, A. 1979. Hearing musicalstreams. Computer Music Journal 3(IV). California.

Paine, G. 2004. Reeds – a responsive sound installation. InProc. Int. Conf. on Auditory Display (ICAD-10). Sydney,Australia.

Sam Fox Museum. 2003. Wooden Shaman Hands, SpecialCollection. Dillingham, AK.

Shatin, J., and Topper, D. 2004. Tree Music: composingwith GAIA. In Proc. New Interfaces for MusicalExpression (NIME). Hamamatsu, Japan.

Tomaselli, K. 1996. Appropriating Images: The Semioticsof Visual Representation. Arhus, Denmark: InterventionPress.

Topper, D. 2001. PAWN and SPAWN (Portable and SemiPortable Audio Workstation). In Proc. Int. ComputerMusic Conf. (ICMC). Berlin, Germany.

Truax, B. (ed.) 1978/1999. Handbook for Acoustic Ecology.Arc Publications, Cambridge Street Publishing, CD-ROM Edition, Version 1.1.

Truax, B. 1994. Discovering Inner Complexity: time-shiftingand transposition with a real-time granulation technique.Computer Music Journal 18(2).

Turner, V. 1969. The Ritual Process: Structure andAnti-structure. Chicago, IL: Aldine Publishing.

Turner, V. 1974. Dramas, Fields and Metaphors: SymbolicAction in Human Society. Ithaca, NY: Cornell UniversityPress.

Turner, V. 1982. From Ritual to Theater: the humanseriousness of play. Performing Arts Journal (PAJ).New York, NY.

Westercamp, H. 2002. Linking soundscape composition andacoustic ecology. Organised Sound 7(1).

Williams, J. 2002. It’s all about angles. Nuts and Volts.http://www.nutsvolts.com/. Parallax Inc.

Xenakis, I. 1992. Formalized Music. New York, NY:Pendragon Press (hardcover revised edition).

Yuxweluptun, L. P. 1996. Inherent Rights, Vision Rights inThe Leonardo Gallery. Leonardo Music Journal 29(1).Boston, MA: MIT Press.

ecoacoustic and shamanic technologies for multimedia …mburtner/files/organized... ·...

Documents