COLOURPOLLUTION

IDAT306 - PRODUCTION OF SPACE

HARRISON BOND10019304

CONTENTS

3 INTRODUCTION

4 IDEATION

6 INSPIRATION SYNESTHESIA7 KANDINSKY

8 VISUALISATION8 REZ9 MAGNETOSPHERE

10 DEVELOPMENT10 COMPUTE SPECTRUM10 PARTICLE SYSTEM11 COMBINING THE ELEMENTS12 SCREENSHOTS

14 CONCLUSION

15 BIBLIOGRAPHY

INTRODUCTION

This project is a study of the ambient acoustics of Pin Lane, Plymouth.

Once a relatively busy trade route to the Barbican area it is now used for little more than a pedestrian walkway and entrance to a car park. One end leads onto a high street while the other opens up to quieter residential areas, and for this reason it offers a very different audio background depending on where you are standing. Often these subtle properties of an area will be overlooked in our everyday lives and in the case of Pin Lane, the majority of the traffic through it will be en route to Co-Op or the surrounding shops. It is only when you stop to listen that you notice the majority of what is going on around you.

The goal of this piece is to expose this ambient sound by visualising it using graphical techniques inspired by synesthesia; a neurological condition that causes sensory interchange in the brain, for example, sound stimuli being perceived as coloured shapes. I will be taking raw sound data from Pin Lane and programming a piece of software capable of extracting this data to create an illustration of the range of sounds present at the location.

Ultimately I intend to offer the possibility of viewing the site in an entirely new way, drawing attention to things that would otherwise remain firmly in the background.

IDEATION

To expand my idea and focus my areas of research and development I analysed various ways and means of overcoming the potential obstacles in my project.

AUDIOThe way I gathered my audio samples from the site would be key to the style and quality of the outcome of the project, so I considered a number of ways of doing this. My main requirement was to get a suitably varied collection in order to effectively show the range of acoustic stimuli, so it was key that I had a substantial recording of the entire lane and particularly the ends, where the difference was most noticeable.

I carried out some tests with a basic dictaphone, taking multiple stationary samples and compiling them in various ways, then going on to take moving recordings in a similar way to soundwalks (see Radio Aporee) etc. The stationary recordings were of marginally higher quality but because Pin Lane is only really used by through-traffic they almost felt irrelevent - because you would rarely have a reason to stand still there - and any transitions between them that I included to simulate movement were jarring due to a lack of a cross-fade.

VISUALA personal interest of mine is synesthesia and as it offered a suitable platform to build an idea of the visual style of this piece, I chose to concentrate my research on contemporary digital art that cited the condition as an influence. Ideally I wanted to provide a visual experience that was made up of simple elements but that was flexible enough to give a clear indication of the sound data it was feeding from, while still being aesthetically interesting. For those reasons I focused almost entirely on animated/interactive rather than static pieces, because by their very nature they contained an extra dimension with which to experiment.

METHODMy options for actually creating this project were quite limited as effective sound visualisers tend to be quite big projects, and with that in mind I needed to choose a system that would allow me to get it done in a relatively short timescale. I entertained the possiblity of using the Processing scripting language, primarily because one of my main sources of inspiration was also created using it, but I had no previous experience with it and eventually after a few tests I settled on ActionScript 3 and it’s ‘computeSpectrum’ method - a platform I was slightly more familiar with, and one which had substantially more documentation to support it.

IDEATION

A basic sketch of my graphical ideas attempting to shows areas where sounds would originate, and the type of blurred aura effect I aimed to achieve with the visualiser.

INSPIRATION

Linking my project with synesthesia was important to keep my ideas grounded, when experimenting with audiovisuals its easy to get carried away with flashy graphics and lose sight of the true meaning of the piece, so it served to give me an appropriate theoretical, artistic and historical context to come back to.

SYNESTHESIAThe condition itself has been a hot topic of debate, particularly in art, for centuries. A well-rounded definition is hard to come by, but essentially synesthetes perceive multisensory experiences from single input stimuli; for example, hearing sounds will trigger the impression of coloured shapes. Auditory to visual synesthesia is one of the more common varieties and since it deals with the two most prominent senses, is also one of the most interesting. In the age of ubiquitous digital media it is also an easy metaphor to fall back on when designing a trans-sensory piece of work, and this is almost certainly why it is becoming such a buzzword in electronic art. Still as is often bemoaned by those who share an interest (see Whitelaw, van Campen etc) in the subject, much of this “synesthetic” art is mundane and irrelevent, and I wanted to try to avoid falling into this trap.

To make sure my graphical style was fairly authentic when related to synesthetic perception I read some accounts from people who experience it, and while there was no discernable consensus there were a few recurring themes. For example many people reported that short, loud noises appeared as circles or rings that expanded ike a ripple in a pond, and that sounds that were drawn out also lingered in their visual representations. From previous reading I knew that synesthesia was highly personal and as such I had no intention of being too strict with my attempt to imitate it, but gathering information from these sources at least helped me to develop a better understanding of what they see and what exactly triggers it.

My attachment to synesthesia might seem excessive, but I viewed it as a gateway to exposing these ambient sounds that litter Pin Lane and fade into insignificance as we pass through. We all know it’s easy to lose a particular sound when its surrounded by others, such as a voice in a crowd, but the addition of visual stimuli can help bring it back to our attention.

”Most people are only familiar with a small number of the provinces of the empire of the senses. It is as though your conscious perception were limited to a little garden in the middle of a jungle. You taste five types of vegetables that you grow in the garden and overlook the exotic fruits in the surrounding jungle.”

van Campen (2007: 157)

INSPIRATION

Above is “Several Circles” by Wassily Kandinsky, an abstract painter widely credited with bringing synesthesia into mainstream art in the early 20th century. In many cases his work and writings cite synesthesia as his inspiration and he claimed to see colours when he heard music, using these perceptions to paint some of his most famous works. He believed that bringing synesthesia to a wider audience would open up possibilities for new and exciting sensory experiences.

VISUALISATION

REZThis was my first visual reference for my project and one of my primary sources of inspiration, both for its graphical style and its effective use of multisensory stimuli. Fusing a classic gameplay mechanic with a contemporary digital art form, the player is heavily linked with the game in ways that weren’t explored before; every action effects not only the situation on the screen but also the style of the pulsing soundtrack, which in turn alters the face of the environment. Static images don’t really do justice to the experience, but as an indication of their intent the developers of Rez pushed their link with synesthesia and its influence on the game to the point where the creative director attempted to christen it a new genre.

Rez was notoriously good at mesmerising the player, keeping them hooked until the action inevitably let up at the end of a level, but for someone watching on it was remarkable to notice how much the electronic soundtrack and digital landscape of the game were intricately linked. Subtle hints at the involvement of the audio were everywhere - such as the waveform-style shape in the image above - but the visual effect that I particularly liked was the way that the sounds and their corresponding graphics blurred and faded into the background at precisely the same moment. That kind of link is something I wanted to achieve with my project, the effective graphical representation of sounds fading away or passing by.

VISUALISATION

MAGNETOSPHEREI’d been a massive fan of this project for a long time and once I had settled on the idea of audio visualisation it became a key inspiration for me when I was developing the graphical style I wanted. It began as a series of Processing tests by Robert Hodgin on his flight404 development blog and ended up being purchased by Apple and packaged with iTunes, such was the popularity of it by the time it reached completion. More than anything else it demonstrates the power of a decent, genuinely reactive music visualiser, especially when the competition is as mundane as the ones included with Windows Media Player, etc.

The particle effects used in this piece are what really attract me to it, they’re actually created by the sounds and react believably to them, bouncing and moving around to provide a mesmerising effect not dissimilar to Rez. Every sound, however big or small, seems to be picked up by the visualiser to an extent and this was really the key point that I wanted to address with my project - the fact that so much tends to be overlooked. In the case of Magnetosphere it almost draws out the hidden bits for you and brings them to the foreground, making them a part of the experience where otherwise they wouldn’t have been.

DEVELOPMENT

SPECTRUM ANALYSISActionScript 3’s computeSpectrum method was integral to the development of my project, it pulled data from the audio track and allowed me to manipulate it in Flash, providing me with more than enough data to create my visualisation.

“This method pulls out 512 samples of amplitude data from the sound at the point at which the method is called - 256 samples for the left stereo channel and 256 for the right stereo channel. The result is 512 samples 20 times per second, or 10,240 samples per second.”

It also provided volume values for both left and right channels which meant I could integrate that, as well as using the data from the array which was constantly updating itself. The initial implementation of the method was a challenge but once I had calculated ways to convert the data into manageable chunks it left me with plenty to work with. Basic code shown below;

PARTICLE SYSTEMHaving noted the success of other visualisers during my research (and particularly Magnetosphere) I created a particle system to form the basis of my visualiser, as it was the most flexible way to combine my sound output data with a graphical element.

DEVELOPMENT

COMBINING THE ELEMENTSCreating a fairly effective and versatile particle system didn’t prove to cause too many issues but the variable input from the sound files had to be conditioned and given boundaries before it could be used to dictate any part of the graphical output.

I wrote a function to output random numbers within a range and another to round up numbers to make them usable as standard values, and used these in conjuction with the data from computeSpectrum to control various aspects of the particles.

The amplitude levels of the samples were used to create the scale of the particles which meant that equal attention was being given to the entire range of frequencies, with bass tones being just as prominent in the visualiser as higher ones. This result could have been avoided through a few tweaks to the code but I felt it was a positive outcome, because in terms of strength a synesthete would have a (roughly) equal visual perception of all sounds regardless of where they fell on the spectrum and I wanted my project to mirror this.

The peak volume values of the left and right channels were combined and used to vary the time it took for particles to fade out once they had spawned, so louder or more constant noises would take longer to disappear. This is in fitting with reports from synesthetes, as mentioned earlier.

All values were used throughout to calculate a range rather than strict numbers and almost everything was used in conjunction with a randomiser, so the experience would be different every time.

I explored various ways to use colour but this was the one thing which I couldn’t really settle on, colour-sound correspondances varied so greatly from person to person that it was impossible to decide how to construct them, so they were set to random as well. Similarly I tried various ways of using the sound data to dictate the direction the particles travelled and the location they spawned in, but every attempt yielded unbelievable or repetitive results that I felt went against the general understanding of synesthetic perception.

Finally, I implemented a blur filter and used bitmapData snapshots to create a more lasting effect when the particles faded, and allowed them to merge in one another in order to mirror another common theme reported by synesthetes. On the next pages are some images of my project throughout its development.

DEVELOPMENT

V1 Particles spawn (up to 100) every time amplitude data is received.

V2 Particles scaling is calculated by computeSpectrum data.

DEVELOPMENT

V3 Colour and spawn-position randomisation implemented.

V4 Channel peak volumes and blur/bitmapData added.

CONCLUSION

COLOUR POLLUTIONAesthetically I think my project is quite appealing, but in a situation without adequate context to back it up I am concerned that my intentions may not be clear enough. For a synesthete the mere mention of the word would make it all clear but for many of us, literally perceiving our senses together like this is still a bizarre concept. I would have liked the connections between audio and graphics to be stronger and more obvious, but I think I was partly limited by the capabilities of my chosen platform and partly by my desire to make something which was still attractive to the casual observer. In the early phases of development I explored some techniques that used sound data to directly draw shapes using lines and fills, but all of my results were very harsh to look at.

I set out to find a delicate balance between durable aesthetic quality and authentic synesthetic correspondance and ultimately, it turned out to be far more of a challenge than I anticipated. Due to the inherently personal nature of the condition its practically impossible to make an imitation or homage to synesthesia that could ever be described as accurate, but used as a metaphor it can be a suitable tool to focus a project and ground it in theoretical and artistic conventions. Having spent so much time working on it, its hard to know whether I’m just seeing what I want to see but I feel like my visualisation does expose some ambient sound in a unique way, if nothing else simply because it literally pulls in every ounce of sound data thats available and represents it on the screen.

Furthermore, I would have liked to have more chances to test various audio recording equipment but I was restricted in that area by both availability and budget. The samples I took with an omnidirectional microphone were considerably lower quality than what I acquired with a basic microphone, and that is an area where I really want to improve this piece. However, while the variables of the visualisaer are currently tweaked to optimise the output from my existing Pin Lane sound sample, there is nothing stopping me from plugging in a better soundtrack at a later date.

A lot of the methods I used in this project were new to me so for that reason I feel that there is plenty of room for improvement, and as it happens to tick several boxes that I am genuinely interested in, I hope I can see it through to a more complete package. However, for what I set out to do, I am reasonably happy with what I ended up with. As a submission for a Production of Space module I can only hope that the connection between what I’ve made and the location we were given is clear enough, despite the fact that I’ve designed something with the potential to be used for numerous other locations, etc.

BIBLIOGRAPHY

DIRECT REFERENCESvan Campen, C (2007) The Hidden Sense: Synesthesia in Art and Science. Cambridge: MIT PressWhitelaw, M (2008) Synesthesia and Cross-Modality in Contemporary Audiovisuals. Senses & Society. 3 (3) 259-276

Rez http://en.wikipedia.org/wiki/Rez http://www.xblarcade.com/files/images/RezArea1a.JPG

Magnetosphere http://www.barbariangroup.com/software/magnetosphere http://www.flight404.com/_videos/magnetosphere/index.html

Wassily Kandinsky http://en.wikipedia.org/wiki/Wassily_Kandinsky http://z.about.com/d/arthistory/1/0/0/t/kandinsky_gugg_0910_25.jpg

AS3 computeSpectrum docs http://livedocs.adobe.com/flash/9.0/ActionScriptLangRefV3/flash/media/SoundMixer.html#computeSpectrum%28%29

Computer Arts particle effects tutorial http://www.computerarts.co.uk/tutorials/new_media/cool_particle_effects_in_flash

ADDITIONAL READINGCyberspace Nova computeSpectrum examples http://mrsteel.wordpress.com/2007/05/07/computespectrum-and-bitmapdata-filters-as3-example/

gotoAndLearn.com Advanced Filter Effects http://www.gotoandlearn.com/play?id=63

Thomas Joos computeSpectrum example http://vilebody.wordpress.com/2008/03/03/as3-computespectrum-lets-visualise-some-particles/

Kirupa.com Colourful Explosion in AS3 http://www.kirupa.com/developer/as3/colorful_explosion_page1.htm