Heat Seeker An Interactive Audio-Visual Project for Performance, Video and Web

Nuno N. Correia Aalto University,

School of Art and Design, Media Lab

IADIS Visual Communication Conference Albert-Ludwigs-Universität Freiburg, 27.7.2010

1. Introduction, Context, Objectives

Introduction – Heat Seeker

•  An audio-visual project by Video Jack – Nuno N. Correia and André Carrilho

•  Developed between 2003 and 2009

•  Audio-visual performance (2004-2008)

–  in Portugal, Poland, Germany and Russia

•  Videos: DVD (2006), YouTube/Vimeo (2007)

– DVD screened in festivals (2007-2008) –  in the UK, Brazil, China and France

•  Interactive web version (2009) – http://www.videojackstudios.com/heatseeker

Contextualization

•  Heat Seeker is related to similar projects combining sound and visuals.

•  In the 1920s, Oskar Fischinger and Walther Ruttman created “visual music” films in Germany – a combination of tinted animation with live music (Moritz 1997).

•  Fischinger was an inspiration to a younger generation of visual music artists, such as brothers John and James Whitney (Moritz 1995).

Contextualization

•  Digital artists such as Golan Levin have explored interconnected audio-visual creative expression (example: Audiovisual Environment Suite, 1998-2000).

•  John Klima’s Glasbead (1999), an “online art work that enables up to 20 simultaneous participants to make music collaboratively via a colorful three-dimensional interface” (Tribe and Jana 2007, p. 54).

•  Toshio Iwai’s Electroplankton (2005) for Nintendo DS, a group of ten interactive audio-visual games.

Golan Levin, Audiovisual Environment Suite (http://www.flong.com/projects/aves/)

Problem

•  The enjoyment of music has always been linked with the experience of “watching a performer physically produce musical sound”. (Austerlitz 2007, p. 11)

•  In laptop-based electronic music performances, the visual impact of physical musical manipulation is usually limited.

Objectives

•  The main objective: –  Combine visuals with sound in an electronic music

performance, –  creating an engaging hypermediated experience for the

audience.

•  In other words: –  to “reunite the separated segments of the musical

experience” (sound and image) realizing “Wagner’s dream of gesamtkunstwerk” (Austerlitz 2007, pp. 11-12 )

Additional Objectives

•  Create a tool for manipulating visuals with adequate flexibility and expression

•  Make the act of manipulating the visuals apparent to the audience

•  Explore other channels to present the project (beyond performances), such as DVD and Web

Additional Objectives

•  Create coherent audio-visual experiences.

•  Meaning resides “not in musical sound, (…) nor in the media with which it is aligned, but in the encounter between them” (Cook 1998, p. 270).

•  Having created the visuals and music of Heat Seeker in articulation with each other, Video Jack hoped to provide the elements for the construction of a coherent audio-visual meaning.

Methodology

•  Practice-based research

•  Researcher as developer and user

•  Questionnaires will soon be conducted to users

2. Description, Development, Spin-Offs

Description of the Software

•  The Heat Seeker software was created originally for performances.

•  It was meant to be used as a visual content management and manipulation system for performers, and also to be projected to the audience.

Description of the Software

•  The graphical interface of Heat Seeker is mainly situated in the edges of the screen, in order to emphasize the animated content in the central area.

•  The interface is visible to the audience, and is part of the visual experience, in order for the audience to see how the visuals are being manipulated in real time.

Content Development

•  Once the software tool was ready, Video Jack started preparing the audio-visual material for their first Heat Seeker performance (in 2004).

•  Nuno N. Correia had already composed part of the music.

•  The preparation of visual content to be used with the software involved a discussion regarding the themes and inspiration behind the music.

Content Development

•  André Carrilho developed animations for use in Heat Seeker based on that discussion, and his own interpretation of the music.

•  He produced additional animations, which in turn served as inspiration for more music.

•  The genres “film noir” and “nouvelle vague” were particularly emphasized, as were concepts related to “heat”.

Content - Similarity with Music Videos

•  Most music videos “do not embody complete narratives or convey finely wrought stories”; otherwise “the song would recede into the background, like film music” (Vernallis 2004, pp. 3-4).

•  The viewer becomes a participant in the music videos, as it is up to him/her to determine the ultimate meaning (Vernallis 2004, p. 10).

•  The meaning of a music video is a “puzzle” for the viewer to solve – “stories are suggested but not given in full” (Vernallis 2004, p. 37).

Project Spin-Offs

•  Performances (2004/2008) –  Two performers, with two computers running different

software: Heat Seeker for visuals; Ableton Live for audio.

•  DVD, YouTube/Vimeo and screenings (2006/2008)

•  Web (2009) –  Motivated by the experience gathered from the following

Video Jack project, AVOL (2007), which was developed as a web-based project.

–  As Heat Seeker was built using Adobe Flash, which is also a web development platform, it could be functional online with some adaptations.

Heat Seeker Performance

Heat Seeker Online

•  In Heat Seeker online, the distinction between user and audience becomes blurred.

•  The user is also audience to his/her own interactions with the software and audio-visual material.

•  It is not possible to manipulate the music.

Heat Seeker Online - Menu

3. Conclusions and Future Developments

Conclusions

•  The aims of conveying a hypermediated experience –  by creating a tool allowing for flexibility of expression,

–  and for transparency of content operation,

•  were achieved (though partly, more could be done).

•  Questionnaires conducted with users will assess conclusions in more detail.

•  With its spin-offs, the project achieved further exposure, and explored different means of involvement with the audience.

Conclusions - Performances

•  In performances, audio and visual manipulation is conducted by two different users, with different software.

•  The integration of sound manipulation with the visuals and GUI would make the sound manipulation more apparent to the viewer.

•  More could be done to better integrate the actions of the performers with the visuals.

Conclusions - Video

•  The video format is very portable and easily distributed on the web.

•  Different audiences want different levels of interaction.

•  Image capture quality could be improved.

Conclusions - Web

•  Sound manipulation capabilities should be implemented, to achieve a higher level of engagement and transparency.

•  Some visual manipulation capabilities, which relied on keyboard shortcuts, were removed for the web version. These functionalities could be implemented in the GUI.

•  A non-interactive, generative version could be implemented online.

Conclusions - Content

•  The integrated development of sound and visuals contributes to a coherent result within each of the “chapters”.

•  However, the visual and music content is not homogenous enough.

•  The project appears more as a “collection of short stories”, and not as a “novel”.

Future Developments

•  Possible additional spin-offs –  Mobile devices

–  Games consoles

–  Interactive TV

•  Follow-up project, addressing limitations identified in Heat Seeker –  Master and Margarita (2009)

–  http://www.videojackstudios.com/masterandmargarita

References

•  Austerlitz, S., 2007. Money For Nothing: A History of the Music Video from the Beatles to the White Stripes. Continuum Books, New York

•  Cook, N., 1998. Analysing Musical Multimedia. Oxford University Press, Oxford

•  Moritz, W., 1995. Color Music – Integral Cinema. In Poétique de la Couleur. Musée du Louvre, Paris.

–  http://www.centerforvisualmusic.org/WMCM_IC.htm

–  Referenced January 24, 2010.

•  Moritz, W., 1997. The Dream of Color Music and the Machines that Made it Possible. Animation World Magazine, Apr. 1997.

–  http://www.awn.com/mag/issue2.1/articles/moritz2.1.html

–  Referenced 24 January 2010.

•  Tribe, M. and Jana, R., 2007. New Media Art. Taschen, Köln.

Thank you!

•  Any questions?

•  www.videojackstudios.com •  www.nunocorreia.com •  mail@nunocorreia.com