GENERATION KILLfor electric guitar, percussion, violin, cello, 4 performers with gamecontrollers, 4 live video’s & live-electronics

Commissioned by SWR for the Donaueschinger Musiktage 2012Dedicated to and premiered by my dear friends of the Nadar Ensemble

Stefan Prins (2012)

i

Generation KillStefan Prins (2012)

I. OPERATING THE GAMECONTROLLER

1. GENERAL FUNCTIONING: ROTATIONS, AUDIO & VIDEO SECTION, TRIGGERING

Only with the Sony Playstation3 gamecontroller will the software work properly.This gamecontroller has several accelerometers built-in, which detect rotations along several axis.Furthermore, it has two X-Y joysticks (J.St.-L & J.St.-R), which can also be pressed as a button. Inthe middle of the controller, you find three buttons: “select”, “start” and the “PS3”-button.All other buttons on the gamecontroller are pressure sensitive.

As shown in Fig.2, the gamecontroller is divided in two sections (not considering the three buttonsin the middle of the controller: “select”, “start” and “playstation” button):

• LEFT half/section: this half is used for manipulations related to video.• RIGHT half/section: this half is used for manipulations related to audio.

The L1, L2 and R1 buttons are the “triggers” (or “shoot”) buttons and activate either the videoimages (L1, L2) or the audio of these images (R1).Audio and video can be activated independently, by pressing either L1/L2 (only video) or R1 (onlyaudio).

2. SELECTING AUDIO- & VIDEO-BUFFERS

Four (plus one) audio/video-buffers are preloaded on startup of the application. The performeralways has to choose which one of the four buffers to use for the playback.Toggling betweenthese different buffers can be done easily during performance using the thumb-pads: the leftthumb pad (Pad-L) and the right thumb pad (Pad-R). Pad-L selects which video-buffer is selected,while Pad-R selects which audio-buffer is used. In both cases, buffer 1 is selected using the upperbutton, the button to the right selects buffer 2, the lower button selects buffer 3, the left buttonselects buffer 4. Note: with the presets are activated (see below) the correct buffers areautomatically loaded.

3. CHOOSING AND PLAYING THE LIVE & PRERECORDED AUDIO / VIDEO

The L1 and R1 buttons activate respectively the selected (with the thumb-pad) prerecorded video-buffers and/or audio-buffer.To activate the live-camera (webcam) of the instrumentalist, the performer presses the L2 button,together with one of the buttons of Pad-L. When pressing the upper button of the left thumb-padwhile the L2 button is pressed, the webcam of the instrumentalist is activated. When pressing theleft button of Pad-L while the L2 button is pressed, the webcam of the gamecontroller-performer

J.St.-L J.St.-R

L2L1

L1L2

R2R1

R1 R2

A

B

LEFT RIGHTFig.1 Fig.2 Fig.3

Pad-L Pad-R

ii

is activated. When pressing the right and/or lower button of Pad-L while the L2 button is pressed,the webcam of instrumentalist X/Y is activated.

4. PLAYBACK SPEED OF AUDIO / VIDEO

As shown in Fig.1, the gamecontroller has two axis around which it rotates:When one rotates around the A Axis, the left point of the playback-loop moves across the buffer(see also fig.4). When rotating the gamecontroller to the left (as seen from the player’sperspective), the cursor will move to the left in the buffer & vice versa.When one rotates around the B axis, the playback speed of the loop is affected. Rotating the“head” of the gamecontroller towards the floor slows down the playback speed, rotating ittowards the sky speeds up the playback speed.

5. RECORDING & PLAYING BACK THE (INSTRUMENTALIST’S) WEBCAM (AND THEAFFILIATED AUDIO)

Both the left and right joysticks have also a button-function: they can be pushed down. Bypushing the right joystick button (J.St.-R) you start recording the input of the first webcam(pointed at the instrumentalist). By pushing it again, you stop the recording. When recording theIdle button in the capture module will turn red and will say recording (see fig.4). Each time arecording is being made, it is concatenated to the previous recording.

At specific points in the piece (with a specific preset) this recording is then loaded and will replaceone of the other four buffers. It can then be used for playback and manipulation as with all theother buffers.

6. SELECTING THE ACTIVE PART OF AN AUDIO/VIDEO LOOP

As you can see in fig.4, there are two waveform viewers for the audio. The upper waveformviewer shows the full preloaded audio buffer. The lower waveform viewer shows the selectedchunk (loop) of this buffer. To move the left (starting) point of the loop to another position in thebuffer, rotate the gamecontroller along the A-axis (as if stearing a car). To move the right (ending)point of the buffer, use the left joystick (J.St.-L) and move to the left (to make the loop shorter) orto the right (to make the loop longer).

7. ZOOMING IN/OUT ON A SELECTION OF THE BUFFER

It’s also possible to zoom in/out on the selected part of the buffer, by pulling the right joystick(J.St.-R) upwards once. On the upper waveform viewer, you will then see on which part of thebuffer you have zoomed in, while on the lower waveform viewer you’ll see the zoomed bufferitself. It is then possible to move within this (zoomed-in) selection as described in 6. It’s alsopossible to zoom-in even more on the part you select in the lower buffer. This can be done untillthe zoomed-in part of the buffer is one sample big. To zoom out all the way again to the wholebuffer, pull down the right joystick once.

8. STORING AND RECALLING BUFFER POSITIONS (WITH ITS RELATED LOOP SIZE)

By pulling the right joystick (J.St.-R) to the right once, you store the current buffer position with itsrelated loop size. This is stored until you pull the joystick to the right again to store another preset.When pulling the right joystick once to the left, this preset is recalled.

9. FORWARDS & BACKWARDS PLAYBACK

When pushing down the left joystick button, the playback direction changes from forwards tobackwards and vice versa.

10. BLACK SCREEN-MODE & FREEZE SCREEN MODE

When pushing down the left joystick-button together with the select button (first select, thenjoystick), you toggle between black screen mode and freeze-screen mode.

11. RESTART & CONTINUE MODE

There are two modes for playback: the so-called “restart” mode (in which every time L1/R1 ispushed again, the buffer is read from the beginning of the selected portion) and the so-called“continue” mode (in which every time L1/R1 is pushed, the selected portion of the buffer is beingread starting from the position where it stopped just before)

12. VIDEO DISTORTIONS

The video-“distortion” can be activated by pushing the R2 button.The “depth” of the distortion of the image is relative to how hard the R2 button is pressed.

iii

II. THE USER INTERFACE1

Each player uses the corresponding standalone application (“GenKill-player1” for player 1 etc.).Player 1 is the partner of the cellist, player 2 of the percussionist, player 3 of the electric guitarist,player 4 of the violinist.

Before opening the program, make sure that the soundcard is connected to the computer andturned on, and that the gamecontroller and 2 webcams are connected to the computer throughUSB-cables (you might need to buy a usb-hub (preferably with external power supply) to makesure you have enough usb-slots available)

After double-clicking the program will open automatically. You’ll see three windows: the “MAX”-window , which reports error messages (you might get several error-messages on start-up, but youcan ignore those), the “GenKill”-window, which is the control room (see image below), andanother window which has a generic name (like u877008625), which is the projected video-image.During performance, you’ll have to watch only the “GenKill” window (you can close the “MAX”window).

The “GenKill” window consists of 10 modules, each one with a specific function.Below some explanation about the modules. Not every meter/button is explained, as some areself-explanatory and others are not important for the performer.

1. Playback module

In the window on the left you can see the projected video-image, in case it comes from one of thepreloaded buffers.The two bigger rectangular boxes V: and A: tell you which video respectively audio-buffer is beingused.On the waveform viewers to the right of this module, you can see both the full (selected) buffer(on top), and the active (zoomed-in) part of this buffer (below).In between these two waveform viewers, there is a speed-indicator (middle = “normal” speed),and three other buttons which change states when you press them:- Continue mode/Restart: when in continue mode, the selected buffer is, when triggered,

being read out starting where it stopped before; in restart mode, the buffer is always beingread out from the beginning when it is triggered again

- Forwards/Backwards: the buffer is being read from left to right (forwards) or vice versa(backwards)

- Blackscreen/Freeze: when the buffer isn’t triggered anymore, the screen will go to blackwhen in blackscreen mode; when in freezescreen mode, the image on the screen will justfreeze in the last frame it had.

2. Output module

1 The programming of the GenKill-application was done by Josiah Oberholtzer (2012) for the composition GenerationKill

On the screen in this module you can see the image which is being projected on the screen (it’s adouble of the other window).DSP button: hit this button to open the DSP-window in which you can select the soundcard (audiodriver) you’re using.The little icon of a speaker on the lower right corner of this module should be turned on (blue)before the audio-processing is activated.Normally should be done automatically when the“INITIALIZATION” cue is triggered in the cuescript module (see below), but it can also be donemanually.

3. Input module

This module deals with the two webcams. On the left, you’ll see the webcam filming theinstrumentalist, on the right the webcam filming yourself.Push getdevices for the module to detect the active cameras. Then push on the open/closebutton, to open the “channel” (and be sure that the ACTIVE toggle is checked on the upper rightcorner).

4. ps3 module

This module visualizes the actions performed on the gamecontroller.In case moving the gamecontroller has no effect on the meters/buttons in this module, theprogram might be “looking” at the wrong device. Therefore it’s important to click getdevices,then select PLAYSTATION(R)3 Controller (or sometimes PLAYSTATION(R)3 Controller 2) from thedropdown menu on top of the module. Once this is done, click the Start Polling button just belowthe menu, to make sure the program “listens”�to the gamecontroller.

5. Capture module

This module monitors the live-recording of the input of the instrumentalist’s webcam. Therecorded video-files are automatically stored in the correct folder inside the program – no need forinteracting with this module!

6. Mixer module

This module, which doesn’t need to be manipulated on screen, routes the audio and video-inputsto “channels”.Channel 4 = the prerecorded buffers (as shown in the Playback module)Channel 5 = the first webcamChannel 6 = the second webcamChannel 7 = the webcam of one of the other instruments (these are being sent to the programthrough the network to whcih all computers are connected)Channel 8 = the webcam of yet another of the instruments (these are being sent to the programthrough the network to whcih all computers are connected)

7. Treatment module

This module, which doesn’t need to be manipulated on screen, monitors the amount of audio orvideo treatment.

iv

8. Config module

This module configurates the network to which all computersare connected. Push DISCOVER once you’re in a network withother computers.

9. Networking module

In this module, you see which IP-address is used by whichperformer, into which network you’re operating and as whichplayer. These settings are automatically loaded, so you don’tneed to interact with this module.

10. Cuescript module

This module contains all the cues (presets) of the piece.In the dropdown menu you can select the correct preset.After starting up the program, choose the menu-item“INITIALIZATION”, which initializes the patch and loads allnecessary presets to make the program performance-ready.After this is done, choose the menu-item which correspondsto the performer you are. If you’re i.e. the partner of thecellist, you choose “VIRTUAL CELLO SETUP”. Then all cellovideo-samples are loaded into the buffers (which might take awhile) and made ready for performance. REMARK: afterhaving done this, It might be necessary again to push “STARTPOLLING” in the ps3 module!

Fig. 4: GenKill window,the “control room”

v

III. CHECKLIST TO GET THE APPLICATION RUNNING

SYSTEM REQUIREMENTS:

This application has been written in MAX5.1.9, makes use of Jamoma 0.5.6 beta7 and worksperfectly on a MacBook Pro 2 GHz intel Core 2 Duo (4 GB 1067 MHz DDR3 RAM & a NVIDIAGeForce 9400M graphics card) running on Mac OSX 10.6.8.

1. connect the computer to the internet router with an ehternet cable & make sure thewireless internet connection is turned off

2. connect soundcard to computer and turn on soundcard3. connect gamecontroller to computer through USB.4. connect the webcams to computer through USB (if necessary, use a USB-hub –with

external power supply- to connect all USB-devices to the computer)5. double-click the GenKill-player1 (or player2, 3 or 4) application to start the application6. do the following in the GenKill application:

a. in the output module: click on DSP and set the audio driver to the soundcard youhave connected to your computer previously

b. press on the ps3 button in the middle of the gamecontroller, to activate thegamecontroller

c. in the ps3 module: i. click on the Get Devices button ii. select PLAYSTATION(R)3 Controller from the dropdown menu on top of the

module iii. click the Start Polling button iv. now any movement or button being pushed on the gamecontroller should be

reflected in this moduled. in the input module:

i. click on the Get Devices button ii. click on the ACTIVE toggle iii. select the correct webcam on the dropdown menu’s iv. click on the open button v. now you should see the video as it is captured by the webcams

e. in the cuescript module: i. select INITIALIZATION in the drop-down menu ii. select VIRTUAL CELLO SETUP / VIRTUAL PERCUSSION SETUP / VIRTUAL E-GTR

SETUP / VIRTUAL VIOLIN SETUP according to your part (NOTE: it might take aminute before all video’s are loaded into RAM at this point)

iii. it might be possible that you need to go back to step 5.c. at this pointm toactivate the gamecontroller again.

f. In the output module: be sure that the button with the speaker-icon is on (blue). Ifit’s not, click on it.

More info: stefan.prins@gmail.com // www.stefanprins.be