collaborative jazz improvisation as an example of cognitive and network extension david mott, john...
TRANSCRIPT
![Page 1: Collaborative Jazz Improvisation as an example of Cognitive and Network Extension David Mott, John Ibbotson, IBM 12 th May 2009](https://reader035.vdocuments.mx/reader035/viewer/2022070305/5514cea555034640138b5f46/html5/thumbnails/1.jpg)
Collaborative Jazz Improvisation as an example of Cognitive and
Network Extension
David Mott, John Ibbotson, IBM
12th May 2009
![Page 2: Collaborative Jazz Improvisation as an example of Cognitive and Network Extension David Mott, John Ibbotson, IBM 12 th May 2009](https://reader035.vdocuments.mx/reader035/viewer/2022070305/5514cea555034640138b5f46/html5/thumbnails/2.jpg)
Structure
• Introduction– extended cognition– jazz playing
• Jazz as collaborative extended cognition
• Jazz as an networkable activity
• Experiments
• Relevance to ITA
![Page 3: Collaborative Jazz Improvisation as an example of Cognitive and Network Extension David Mott, John Ibbotson, IBM 12 th May 2009](https://reader035.vdocuments.mx/reader035/viewer/2022070305/5514cea555034640138b5f46/html5/thumbnails/3.jpg)
Introduction
![Page 4: Collaborative Jazz Improvisation as an example of Cognitive and Network Extension David Mott, John Ibbotson, IBM 12 th May 2009](https://reader035.vdocuments.mx/reader035/viewer/2022070305/5514cea555034640138b5f46/html5/thumbnails/4.jpg)
What is Cognitive Extension?
• Coupling with external artefacts– counting on fingers
• Socially extended– language to share thoughts, refine concepts, execute
plans
• Sense of self– affected by interaction with others– confidence as an improver of performance
[The Extended Mind, Clark and Chalmers]
![Page 5: Collaborative Jazz Improvisation as an example of Cognitive and Network Extension David Mott, John Ibbotson, IBM 12 th May 2009](https://reader035.vdocuments.mx/reader035/viewer/2022070305/5514cea555034640138b5f46/html5/thumbnails/5.jpg)
Examples that convince me of Cognitive Extension
• Driving a car:– the car “drives itself”– I have the same “feeling of vulnerability” when
I turn that I have with my own body
• Playing jazz music– (on a good day) the piano “plays itself”
• Consider the jazz example …
![Page 6: Collaborative Jazz Improvisation as an example of Cognitive and Network Extension David Mott, John Ibbotson, IBM 12 th May 2009](https://reader035.vdocuments.mx/reader035/viewer/2022070305/5514cea555034640138b5f46/html5/thumbnails/6.jpg)
Leaving artefacts in the environment
• augment memory– memory == existence of an entity in a place– keys in same place– (plant triggers in memory)
• reminders/requests to do things– notes, paper trails
• for self and others
![Page 7: Collaborative Jazz Improvisation as an example of Cognitive and Network Extension David Mott, John Ibbotson, IBM 12 th May 2009](https://reader035.vdocuments.mx/reader035/viewer/2022070305/5514cea555034640138b5f46/html5/thumbnails/7.jpg)
Using tools
• notepads
• calculators
• hoey [cognition in the wild]
![Page 8: Collaborative Jazz Improvisation as an example of Cognitive and Network Extension David Mott, John Ibbotson, IBM 12 th May 2009](https://reader035.vdocuments.mx/reader035/viewer/2022070305/5514cea555034640138b5f46/html5/thumbnails/8.jpg)
Is jazz cognitive and capable of cognitive extension?
![Page 9: Collaborative Jazz Improvisation as an example of Cognitive and Network Extension David Mott, John Ibbotson, IBM 12 th May 2009](https://reader035.vdocuments.mx/reader035/viewer/2022070305/5514cea555034640138b5f46/html5/thumbnails/9.jpg)
Jazz group
• instruments are tools
• need to collaborate
• need to problem solve
![Page 10: Collaborative Jazz Improvisation as an example of Cognitive and Network Extension David Mott, John Ibbotson, IBM 12 th May 2009](https://reader035.vdocuments.mx/reader035/viewer/2022070305/5514cea555034640138b5f46/html5/thumbnails/10.jpg)
Use of patterns
• Some aspects of performance is realtime– no time to construct totally new melody lines– people use patterns previously learnt– but not necessarily exact note-for-note, so
cognition is still required
• Other aspects are less realtime– who is doing next solo– this can be done by problem solving
![Page 11: Collaborative Jazz Improvisation as an example of Cognitive and Network Extension David Mott, John Ibbotson, IBM 12 th May 2009](https://reader035.vdocuments.mx/reader035/viewer/2022070305/5514cea555034640138b5f46/html5/thumbnails/11.jpg)
Is playing jazz NEC?• Is it Cognitive?
– a complex problem solving activity involving sequencing, tempo, structure (you only have abstract chords for guidance), in real time
– Improvisation requires more than just muscle memory• Is it network?
– must synchronise, respond, and lead other members of the group (different roles for same player at different times in an ad hoc way)
– If you cant hear, you cant play– If the “groove” is good, you play better (mismatch of different player’s rhythms can destroy
playing)– If the “cross-musician feeling” is good you play better (eg take up of others improvisation and
rhythmic ideas)• Does the tool (the piano) assist?
– out of tune means you cant play– good key action leads to better playing– good sound quality leads to better playing– there is an intimate connection between tool and mind
• Do the tools interact?– bad matching (eg guitar and piano in same “tonal register”) can destroy playing
• Therefore playing jazz is definitely NEC!– The sum is significantly greater than the whole, and the network and tools are essential– (I am talking here about quality of playing - achieving a “jazz feel”, not basic note work)
![Page 12: Collaborative Jazz Improvisation as an example of Cognitive and Network Extension David Mott, John Ibbotson, IBM 12 th May 2009](https://reader035.vdocuments.mx/reader035/viewer/2022070305/5514cea555034640138b5f46/html5/thumbnails/12.jpg)
Do jazz performers leave artefacts?
• Does one performer leave something for later use?– to another performer– to himself– to the “environment”
• e.g. start a song with a certain rhythm, tempo?– really setting up a pattern
![Page 13: Collaborative Jazz Improvisation as an example of Cognitive and Network Extension David Mott, John Ibbotson, IBM 12 th May 2009](https://reader035.vdocuments.mx/reader035/viewer/2022070305/5514cea555034640138b5f46/html5/thumbnails/13.jpg)
Instruments as tools
• Does the particular layout of the instrument enhance the cognitive extension?
• Most jazz tools have a visual component (keyboard “lays out” chords spatially but different transpositions have different patterns)
![Page 14: Collaborative Jazz Improvisation as an example of Cognitive and Network Extension David Mott, John Ibbotson, IBM 12 th May 2009](https://reader035.vdocuments.mx/reader035/viewer/2022070305/5514cea555034640138b5f46/html5/thumbnails/14.jpg)
Do performers “play” other instruments (use other tools)
• Does piano player “play” the drums?– piano player does play rhythmically– if the piano is “heard” as drums then this is
like playing a drum?– eg the passing drum rhythm to drummer
• Does piano player “play” the bass?– e.g. if bass is lost then piano can play the
bass line to resynch
![Page 15: Collaborative Jazz Improvisation as an example of Cognitive and Network Extension David Mott, John Ibbotson, IBM 12 th May 2009](https://reader035.vdocuments.mx/reader035/viewer/2022070305/5514cea555034640138b5f46/html5/thumbnails/15.jpg)
Jazz performance - Tactical and Strategic
• Strategic– Book the hall and the group– Advertise
• Tactical– perform a piece
![Page 16: Collaborative Jazz Improvisation as an example of Cognitive and Network Extension David Mott, John Ibbotson, IBM 12 th May 2009](https://reader035.vdocuments.mx/reader035/viewer/2022070305/5514cea555034640138b5f46/html5/thumbnails/16.jpg)
Can jazz be network extended?
![Page 17: Collaborative Jazz Improvisation as an example of Cognitive and Network Extension David Mott, John Ibbotson, IBM 12 th May 2009](https://reader035.vdocuments.mx/reader035/viewer/2022070305/5514cea555034640138b5f46/html5/thumbnails/17.jpg)
Utility of the Network
• To achieve the quality of face to face?– when communication MUST be distributed, with
reduced communication, can the network rise to the same quality as face to face?
• To improve over face-to-face?– Are there situations where the network can enhance
performance eg• training• direction• better awareness of what others are doing• better awareness of holistic view of performance
![Page 18: Collaborative Jazz Improvisation as an example of Cognitive and Network Extension David Mott, John Ibbotson, IBM 12 th May 2009](https://reader035.vdocuments.mx/reader035/viewer/2022070305/5514cea555034640138b5f46/html5/thumbnails/18.jpg)
What does NEC give you?
• People playing jazz together play better than on their own
• Other players contribute to creativity
• But group must also be disciplined for this to happen– some “maverick” behaviour (strange chords,
rhythms, phrases) may add to creativity, but too much is disruptive
![Page 19: Collaborative Jazz Improvisation as an example of Cognitive and Network Extension David Mott, John Ibbotson, IBM 12 th May 2009](https://reader035.vdocuments.mx/reader035/viewer/2022070305/5514cea555034640138b5f46/html5/thumbnails/19.jpg)
Confidence
• Confidence is key to quality of playing and integration
• Easy to lose confidence if group not working together, not synchronised
• Can the players help increase confidence
• Can the network help increase confidence
• Might a bad network reduce confidence?
• Rhythmic stability seems very important
![Page 20: Collaborative Jazz Improvisation as an example of Cognitive and Network Extension David Mott, John Ibbotson, IBM 12 th May 2009](https://reader035.vdocuments.mx/reader035/viewer/2022070305/5514cea555034640138b5f46/html5/thumbnails/20.jpg)
Rhythmic collaboration (1)
• Ending a song• Actions
– looked at the drummer and played a rhythm– taken over by drummer– subsequently mutual ending
• Needed– visual communication– music communication– shared schema/pattern
![Page 21: Collaborative Jazz Improvisation as an example of Cognitive and Network Extension David Mott, John Ibbotson, IBM 12 th May 2009](https://reader035.vdocuments.mx/reader035/viewer/2022070305/5514cea555034640138b5f46/html5/thumbnails/21.jpg)
Rhythmic collaboration (2)
• Guitar lost synchronisation– Save the Last Dance for Me has unusual 5-bar
phrase lengths– Guitarist changed chord too early– Pianist emphasised new chord at the right place and
guitar resynchronised
• Needed– music communication (both ends)– musical “validation” ability– shared memory of song
![Page 22: Collaborative Jazz Improvisation as an example of Cognitive and Network Extension David Mott, John Ibbotson, IBM 12 th May 2009](https://reader035.vdocuments.mx/reader035/viewer/2022070305/5514cea555034640138b5f46/html5/thumbnails/22.jpg)
Schemas
• Schemas are typical patterns/stories we expect to happen– “eating at restaurant”– “dodgy accountants” – enemy movements?
• Musical Schemas– 4 bar phrases– “Surprise Symphony” [play example]– jazz structure: tune, solo, tune
![Page 23: Collaborative Jazz Improvisation as an example of Cognitive and Network Extension David Mott, John Ibbotson, IBM 12 th May 2009](https://reader035.vdocuments.mx/reader035/viewer/2022070305/5514cea555034640138b5f46/html5/thumbnails/23.jpg)
What is communicated?• Organisational
– who is to solo next– when a solo is finished– timing of piece– when to complete a piece
• Leadership– leadership passes between the players, semi ad hoc
• Musical Patterns– Tempo, eg change to double time– Style, eg “bebop”, “latin”– Phrases, specific phrases may be echoed, or modified– Harmony, eg player may lay down a chord substitution
and this is taken up by others
![Page 24: Collaborative Jazz Improvisation as an example of Cognitive and Network Extension David Mott, John Ibbotson, IBM 12 th May 2009](https://reader035.vdocuments.mx/reader035/viewer/2022070305/5514cea555034640138b5f46/html5/thumbnails/24.jpg)
Communication of Information
• Rhythmic Metre (beat), the gross timing of the piece sync the “tapping of feet”
• Rhythmic Style, the gross “type” of piece, eg Latin, Swing, Rock selection of accompaniment pattern
• Rhythmic Confidence/Stability, whether the drummer is reliable personal confidence need to “lead the rhythm”? sparseness of accompaniment
• Harmonic Progression, roots sync the place in the sequence more space to play richer “higher voicings”
• Harmonic Progression “where I am”???
![Page 25: Collaborative Jazz Improvisation as an example of Cognitive and Network Extension David Mott, John Ibbotson, IBM 12 th May 2009](https://reader035.vdocuments.mx/reader035/viewer/2022070305/5514cea555034640138b5f46/html5/thumbnails/25.jpg)
Communication of Information
• Rhythmic Metre (beat), the gross timing of the piece sync the “tapping of feet”
• Rhythmic Style, the gross “type” of piece, eg Latin, Swing, Rock selection of accompaniment pattern
• Rhythmic Confidence/Stability, whether the drummer is reliable personal confidence need to “lead the rhythm”? sparseness of accompaniment
• ?? “easier to play if the piano sound is good” ??
![Page 26: Collaborative Jazz Improvisation as an example of Cognitive and Network Extension David Mott, John Ibbotson, IBM 12 th May 2009](https://reader035.vdocuments.mx/reader035/viewer/2022070305/5514cea555034640138b5f46/html5/thumbnails/26.jpg)
Other’s tools
• Is it possible to extend the mind into others instruments?
• In a genuinely collaborative situation can this achieve a deeper common understanding?
• Would need intimate feedback between other tool and ones brain, via network
![Page 27: Collaborative Jazz Improvisation as an example of Cognitive and Network Extension David Mott, John Ibbotson, IBM 12 th May 2009](https://reader035.vdocuments.mx/reader035/viewer/2022070305/5514cea555034640138b5f46/html5/thumbnails/27.jpg)
Key collaborative performance aspects
• Synchronised rhythm– style (accents, patterns) varies (swing, latin)– key aspect for a good performance– key for confidence
• Harmonic structure– where are we in the chord sequence?
• Organisational structure– when do we end and change solos?
![Page 28: Collaborative Jazz Improvisation as an example of Cognitive and Network Extension David Mott, John Ibbotson, IBM 12 th May 2009](https://reader035.vdocuments.mx/reader035/viewer/2022070305/5514cea555034640138b5f46/html5/thumbnails/28.jpg)
Creating jazz NEC
![Page 29: Collaborative Jazz Improvisation as an example of Cognitive and Network Extension David Mott, John Ibbotson, IBM 12 th May 2009](https://reader035.vdocuments.mx/reader035/viewer/2022070305/5514cea555034640138b5f46/html5/thumbnails/29.jpg)
How do we create a Jazz NEC?• MIDI instruments:
– keyboard– midi drums– midi guitar (or use keyboard)
• MIDI is a stream of digitised events that can be communicated, instrumented and recorded
• Setup a MIDI “group” across the network with remote players, and play!– connect via sensor fabric– build OpenSim module to communicate MIDI?
![Page 30: Collaborative Jazz Improvisation as an example of Cognitive and Network Extension David Mott, John Ibbotson, IBM 12 th May 2009](https://reader035.vdocuments.mx/reader035/viewer/2022070305/5514cea555034640138b5f46/html5/thumbnails/30.jpg)
NEC over the Sensor Fabric
Fabric Node
Sensor Client
Sensor ClientSensor
Client
SynthSynth Synth
Plugins
Dashboard
Control
Data
Control
![Page 31: Collaborative Jazz Improvisation as an example of Cognitive and Network Extension David Mott, John Ibbotson, IBM 12 th May 2009](https://reader035.vdocuments.mx/reader035/viewer/2022070305/5514cea555034640138b5f46/html5/thumbnails/31.jpg)
Experiments
![Page 32: Collaborative Jazz Improvisation as an example of Cognitive and Network Extension David Mott, John Ibbotson, IBM 12 th May 2009](https://reader035.vdocuments.mx/reader035/viewer/2022070305/5514cea555034640138b5f46/html5/thumbnails/32.jpg)
How do we evaluate a Jazz NEC?
• Evaluation is “quality of jazz feel”– must be done by other musicians (or keen listeners)
subjective feel– BUT I believe you will get a good degree of
agreement (you can tell when a band is “together” or “tight”)
• Could you analyse the MIDI stream mathematically?– Not sure, you might pick up “gross defects” (mismatch
of rhythm, complete collapse of group), but for the “jazz feeling” I suspect this is elusive
![Page 33: Collaborative Jazz Improvisation as an example of Cognitive and Network Extension David Mott, John Ibbotson, IBM 12 th May 2009](https://reader035.vdocuments.mx/reader035/viewer/2022070305/5514cea555034640138b5f46/html5/thumbnails/33.jpg)
What can be measured?
• Degree of “music playing itself”– subjective judgement by player
• post hoc listening and recall (there I was playing well, there I got lost)
• additional effector (foot button?) pressed to indicate “im now in the flow”
• Emotional State– Emotive-like devices, measuring emotion, mood, physiological
factors
• Group review by players– post hoc listening and recall– discussion of what was going on, issues (I couldn’t hear the
guitar, the drums were lagging, I felt the group was not together)
• …
![Page 34: Collaborative Jazz Improvisation as an example of Cognitive and Network Extension David Mott, John Ibbotson, IBM 12 th May 2009](https://reader035.vdocuments.mx/reader035/viewer/2022070305/5514cea555034640138b5f46/html5/thumbnails/34.jpg)
Experiments
• Effect of network quality– can artificially reduce MIDI signals between players
• reduce volume so cant hear drummer• simulate bandwidth reduction by dropping packets
• Effect of face-to-face presence– playing over headphones vs being in visual contact
• Effect of injecting information– put drum “double time rhythm” onto the network to
see if it is taken up by others
• If things go wrong, can the group recover?
![Page 35: Collaborative Jazz Improvisation as an example of Cognitive and Network Extension David Mott, John Ibbotson, IBM 12 th May 2009](https://reader035.vdocuments.mx/reader035/viewer/2022070305/5514cea555034640138b5f46/html5/thumbnails/35.jpg)
QOI
• Quality of Information measures could be applied to performance?
![Page 36: Collaborative Jazz Improvisation as an example of Cognitive and Network Extension David Mott, John Ibbotson, IBM 12 th May 2009](https://reader035.vdocuments.mx/reader035/viewer/2022070305/5514cea555034640138b5f46/html5/thumbnails/36.jpg)
Measuring vs changing
• Could just measure a performance
• Could introduce a change and see the effects
• It may be premature to experiment with the effects of introduced change
![Page 37: Collaborative Jazz Improvisation as an example of Cognitive and Network Extension David Mott, John Ibbotson, IBM 12 th May 2009](https://reader035.vdocuments.mx/reader035/viewer/2022070305/5514cea555034640138b5f46/html5/thumbnails/37.jpg)
Simple intervention – play to click
• Rising to f-f:– prioritise click over full drums, given reduced
bandwidth– agent to approximate drums and send to players
• Improving f-f– already used in real performances– would click give confidence to new players– intervene with double time click?– auto quantise drums – would this increase
confidence?
![Page 38: Collaborative Jazz Improvisation as an example of Cognitive and Network Extension David Mott, John Ibbotson, IBM 12 th May 2009](https://reader035.vdocuments.mx/reader035/viewer/2022070305/5514cea555034640138b5f46/html5/thumbnails/38.jpg)
BeatFinder
• Locate “beats” in music– 1, 2, 3, 4, 1, 2, 3, 4– a complex task, lets assume we can do it
• Show coloured screen for “in time”, “early” and “late”?
• Detect individual accuracy of beats• Detect collaborative accuracy of beats• Work out and display where in the music
we are
![Page 39: Collaborative Jazz Improvisation as an example of Cognitive and Network Extension David Mott, John Ibbotson, IBM 12 th May 2009](https://reader035.vdocuments.mx/reader035/viewer/2022070305/5514cea555034640138b5f46/html5/thumbnails/39.jpg)
Displaying place in tune
• use beat tracker to calculate place in tune– jazz navigational structure usually fairly
simple– would need to visualise place in context?
• would this improve confidence in new player?
• or take away the skills learning?
![Page 40: Collaborative Jazz Improvisation as an example of Cognitive and Network Extension David Mott, John Ibbotson, IBM 12 th May 2009](https://reader035.vdocuments.mx/reader035/viewer/2022070305/5514cea555034640138b5f46/html5/thumbnails/40.jpg)
Experiments with beat finder
• Surely groups already synchronise their playing?– NO, not if nervous or multiple versions of rhythm and
not clear where beat is
• Possible experiments:– For individual practise– Dashboard view of collaborative performance– See other players beats (all, or selected)– One player (drummer, trainer) pushes a
synchronisation pulse to other players
• What effects would this have?
![Page 41: Collaborative Jazz Improvisation as an example of Cognitive and Network Extension David Mott, John Ibbotson, IBM 12 th May 2009](https://reader035.vdocuments.mx/reader035/viewer/2022070305/5514cea555034640138b5f46/html5/thumbnails/41.jpg)
Using network for training
• transcription/copying is very important for learning
• could a trainer play examples to copy?– in the context of a networked performance– note-for-note copy or approximate patterns
• could a trainer give real-time cues?
![Page 42: Collaborative Jazz Improvisation as an example of Cognitive and Network Extension David Mott, John Ibbotson, IBM 12 th May 2009](https://reader035.vdocuments.mx/reader035/viewer/2022070305/5514cea555034640138b5f46/html5/thumbnails/42.jpg)
The relevance to ITA
![Page 43: Collaborative Jazz Improvisation as an example of Cognitive and Network Extension David Mott, John Ibbotson, IBM 12 th May 2009](https://reader035.vdocuments.mx/reader035/viewer/2022070305/5514cea555034640138b5f46/html5/thumbnails/43.jpg)
What is the point of jazz in the ITA?
• An example of extended mind– well understood, easy to relate to (for some)– an articulation of the abstract cognitive
concepts
• NATO uses it as an example of C2– ????
• Realistic test bed of some complex experiments
• Stimulation of new ideas
![Page 44: Collaborative Jazz Improvisation as an example of Cognitive and Network Extension David Mott, John Ibbotson, IBM 12 th May 2009](https://reader035.vdocuments.mx/reader035/viewer/2022070305/5514cea555034640138b5f46/html5/thumbnails/44.jpg)
Metaphors
• (Lakoff) Cognitive metaphors bridge from concrete to abstract– LIFE IS A JOURNEY– TIME FLOWS FORWARDS– EMOTIONS GO UP AND DOWN
• (Saslow) Musical Metaphors– MUSIC IS A PATH– MUSIC IS A FORCE
![Page 45: Collaborative Jazz Improvisation as an example of Cognitive and Network Extension David Mott, John Ibbotson, IBM 12 th May 2009](https://reader035.vdocuments.mx/reader035/viewer/2022070305/5514cea555034640138b5f46/html5/thumbnails/45.jpg)
The FORCE Metaphor
MUSICAL EXAMPLE Military TERMSDEFENCE, SEIZE
![Page 46: Collaborative Jazz Improvisation as an example of Cognitive and Network Extension David Mott, John Ibbotson, IBM 12 th May 2009](https://reader035.vdocuments.mx/reader035/viewer/2022070305/5514cea555034640138b5f46/html5/thumbnails/46.jpg)
Jazz as extended mind
• coupling with external artefacts– finger positions on keyboard to calculate harmonic
progressions
• socially extended– other people as memory of tunes, styles, rhythms,
harmonic progressions– synchronisations of rhythms
• sense of self– jazz playing in social context affects sense of self