1

CSP01-02 - Introduction

Introduction

Lecturer:Smilen Dimitrov

Cross-sensorial processing – MED7

2

CSP01-02 - Introduction

About the course and communication

• Course: Cross-sensorial processing – MED7• Course lecturer – Smilen Dimitrov• Contact per e-mail welcome: sd@imi.aau.dk (smilen@smilen.net)

– When writing, you are welcome to use your group e-mail– However, if you do not use the group e-mail, but your own

individual addresses, please include the other group members’ addresses in Carbon Copy (CC) when writing the e-mail

– Course website: http://www.smilen.net/csp/

• Course requirements – PE course– Provide an illustration of cross-sensory processing in a

technical context, and provide inspiration for the projects– Semester theme - understand and develop interactive systems

which combine audio, vision and tactile information. – Implement content as studied in the the other semester

courses – Cross-modal processing and Digital Culture

3

CSP01-02 - Introduction

Introduction

• What’s in a name? – The term "cross-sensory processing" - origin in neuro-science.

• Natural origins of cross–sensory processing in living beings – to allow for better interaction within the environment

4

CSP01-02 - Introduction

In biology / neuroscience

• Sensory systems exist to help support an animal (or animat) in its ecological niche. They exist to provide the input that enables the animal to determine its actions.– When the sensory system is working effectively, cross-modality

improves our responsiveness, and interaction with our environment.

Example: For a cricket, audition is primarily to allow the assessment and finding of a mate: each male makes a noise, and the females locate suitable males. Thus, for crickets, what matters is very specific: they require to assess a song, and to locate the singer!

5

CSP01-02 - Introduction

In biology / neuroscience

• Cognition a general concept that is concerned with acquiring information about the world, representing and transforming this information as knowledge, and using this knowledge to direct our attention and behavior.

• Three components of cognition: – Acquisition – Representation – Use

• Perception is a cognitive process that is directly involved with the detection and interpretation of sensory information. Perception requires a direct connection between a person and the object being perceived so that sensory signals can be directly processed.

– Hearing an animal in a forest environment running ahead of you and to your right is a direct sensory experience and, therefore, perception.

– Thinking about how long it would take you to drive to the mall is not perception.

6

CSP01-02 - Introduction

In biology / neuroscience

Some research areas:

• The role of attention in perception - brain mechanisms of attention operate, at least in part, by selectively monitoring inputs arising from particular subgroups of peripheral nerve fibers.

• Interactions between stimuli presented to different sensory modalities.– functional brain imaging studies have shown how one type of

sensory stimulus (for instance touch) activates a different sensory area in the brain.

– irrelevant stimulation in one modality may affect the speed and accuracy of responses made to stimuli in another.

• The role of stimulus context in perception - contextual effects can reflect adaptation-like changes in perceptual systems.

7

CSP01-02 - Introduction

In biology / neuroscience

• The brain's task is to sort through the massive and multiple streams of information it receives and to couple those signals that, regardless of their modality, should be related (bound) to one another because they are derived from a common event – to weigh the inputs it receives from the different senses in

producing a final perceptual output or experience.– Gestalt grouping principles - coincidence in space & time

are two of the cues used by the brain to solve the binding problem - BUT there are a number of others

Spatial coincidence controls overt orienting responses (space is inherent to its functioning)• Spatial coincidence plays an important role in behaviors (tasks) that involve some kind of spatial discrimination (explicit or implicit).

• Spatial coincidence sometimes plays a small role in other tasks (eg AV motion integration; AV & VT temporal perception).

• For some effects of audiovisual integration, spatial coincidence doesn't matter - McGurk effect

8

CSP01-02 - Introduction

Technologic implementation

• Technologic simulation – why? – whereas speech recognition systems have achieved real-time continuous

operation, artificial systems, designed for vision or olfaction are far less advanced, yet the combination of different information sources, or senses, may help overcome some of the processing limitations.

• Implementation in technology – start with a similar goal – Integration of multiple types of sensor data – in order to allow an

(computing) entity to better interact with (respond to) its environment

– Must simulate the entire entity, as well as conceptualize its interactions with the environment

– Must weigh (discriminate between) the inputs received from the different sensors, in producing a final perceptual output or experience.

– Two major problems – localization (where is it?) and identification (what is it?) of an event

– Must define ourselves what a common event is, and what it’s respective sensory cues would be

9

CSP01-02 - Introduction

Base exercise

• The technological implementation of a cross-sensory perception system will be discussed through a single base exercise, to be performed in class during the semester

• The base exercise is about implementing a immobot (immobile robot), equipped with a “head” made of camera (or pair) as visual sensory system, and a pair of microphones as an auditory sensory system

10

CSP01-02 - Introduction

Base exercise

• Each sensory system will have its own dedicated PC to perform the given sensory processing

• Raw sensory data needs to be converted to messages that give information about detected events – sent through a network

• These messages are sent to the “cognitive process” program on the “brain” computer, which determines a state based on them. States:

-“nothing is happening”, - “something is happening” -“event identified”

11

CSP01-02 - Introduction

Base exercise

• The state determined by the cognitive program needs to be output – actuation

• Media actuation – showing text, rendering real-time animation or sound based on the input – possibility for implementing different content• Motor actuation (possibly) – where a given state would activate a programmable motor, and cause the “head” to rotate – interaction with environment.

12

CSP01-02 - Introduction

Base exercise

• Important – to determine proper sensory processing algorithms, and to define what constitutes an event.– A simple example – a defined event could a wooden cube of a

given color, producing sound when being hit with a stick

• Audio processing – identifying if a sound is a wooden cube click, and sound (audio) localization, and providing the space coordinates of this auditory cue

• Video processing – performing feature extraction, identifying if the features (color) match the wooden cube, and providing the space coordinates of this visual cue

13

CSP01-02 - Introduction

Base exercise

• Each type of identified cue, on its own could move the state from “nothing is happening” to “something is happening” – However, only when both auditory and visual cue can be

related (bound) to each other, can the process conclusively state that “event is identified”

• Can be understood through this simple block model

– In the project, you have free choice of sensory systems and possibilities for actuation

• Tasks– Processing algorithm for visual data (localization) – Processing algorithm for auditory data (localization) – Implementation of a networking protocol – Implementation of a state-deciding cognitive process – Further actuation of the results of the cognitive process (as media or motor

output)

14

CSP01-02 - Introduction

Base exercise relations

• In the real world, cross-sensory processing increases effectivity in responding to the environment – not the goal here

• In the real world, cross-sensory processing might mean different allocation of resources to different sensory processes (attention) – not the goal here, but could be simulated

• Each of our senses has at least one analyzer associate with it that processes raw sensations (Visual sensory analysis processes information related to

such thigs as color, shape, size, orientation, location and motion) – exemplified by having separate PCs for different types of media stream processing

15

CSP01-02 - Introduction

Base exercise relations

• Cognitive mapping, is the process of acquiring, coding, using, and storing information from the multitude of environments external to the mind.

• A cognitive map is therefore a person 's model of objective reality – the exercise aims to simulate the cognitive map of the immobot.

• Cognitive mapping processes encode in memory the existence of objects, their characteristics, and known locations in space.

16

CSP01-02 - Introduction

Base exercise relations

• Relation to neural networks:

• An artificial neural network processes information similar to the way a brain processes information.

• This is quite different from the way a standard desktop computer processes information. – Desktop computers have a central processing unit that acts on

instructions usually provided by a software program. The various instructions in the program are executed very quickly but one at a time in a serial process.

17

CSP01-02 - Introduction

Base exercise relations

• Relation to neural networks:

• The mind is not like a general purpose computer that runs all kinds of different software; it more closely resembles a huge number of dedicated computers that are wired up to perform specific tasks.

– There is no need for software since each computer (neuron) is wired to do one specific task. Artificial neural networks are refereed to as parallel distributed processing models because the information is distributed among many individual processors

18

CSP01-02 - Introduction

Base exercise relations

• Relation to neural networks:

• We will not directly deal with implementing an artificial neural network in software.

• However, the whole system of networked PC’s specialized for a certain task, could be seen as resembling a “giant” (set of) neuron(s), or a network with three nodes

19

CSP01-02 - Introduction

Conclusion

• Overview of the base exercise system in this course:

· Visual processing will be done on PC 1,

implemented as a Max/MSP patch. The patch will be able to detect a single object based on its color, and estimate its 3D position relative to the pair of webcams.

· Auditory processing will be done on PC

2, implemented as a Max/MSP patch. The patch will only be able to estimate the direction of (the loudest) sound source in the environment, detected by the pair of

microphones. · PC 3, as the 'brain', will receive messages produced by the first two PCs, and based on them, will reproduce a 3D representation of the incoming

data - implemented as a Virtools patch. Information from both sensory pathways will be visualized, and then compared to derive and display a state.

20

CSP01-02 - Introduction

Survey of HW and SW possibilities for a MED7 project

• Hardware options– Magnetic tracker– HMD – Head Mounted Device– Haptic device (phantom)– 8 channels surround– Wireless data acquisition boards (A/D conversion of sensor

data - from Le Kitchen and custom made) – Making things data acquisition boards (A/D conversion of

sensor data)

• Software options– Max/MSP and Jitter (Cycling ’74)– Virtools – Pure C++ engine (for the haptic device, or custom

programming)