Active Optical Markers

CREATING NEW SENSORIAL MODALITIESBy Alvaro CassinelliResearcher, Toy Maker, Functionalist, Media artist, and Destroyer of (non-existing) Phylosophical Zombieswww.alvarocassinelli.comA practical recipeA philosophical enquire,

... And yes, it would be possible to know what it is like to be a Bat.1

The Nigthmare, by Henry Fuseli (1781)

Dreams, illusions, hallucinations, delusions. What is real? What is unreal?

Example that everybody can understand: Dreams. Hyponopompic, hypnagogic hallucinations and altered states of the mind.

Personal account: sleep paralysis. Happens to me since chilhood, once or twice a month, when I disrupt my cyrcdian rythms

Loose percepction of time, space and even identity. Anything outside become hallucinated into something in my dream

-> Ganzfeld-> sensory DEPRIVATION

-> the origin of the supernatural in human history: an interpretation of a natural phenomena2

Delia Malchert, Scintillating scotoma, 2003. 2003 Delia Malchert

Another thing that get me started on the subject since chilhood: Migraine with Aura (Todd's syndrome)Alice in Woderland syndrome, Aphasia

3

4

http://deepdreamgenerator.com

and now machines can also hallucinate too!!

What it is to see? HallucinationsExperimental models 5

Commonsense realism: world, stimuli and perceptsCentrality of the body-mind problem Qualia: an obstacle to the creation of artificial senses? Some examples Early experiments, TVSS, and clinical psychologyInterfaces or extensions of the self? (embodied cognition)Example of a new sensorial modality

Approaching the problem of Perception

world [ stimulus, input ] -> perceiver [ model ] -> world [ action, output ]Nave realism & the input-output theory of perceptionWe clearly means by perception something of the following:

a. The way we process the input to give an outputb. The way we model the world the knowledge we form c. or some strange epiphenomena?

Sensing AttentionRecallPerception?ProcessingIn fact, (a) and (b) do not differ very much, unless we also suppose that (c) does happens, and that this epiphenomenal percept is produced by either the model of the world, or the sensation of the world (direct or indirect realism).

direct and indirect perception

Dualism: percepts (including our own sense of self) are are real as the world that seems to make them appear. Is there causality, simultaneity (fortuitous?), independence between these worlds?

The gordian knot seems to be the position of the perceiver in relation with the world and the nature of this knowledge or model

7

From: http://www.simplypsychology.org/cognitive.htmlAttempt to study perception as (a) or (b) produced lots of interesting results and theories: ethology, artificial intelligence, theories of the mind, robot vision and even artificial prosthetics while attemp to study perception from the point of view of (c) produced mostly tons of books and heated metaphysical discussions. Lets leave it asside for the moment.

Shakey the Robot (1966-1972)

(the hard problem... for some people. For me this is hard as it was hard to understand Life. There is indeed a problem of complexity of information processes, but not hard in the sense of impossible )

Remember Leibnitz and the gap problem (philosophy of mind)?

It must be confessed, moreover, that perception, and that which depends on it, are inexplicable by mechanical causes, that is, by figures and motions, And, supposing that there were a mechanism so constructed as to think, feel and have perception, we might enter it as into a mill. And this granted, we should only find on visiting it, pieces which push one against another, but never anything by which to explain a perception. This must be sought, therefore, in the simple substance, and not in the composite or in the machine.Gottfried Leibniz, Monadology

... And Daniel Dennet view of it: not being able to explain (inexplicable by mechanical causes) is no real argument. Our minds may not be able to understand themselves (all the gears!). This is by no means means they cannot be mechanical not even that they cannot be understood at some point, by developping adequate tools (mathematical, phylosophical...). Cognitive Sciences

In fact (a) and (b) are not very different: -Naive realism / direct perception: properties of things are independent of us (challenged by QM?). And we perceive the world as it it really is - Scientific realism / indirect perception: we perceive things thanks to these sensations + a model

8

And how can we possibly model something we never saw before? 2) A model of what? Enters again the mind-body problem.Young man, in mathematics you don't understand things. You just get used to them.John von NeumannHow can we form a model of a world out there? Out of what? Are we outside?

Meno: And how are you going to search for [the nature of virtue] when you don't know at all what it is, Socrates? Which of all the things you don't know will you set up as target for your search? And even if you actually come across it, how will you know that it is that thing which you don't know?Dialogue between Socrates and Meno (in Platos Meno)Perceptual bistability is a proof that we are not modeling an external, immutable reality, but we are hallucinating something what and why? Can we see and identify one piece of cake if we always saw entire cakes?

What does it mean to undertand?

Getting used to something (concepts, maths.. and REALITY). Reality does not need to be accommodate to our modest brains. We do understand intuitively a lot, but only the lot that concerns our survival. Quantum Mechanics or Relativity are not weird because they challenge notions of time, space, measurement, etc. What is actually weird is that we can get along in life without having an intuitive understanding of these more fundamental principles. Herein lies all the difficulty and beauty of Physics...

"It would be a mistake, however, to suppose that cognition consists simply of building maximally accurate representations of input information...the gaining of knowledge is a stepping stone to achieving the more immediate goal of guiding behavior in response to the system's changing surroundings." Marcin Milkowski: Explaining the Computational Mind, p. 49

For reasons of survival, we are machines prepared to learn about certain things pertaining to the way the world and us relate to each other. Action in Perception Theory (action includes directing attention, and attention even to our own mental processes).Discovery of Invariants in the Sensory-Motor Relations Sensory Motor Contingencies (SMC) and it also explains what this knowledge may be: it is practical knowledge, an enacted knowledge - embodied cognition, Erschlossenheit? (Heidegger)3) Action in Perception: an evolutionary explanation?Evolutionary psychology may be the best framework to understand (1) why we can know something the world, (2) what this knowledge may be, and finally, (3) what it is to perceive (to enact that knowledge)

Gibson optic array (1966) is a constructivist theory of visual perceptionThis somehow solves the problem of nave realism/indirect realism: we are part of the world, and we already know something about it (albeit most of the time not without words... )

The shaping forces of intuitive perception are evolutionnary (phylogenetic) or ontogenetic in nature. We are not born like a blank slate (not for language, not for mathematics and numbers, and most important: not for how to learn). The goal is not necessarily perfect knowledge, not even a good model at every level, but survival! We FILTER reality like crazy

Example: Trying to understand the implications of the battle of waterloo understood through chemistry (reductionistic approach) is a mistake - or at least a very tortuous way to model high-level concepts. (The Fabric of Reality, David Deutsh); The lion, the principle of intentional stance (Daniel Dennet).

Enacted knowledge: .. Even at the level of a non-self aware being: autopoiesis

10

.. And more recently, a robot with broken legs re-learn new Sensory-Motor mappings. It has developed a real sense of proprioception.

Would you argue it is not perceiving something? Why?A sad case of experiencial blindnessThe inverted goggles experiment reshape the spatial sensory-motor aspects of perception, circa 1954

Paul Bach-y-Rita's Tactile Vision Sensory Substitution (TVSS), circa 1960

WHY THE I/O was not promptly superseeded by the action-in-perception theory even if the inverted goggles experiment was done in the fifties? BECAUSE AT THE TIME WE HAD BASIC COMPUTERS and not ROBOTS.So we tried to TEACH them (i.e., program them) with very basic tailored SMC. It did not appear then that this was perception, but just a model of the world, an abstract, mathematical world. But REAL perception is a complex entangled web of SMC, even relating to the way we examine our own minds and the way we expect the recall to happen.

NOTE: MIRRORS are an incredible things, and the fact we can adapt to their distortion that radically alter the spatial aspect of the SCM is a proof of that the enacted view of perception is right, if not yet functional enough.

The sensory motor contingencies pertain to different aspects of the sensing or body movements.

Dorsal and ventral streams

The case of vision-based proprioception

11

3) The (problem?) of the Qualia.Redness of redInverted SpectrumMary the ScientistThose who say the robot does not really perceive the world, may argue that it is impossible to create real artificial senses..Percept? Are really there percepts independent of the model or SMC, and are these important to create an artificial sense?

I wont go there! (At best, if qualia really exists, they are by definition epiphenomena and cannot contribute to any kind of knowledge of the world - see Dennet read all Dennet).

The real question may be psychological after all. As Minky put it, it is mistaking complexity for simplicity. Redness is an extremely complex entangled bunch of lots of different SMC...

The interesting question is then: why we feel there is something lacking in our (own) verbal account of phenomenological experiences? Neurosciences may gives an answer: speed of processes, subitization. Effortless processing we cannot describe. A sort of savant syndrome we evolved and share (same perhaps with the sense of Self and thus consciousness). What they could be possible trying to say?

Hillary Putnam: "... Being an apple is not a natural kind in physics, but it is in biology, recall. Being complex and of no interest to fundamental physics isn't a failure to be "real". I think green is as real as applehood.".

Parenthetically: the correlates of consciousness arethe crypto-dualists! (see my blog).

Qualias are ineffable seeing magnetic fields (Asimov). But then, VERY few things in the world are amenable to verbal description!!! It looks otherwise because most of the time we live in a world... of language.

12

Conclusion:It is about creating artificial Sensory-Motor dependencies. Qualia does not need to have any consequence on our capacity to create perfectly genuine new artificial senses. Our frustration (over the hard problem!) is just evidence of the poverty of verbal first-person accounts of experience (language did not evolve to tackle those problems, just like our intuitive physics did not evolve to account for quantum entanglement or the relativity of simultaneity get used to the weirdness of being) Modifying or creating new sensorial modalities is producing by artificial means (technological means or whatever hypnosis perhaps) a repeatable response to stimuli with respect to the actions of the observer (that will then enactively learn it - and find natural, obvious, and ineffable).

and therefore, in many senses you could one day know what it is to be a bat!

Interfaces or extensions of the Self?

EXAMPLES: Augment Humans by designing artificial sensory motor systems based on action-perception theory

My approach to design affordances is based on the concept of SENSORY MOTOR CONTINGENCIES (action-perception theory): the given natural world is just one possible instantiation of a coherent set of SCMs. We are not obliged to rely on them, but we must create systems that are coherent even if completely unreal. Reality and the world (physical, psychological) is just a coherent set of physical/psychological RULES.

So, how to design interfaces which do not present the user with paradoxes? ALTERNATE REALITIES!!!

Note1: Eva Hornecker: While we should not give up on exploiting the benefits of physicality, we also need design strategies that acknowledge the hybrid nature of systems. [] A change of focus is advocated: from apparent immediacy to the de- sign of seamful mappings and support of reflection, ena- bling appropriation and a better understanding of systems.

Leveraging the knowledge of the physical world affordances and copying it may NOT be a good idea (it will lead to contradictions, unless it is a perfect simulation or a perfect isomorphic real world). My point is: only the LOW level affordances may be useful as an entry point, but the rest may be an abstract metaphor (I say abstract very much like in the sense employed in mathematics: abstract objects manipulated by abstract rules that may or may not become second nature).

Note: We are indeed trained since childhood to read many physical affordances, and therefore it may be difficult to transcend metaphors when they break; however, we can train like a mathematician trains her/his mind on manipulating abstract "objects Note this interesting quote by John Von Newman: in mathematics you dont understand: you get used to.

When the interface becomes second nature, transparent, ready at hand (ex: distal attribution):Action in Perception (this informs and inspires my work as an artist and researcher) =>such interfaces reinforce the action-perception loop using multiple modalities.procedural memory vs. declarative memory Being IN the world, vs. describing the world

Note: The opposite is equally important for elucidating the nature of the self (construct), and it is also interesting for its peculiar effects (optical illusions, distortions, altered perception): disconnecting/disrupting/forgetting (Check my other works: TBNTB, boxedEgo, blended identitiies). I am interested in this area as a media artist and researcher.

14

Seeing without eyes: extending the body for spatial awareness

new sensorial modality (this is not TVSS) Extension of the body (sensorial extension, 360 degrees)

Example: Electronic Travel Aid for the blind. Augmented perception by creating a new sensorial modalityExtension of the body (sensorial extension, 360 degrees)

Examples: Haptic Radar / Haptic Mask / Haptic Car / Light Arrays

The METAPHOR: cilia for machines or humans. NO COMPUTER VISION

Light as a tangible substance: tangible RAYS

15

Real experiments in Rio de Janeiro, with 50 blind peopleIn collaboration with Eliana Sampaio (a former collaborator to Bach-y-Rita)

http://www.k2.t.u-tokyo.ac.jp/perception/HapticRadar/

Qualitative results were extraordinary (semi-structured interviews & ANOVA analysis of anxiety trait/state)Study of spatial cognition in real or virtual environment

References: A. Cassinelli, E. Sampaio, S.B. Joffily, H.R.S. Lima and B.P.G.R. Gusmo, Do blind people move more confidently with the Tactile Radar? IOS Press, Technology and Disability, Vol. 26, N.2-3, pp:161-170, (2014) [PDF-775KB]Cassinelli Alvaro, Reynolds Carson and Ishikawa Masatoshi : Augmenting spatial awareness with Haptic Radar, Tenth International Symposium on Wearable Computers(ISWC) (Montreux, 2006.10.11-14)/Short paper(4 pages) [PDF-103KB] Slide presentation [ PPT-6.4MB]. Unpublished long version (6 pages) [ PDF-268KB]Cassinelli Alvaro, Reynolds Carson and Ishikawa Masatoshi : Haptic Radar, The 33rd International Conference and Exhibition on Computer Graphics and Interactive Techniques(SIGGRAPH) (Boston, 2006.8.1)/ [PDF-202KB, Large Quicktime Video, Small Quicktime Video, MPG-4]More detailed project web page : http://www.k2.t.u-tokyo.ac.jp/perception/HapticRadar/HapticRadar_LongPage.html

16

Seeing is a way of actingJ.K ORegan and A. Noe

See how I move (like a dolphin using a sonar). My vision needs proprioception. 17

What it is like to be a CAT?

Artificial cat's whiskers to feel feline and luminous. This wearable device possesses rangefinders at the locations of the cat's whiskers as well as motor vibrators (similar to that of the Haptic Radar, but the range data is sent wirelessly to control sound). Note: I fabricated it for actress Joanna Lumley, for the TV serie Catwoman.

18

Extending the body, distal attribution and what it is to be a CAR?

!!

http://www.k2.t.u-tokyo.ac.jp/perception/haptiKar

This research extends and complements the Haptic Radar project. The goal of the Haptic Radar was to augment an individual spatial awareness through a set of invisible, sensitive "light hairs" that extended the body outwards, and into regions not directly not covered by the eyes. The HaptiKar experiment consist on placing the sensors over the surface of a car, while mantaining the actuators over the body of the driver. This way, the driver would feel the surrounding of the car: obstacles on the car blind spots, proximity of the cars behind as "annoying" pressure on the back of the head, etc.

19

force field

Virtual Haptic RadarHaptic interaction with virtual 3d objects embedded in the real world

An example beyond projection mapping: tactile feedback from virtual objects. This is like feeling ghosts.

Details: tracking is done using ultrasound (time differences between pulses).

I will show a newest experiment related to the Haptic Radar later.

20

Artificially out-of-body experience?

Manipulation the perception of the SELF!Boxed Ego

And what it is to be somewhere?http://www.k2.t.u-tokyo.ac.jp/members/alvaro/boxedEgo

BoxedEgo seeks to combine several pre-cinematographic techniques in order to create a magical out-of-body experience (stereoscope, diorama, peep-show box and synthetic pepper ghost effect). From the research perspective, this work can be seen as a preliminary experiment on the cognitive (and possible practical) aspects of time-delayed artificial autoscopy/heautoscopy with micropsia (see paper at DAP2008).

21

- Playing with SPACE and location profoundly affects the perception of the self.

- Alice in Wonderland Syndrome / migraines

- Technology helps understand the range of human experiences (it can be done by writing of course Oliver Sacks)22

Minimal displays & supernormal stimuliHeres a thought experiment: suppose you hold up a smartphone and say, wheres my shoe? and a laser-pointer jumps out of it and focuses a hot red dot on your lost shoe []. Maybe we need a definitional category that involves humble little augment apps that dont inflict major interventions on reality.Bruce Sterling, (Wired Blog)

Laser Aura / Light Robot / Fluid Suit

What is ATTENTION? Do we need words to summon the user?

Examples: The Laser Aura (for augmented expression)the robot made of light or light pet. Interesting: FAST RESPONSE makes the user feel that the agent is really there: its NOT a projection23

Laser Aura: externalizing emotions (for use in the office)Example of a minimal display inspired by manga graphical conventions

http://www.k2.t.u-tokyo.ac.jp/perception/laserAura

Example of MINIMAL DISPLAYS: cartoon-like graphics

We are studying the use of a 'laser aura' as a (wearable) prosthesis for enhanced emotional expression. The goal of such wearable display is to externalize subtle psycho-physiological states of the user by projecting minimalistic imagery in the immediate surrounding for others to see. In the present configuration, a 'laser aura' or 'laser halo' change its shape and dynamic behavior as a function of the user stress. Such iconic imagery (inspired by manga graphical representations) may give others an instant cue about the person psychological state (and thus function as the equivalent of online availability status in the real world). It may as well function as a biofeedback device, and help regain control of one's own body in stressful situations.

24

Cassinelli, A., Zhou, Y., Zerroug .A and Ishikawa, M.: The Laser Aura: a prosthesis for emotional expression, SIGGRAPH ASIA 2011, Technical Sketches and Posters, Hong Kong, 12-15 Dec. (2011), one page: [PDF-1MB]/poster [PDF-1MB]/two pages long abstract (additional material) [PDF-1.3MB] / video [WMV-15MB]Yuko Zou, Alvaro Cassinelli, Masatoshi Ishikawa: Extroverting Interface, Entertainment Computing 2011, (EC2011), 06B-07, Oct. 7-10 (2011) (non-peer reviewed)

25

artificial synesthesia real-time interaction new interfaces for musical expression

scoreLight: a human sized pick-up head

http://www.k2.t.u-tokyo.ac.jp/perception/scoreLight

This installation is an artistic approach to artificial sensory substitution research and artificial synesthesia very much along the lines of Golan Levin's works in the field [3]. I particular, it can be seen as the reverse (in a procedural sense) of the interacting scheme of Golan Levin & Zach Liebermann "Messa di Voce" [2], in which the speed and direction of a curve continuosly being drawn on a screen is controlled by to the pitch and volume of the sound (usually voice) captured by a microphone nearby.

Very high response time: PERFORMANCE, SMC.. I tried this with a blind person who could recognize by the sound the different shapes (triangle, square, smoothness...)

Cassinelli A., Manabe D., Perrin S., Zerroug A. and Masatoshi I.: scoreLight & scoreBots, ACM CHI'12 (Interactivity), May 5-10, 2012, Austin, Texas, USA (2012) [PDF-1MB]A. Cassinelli, Y. Kuribara, A. Zerroug, D. Manabe and M. Ishikawa, scoreLight: playing with a human sized laser pickup, International Conference on New Instruments for Musical Expression (NIME2010), 15-18th June 2010 Sydney, Australia, pp:144-149, (2010) [PDF-2.5MB], Slide presentation [PDF (no videos)-11MB].A. Cassinelli, Y. Kuribara, D. Manabe and M. Ishikawa, scoreLight, Digital Content Expo 2009 Symposium (25 Oct. 2009, Miraikan - Museum of Emerging Science and Innovation, Tokyo): Slide presentation [PPT].A. Cassinelli, Y. Kuribara, D. Manabe and M. Ishikawa: scoreLight: a laser-based synesthetic experience, additional documentation for SIGGRAPH ASIA 2009 (Art Gallery), [PDF-1.7MB].

26

Augmented Sensing: Earlids

Voluntary control of auditory gain by contraction of mastication muscles

This is another example of a new SMC, not sensorial substitutionhttp://www.k2.t.u-tokyo.ac.jp/perception/earlids

"Earlids" is a wearable device enabling the semi-voluntary control of auditory gain. Artificial earlids represent to the ears what natural eyelids are to the eyes: a fast and efficient reflex mechanism for protecting delicate sensory organs.

A. Cassinelli, EARLIDS & entacoustic performance, Third Workshop on Devices that Alter Perception (DAP 2010) in conjunction with ISMAR 2010, October 13th, 2010, Seoul, South Korea [PDF-1.2MB]Devices that Alter Perception 2010. ISBN-10: 1463664249. Reynolds, C. (Ed.) (2011). Charleston, SC: CreateSpace.

27

http://www.iflscience.com/brain/master-art-ignoring-psychology-experiment

or diminished sensing? (how to ignore?)

http://www.k2.t.u-tokyo.ac.jp/perception/lightArraysTHANK YOU

Wearables I worked on with the purpose to augment proprioception (for dancers)

http://www.k2.t.u-tokyo.ac.jp/perception/lightArrays/index-e.html

Wilde D., Cassinelli A.., Zerroug A.:LightArrays, ACM CHI'12 (Interactivity), May 5-10, 2012, Austin, Texas, USA (2012) [PDF-4.2MB] [wmv-46MB]Wilde, D., Cassinelli, A., Zerroug, A., Helmer, R J N., Ishikawa, M. Light Arrays: a system for extended engagement. Proc. ICDVRAT with ArtAbilitation Via del Mar/Valparaso, Chile. September 2010 [PDF-2MB]Yebisu International Festival of Art and Alternative Visions (18-27/2/2011) [website][Blog], Panel presentation [PDF-37MB]

29