![Page 1: Hyper Communication: Toward the Creation of New Media for ... · Communications Filling the Gap of Distance, Time, and Culture ... (Agent Generation) (Nonverbal Communications) (Communication](https://reader036.vdocuments.mx/reader036/viewer/2022071007/5fc4b0e3bc69ba056136ab50/html5/thumbnails/1.jpg)
Hyper Communication: Toward the Creation
of New Media for the Information Network
Ryohei Nakatsu
ATR Media Integration & Communications
Research Laboratories
![Page 2: Hyper Communication: Toward the Creation of New Media for ... · Communications Filling the Gap of Distance, Time, and Culture ... (Agent Generation) (Nonverbal Communications) (Communication](https://reader036.vdocuments.mx/reader036/viewer/2022071007/5fc4b0e3bc69ba056136ab50/html5/thumbnails/2.jpg)
Contents
Hyper Communication
Approach toward Hyper Communication
Technologies for Hyper Communication
Examples of Hyper Communication System
![Page 3: Hyper Communication: Toward the Creation of New Media for ... · Communications Filling the Gap of Distance, Time, and Culture ... (Agent Generation) (Nonverbal Communications) (Communication](https://reader036.vdocuments.mx/reader036/viewer/2022071007/5fc4b0e3bc69ba056136ab50/html5/thumbnails/3.jpg)
Hyper Communication
![Page 4: Hyper Communication: Toward the Creation of New Media for ... · Communications Filling the Gap of Distance, Time, and Culture ... (Agent Generation) (Nonverbal Communications) (Communication](https://reader036.vdocuments.mx/reader036/viewer/2022071007/5fc4b0e3bc69ba056136ab50/html5/thumbnails/4.jpg)
Telephone, Teleconferencing
Multimedia Communications
Future Media
Movies
Video Game
TVNovel, Poem
Toward the Realization of Future Media
![Page 5: Hyper Communication: Toward the Creation of New Media for ... · Communications Filling the Gap of Distance, Time, and Culture ... (Agent Generation) (Nonverbal Communications) (Communication](https://reader036.vdocuments.mx/reader036/viewer/2022071007/5fc4b0e3bc69ba056136ab50/html5/thumbnails/5.jpg)
New Trend in Various Kinds of Media
Communications /Entertainment
in Cyberspace
Telecommunications (Communications)
Movies (Entertainment) Video Game (Amusement)
Internet
Roll Playing GameCG Movies, Digital Movies
![Page 6: Hyper Communication: Toward the Creation of New Media for ... · Communications Filling the Gap of Distance, Time, and Culture ... (Agent Generation) (Nonverbal Communications) (Communication](https://reader036.vdocuments.mx/reader036/viewer/2022071007/5fc4b0e3bc69ba056136ab50/html5/thumbnails/6.jpg)
in Cyberspace
Communications Filling the Gap of Distance, Time, and Culture
Natural Communications with Computers
Hyper Communication
![Page 7: Hyper Communication: Toward the Creation of New Media for ... · Communications Filling the Gap of Distance, Time, and Culture ... (Agent Generation) (Nonverbal Communications) (Communication](https://reader036.vdocuments.mx/reader036/viewer/2022071007/5fc4b0e3bc69ba056136ab50/html5/thumbnails/7.jpg)
• ECreation of Cyberspace (Cyberspace Generation)
• EWarp into Cyberspace (Metamorphosis)
(Agent Generation)
(Nonverbal Communications)
(Communication Support)
• EOperation of Materials in Cyberspace (Cyberspace Handling)
• ECommunication with Characters in Cyberspace
Technologies for Hyper Communication
![Page 8: Hyper Communication: Toward the Creation of New Media for ... · Communications Filling the Gap of Distance, Time, and Culture ... (Agent Generation) (Nonverbal Communications) (Communication](https://reader036.vdocuments.mx/reader036/viewer/2022071007/5fc4b0e3bc69ba056136ab50/html5/thumbnails/8.jpg)
Art and Technology
![Page 9: Hyper Communication: Toward the Creation of New Media for ... · Communications Filling the Gap of Distance, Time, and Culture ... (Agent Generation) (Nonverbal Communications) (Communication](https://reader036.vdocuments.mx/reader036/viewer/2022071007/5fc4b0e3bc69ba056136ab50/html5/thumbnails/9.jpg)
Artists
Engineers-Designers
Artists-Engineers
Collaboration between Engineers
and Artists
Engineers
Engineers
Artists
![Page 10: Hyper Communication: Toward the Creation of New Media for ... · Communications Filling the Gap of Distance, Time, and Culture ... (Agent Generation) (Nonverbal Communications) (Communication](https://reader036.vdocuments.mx/reader036/viewer/2022071007/5fc4b0e3bc69ba056136ab50/html5/thumbnails/10.jpg)
Communications Channels
Verbal Channel
Nonverbal Channel
![Page 11: Hyper Communication: Toward the Creation of New Media for ... · Communications Filling the Gap of Distance, Time, and Culture ... (Agent Generation) (Nonverbal Communications) (Communication](https://reader036.vdocuments.mx/reader036/viewer/2022071007/5fc4b0e3bc69ba056136ab50/html5/thumbnails/11.jpg)
Communication Model
Communication
(Language)
Interaction
Reaction
A I
![Page 12: Hyper Communication: Toward the Creation of New Media for ... · Communications Filling the Gap of Distance, Time, and Culture ... (Agent Generation) (Nonverbal Communications) (Communication](https://reader036.vdocuments.mx/reader036/viewer/2022071007/5fc4b0e3bc69ba056136ab50/html5/thumbnails/12.jpg)
Interactive Art
Traditional Art
Interactive Art
Artists
Audiences
AudiencesArtists
Artworks
Artworks
![Page 13: Hyper Communication: Toward the Creation of New Media for ... · Communications Filling the Gap of Distance, Time, and Culture ... (Agent Generation) (Nonverbal Communications) (Communication](https://reader036.vdocuments.mx/reader036/viewer/2022071007/5fc4b0e3bc69ba056136ab50/html5/thumbnails/13.jpg)
Artistic Approach v.s. Engineering Approach
Art
Technology
based on 'kansei' (sensitivity) and
intuition of artists
based on analysis of human behaviors and
utilization of image/speech processing
technologies
![Page 14: Hyper Communication: Toward the Creation of New Media for ... · Communications Filling the Gap of Distance, Time, and Culture ... (Agent Generation) (Nonverbal Communications) (Communication](https://reader036.vdocuments.mx/reader036/viewer/2022071007/5fc4b0e3bc69ba056136ab50/html5/thumbnails/14.jpg)
Engineering Approach vs.
Artistic Approach
An Example of
an Artistic Approach
•i"MIC" by Naoko Tosa•j
Surprise
Happy
An Example of
an Engineering Approach
![Page 15: Hyper Communication: Toward the Creation of New Media for ... · Communications Filling the Gap of Distance, Time, and Culture ... (Agent Generation) (Nonverbal Communications) (Communication](https://reader036.vdocuments.mx/reader036/viewer/2022071007/5fc4b0e3bc69ba056136ab50/html5/thumbnails/15.jpg)
Cyberspace Generation
Realistic Environment Generation
Hyper-Realistic Environment Generation
![Page 16: Hyper Communication: Toward the Creation of New Media for ... · Communications Filling the Gap of Distance, Time, and Culture ... (Agent Generation) (Nonverbal Communications) (Communication](https://reader036.vdocuments.mx/reader036/viewer/2022071007/5fc4b0e3bc69ba056136ab50/html5/thumbnails/16.jpg)
![Page 17: Hyper Communication: Toward the Creation of New Media for ... · Communications Filling the Gap of Distance, Time, and Culture ... (Agent Generation) (Nonverbal Communications) (Communication](https://reader036.vdocuments.mx/reader036/viewer/2022071007/5fc4b0e3bc69ba056136ab50/html5/thumbnails/17.jpg)
Merging of Real and CG Images
Real Scene Image CG Scene Image
![Page 18: Hyper Communication: Toward the Creation of New Media for ... · Communications Filling the Gap of Distance, Time, and Culture ... (Agent Generation) (Nonverbal Communications) (Communication](https://reader036.vdocuments.mx/reader036/viewer/2022071007/5fc4b0e3bc69ba056136ab50/html5/thumbnails/18.jpg)
![Page 19: Hyper Communication: Toward the Creation of New Media for ... · Communications Filling the Gap of Distance, Time, and Culture ... (Agent Generation) (Nonverbal Communications) (Communication](https://reader036.vdocuments.mx/reader036/viewer/2022071007/5fc4b0e3bc69ba056136ab50/html5/thumbnails/19.jpg)
Interactive Environment
• EEnvironments are fixed
• EHumans select environments
depending on communications.
• EEnvironments are flexible
• EEnvironments vary depending
on communications
Interactive
Environment
Communication
Environment
![Page 20: Hyper Communication: Toward the Creation of New Media for ... · Communications Filling the Gap of Distance, Time, and Culture ... (Agent Generation) (Nonverbal Communications) (Communication](https://reader036.vdocuments.mx/reader036/viewer/2022071007/5fc4b0e3bc69ba056136ab50/html5/thumbnails/20.jpg)
(Laurent Mignonneau & Christa Sommerer for ATR MIC Lab, 1995)
Concept of Interactive Environment
![Page 21: Hyper Communication: Toward the Creation of New Media for ... · Communications Filling the Gap of Distance, Time, and Culture ... (Agent Generation) (Nonverbal Communications) (Communication](https://reader036.vdocuments.mx/reader036/viewer/2022071007/5fc4b0e3bc69ba056136ab50/html5/thumbnails/21.jpg)
Interaction in Interactive Environment
![Page 22: Hyper Communication: Toward the Creation of New Media for ... · Communications Filling the Gap of Distance, Time, and Culture ... (Agent Generation) (Nonverbal Communications) (Communication](https://reader036.vdocuments.mx/reader036/viewer/2022071007/5fc4b0e3bc69ba056136ab50/html5/thumbnails/22.jpg)
Environment
Interaction Interaction
Interaction
Environment : Mirror of our mind, Our 'doppel'
Communication and Environment
![Page 23: Hyper Communication: Toward the Creation of New Media for ... · Communications Filling the Gap of Distance, Time, and Culture ... (Agent Generation) (Nonverbal Communications) (Communication](https://reader036.vdocuments.mx/reader036/viewer/2022071007/5fc4b0e3bc69ba056136ab50/html5/thumbnails/23.jpg)
Image Communication System
Camera
Microphone Array
Screen
Projector
Speaker
Computer
![Page 24: Hyper Communication: Toward the Creation of New Media for ... · Communications Filling the Gap of Distance, Time, and Culture ... (Agent Generation) (Nonverbal Communications) (Communication](https://reader036.vdocuments.mx/reader036/viewer/2022071007/5fc4b0e3bc69ba056136ab50/html5/thumbnails/24.jpg)
Metamorphosis Technology
![Page 25: Hyper Communication: Toward the Creation of New Media for ... · Communications Filling the Gap of Distance, Time, and Culture ... (Agent Generation) (Nonverbal Communications) (Communication](https://reader036.vdocuments.mx/reader036/viewer/2022071007/5fc4b0e3bc69ba056136ab50/html5/thumbnails/25.jpg)
Virtual KABUKI
Facial Expression
Detection/
Reproduction
Virtual KABUKI
Reproduction of
"Realistic" Facial Expressions
Mapping on KABUKI Actor's Face
("Metamorphosis")
![Page 26: Hyper Communication: Toward the Creation of New Media for ... · Communications Filling the Gap of Distance, Time, and Culture ... (Agent Generation) (Nonverbal Communications) (Communication](https://reader036.vdocuments.mx/reader036/viewer/2022071007/5fc4b0e3bc69ba056136ab50/html5/thumbnails/26.jpg)
Flowchart for Reconstruction
of Facial Images
Original Image
Detection of
Facial Expression
Estimation of
Facial Expression
Deformation of
3D WFMTexture Mapping
Reconstruction of Facial Image
3D WFMColor
Texture
Modeling
ReconstructionDetection
![Page 27: Hyper Communication: Toward the Creation of New Media for ... · Communications Filling the Gap of Distance, Time, and Culture ... (Agent Generation) (Nonverbal Communications) (Communication](https://reader036.vdocuments.mx/reader036/viewer/2022071007/5fc4b0e3bc69ba056136ab50/html5/thumbnails/27.jpg)
Reproduction of
Facial Expressions
![Page 28: Hyper Communication: Toward the Creation of New Media for ... · Communications Filling the Gap of Distance, Time, and Culture ... (Agent Generation) (Nonverbal Communications) (Communication](https://reader036.vdocuments.mx/reader036/viewer/2022071007/5fc4b0e3bc69ba056136ab50/html5/thumbnails/28.jpg)
Mapping on
a KABUKI Actor's Face
![Page 29: Hyper Communication: Toward the Creation of New Media for ... · Communications Filling the Gap of Distance, Time, and Culture ... (Agent Generation) (Nonverbal Communications) (Communication](https://reader036.vdocuments.mx/reader036/viewer/2022071007/5fc4b0e3bc69ba056136ab50/html5/thumbnails/29.jpg)
Mapping on
a KABUKI Actor's Face
![Page 30: Hyper Communication: Toward the Creation of New Media for ... · Communications Filling the Gap of Distance, Time, and Culture ... (Agent Generation) (Nonverbal Communications) (Communication](https://reader036.vdocuments.mx/reader036/viewer/2022071007/5fc4b0e3bc69ba056136ab50/html5/thumbnails/30.jpg)
Detection of Body Motions
Thermographical Image Detection Result
•@
![Page 31: Hyper Communication: Toward the Creation of New Media for ... · Communications Filling the Gap of Distance, Time, and Culture ... (Agent Generation) (Nonverbal Communications) (Communication](https://reader036.vdocuments.mx/reader036/viewer/2022071007/5fc4b0e3bc69ba056136ab50/html5/thumbnails/31.jpg)
Metamorphosis
into a KABUKI Actor
![Page 32: Hyper Communication: Toward the Creation of New Media for ... · Communications Filling the Gap of Distance, Time, and Culture ... (Agent Generation) (Nonverbal Communications) (Communication](https://reader036.vdocuments.mx/reader036/viewer/2022071007/5fc4b0e3bc69ba056136ab50/html5/thumbnails/32.jpg)
Manipulation in Cyberspace
Glove device Intuitive manipulation
Difficulty in fine manipulation
Force feedback PDDM
Cooperative object manipulation
Hand gesture recognition
![Page 33: Hyper Communication: Toward the Creation of New Media for ... · Communications Filling the Gap of Distance, Time, and Culture ... (Agent Generation) (Nonverbal Communications) (Communication](https://reader036.vdocuments.mx/reader036/viewer/2022071007/5fc4b0e3bc69ba056136ab50/html5/thumbnails/33.jpg)
P D D M
(Palmtop Display for Dextrous Manipulation)
Structure:Small LCD
•{
Force Display
with
Torque Controlled
Ultrasonic Motor
Control Phases:Observing Phase
Handling Phase
![Page 34: Hyper Communication: Toward the Creation of New Media for ... · Communications Filling the Gap of Distance, Time, and Culture ... (Agent Generation) (Nonverbal Communications) (Communication](https://reader036.vdocuments.mx/reader036/viewer/2022071007/5fc4b0e3bc69ba056136ab50/html5/thumbnails/34.jpg)
Manipulation by PDDM
![Page 35: Hyper Communication: Toward the Creation of New Media for ... · Communications Filling the Gap of Distance, Time, and Culture ... (Agent Generation) (Nonverbal Communications) (Communication](https://reader036.vdocuments.mx/reader036/viewer/2022071007/5fc4b0e3bc69ba056136ab50/html5/thumbnails/35.jpg)
Cooperative Object Manipulation
Multi-hand Manipulation
Multi-user Manipulation
Precise object manipulation
Extentional slave hand
Integration of operations by multiple users
Improvement of presence with cooperative manipulation
![Page 36: Hyper Communication: Toward the Creation of New Media for ... · Communications Filling the Gap of Distance, Time, and Culture ... (Agent Generation) (Nonverbal Communications) (Communication](https://reader036.vdocuments.mx/reader036/viewer/2022071007/5fc4b0e3bc69ba056136ab50/html5/thumbnails/36.jpg)
Multi-user Manipulation
![Page 37: Hyper Communication: Toward the Creation of New Media for ... · Communications Filling the Gap of Distance, Time, and Culture ... (Agent Generation) (Nonverbal Communications) (Communication](https://reader036.vdocuments.mx/reader036/viewer/2022071007/5fc4b0e3bc69ba056136ab50/html5/thumbnails/37.jpg)
Realtime Recognition of
Hand Gestures
Rotation Angle Detection Finger Bending Detection
![Page 38: Hyper Communication: Toward the Creation of New Media for ... · Communications Filling the Gap of Distance, Time, and Culture ... (Agent Generation) (Nonverbal Communications) (Communication](https://reader036.vdocuments.mx/reader036/viewer/2022071007/5fc4b0e3bc69ba056136ab50/html5/thumbnails/38.jpg)
Virtual Object Handling
by Hand Gesture Recognition
![Page 39: Hyper Communication: Toward the Creation of New Media for ... · Communications Filling the Gap of Distance, Time, and Culture ... (Agent Generation) (Nonverbal Communications) (Communication](https://reader036.vdocuments.mx/reader036/viewer/2022071007/5fc4b0e3bc69ba056136ab50/html5/thumbnails/39.jpg)
Generation of
Autonomous Agent
![Page 40: Hyper Communication: Toward the Creation of New Media for ... · Communications Filling the Gap of Distance, Time, and Culture ... (Agent Generation) (Nonverbal Communications) (Communication](https://reader036.vdocuments.mx/reader036/viewer/2022071007/5fc4b0e3bc69ba056136ab50/html5/thumbnails/40.jpg)
Automatic Generation of
Walking Amimations
Animation generation by animators
Automatic generation of animations
![Page 41: Hyper Communication: Toward the Creation of New Media for ... · Communications Filling the Gap of Distance, Time, and Culture ... (Agent Generation) (Nonverbal Communications) (Communication](https://reader036.vdocuments.mx/reader036/viewer/2022071007/5fc4b0e3bc69ba056136ab50/html5/thumbnails/41.jpg)
Multi-layer Architecture
for Agent Behaviors
![Page 42: Hyper Communication: Toward the Creation of New Media for ... · Communications Filling the Gap of Distance, Time, and Culture ... (Agent Generation) (Nonverbal Communications) (Communication](https://reader036.vdocuments.mx/reader036/viewer/2022071007/5fc4b0e3bc69ba056136ab50/html5/thumbnails/42.jpg)
Generation Tool for
Walking Animations
![Page 43: Hyper Communication: Toward the Creation of New Media for ... · Communications Filling the Gap of Distance, Time, and Culture ... (Agent Generation) (Nonverbal Communications) (Communication](https://reader036.vdocuments.mx/reader036/viewer/2022071007/5fc4b0e3bc69ba056136ab50/html5/thumbnails/43.jpg)
Automatic Generation of
Animations
![Page 44: Hyper Communication: Toward the Creation of New Media for ... · Communications Filling the Gap of Distance, Time, and Culture ... (Agent Generation) (Nonverbal Communications) (Communication](https://reader036.vdocuments.mx/reader036/viewer/2022071007/5fc4b0e3bc69ba056136ab50/html5/thumbnails/44.jpg)
Nonverbal Communications
![Page 45: Hyper Communication: Toward the Creation of New Media for ... · Communications Filling the Gap of Distance, Time, and Culture ... (Agent Generation) (Nonverbal Communications) (Communication](https://reader036.vdocuments.mx/reader036/viewer/2022071007/5fc4b0e3bc69ba056136ab50/html5/thumbnails/45.jpg)
Emotional Agents "MIC"
Computer Agents
Emotional Agents
Logical Processing
Verbal Communications
Emotions Processing
Nonverbal Communications
![Page 46: Hyper Communication: Toward the Creation of New Media for ... · Communications Filling the Gap of Distance, Time, and Culture ... (Agent Generation) (Nonverbal Communications) (Communication](https://reader036.vdocuments.mx/reader036/viewer/2022071007/5fc4b0e3bc69ba056136ab50/html5/thumbnails/46.jpg)
Emotions Recognition
Emotions
Joy, Anger, Surprise, Sadness, Disgust, Teasing, Fear, Normal
Speaker-independent, Content-independent
Emotions Recognition
Algorithm : Neural Network
Multi-speaker, Phoneme-balanced Speech Data
![Page 47: Hyper Communication: Toward the Creation of New Media for ... · Communications Filling the Gap of Distance, Time, and Culture ... (Agent Generation) (Nonverbal Communications) (Communication](https://reader036.vdocuments.mx/reader036/viewer/2022071007/5fc4b0e3bc69ba056136ab50/html5/thumbnails/47.jpg)
An Example of
MIC's Reactions
![Page 48: Hyper Communication: Toward the Creation of New Media for ... · Communications Filling the Gap of Distance, Time, and Culture ... (Agent Generation) (Nonverbal Communications) (Communication](https://reader036.vdocuments.mx/reader036/viewer/2022071007/5fc4b0e3bc69ba056136ab50/html5/thumbnails/48.jpg)
Block-diagram of
Emotional Agent
Voice Input
Emotion
to
Reaction
Conversion
Animated
Output Face
Output
Sound
Emotion
Recognition
![Page 49: Hyper Communication: Toward the Creation of New Media for ... · Communications Filling the Gap of Distance, Time, and Culture ... (Agent Generation) (Nonverbal Communications) (Communication](https://reader036.vdocuments.mx/reader036/viewer/2022071007/5fc4b0e3bc69ba056136ab50/html5/thumbnails/49.jpg)
Block-diagram for
Emotions Recognition
Speech feature calculation
Extraction of speech period
Speech feature extraction
Training of neural networkEmotion recognition
using neural network
Mapping on emotion plane
Generation of reaction
Generation of output speech
Speech input
Recognized emotion
Speech Processing
Emotion Recognition
Reaction Generation
![Page 50: Hyper Communication: Toward the Creation of New Media for ... · Communications Filling the Gap of Distance, Time, and Culture ... (Agent Generation) (Nonverbal Communications) (Communication](https://reader036.vdocuments.mx/reader036/viewer/2022071007/5fc4b0e3bc69ba056136ab50/html5/thumbnails/50.jpg)
Configuration of
Emotion Recognition Part
Decision logic
Speech feature parameters
Output
Sub-neural
network
(neutral)
Sub-neural
network
(sadness)
Sub-neural
network
(anger)
![Page 51: Hyper Communication: Toward the Creation of New Media for ... · Communications Filling the Gap of Distance, Time, and Culture ... (Agent Generation) (Nonverbal Communications) (Communication](https://reader036.vdocuments.mx/reader036/viewer/2022071007/5fc4b0e3bc69ba056136ab50/html5/thumbnails/51.jpg)
Training Data
Words: 100 phoneme-balanced words
Speakers: five males and five females
Utterances: 800 for each speaker
Training and Recognition
Recognition Experiment
Recognition rate : about 60%