immersive multimedia and virtual crowds -...
TRANSCRIPT
Immersive Multimedia and
Virtual Crowds
Farzad Safaei
Farzad Safaei
Slide 2
Physical and Virtual
Mixed reality continuum
The boundary between physical and virtual is
being blurred
Augmented Reality Augmented Virtuality Real Virtual
Farzad Safaei
Slide 3
Augmented reality
Augmentation of physical with virtual
Farzad Safaei
Slide 4
Augmented virtuality
Augmentation of virtual with physical
Farzad Safaei
Slide 5
Augmented virtuality challenges
• Presentation of video and audio within a virtual
space
• New methods of rendering
• New modes of consumption
• Complexity of assessing quality
Farzad Safaei
Slide 6
Examples
• Scalable video collaboration
• Free viewpoint video
Scalable Video Collaboration
Farzad Safaei
Slide 8
iSee
• Combines virtual spaces with video conferencing
• Emulates a natural human gathering
• Can scale to very large meetings
Farzad Safaei
Slide 9
Why augmented virtuality?
Farzad Safaei
Slide 10
Technical scalability
The required capacity grows
as the square of number of
participants
Farzad Safaei
Slide 11
Current solution
Farzad Safaei
Slide 12
Current solution
Farzad Safaei
Slide 13
Cognitive scalability
Utility will diminish if there
are too many video and audio
streams
Farzad Safaei
Slide 14
iSee scalability
• By emulating a natural human gathering,
iSee has both technical and cognitive
scalability
• But also smaller meetings become more
usable
demo
Farzad Safaei
Slide 15
iSee scalability
• Uses much less bandwidth than conventional
tools – 70% - 90% less bandwidth
– The bandwidth saving will be more for larger meetings
Farzad Safaei
Slide 16
iSee scalability
• Allows easy interaction in large
gatherings
– Multiple simultaneous conversations
– Moving from one conversation to another
Farzad Safaei
Slide 17
Key methods for technical scalability
Visibility Culling
• Only receive those video streams that are
within the visual field of view of this user
• Distance
• View frustum
• Back face
• Occlusions
P. Pourashraf, F. Safaei, and D. R. Franklin, "Distributed Area of Interest Management for Large-
Scale Immersive Video Conferencing," in Multimedia and Expo Workshops (ICMEW), 2012 IEEE
International Conference on, 2012, pp. 139-144.
Farzad Safaei
Slide 18
Visibility Culling
• More than 70% reduction with 25 participants – The saving is more significant as the number of users increases
Farzad Safaei
Slide 19
Visibility Culling
• Dealing with movement
– Predicting future visibility based on motion vectors
Farzad Safaei
Slide 20
Key methods for technical scalability
Quality Culling
• Only receive the video quality that is needed based on the
perspective of the user
• Virtual distance
• Virtual orientation
• Our technique for this is called Perceptual Pruning
F. Safaei, P. Pourashraf, and D. Franklin, "Large-scale immersive video
conferencing by altering video quality and distribution based on the virtual context,"
Communications Magazine, IEEE, vol. 52, pp. 66-72, 2014.
Farzad Safaei
Slide 21
Quality Culling
• The required quality is dependent on orientation and
distance of the avatars
Farzad Safaei
Slide 22
Quality Culling
Texel to pixel mapping • Video is rendered as texture on the front surface of the avatar
• Will result in texture minification and magnification
Farzad Safaei
Slide 23
Quality Culling
Perceptual pruning
• DCT block down-sampling based on texture minification
Farzad Safaei
Slide 24
Quality Culling
• Significant bandwidth saving in addition to visibility culling
Farzad Safaei
Slide 25
Quality Culling
• 61% saving in worst-case scenario (lecture theatre)
Farzad Safaei
Slide 26
Key methods for technical scalability
Saliency ranking
• During congestion, reduce quality of less salient videos
• Saliency measured by a number of attributes
• Virtual distance
• Virtual orientation
• Voice activity
Farzad Safaei
Slide 27
Scalability tests
• Tests conducted during
2014
Farzad Safaei
Slide 28
Problems solved
• Scalability, and more …
• Easy for the students
– Natural interaction and socialisation
– Peer-based learning
• Easy for the instructors
– No need to adapt to a new mode of delivery
Farzad Safaei
Slide 29
Case studies
• DoE debating competition
• DoE Aurora College – https://drive.google.com/a/iseevc.com/file/d/0B9wPZZq-45gIM3FRclBsTWFEQkE/view?usp=sharing
• OTEN classes by invitation
• AIS art gallery – http://youtu.be/wfOfUWe58-4
• UOW Engineering Project Pitch – https://drive.google.com/file/d/0BzDp2xswG__IX0lKMTcySVctRTQ/view?usp=sharing
• UOW Alumni Fair
Farzad Safaei
Slide 30
OTEN Health Class
Farzad Safaei
Slide 31
UOW Engineering Pitch
Farzad Safaei
Slide 32
Some of the feedback … UOW pitch
• 90% of students and 100% of industry participants indicated they wanted
to participate in a similar activity again – 80% of industry participants said they would not have been able to participate in the event had it
been held on-camps
Lee, Mark, et al., “Engaging Industry in Student Projects Through a Video-Augmented 3D Virtual Trade Fair”,
Under review, IEEE Transactions on Education – contact me for a copy of current manuscript.
Farzad Safaei
Slide 33
Some of the feedback … UOW pitch
• Improvements in pitch presentation was significant
Lee, Mark, et al., “Engaging Industry in Student Projects Through a Video-Augmented 3D Virtual Trade Fair”,
Under review, IEEE Transactions on Education – contact me for a copy of current manuscript.
Farzad Safaei
Slide 34
Some of the feedback … UOW pitch
Nikolic S, et al., “2D versus 3D Collaborative Online Spaces for Student Team Meetings: Comparing a Web
Conferencing Environment and a Video-Augmented Virtual World”, To appear, Proceedings of Australasian
Association for Engineering Education Conference, AAEE2015.
Farzad Safaei
Slide 35
UOW Fair
Farzad Safaei
Slide 36
Some of the feedback … Alumni Fair
• Student responses:
Farzad Safaei
Slide 37
Some of the feedback … Alumni Fair
• Industry responses:
Farzad Safaei
Slide 38
Summary
• Intersection between virtual and physical will
be of significant social and commercial
importance
• Potential for new industries (and leaders) to
emerge
ICT Research Institute University of Wollongong