Immersive Media VisionCreate seamless immersive environment for distributed, interactive real and virtual eventsReproduce audio, video and other senses with fidelity approaching the limits of human perception

Relevant to any type of human interaction: entertainment, education, communications

IMSC Entertainment Vision:RMIA fusion of internet browsing with a theater-like immersive experienceHD Video at up to 45 Mbits/sec10.2 ch Immersive Audio (16 Mbits/sec)Streaming over the Internet on-demand to a mouse click (20 min prog)

RMI demonstrates IMSC innovations in several critical technical areas: Immersed in a college football gameDoctors assisting in a remote procedureBusiness people negotiating like they are in the same roomStudents visiting an aquarium a thousand miles awayStreaming media serversImmersive audio capture and renderingNetwork protocols for error correctionSynchronizationApplications

Premier Demonstration May 9, 2002Cross-country streaming from ISI-East in VirginiaNY Times coverage in Circuits sectionNBC-TV and KTLAhttp://imsc.usc.edu/rmi/http://www.east.isi.edu/NGI-S/

Internet2 Fall 2002 Member MeetingAudio: 10.2 channel, immersive sound systemVideo: HDTV 1920x1080iNew World Symphony, Miami Beach550 seat USC Bing Theater

RMI Experimental SetupSynchronized immersive audio and HDTV streamed playback from Yima server over Internet2Control of end-to-end process: capturing, network interface, transmission, renderingCENIC 2003 Gigabit or Bust Award

University of Maryland ServerUSCUSC Client New World Symphony ServerGeorgia Tech Server

Remote Master ClassLA Philharmonic cellist located at our USC labStudent at New World Symphony in Miami BeachThree hour session with MPEG2 video and 10.2 immersive audioTeacher reports that student "was really there" with immersive audioMany psychophysical, perceptual and artistic tests to be done

Recent Developments...Two-way audio communication to New World Symphony in Miami Beach, Florida8-channel audio low latencyLive video capture/transmission (DVCam) Near term prospectsRMI theater at Inha University (Korea)2-way live HD-video/immersive audio streaming for Internet collaboration

Distributed Immersive PerformanceCreate seamless immersive interactive environmentDistributed musicians, conductor (active) and audience (passive)Q: Why a musical performance? A: A distributed orchestra is one of the most challenging immersive, interactive environmentsScenario:Multiple soloists scheduled in different citiesUnder served areas can be reached

IMSC Communication VisionInterAct!UWB console captures 3D model and problem detailsPDA provides spatial array of participants and sonic translation of detailsImmersive workstation coordinates video, avatars, graphics, audio, and haptics to facilitate problem resolution

Research Highlights

immersive audiomultichannel and HRTF approaches - holistic DSP approach

streaming serversdistributed and scalable architecture

computer visioncomputational framework for grouping based on tensor voting, tracking for augmented realities and SFX

graphics & animation3D DSP mesh processing, compression, mesh operations, hair modeling and animation

multimodal emotive interfacesSpeech and dialog, vision sensing of body and hands, facial expressions analysis and expressive avatars

virtual reality -- chair of VR2003 in LAapplications to psychology (ADD diagnosis), haptics applications, and user studies

perception & cognitive modelingtheoretical foundations for understanding benefits of media immersion

UWB wirelessleadership in technology, FCC regulation, and commercializationIMSC has produced ground breaking fundamental research in:

Sensory Interfaces (SI)Robust Vision SystemsSegmentation, motion, 3D body and hand tracking, tensor voting, activity analysis, recognition and trackingSpeech Recognition and SynthesisEmotive dialog, translation, emotion analysisImmersive AudioSonic visualization, spatialization, autocalibrationFacial Analysis -- Gestures and AnimationExpression signatures for emotion and recognitionVideo fusion with 3D models for SurveillanceStructure extraction from LiDAR and texture projectionDigital Geometry (3D) ProcessingCompression, surface and topology smoothing, mesh simplification